WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
Authors: Bill Yuchen Lin
Yuntian Deng
Khyathi Chandu
Abhilasha Ravichander
Valentina Pyatkin
Nouha Dziri
Ronan Le Bras
Yejin Choi
Venue: ICLR 2025 Spotlight (2025)
Keywords: LLM
Evaluation
Benchmarking
Primary area: datasets and benchmarks
TL;DR: WildBench evaluates LLMs with hard and real tasks from users with metrics that are highly correlated with human-voted Elo.

Abstract: We introduce WildBench, an automated evaluation framework designed to benchmark large language models (LLMs) using challenging, real-world user queries. WildBench consists of 1,024 tasks carefully selected from over one million human-chatbot conversation logs. For automated evaluation with WildBench, we have developed two metrics, WB-Reward and WB-Score, which are computable using advanced LLMs such as GPT-4-turbo. WildBench evaluation uses task-specific checklists to evaluate model outputs systematically and provides structured explanations that justify the scores and comparisons, resulting in more reliable and interpretable automatic judgments. WB-Reward employs fine-grained pairwise comparisons between model responses, generating five potential outcomes: much better, slightly better, slightly worse, much worse, or a tie. Unlike previous evaluations that employed a single baseline model, we selected three baseline models at varying performance levels to ensure a comprehensive pairwise evaluation. Additionally, we propose a simple method to mitigate length bias, by converting outcomes of “slightly better/worse” to “tie” if the winner response exceeds the loser one by more than K characters. WB-Score evaluates the quality of model outputs individually, making it a fast and cost-efficient evaluation metric. WildBench results demonstrate a strong correlation with the human-voted Elo ratings from Chatbot Arena on hard tasks. Specifically, WB-Reward achieves a Pearson correlation of 0.98 with top-ranking models. Additionally, WB-Score reaches 0.95, surpassing both ArenaHard’s 0.91 and AlpacaEval2.0’s 0.89 for length-controlled win rates, as well as the 0.87 for regular win rates.

Published as a conference paper at ICLR 2025
WILDBENCH: BENCHMARKING LLMs WITH CHALLENGING TASKS FROM REAL USERS IN THE WILD
Bill Yuchen Lin 90
Yuntian Deng
Khyathi Chandu ?
Faeze Brahman Abhilasha Ravichander 9
Valentina Pyatkin 9
Nouha Dziri
Ronan Le Bras
Yejin Choi
Allen Institute for AI
University of Washington
https://ht co/ spaces https////waenanannanaaeeeeee
ABSTRACT
We introduce WildBench, an automated evaluation framework designed to bench- mark large language models (LLMs) using challenging, real-world user queries. WILDBENCH consists of 1,024 examples carefully selected from over one million human-chatbot conversation logs. For automated evaluation with WILDBENCH, we have developed two metrics, WB-Reward and WB-Score, which are computable using advanced LLMs such as GPT-4-turbo. WILDBENCH evaluation uses task- specific checklists to evaluate model outputs systematically and provides structured explanations that justify the scores and comparisons, resulting in more reliable and interpretable automatic judgments. WB-Reward employs fine-grained pair- wise comparisons between model responses, generating five potential outcomes: much better, slightly better, slightly worse, much worse, or tie. Unlike previous evaluations that employed single baseline model, we selected three baseline mod- els at varying performance levels to ensure comprehensive pairwise evaluation. Additionally, we propose a simple method to mitigate length bias by converting outcomes of "slightly better/worse" to "tie" if the winner's ' response exceeds the loser's by more than characters. WB-Score evaluates the quality of model outputs individually, making it fast and cost-efficient evaluation metric. WILD- BENCH results demonstrate a strong correlation with the human-voted Elo ratings from Chatbot Arena on hard tasks. Specifically, WB-Reward achieves a Pearson correlation of 0.98 with top-ranking models. Additionally, WB-Score reaches 0.95, surpassing both ArenaHard's 0.91 and AlpacaEval2. 0's 0.89 for cength-contoolled win rates, as well as the 0.87 for regular win rates.
INTRODUCTION
Large language models (LLMss) have become integral to wide range of real-world applications due to their strong generalization capabilities across diverse tasks. However, effectively evaluating their performance remains challenging problem, particularly when striving for an automated and cost-effective solution. Traditional benchmarking datasets like MMLU (Li et al., 2023a) focus primarily on assessing the reasoning abilities of LLMs using multiple-choice questions, which fall short in evaluating the more open-ente open-ended ended problems that real-world users pose. Chatbot Arena (Chiang al., 2024) provides an online platform where human preferences are collected to judge pairs of model outputs, subsequently ranking L.MSs using Elo ratings. While this human-based evaluation method offers valuable insights into user preferences, it has notable limitations, such as high labor costs, the inability to deliver real-time results, lack of data transparency, and the challenge of fairly evaluating all models with the same data.
Several automated benchmarks such AlpacaEval (Li et al., 2023b), MT-bench (Zheng et al., 2024), and ArenaHard (Li et al., 2024) employ advanced LLMs like GPT-4-Turbo to assess the quality of model responses. Comparative analyses of these benchmarks are presented in Table and Figure 3. These existing benchmarks exhibit significant shortcomings in task composition and skill coverage, particularly in mirroring the natural distribution of real-world user tasks. MT-hench, comprising

---

Published as a conference paper at ICLR 2025
What capital of Australia?
Please provide me python code go through directory and its subdirectories and delete images that are not horizontal.
What some cool music from the 1920s? wrap present neatly?
Can you write code?
recipe generation tacles
AlpacaEval
WildBench Diverse tasks from real users!
hey can you write an essay on the impact of the G20 summit on the global economy, trade, development and the role young people in shaping the future of the world, it has have more than 1200 words. Write beautiful and poetic. Use extensive vocabulary. Use lot of factual and empirical data. Use some, ancient indian historical references.
I want create open source, highly realistic and grounded text- based business simulation game that played in the terminal, with a large range of different features that make the game realistic simulation possible. In light of this the game should not have set values for anything because that unrealistic - real life isn't like that; the sim should be as close reality as possible. will host it on Github. Please create a FULL, COMPLETE file structure for the game's Github repo
Figure 1: Example tasks sampled from AlpacaEval (Li et al., 2023b) and WILDBENCII. Tasks in WILDBENCII are more diverse and challenging, which are collected from real users in the wild. Complex real-user tasks usually have multiple constraints and require higher-order reasoning skills, which are well represented in WILDBENCH.
only 80 hand-crafted examples, lacks sufficient breadth for a comprehensive evaluation. Meanwhile, AlpacaEval, with 805 tasks derived from multiple alignment datasets, includes relatively simple tasks, such as "Whal is the capital of Australia?" and suffers from low task diversity; for instance, over 20 tasks redundantly assess recipe generation skills (c.g., "can you provide a recipe for ...?"). We show a few examples in Figure to illustrate the differences between AlpacaEval and our WILDBENCH.
AlpacaEval mostly focuses on information-seekingg tasks, containing merely 6% coding and 3% mathematics tasks. Conversely, ArenaHard, sampling 500 tasks from ChatbotArena, displays an excessive concentration on coding and debugging tasks, accounting for over 57% of its content. Most existing benchmarks do not sufficiently challenge the models with the varied and unexpected nature of user inquiries in practical settings, thus limiting their overall effectiveness in providing a holistic evaluation. This issue highlights the necessity for more comprehensive benchmarks that can better simulate the wide range of tasks from real users.
In this paper, we introduce WILDBENCH, an automated evaluation framework designed for assessing LLMs using complex tasks from real-world users. The examples in WILDBENCH are periodically updated, with the current version (V2) comprising 1,024 tasks carefully curated from real user-chatbot dialogs provided by the A12's WildChat project (Zhao et al., 2024). We engage multiple advanced LLMs to process a filtered selection from WildChat, tasking them with the analysis of the requisite knowledge and skills for each task and subsequently labeling the difficulty level. Tasks considered as easy by all models are excluded. We ensure the distribution of tasks mirrors the original WildChat data, such that the task distribution of WILDBENCH is still natural (Figure 3). Additionally, all finalized tasks undergo manual review. Further details are provided in Section 2.
As shown in Figure 1, WILDBENCH presents significantly harder challenge due to the complexity, depth, and realism of the tasks involved. WILDBENCH is sourced from real-world user interactions and has been carefully curated to ensure diversity and challenge. The tasks in WILDBENCH typically demand higher-order reasoning, such as writing and/or debugging code with specific constraints, creative writing with multiple constraints on the style and content, or designing software system with complex requirements. These tasks often require critical thinking, creativity, and technical expertise, making WILDBENCH substantially more challenging than AlpacaEval, where simpler, factual, or surface-level tasks dominate.
WILDBENCI evaluation is illustrated in Figure 4. To design a reliable automatic evaluation, we employ two key designs for using LLMs as judges. Drawing inspiration from how humans evaluate responses to open-endee apen-ended questions, we develop task-specific checklists. These checklists guide LLMs in generating consistent and reliable judgments, with each checklist comprising questions focused on specific criteria. Similar to the zero-shot Thaon-of-Thoughts (CoT) prompting (Kojima et al., 2022), we prompt LLMs to provide step-by-step, structured analyses of each LLM response. This method encourages a detailed, fine-grained evaluation process, culminating in a well well-justified final decision.
We employ two primary metrics: WB-Reward for pairwise comparisons and WB-Score for individual scoring. -ew-ad is based on pairwise comparisons between LLMs, with five possible outcomes: "A is much/slightly better/worse than B" or "Tie." Notably, we used three baseline models compare with each testing model instead of using a single baseline model, as most prior works do. This approach provides more comprehensive assessment based on different levels of model performance.
2

---

Published as a conference paper at ICLR 2025
WB-Score measures the quality of each model's generation individually, offering a quicker and more cost-effective evaluation. To mitigate the bias towards longer outputs, a common issue in LLM-ae-a-judge evaluations (Dubois et al., 2024), we introduced simple length-penaale method, converting slight wins/losses to ties when the winner's output is significantly longer than the loser's.
Both metries have demonstrated strong correlations with human judgments, evidenced by Pearson correlation of 0.98 for WB-Reward and 0.95 for WB-Score against the human-voted Elo rating from Chatbot Arena on the top-ranking models. These scores significantly surpass other benchmarks, such as ArenaHard(Li et al., 2024)'s 0.91 and AlpacaEval2.09 0.87 (0.89 for the length-controlled version) (Li et al., 2023b; Dubois al., 2024), validating WILDBENCH's effectiveness and alignment with cumann-based evaluation. More details are shown in Table 3 in Section 4.
2 WILDBENCH DATA CURATION
In this section, we describe the data curation process for the tasks used to evaluate LLMs in WILD- BENCH Our goal is to ensure that the selected tasks not only represent real-world use cases but are also challenging enough to distinguish the varying capabilities of LLMs.
Table I: Statistical comparison of LLM alignment benchmarks. Length are in characters.
Dataset
TTasks
#Turns
Chatllistory
QueryLen
PromptLen
RealUser
TaskTag
Evaluation
M'T-Bench
80
2
v Dynamic
202.2
Dynamic
x
Score
AlpacaEval
805
1
x 164.9
164.9
x
X
Pair (ref=1)
ArenaHard
500
1
x
406.4
406.4
x
Pair (ref=1)
WILDBENCH
1,024
<5
Static
978.5
3402.1
Scorc+Pair (ref=3)
303
alpaca_es Leaca_
arena hard
wildbench
23
300
153
109
20
303
1000
1000
2000
2030
305
1000
15:0
2000
2000
300
10.0
12.00
2000
2030
Figure 2: Distribution of query lengths in AlpacaEval, ArenaHard, and WildBench.
2.1 MINING CHALLENGING TASKS FROM WILDCHAT
We sourced tasks from the WildChat dataset (Zhao et al., 2024), which comprises million human-chatbot conversations from real users. This dataset is particularly suited for conversion into an evaluation benchmark because it contains diverse array of tasks that users expect LLMMs to perform, such as writing assistance, coding, mathematics, data analysis, role playing, and planning.
Basic filtering. To control the quality and diversity of the selected tasks, we applied several filtering steps. First, we removed user queries that were either too short (less than 10 tokens) or excessively long (more than 3,000 tokens). We also excluded conversations with more than five user-chatbot turns to maintain focus and coherence in the tasks, as conversations exceeding five turss tend to contain multiple topics. Furthermore, we focused on English data and filtered out non-English tasks. Since our focus is more evaluating the capabilities of L.MSs rather than content moderation, we also removed toxie conversations. To ensure task diversity, we used sentence embeddings from BeRtenceBERT (Reimers & Gurevych, 2019) to calculate the cosine similarity between queries, discarding those with a high similarity score above 0.9. The threshold determined by manual inspection. Lastly, to further enhance task diversity, used diverse user pool by retaining only the last conversation for each unique device, thus removing tasks from the same user that might require similar underlying skills.
Difficulty annotation. To identify challenging tasks that can distinguish the performance different LLMs, used GPT-4-Turbo (OpenAI, 2023), Claude-3-Sonnee and Opus (Anthropic, 2024) to
3

---

Published as a conference paper at ICLR 2025
AlpacaEval (805)
ArenaHard (500)
WildBench (1024)
Cedtig Dabusping
Creative Writing
Coding & Debugging
16%
Information seeking
Information seeking
Information
1/%
10%
seeking
Coding & Debugging
6%
Advice sooking brainstorming Role playing
Role
Advice seeking
Brrmndorming
Wessoringg
1225
Advice seeking Br brainstorming
Creative Writing
10%
6%
Editing
Data Anwlyss
Creative winting Reasoning Phanning
5%
Editing
Deta Analysis
Panning
11%
0%
8%
Editing
Data Analysis
Role playing
Reasoning
Planning Matth
Math
Figure Distribution of task categories AlpacaEval, ArenaHard, and WildBench.
analyze the required background knowledge and reasoning capabilities for each task. These models assigned a difficulty rating on a five-point scale (from "very easy" to "very hard"). Tasks rated as "very easy" "easy" by all models excluded. From the remaining pool, randomly sampled 1,500 tasks to ensure that the distribution of task categories is similar to the original dataset.
Human annotation. To improve the quality of selected tasks, human annotation used for quality control. We first used GPT-4-Turbo to summarize the intent of each query. These summaries were then used to help human reviewers remove nonsensical tasks. Finally, we retained 1,024 tasks for WILDBENCH. We also manually reviewed the tasks to ensure that they were challenging and diverse, covering a wide range of task categories. For the checklist questions, we verified that they were clear, interpretable, and relevant to the evaluation of LLM responses.
Dynamic updates and data leakage prevention. WILDBENCH is designed be a dynamic benchmark that is updated regularly to reflect new types of user interactions. In fact, we have already released two versions the benchmark (V1 in 2024 March and V2 2024 May), with similar curation process but on different iterations of WildChat data. To prevent potential data leakage for LLMs that use WildChat as part of their training or alignment, we coordinated with the WildChat team to ensure that the tasks sample will not be publicly available in the WildChat dataset.
2.2 WILDBENCH STATISTICS
To better understand the composition of our evaluation, we analyze basic statistics and task categories.
Basic statistics. Table compares the statistics of WILDBEECH to existing benchmarks AlpacaE- val (Li et al., 2023b; Dubois et al., 2024), MT-Bench (Zheng et al., 2024), and ArenaHard (Li et al., 2024). Among these benchmarks, only ArenaHard and WILDBENCI are sourced from user queries in the wild ("RealUser"), rather than being curated by experts or through crowdsourcing. The difference between ArenaHard and our WildBench is that data distribution aligns with real users' task categories, rather than overly focusing on coding and debugging as ArenaHard does.
Long-context tasks. WILDBBENC includes conversation histories of up four turns per conversa- tion, reflecting complex and extended user interactions that are facilitated by recent advancements in LLMs, with over 20% of conversations having more than two or more turns shown Figure 8. Ad- ditionally, as shown in Figure 2, WILDBENCH has longer query lengths, attributable to the extensive context provided by real user interactions captured in the dataset. This because that GPT-4-Turbo, one of the chatbots behind WildChat, supports up to 128K context tokens and 4K output tokens. This capability exemplifies the importance of dynamic, In-the-wild benchmark: as models evolve, they unlock new user applications. Thanks to these realistic user activities, WIL.DBENCH is more suitable benchmark for testing the long-context problem solving abilities of L.LMs.
Task categories. To enable fine-grained analysis of LLM capabilities across varied tasks, we categorize the tasks into 12 categories based on previous analysis of ShareGPT queries (Ouyang et al., 2023) and our intent annotation of the tasks. Detailed descriptions about the 12 task categories are shown in Appendix A. The distribution of the task categories is shown in Figure 3. In this figure, we also compare AlpacaEval and ArenaHard. Notably, W WILDBENCH is more balanced compared to AlpacaEval and ArenaHard, which have over 50% of their tasks Information seeking and Coding & Debugging categories, respectively.
4

---

Published as a conference paper at ICLR 2025
History Query
LLM response
Json_output
"onalysts "[onslystx of Resporse
Pairwise
Model X vs (Baseline)
WB- Re-aaad
"analysts "[anslysia tesporse
+1 when KosY; +0.5 when X-Y;
"reason of Aad"; [[heere Response and perfore: equntyy"'
-1 when Xeef; -0.5 when X-t;;
Checklist
LEM 8 response
"[there Beagonte better than Al", Bee]" Aee mouns such zetver, An ncors alighely Better....
"reason of ava": "[herre Besponse is better than a)",
"reason BxA":
Length
"croice": "Dt+- As
Baseline Modala
RETAA L1aTe-2-203
History
Checklist
jsen_output
Individual
"ncainesses": "Conelysis strangthss",
"[anslysis
H WB-Score
Query
LLM response
nedenessess"
"store": "[2-18]"
Score 5-6: The response is Tate but has somc Essues (s.g... foctuol ornors, nolluctnstion stssiing key Informtion)). ..4
WildBench
swry)
Coding Dal waiiiggiingggg..............o 200 poi
3 User iannt fommu that notte
Chacklist
attion
0.964
retare aeavilila toomche sou couume 6 delb...
@ Doss the allerr formala provided correctly address the wsuts recd to fine the loo matching vakue specified solumn and returna
Charren 1ap Mard ww 1T:
Usare you proo ded the usset sammal
B the alemmatie oe formula syntacticol by cerrect and comp putmppbill with sprcedafest softwere such Microsof' Excel Google Shoets?
0.382
Areea
9.989
Score
Lowand
(Brmua is there an bermat farmua
Figure 4: Evaluation framework for WILDBENCH. There are two metrics: WB-Score for individual evaluation and WB-Reward for pairwise evaluation. The checklist used to guide the evaluation process. The length penalty is used to mitigate the length bias. WR-Reward and WB-Score both have strong correlations with human- based ranking of I.I.Ms on Chatbot Arena.
3 AUTOMATIC EVALUATION WITH WILDBENCH
In this section, we introduce the evaluation process of LLMs using WILDBENCH. We first explain how we generate checklist for each test query enhance interpretability and reduce evaluation ambiguity in WILDBENCH. Then, we introduce two automatic metrics: WILDBENCH- and WILDBENCH- Reward. Finally, we discuss how we mitigate the length bias in the evaluation process.
3.1 COTTANE-SPPECIEE CHECKLISTS
Powerful LLMs have been widely used judges evaluate the quality of LLM outputs in many automatic evaluation methods, such as AlpacaEval (Li et al., 2023b). However, even asking humans to judge which of the given two model outputs is better can be subjective and ambiguous. Moreover, such judgements provide limited information about the quality of the models. Without constant, interpretable, and comprehensive evaluation standard, the results can be noisy and hard interpret.
To address this issue, we generate a checklist for each test query in WILDBENCH to comprehensively evaluate the responses of different models. The checklist consists of 5-10 questions that are designed to be interpretable and casy to verify. We combine the responses of GPT-4-Turbo and Claude-3-Opus to finalize the checklists, thereby mitigating the bias of using a single LLM as the evaluator. These checklists have been manually reviewed and are used as part of the prompts for LLM judges to evaluate the responses of different models. An example of the checklist can be found in Figure Taking the G20 example in Figure 1, here is subset of checklist questions for the task:
Example checklist for the G20 task example Figure 1.
Does the essay contain more than 1200 words as requested by the user? Is the language of the essay beautiful and poetic, incorporating extensive vocabulary as specified?
Does the essay include significant amount of factual and empirical data related to the impact of the G20 summit on the global economy, trade, and development?
Are there references the role of young people in shaping the future of the world within the context of the G20 summit?
Does the essay include ancient Indian historical references as requested by the user? Is the essay structured in clear and logical manner, facilitating an easy understanding of the discussed topics?
3.2 PAIRWISE EVALUATION WITH
METRIC
WB- -BB-Rwaard is based on pairwise evaluation. which uses a GPT-4-Turbo judge to compare the responses of two LLMs to determine which one performs better given task, using structured
5

---

Published as a conference paper at ICLR 2025
checklist to guide the comparison. This metric provides straightforward comparisons among models and the intermediate outcomes of win/lose rates are easy to interpret.
Step-by-step evaluation process. In Figure 4, we detail the step-by-step evaluation process for pairwise comparison. First, we provide a chain of evaluation questions to guide the LLM judge to analyze the query and the conversation history. The LLM then evaluates the two responses and also analyze where and why one is better than the other. Finally, we ask the LLM to make a final judgment on which response is better and why. This method is inspired by the evaluation process in human evaluation, where human judges are asked to provide detailed feedback on the quality of the responses before making a final decision. The full evaluation prompt can be found Appendix D
WB-Reward metric. To compute the WB-Reward for a test model X against a baseline model Y, we assign rewards based on the comparison result: +1 X much better than Y, +0.5 if X is slightly better than Y, O for a tie, -0.5 for X is slightly worse than Y, and -1 for X is much worse than Y.
Baseline LLMs for pairwise evaluation. Using a single baseline model for pairwise evaluation can lead to noisy and biased evaluations. To mitigate this issue, we use three baseline models TGPT-4-Turbo-0429, Claude-3-Haiku, and -lamm-2-70B-chat (Touvron al., 2023)) to compute the rewards for each model. Our metric WB-Reward (Mix) is the average of the rewards from these three baselines on 1024 examples, providing a more robust performance evaluation on WILDBEENH.
Mitigating length bias with margin for ties. Previous studies have shown that LLM judges tend to prefer longer responses (Dubois et al., 2024). To mitigate this bias, we propose a simple and intuitive length penalty method. If the winning response is longer than the losing one by a certain threshold (K characters), we convert Slightly Win/Slightly Lose to a Tie. K can be customized via our leaderboard web-page for personalized configuration. Setting K = will disable the length penalty, We designed this feature to support a more personalized and flexible leaderboard. For example, users who prefer shorter and more concise outputs perfectly with the general human-basee model rankings on ChatbotArena, This choice allows for customized leaderboard experience depending on user preferences.
smaller K if they do not prioritize correlating
3.3 INDIVIDUAL EVALUATION WITH WB-SCORE METRIC
Although pairwise evaluation provides direct comparison between LLMs, it is usually more expensive and time-consuming than grading each individual LLM generation. To individually evaluate the performance each model on WILDBENCH, we prompt GPT-4-Turbo to assign a score from to 10 for each model's response. The full evaluation prompt can be found Appendix E.
Score definition. To ensure a stable and consistent evaluation, we ask GPT-4-Turbo to evaluate the quality each response based on the checklist and provide detailed strengths and weakness of each output before giving a score from I to 10. The scores are defined as follows:
Score 1-2: The response very poor and does not make sense all.
Score 3-:: The response is poor and does not help the user solve the prohlem meaningfully.
Score 6: The response is fair but has issues (e.g.. factual errors, hallucinations. missing key information).
Score 7-8: The response good but could be improved.
Score 9-10: The response is perfect and provides helpful information solve the problem.
Score rescaling. The WILDBENCH--S
is calculated as the average of the scores on all examples
tested, where each score is first subtracted by 5 and then multiplied by 2 (i.e., S' = (S -) x 2). A score of 5 represents a borderline acceptable response, so this rescaling can help to better differentiate the performance of models that can effectively solve the tasks.
RESULTS & ANALYSIS
We analyze the performance of different models on WILDBENCH. We first present the leader- board analysis, then examine the length bias issue in the evaluation process, and finally discuss the correlation between WILDBENCH-SEE Score and ChatbotArena Elo rating.
Leaderboard features. In Table 2, present a subset of the results from our live leaderboard demo. For the most up-to-date results and more interactive features, such as customizing length penalties and viewing the detailed task-wise performance of each model, please refer to our live leaderboard. Our

---

Published as a conference paper at ICLR 2025
Table 2:
Evaluation results (subset) of LLMs using WILDBENCH and other benchmarks. Please
refer to Figure 6-7 and demo website to view and interact with the full results.
WB-Reward (no length penalty)
WB-
Arena
Arena- Hard
AlpacaEval2
Model names
Mix *GPT4T *Haiku #Llama2
Score
LC
WR
GPT-4o-0513 A
35.7
1.5
46.3
59.3
65.3
1293
57.5
51.3
GPT-4-Turbo-0409 A
34.6
0
45.3
58.4
64.7
1251
82.6
55.0
46.1
GPT-4-Turbb-0025 A
29.9
-4.4
38.8
55.2
63.3
1239
78.0
Gemini-1.5-Pro A
27.8
-44
37,9
50
55.7
5
-lama-3-70B-Iie Inst
21
-19
31.9
50.2
60.4
1213
41.1
34.4
33.2
Claude 3 Opus
20.1
-20.4
34.3
46.3
63.1
1232
60.4
40.5
29.1
7
Gemini-1.5-Flash A
17.4
-16.6
26.3
42.5
53.1
8
Yi-1.5-34B-Chat
16.8
-18.3
24.1
44.5
57.8
10
Llama3-Ixt-8B-SmmmPP
14
-22.5
18.9
45.7
53.9
33.8
44.7
40.5
Claude 3 Sonnet A
7.2
-31.6
19.4
33.9
55.5
1187
46.8
34.9
25.6
@wenl.5--2B--hhir
4.4
-34.8
13.1
34.7
56.5
1143
36.1
36.6
26.5
17
Command R- -lPlus
0.4
-36.3
7.4
30.2
51.4
1155
33.1
20
Claude 3 Haiku
-8.5
-46.9
21.4
50.4
1169
41.5
Mistral-Lane Large
-0055
-48.1
20.5
54.2
37.7
32.7
21.4
StarlingI MM-7B-htte
-11.9
-48.7
18
46.8
23.0
Jlama-3-8B-Ine Inst
14.6
-49.8
-9.7
15.7
45.7
1144
20.6
22,9
22.6
25
Command-F R A
-16
-48.4
-12.7
13.1
45.7
1106
17.0
26
Mixtral-8x7B Inst
-18.8
-53.4
13..5
10,4
47.8
1114
23.4
23.7
18.3
27
DBRX Inst
-2.66
-57.3
-16.3
8.7
48.9
1106
23.9
25.4
18.4
29
Yi-1.5-6B-CII -24.3
-55
19,99
2.1
39.6
30
Mistral-7B-IImt-v0..2
-25
-58.1
-22.4
5.5
43.4
1071
17.1
14.7
Tulu-2-dpo-70b
-25.4
-59.3
20.3
3.3
45.2
1099
15.0
21.2
16.0
33
Llama- Llama-2-700-ce 2-70B-chat
-26.8
-56.9
-2.66
0
39.2
1070
11.6
14.7
13.9
34
Qwenl Qwen1.5-7B-CHB 5--7B-hat
-27
-57.7
-23
-0.2
40
1059
14.7
11.8
35
Phi-3-medium-1222
128k
-33.3
-66.4
-30
-3.6
42.1
trr3.55turbo-0125
-33.5
-66.3
-4.1
42.1
1105
23.3
Llama 2-7B-chat
.48
-71.8
-44.6
-27.8
27.6
4.6
5.4
5.0
Giemaa-7B-.
-57
-78.4
-55.8
-36.8
23.9
1047
7.5
10.4
6.9
40
Gemma-22B-
-74.1
-87.8
-73.6
-60.8
6.2
980
3.0
5.4
3.4
live leaderboard also supports exploring data and comparing model outputs side by side to understand the strengths and weaknesses of each model.
By using three baseline models of varying performance levels (GPT-4-Turbo Claude 3 Haiku > 7-0-cch-) we observe that the tested models can naturally grouped into three tiers based on their performance. Tier models outperform Claude 3 Haiku, Tier models outperform Llama-2- 70B-chat but are worse than Claude 3 Haiku, and Tier models are worse than -lama-2-70B--n-
4.1 LEADERBOARD ANALYSIS
Where are the gaps between models? A unique feature the WILDBENCH leaderboard the ability compare models across different task categories, which enables us identify the strengths and weaknesses of each model on different types tasks. In Figure 5, we select a set of popular models for analysis: Llama-3-BB-Ins (Meta, 2023), lama-3-8B-Inst-Smmlee Yi-1.5-34B-chal (AI et al., 2024), Llama-3-70B-Ing
(Meng et al., 2024b), GPT-4-Turbo-0400 and Claude 3 Opus. We show their performance in WB-Score across five task categories (merged from the 12 categories shown in Figure 3). Larger models like (1r-4-Turbo-000 and Claude 3 Opus perform well across all task categories. while open LLMs like Lamma-3-BB-ll and F1-1.5-34B-char show weaker performance on coding and math-related tasks.
Will an 8B model outperform 70B model? On the 21aacaEval-2.0 leaderboard, Llama-3-8B. Inst-SimPO (LCC=4.7%% significantly outperforms Llama-3-70B- (LC=34.4%) (Meng et al., 2024a), which surprising and differs from our results. As shown both Table and Figure 5, our results indicate that Llama-3-8B -aana---B-IIns----- Inst-n---SimPO is generally still worse than Yi-34B -catt and Llama-3- 70B-Inst. However, on information seeking and creative tasks, Llama-3-8B. --na----BB-mmm- L-s----st-SimPP performs comparably to Llama-3-70B- -nst.nst. Thus, believe AlpacaEvall 's evaluation results underestimate the performance of Alama-3-77B- due to task selection bias in addition to the weakness their evaluation prompting method. While the performance of -lama-3-8B ---------------- --s-BB-----tiim Inst-SimPO is not good it

---

Published as a conference paper at ICLR 2025
Table 3: Correlation with Chatbot ArenaElo Hlo (H4rd-En-240520) of alignment benchmarks.
Metric
p-Corop
P-Coran
S-Corall
K-Corai
Metric
P-Corop
P-Corai
S-Coral
ArenaElo (Hard-En)
1.000
1.000
1.000
1.000
Avg Length
0.472
0.554
0.376
Arena-Hard
0,909
0.925
0.965
0.890
W WB-Reward flima
0.976
0.965
0.965
AlpacaEva12-LC
0.892
0,951
0,924
0.818
WB Reward
0.974
0.961
0.965
AlpacaEval2
0.865
0.952
0.960
0.868
WB-Reward hrik
0.985
0.974
0,982
WB-Score
0.955
0.940
0.943
0.846
WB- Re
0.977
0.969
0.961
WB-Reward
0.984
0.973
0.978
0.912
W WB-Reward5j0o
0.992
0.973
0.969
WB RB-Reward500
0.984
0.976
0.974
0.912
WB- 1B-Rewardg00
0.973
0.976
0.974
seems on AlpacaEvvl-2.0, it is indeed the best 8B model in our evaluation and outperforms some 3--8B-Int--SmPOO consistently improves the performance of Alama-3-8B-Ing --stttt on all task categories, resulting in similar shape on the radar plot in Figure 5.
other larger models. Interestingly,
Are longer responses always better? WILD- BENCH is robust to length bias. For example,
Reasoning & Planning
have similar output lengths (2,965 vs 2,983 chars), yet Llama-3-70B-Ins ranks 5th while Llama-2- 70B-chat ranks 33rd on the leaderboard of 40 models. Additionally, Yi-1.5-6B s output length is the 4th longest among the 40 models (3,322 characters), but it ranks 29th on the leaderboard. This suggests that the WILDBENCH evaluation is not biased towards longer responses, with re- sponse quality being the most important factor in the evaluation process. Additionally, we use length penalty to ensure that longer responses are not always favored, and users can customize the length penalty to adjust the trade-off be- tween response length and quality according to their needs. This feature is available on our live leaderboard and is illustrated in Figure 6.
-lama-2-70B-chhe and Llama-3-70B-
32
Creative Tasks
Info Seeking
Coding Debugging
Limra 3-100immmr
Figure 5: Performance breakdown by task category of 6 models on WILDBENCH.
4.2 CORRELATION TO HUMAN JUDGMENT
To analyze how well WILDBENCH evaluation
correlates with human judgment, we compare our results to the ChatbotArena Elo rating generated by large-scale online human evaluations. Focusing hard prompts, use the Elo ratings from the Hard-Enn Hard-Englishh version released on May 20, 2024.
We compare our WB-Reward and WB-Score with three other metries: AlpacaEval winrate (WR), length-controlled winrate (LC), and ArenaHard scores. We use three correlation metrics: Pearson correlation (P-Cor), Spearman correlation (S-Cor), and Kendall's a fair comparison, we consider all models that have all four metrics available in Table 2, which results in 14 models. To distinguish the top top-performing models, we also consider the top 6 models, denoting their correlation metrics as P-Corup- and P-Coran respectively. The reason why we care about the correlation on top-ranking models is that models released in the future are likely to compete with the top models, so the Pearson correlation in this range is more important from the perspective of predicting the future application of a metric. The analysis results are shown in Table 3.
correlation (K-Cor). To ensure
Both WB-Reward and WB-Score show strong correlations with the basen Elo rating, par- ticularly for the top- peppefforrmng models, achieving the best correlation among all other automatic metrics. Among using different baseline models for pairwise evaluation, find that using Haiku as the baseline model yields the best correlation. These results suggest that the WILDBENCH evaluation correlates well with human judgment in ranking model performance automatic metric.
8

---

Published as a conference paper at ICLR 2025
4.3 ABLATION STUDIES AND DISCUSSIONS.
Checklists. In our ablation study on the impact of checklists, we compared model performance with and without checklists by removing the associated parts from the prompt templates. The results indicate that incorporating checklists improves the final correlation with human preferences. Specifically, the WB-Score without checklists achieves a Pearson correlation of 0.905 (for all models), which is lower than the 0.925 correlation achieved when using checklists.
Length penalties, We experimented with different K (100, 200, 500, 1000, inf) in the length penalty method. We found that K 500 is the best choice, as it achieves the highest correlation with human judgments. This result suggests that the length penalty method is effective in mitigating the length bias in LLM evaluations.
Do multiple LLMs as judges help? How much do multiple LLMs help? We experimented with using GPT-4 Claude 3 Opus, and Mistral-Large as LLM judges. Our experiments revealed that these LLM judges produced very similar results, thereby exerting minimal influence on the final relative ranking of LLMs. Considering to reduce the cost of evaluation and faster turnaround time, we recommend using a single LLM judge in practice. In the future versions, we will explore more efficient ways to use multiple L.MSs as judges, for example, by using different judge L.MSs for different tasks that are best suited their strengths.
Data distribution. How do we explain that WildBench has a different distribution compared to ChatbotArena's 's platform but still shows strong correlation, even better than ArenaHard? The objective of WildBench is to evaluate LLMs on challenging tasks from real users. The ArenaElo we use for comparison is derived from the Bard-Englishi split in ChatbotArena, where human users submit tasks and vote. Thus, both WildBench and ChatbotArena aim to address the same goal. While it is practically impossible to match the exact distribution of users and tasks between the two-given that WildChat users are anonymous and ChathotArena does not publicize its data-both are sourced from real users on the web. Consequently, this represents the best possible approach for correlating our LLM ratings with human-based ratings.
Two complementary metrics: WB-Reward & WB-Score. Both metrics use checklists and a CoT--ttyle prompt for evaluation, utilizing the same testing data. The key differences are in their methodologies: WB-Score: Evaluates each model's outputs individually on a scale of 1-10, with detailed explanations for each score (see Appendix); WB-Reward: Compares model's outputs to those of three baseline models at different performance levels for a comprehensive evaluation. Pairwise evaluations can be coarse, but using three baseline models and refined pairwise choices (e.g.. much better or slightly better) mitigates this. WB-Score provides a universal score comparable across models using the same evaluation templates and checklists. Additionally, WB-Score is cheaper and faster to run (10 minutes, $5) compared to WB-Rewar Reward, which requires 3-4 times the cost due to multiple baselines. Both metrics have their strengths and weaknesses. We use both build our official leaderboard, allowing users to choose the most suitable metrics for their experiments.
RELATED WORKS
Close-ended benchmarks. Close-ended benchmarks typically consist of multiple-choice questions and have been widely used to evaluate LLMs authors (2022). For example, MMLU (Hendrycks et al., 2020) includes multi-choice questions across various subject areas. Its variants include CMMLU (Li et al.. 2023a) for Chinese, KMMLU (Son et al., 2024) for Korean, and MMLU-Pro (Wang et al., 2024) for more challenging evaluation. GPQA (Rein et al., 2023) is another close-ended benchmark designed to be challenging even for humans with internet access. Specialized benchmarks with ground-truth answers, such as GSM8K (Cobbe al., 2021) and MATH (Hendrycks et al., 2021), 1), also fall into this category. While these benchmarks focus close-form answers, our work evaluates LLMs' ability to generate free-form responses and engage in conversations with users.
Expert-curated and crowdsoureed data. Several open-ended generation benchmarks rely on data curated by human experts or crowdsourcing workers. For instance, MT--enod (Zheng et al., 2024) manually creates examples for predefined categories. AlpacaEval (Li et al., 2023b) is based on written examples (Dubois et al., 2023; Taori et al., 2023; Wang et al., 2022), which primarily consists of simple instructions such as rewriting tasks.
9

---

Published as a conference paper at ICLR 2025
In-the-wild data. A key feature of our work is that its underlying data is sourced from real- woold use cases, ensuring alignment with actual LLM use cases. Notable benchmarks using real-world data include ChatbotArena (Zheng et al., 2024; Chiang et al., 2024), where users input their questions and choose the better response from two LLMs. However, ChatbotArena relies on extensive human feedback. WildVision (Lu et al., 2024) is similar project but designed for vision language models. ArenaHard (Li et al., 2024) is another work that selects user queries from ChatbotArena to construct a benchmark for automatic evaluation.
Evaluation methods. Evaluating open-ended generation poses challenges due to the lack of a single valid ground truth. Human evaluation, though reliable, is expensive and consumin.. To reduce costs and enable fast evaluation, powerful LLMs are often used as judges, as seen in benchmarks like MTT-Bench, AlpacaEval, ArenaHard, and our own. Evaluation methods include single-system grading, which assigns scores to individual outputs, and pairwise comparisons, which compare outputs of two systems to compute win rates. Pairwise comparisons, while more expensive, can highlight subtle differences across systems (Zheng et al., 2024). To mitigate self-selection bias where an LLM prefers its own outputs (Panickssery et al., 2024), we use checklists generated from multiple LLMs, similar to InfoBench (Qin al., 2024). In addition, we ask LLM judges generate structured explanations that enable human verification for further calibration, inspired by Just-Eval (Lin et al., 2023). There also local evaluators that be used to evaluate LLMs with our WILDBENCH with open-weight LLMs, such as TIGERScore (Jiang et al., 2023) and Prometheus (Kim et al., 2024).
Data leakage prevention. Publicly available benchmarks risk contamination from LLMs trained on such data. GPQA includes a special string to help LLM developers filter out its data (Rein et al., 2023), yet indirect leakage through cited examples remains possible. To mitigate this, we reserve subset of WildChat that is never released publicly, which keeps its expert-curated evaluation data private. However, WILDBENCH provides public validation set and details the benchmark construction process for greater transparency.
Other dimensions for evaluation. While our focus on evaluating LLM capabilities, other evaluation dimensions, such safety (Mazeika et al., 2024; Jiang et al., 2024), fairness (Gallegos et al., 2024), logical reasoning (Lin et al., 2024), agentic planning (Liu et al., 2023; Mialon et al., 2023; Lin et al., 2022), and hallucination detection (Min et al., 2023; Mishra et al., 2024; Hong et al., 2024), are equally important.
6 CONCLUSION AND FUTURE DIRECTIONS
In this work, we introduced WILDBENCH, benchmark designed to evaluate LLMs using real- world user queries. An important feature of WILDBENCH data is the nature of in-the-wild user queries with natural task distribution. To evaluate LLM performance using the collected data, we introduced CoT--like LLLMassjudge method to improve the interpretability of evaluations and reduce ambiguity. We also incorporated a length penalty method to mitigate the length bias in LLM-as-judge evaluations. Experiments show that our primary metrics, WB-Reward and WB-Score, have very strong correlations with human judgments, surpassing existing evaluations.
We present extensive experiments and analyses, showcasing the performance of a wide range of 40 LLMs, including both proprietary and public ones, on the WILDBBENC benchmark. By providing a detailed breakdown of scores across different task categories, WILDBENCH offers insights the strengths and weaknesses of different models. By introducing WILDBENCH, we aim to provide realistic, dynamic, and contamination-reslleen evaluation framework that accurately reflects the capabilities of LLMs. We will actively maintain the project for continually evaluating new LLMs with unseen tasks over time.
REFERENCES
01. AI. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.
10

---

Published as a conference paper at ICLR 2025
Anthropic.
The claude 3 model family: Opus, sonnet, haiku. https: https://www-cdn ant.hooppi.. anthropic. com/ de aba 9b01 f 3 00011551111111110111111177777771 2at.://11888666222oeeeeeeeee Caau_ pdf, 2024.
The BigBench authors. Beyond the imitation game: Quantifying and extrapolating the capabiliitiss of
language models. ArXiv, abs/2206.04615, 2022. URL httttt////ppp semant
org/Corpus 26622881..
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng. Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang. Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena: An open platform for evaluating IIms by human preference, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Relichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint
I..1::2100.1466,, 2021.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305. 14387, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2024.
Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernon- court, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and fairness in large language models: A survey, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint wrXiv:2009.03300 2020.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 10t::::003..877... 2021.
Giwon Hong. Aryo Pradipta Gema, Rohit Saxena, Xiaotang Du, Ping Nie. Yu Zhao, Laura Perez- Beltrachini, Max Ryabinin, Xuanli He, Clémentine Fourrier, and Pasquale Minervini. The hal- lucinations leaderboard an open effort to measure hallucinations in large language models, 2024.
Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. Tigerscore: Towards building explainable metric for all text generation tasks. Transactions on Machine Learning Research, 2023.
Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. ArXiv, abs/2406. 18510, 2024. URL htt.h:tp////... //ppi.appp///////////
org/ Corpus 270738096.
Scungone Kim, Juyoung Suk. Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. ArXiv, abs/2205. 11916, 2022. URL https: //api.
semant wwchhllr. org/ httgg//orpuuuaa ID: 9017743.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.0021 /2, 2023a.
Tianle Li, Wei-Lin Chiang. Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard hard pipeline, April 2024, URL https: / 1mays org/b ooloog//2000 220/22224-2222 04 100--19aaree aaraaaahaaad
11

---

Published as a conference paper at ICLR 2025
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction instruction-followinn models. https://githhli htt.:/aa....m/ta.aaaaaaaaaaaaaaa
_eval, 2023b.
Bill Yuchen Lin, Chengsong Huang, Qian Liu, Wenda Gu, Sam Sommerer, and Xiang Ren. On grounded planning for embodied tasks with language models. ArXiv, abs/2209.00460 00460, 2022. URL https: / /api. semann et csschol .r. Co/ Corpus TD 251 979509.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Raghavi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base Ilms: Rethink- ing alignment via in-context learning. ArXiv, abs/2312.01552 2023. URL https: //api.
semant wwc.nticscclaaa.. wwwpollarrrrrgoppooiooeeo wtrg//Corpssipa 265:::I:::66688000
Bill Yuchen Lin, Ronan Le Bras, and Yejin Choi. Zebralogic: Benchmarking the logical rea-
soning ability of language models, 2024. URL htttps://h.. co/ spaces/ lenai/
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Yuxian Gu, Hangliang Ding, Kai Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Shengqi Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating Ilms agents. ArXiv, 0le/2308.03666 2023. URL https: semant www.cccccoa....oooooppooeeeeeeee 260682249.
Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wildvision: Evaluating vision-language models in the wild with human preferences. arXiv preprint 24K1::2006.10666 2024.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standard- ized evaluation framework for automated red teaming and robust refusal, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. 2024a. URL https: //aps:///pp....e swwanti www.ccccoll...oooooooooooo ar 269983560.
Corg/CoppssI::
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference- free reward, 2024b.
Meta. Introducing Meta Llama 3: The most capable openly available LLM to date. https:
//ai. meta wtt.cccmmolll.. wwwmbo/g///etaaaaa///ooooo/oaaaa 11ama 3/ 3/, 2023.
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann André LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. ArXiv, abs/2311.12988, 2023. URL
https: /ttt::://ap..semmniiiccooo
exemanticscher ar. orgg/oorpsID::653333 664.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305. /4251, 2023.
Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grained hallucination detection and editing for language models, 2024.
OpenAl. Gpt-4 technical report, 2023.
Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, and Jiawei Han. The shifted and the overlooked: A task-oriented investigation of user-GPT interactions. The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https:ttp............ / reperreviie ew .wt. forum? d=qS1 p2dGHO.
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Lim evaluators recognize and favor their own generations, 2024.
12

---

Published as a conference paper at ICLR 2025
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang. Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models, 2024.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert-networks, 2019.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-prodt q&a benchmark, 2023.
Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. Kmmlu: Measuring massive multitask language understanding in korean. arXiv preprint a5Xiv:2402.11548, 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An moslluction-followingg Ilama model. https: /ttg//itttti github.hhbb...e com/ tatsu-lab/ stanford_
2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith. Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint 22 10560, 2022.
Yubo Wang, Xueguang Ma, Ge Zhang. Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang. Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: more robust and challenging multi-task language understanding benchmark, 2024.
Wenting Zhao, Xiang Ren, John Frederick Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wild- chat: Im chatgpt interaction logs in the wild. 2024. URL htts:::: /api.. semant icolar org/CorpusID: : 269390491
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging Ilm- judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Appendix
A TASK CATEGORIES
In Section 2.2 we mentioned that tasks are categorized into 12 categories to enable fine-grained analysis of LLM capabilities. The definition these task categories are as follows.
Information seeking Users ask for specific information facts about various topies.
Reasoning- Queries require logical thinking, problem-solving. or processing of complex ideas.
Planning Users need assistance in creating plans strategies for activities and projects.
Editing Involves editing, rephrasing, proofreading, or other tasks related to the composition of general written content.
Coding & Debugging- Users seek help with writing, reviewing, fixing code in programming.
Math Queries related mathematical concepts, problems, and calculations.
Role playing Users engage in scenarios requiring ChatGPT to adopt a character persona.
Data Analysis- Requests involve interpreting data, statistics, or performing analytical tasks.
Creative Writing Users seek assistance with crafting stories, poems, or other creative texts.
Advice seeking Users ask for recommendations or guidance on various personal or professional
issues.
Brainstorming Involves generating ideas, creative thinking, or exploring possibilities. of miscellaneous nature.
Others Any queries that do not fit into the above categories
We consolidate the original categories into five major groups for easier task-wise analysis. Specifically, we combine "Information seeking" and "Advice seeking" into "Info Sceking": "Math" and "Data Analysis" into "Math & Data"; and "Reasoning" and "Planning" into "Reasoning & Planning." The remaining types are grouped under "Creative Tasks." " These consolidated groups are illustrated in Figure 5.
Please note that the following links are allenai for double-blind review, which we will update after the review process. The supple- mentary zip file contains the source code for the evaluation scripts, the leaderboard, and the data.
Figure Distribution of number of turns in WildBench.
700
B MORE INFORMATION ON WILDBENCH DATA
500
The distribution of the number of turns in WILDBENCH can be found in Figure The dataset documentation, metadata, and the public sub-
400
set of WILDBENCH can be found https: www.ggggggaaeeeeeeeeeeeeo co / /adaaaaaeee//eeeeee saaaetssaall a lena /Wi dBench/vv ewer/v2. We re- low-risk artifact,
300
lease the data under AT2's ImpACT license and we bear all responsibility in case of rights violations. We will ensure that the dataset will be available for a long time and maintain the data by continuously updating it.
200
100
1
4
5
C MORE INFORMATION ON WILDBENCH EVALUATION
Our evaluation results on the public subset of WILDBENCH can be reproduced using evaluation scripts
available https: / /ii tuub. cwm/ homm/lleeaaii Metenai/Wille http///eidd/c///////c/emmem We have included generation script for each model under the folder https: githhub. com/ allanai, /tw./W/Beeennn miildench/ttre/ main/ main/scripts. and the scripts for evaluating generations can be found at hattps://giii com/allenai/Wal http:///wwiieechheeeeeee rre /ttee/aannn//eeeeeeeeeeeeeeeeeee wwt.n//vallaaiionoo
D PROMPT TEMPLATE FOR PAIRWISE EVALUATION METRIC WB-REWARD
The prompt template for pairwise evaluation shown below. can be divided into three sections: the first section provides the high-level instruction, the task be tested, and two model outputs; the
14

---

Published as a conference paper at ICLR 2025
second section specifies the checklist and the rules; and the last section instructs the LLM judge to follow the step-by-step evaluation process as detailed in Section 3.2
Instruction on
You are expert evaluator. Your task is evaluate the quality of
the responses generated by two AT models. we will provide you with
the user query and pair of AI-generated responses (Response A and
B). You should first read the user query and the conversation analyzing the task, and then evaluate the quality of the responses based on and rules provided below.
history carefully
Conversation between User and AI
History
(lbegin_of_biston
(Shistory) clent_os_hissorrl>
Current User Query
<lbegin_ofi_ueryl> (Suser_query) oeen_yy__uurr/>>>>
# Response
< [bbegin_of_respons__A11 (Scandidate_A) <end_oo__rsspoos__Al>>>>>>
# Response
< [Scandidate_B)
_eed__o_______ send_of_reeppee siddoofreepoose__ B|>
Evaluation
# Checklist
<|begin_of_chese |>
(Schecklist)
<|cnd_of_checkkist
Please this checklist guide your evaluation, but do not 1imit your assessment the checklist.
# Rules
You should compare the above responses based your analysis of the user queries and the conversation history, You should first + write down your analysis and the checklist that you used for the evaluation, and then previde your assessment according to the five choices to give your final assessment: ["At*", "At", "A=B", "Bt", "B++"], which correspond to the folloving meanings:
checklist, There
AA++; Response A is much better than Response 3.
"A+": Response is only slightly better than Response
"A=B": : Response and are of the same quality. Please use this choice sparingly.
"B+": Response B is only slightly better than Response A.
"B++": Response is much better than Response A.
15

---

Published as a conference paper at ICLR 2025
# Output Format
First, please output your analysis for each model response, and then summarize your assessment to three aspects: "reason A-B", "reason A>B", and "reason B>A", and finally make your choice for the final assessment.
Please provide your evaluation results in the following ison format by filling in the placehelders in []:
"analysis of A": "[analysis of Response Al",
"analysis of =": "[analysis of Response s]",
"reason of A-B": "[where Response and в perform equally welll",
"reason of A>B"; "[where Response is better than Response B]",
"reason of B>A": "[where Response is better than Response A]
"choice": "[A++ or At or A=B or B+ or B++]",
E PROMPT TEMPLATE FOR INDIVIDUAL EVALUATION METRIC WB-SCORE
The prompt template for individual evaluation shown below. It can be similarly divided into three sections: the first section provides the high-level instruction, the task to be tested, and the model output; the second section specifies the checklist and the rules; and the last section instructs the LLM judge to follow the step-by-stey bt-ptep evaluation process as detailed in Section 3.3.
# Instruction
You are an expert evaluator. Your task to evaluate the quality + the responses generated by AI models.
We will provide you with the user query and an AI-generated responses. You should first read the user query and the conversation history
carefully for analyzing the task, and then evaluate the quality of + the responses based on and rules provided below.
# Conversation between User and AI
History <lbegin_of_historyy>>
(Shistory)
<lent_of_historryl>>>>
## Current User Query <Ibegin_of_quezy)
(Suser_query)
< <lend of >
## AI Response < <f
(Smodel_oupput)
<lend__o_reesponse]]
16

---

Published as a conference paper at ICLR 2025
Evaluation
|| Checklist
<beggin_ot_chccklsstt
($checklist)
< [f
Please use this checklist to guide your evaluation, but do not limit + your assessment to the checklist.
Rules
You should compare the above response based on your analysis of the user queries and the conversation history.
You should first write down your analysis and the checklist that you
+ used for the evaluation, and then provide your assessment according + to the checklist.
The scores are in the range of 1-10, where means the response is very poor and 10 means the response is perfect. Here more detailed criteria for the scores:
Score 1-2: The response is very poor and does not make sense at all.
Score 3-4: The response is poor and does help user solve the problem in meaningful way.
Score 5-6: The response is fair
has some issues (e.g., factual
errors, hallucinations, missing key information).
+
Score 7-8: The response is good enough but could be improved in some ways.
Score 9-10: The response is perfect and provides helpful information that can help user solve the problem.
## Output Format
Pirst, please output your analysis for the model response, and then summarize your assessment to two aspects: "strengtss" and "weaknesses"; Finally, please write down your rating for the assessment.
Please provide your evaluation results in the following jsen format by filling in the placeholders in []:
"strongths"; "[analysis for the strengths of the respense]", "weaknesses"; "[analysis for the weaknesses of the response]", "score": "[1-10]"
F FULL WILDBENCH LEADERBOARD
The full WILDBENCH leaderboard as of Jun 5, 2024 can be found in Figure 6; The updated leader- board as of Sept 2024 can found Figure 7. Note that we used new metric named WB-Elo that is based on merging WB-Reward and WB-Score to collection of pairwise comparisons and perform Elo rating updates on top existing LMSYS Elo rating, thus we can have a faster and more stable leaderboard update. You view and interact with the latest results website at https: //hgghiuan in ng aace. co/ spaces / a lenai / dBench
Ieaderboard on
17

---

Published as a conference paper at ICLR 2025
WildBench: Benchmarking LLMs with Challenging Tasks from Real Users the Wild
GitHuus HurgiinnEasee
[ Discussions
Version: V2 Examples: 1024
Models:
Leaderbbordd
W Details
Q Explore
Evaluate
About Us
Main
o Llamaz-708
WB Reward for each pairwise compatison. reward for is +/4l is much better/worse then and +/-0.5 if slightly better/worse then 0 Tie. base ine models are GPT4-Turbo, Helku, and Lleme2-70B, and Mix average thrre. Score indikidually ly scores model on Evaluator CPT-4-Turro
Langth Margin
pensitty)
To miligate the length bits, we cons der it Tie only alightly better than langer than more then chowr
Type
Open Source Models Cnlyy
Goward
Mocroligg
newlyy
Vodel
Remard- (Avg)
Reward-
2 WB
or
@'
LMSYS
Aren
AE2-
AE2-
Scoze
Bit:-lo-200::00::
35.7
06.3
1-5
99.3
1293
57.5
$1.3
3496
Aeminh..A.
27.8
20.9
55.7
-6.0
37.9
2968
21
22.7
68.4
-9
22.9
39.2
1213
41.1
34.4
33.2
2965
Cloude. 3.ao
20.1
21.7
63.1
-20.4
$4.3
46.3
1232
60.4
48.5
29.1
2606
Gendod, J..5 Flash
27.4
19.9
53.1
-15.6
26 3
42.6
3184
57.8
-18.
24.1
Yi-Large
Josta::
2585
Llenad:Cost: GioED
12.5
10.5
63.5
42.5
2470
12
22.5
12.3
68.2
-24.5
22.8
48.3
2706
13
Clause. J. Smcont
2.2
9.6
56.5
-31.6
10.4
33.9
1::07
16.0
31.9
35.4
2536
14
Owend, G-77
2.2
36.9
-37.3
12.6
36.6
1:43
36.1
30.4
26.5
2303
Ce902:748:44kk Lost
9.1
56.8
-35.7
12.7
33.4
2764
-35.5
7.4
3859
15.6
-7.3
47.8
-05.8
1.7
25.2
2762
19
SELM (Zeehyx: ARt33)
66.8
+9.8
46.9
-39.6
-3.1
22.2
24
2736
28
Glasode... 3. Huikut
8.55
16.9
58.4
-45.9
8
21.4
11.99
41.5
2442
21
sisstral:Large
-1..6
-1..2
64.2
-08.1
-4
28.6
11:8
37.7
22.7
21.4
2464
-11.3
-12.2
48.2
-47.9
-6.6
29.7
LJamo.2:99:I
-14.6
+89.8
15.7
11:66
2834
25
GommanddJ
-16
18..6
45.7
-65.4
-32.7
13.1
11.05
2748
26
Mixteal wtw.alll-lMll-eetttttooo
-18
155
47.8
-51
-33.5
14.4
1:14
21.4
23.7
38.1
7558
27
0808.Inxtaae Instauct
21.5
22.4
48.9
57.3
126.3
0.7
1106
23.9
25.4
18.4
2525
-2..5
22.6
45.1
+57.9
137.2
2630
IA:11 S.6.:Chttt
22.33
25
39.6
-29.9
3322
-55
-25.4
-44
46.1
-59.5
-28
1048
269h
22
1:d0:2:002.999
-25.4
26.5
45.2
-99.3
-28.3
3.3
1099
21.2
15
2658
33
L. Llana-2-70D-chat
-26.3
25.66
39.2
-56.9
-23.6
0
1070
11.6
24.7
23.9
2983
Depa1 Sesa1_.::8::00 -6:78:0095
-27
22 2
48
+57.1
-23
+8.2
1059
14.7
11.8
2474
-33.3
-0.2
42.1
-66.4
-30
-3.6
3572
502:3 uo::9..:fucco -tt.:bo--ooo0002
-33.5
-2.77
42.1
-65.3
-39
Llamn shetttt
-40
.51
27.4
-71.8
-44.6
-27.0
1012
5.4
2037
59
Semm::2::1
-57
157
23.9
-78.4
+55.8
-36.8
1047
7.5
18.4
6.9
1724
40
-14.1
-8.4
6.2
-87.8
-73.4
-60.8
V8U
b.9
3.4
15/8
Figure 6: Leaderboard of WildBench (2024 Jun 5th)
18

---

Published as a conference paper at ICLR 2025
id Natth
Model
KB-Elo (LC)
Indo Seak
Creative
Code
Debug
Data
Roason
Scoze
(Row)
ND-E2o
Lon
5227.4
62.0
60.5
57
00.2
59.3
3226.7
3723
Clasde Noonet
1235.4
56.5
30.2
59.5
54.7
3223.7
2911
gat. Buzzoo 2024 dd
1299.0
57.2
98.7
55.1
53
90.2
15.2
2237.1
3993
1300.9
57.7
07.4
64.3
2226.1
2012
Dossteoki 10022
API)
32.7
35.4
1287/2
5052
1397.3
64.4
67.4
$2.4
45.8
53.5
$2.3
3766.
3205
2280.5
Aemtol, Clasb
48.7
81.7
48.7
48.9
3306.3
2404
LLame 728 Iraiauct
1387.9
52.3
24.3
44.7
42.1
47.8
2275.0
3346
22
Yi-Leage
1183.9
53
91.8
47.7
44.5
92.3
45.9
2386.
1995
53
5169.7
69.6
2196.
1105
Remalta AJSASS Joal
53
55.3
46.3
48.8
47.7
1382.
2154
25
P Gertacã-2tleitt B
1576.4
$2.5
53.4
47
43.9
90.4
48.5
1101
2004
Ctauor.1 Scroet
47.1
66.3
46.1
48.6
45.6
3376.4
2004
Auruinn 2004 15
1375.5
37.4
66.3
51.2
57.1
2395.2
1640
Deenn Inst
1376.7
2666
23
@ Jaba, Gasu
1370.4
32.3
23.5
48.0
40.3
48
43.9
3379,1
2592
$566.6
66.6
G1
08.9
48.6
12.2
5
21
gerna 2:70
1366.6
52.2
20.1
20.5
40.1
53.2
1386.4
3902
23
YL-1 Jiat
1359.6
58.3
63.6
42.1
39.4
49.1
45.6
1164.4
1623
25
Nistral: dest. (1291
1158.6
$3.9
$4.6
19.7
86.6
47.4
44.4
1366. 9
8558
28
2257
46.1
47.7
13.7
35.9
2259.3
2014
Sewao-2:00:11
21
Crerazz: Bu7taa
1151.4
41.2
52.4
20.4
23.5
40.5
16.0
2353.3
3293
OLtc stat
2363,9
32
48.9
49.2
33.7
39.3
1354,0
3307
31
Mir scttuchat
3348
42.6
85.6
32.2
42.4
16.7
3166.2
1468
33
Llannd trat 49 -0
1347.4
47.9
91.8
21.5
24.4
68.3
37.2
3381.3
2633
21
Zwend Aoto
2247.4
48.2
59.4
25.4
28.6
30.9
2150
2392
35
SpMt. (1J000 SRR Abuai:
3344
46.1
51.1
27.3
23.5
37.5
35.3
3340.
2013
31
Liatee Blaoec--t-rrddaus
39.3
43.6
30.2
3338.6
2015
34
Mar Aarmot-2-Thet Thet--- 1oma-1-89
1539
41.6
84
28.1
14.9
33.3
36.0
1189 7
3747
41
SE1s :Dostyo:00:iteet
1534.
43
44.7
11
12.9
31.5
25.1
5326. 5
2009
42
2232.7
41.5
42.4
22.1
22.5
35.4
1132.1
2103
42
Comuu
39.9
45.6
27.9
25.8
27.8
33477
41
Aet:alGetuuthee0122
1329.2
36.5
37.4
26.5
23.6
33.4
30
1322.7
1004
Jestauct
1329.4
41
12.0
33.4
23.4
3116
4100
47
Seu 28 Itoinuck
1326.2
35.5
90.5
$1.4
25
1122.1
5735
47
@ 13
1325.6
44.1
47.4
29.3
26
34.5
29.5
3325.3
2019
52
2324.7
41.9
42.8
31.5
3322.4
2603
51
YA. 5 Chok
31.4
91.1
26.6
26.8
3338.3
3999
52
2321
42.7
42.7
28.7
34.6
32.3
28
2319.1
2400
54
43.1
43.3
18.4
25.6
3306.1
2002
23
Llane chut
1353.9
38.3
42
9.3
20.7
3099.2
3130
3055.8
$9.8
57.9
28
23.8
$4.2
39.7
3885.0 0
2814
54
3696.5
2486
3079.2
12.7
21.2
1.8
3054.3
3726
5852.6
37.7
29.8
-6.8
-3.2
15.4
8.3
1864
2065
Figure 7: Leaderboard of WildBench (2024 Sept 1st)
19


Reviews:
[{"review_id": "gbSW2lk8cX", "reviewer": "Reviewer_LYdL", "summary": "The paper introduces WILDBENCH, a novel benchmarking framework designed to evaluate large language models (LLMs) using challenging, real-world user queries. It consists of 1,024 examples selected from over one million human-chatbot conversation logs. The framework employs two evaluation metrics, WB-Reward and WB-Score, to provide reliable and interpretable assessments of LLM performance. By using task-specific checklists and a dynamic update mechanism, WILDBENCH aims to reflect real user interactions and mitigate data leakage concerns. The results demonstrate a strong correlation with human judgment, surpassing existing benchmarks and offering a more accurate reflection of LLM capabilities.", "questions": "See weakness part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. **Real-World Evaluation**: utilizes real user queries to provide a more reliable assessment of LLM performance. This approach is more cost-effective than human evaluations and offers greater authenticity compared to traditional machine evaluations.\n\n2. **Robust Evaluation Metrics**: introduces WB-Reward and WB-Score, which enhance accuracy through the use of checklists and a CoT-style evaluation process. These metrics effectively mitigate length bias, ensuring fairer comparisons.\n\n3. **Dynamic Updates and Data Leakage Prevention**: update regularly, reflecting new user interactions and preventing data leakage, which enhances the benchmark's relevance and security.", "weaknesses": "1. **Cost Comparison**: The paper does not provide a detailed comparison of the llm api costs between WILDBENCH and previous evaluation methods, which could help in understanding its efficiency.\n\n2. **Cheaper Version with Consistency Check**: Offer a cheaper version of the benchmark, such as using 30% of the samples, and verify it's the consistency and reliability with the full evaluation results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4YsxDBGZfT", "reviewer": "Reviewer_EFf3", "summary": "This paper carefully picked out data from WildChat to construct an evaluation benchmark with in-the-wild user queries. \n\nThis paper proposes two kinds of evaluation strategies, namely, checklists and structured analysis, to encourage more detailed and fine-grained evaluation.\n\nThis paper proposes two primary metrics for both pairwise and pointwise evaluation and further mitigates the bias towards longer outputs.", "questions": "In the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The first benchmark for LLM evaluation with in-the-wild user queries.\n- A bunch of tricks for a more accurate and debiased evaluation:\n    - pairwise + pointwise\n    - checklist-based\n    - multiple baseline models for pairwise evaluation.\n    - structured analysis before concluding the rewards.\n    - mitigating the length bias with a margin term $K$ for ties, which is customizable according to the users’ needs.\n- Interesting experimental findings such as the re-evaluated performance of LLaMA3-Inst-8B-SimPO.", "weaknesses": "- The novelty of this work is quite limited despite its richness of evaluation and wild data construction practices.\n- The handling of length penalty in this work is a bit crude, because sometimes a longer response is indeed better. Performing uniform processing directly at the entire dataset level fails to take into account the specific situations of each sample.\n- The experimental findings offer limited novelty compared to existing benchmarks like AlpacaEval and Chatbot Arena. The comparative performance among different models remains largely consistent with previous evaluations. Additionally, this work lacks sufficient analysis from the perspective of \"wildness\" in the data. It fails to address a crucial question: What unique insights emerge when introducing \"wildness\" into the evaluation process?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G2AdFKQcaG", "reviewer": "Reviewer_vPGb", "summary": "The authors introduced WildBench, a benchmarking framework for evaluating large language models (LLMs), using 1,024 complex tasks selected from real user conversation. These tasks reflect the natural distribution of real user tasks and are diverse, allowing for a more realistic evaluation of model capabilities. The framework incorporates detailed checklists to ensure objective evaluation. Experiments show that WildBench has a high correlation with human-voted Elo rankings from Chatbot Arena, outperforming other existing automated evaluation frameworks.", "questions": "1. WildBench's feature of updating its task set over time is valuable. However, since the baseline models are fixed, have the authors considered whether the current baseline models can effectively evaluate models with updated training data and more advanced performance, as both the models and task sets evolve over a long period?\n2. The responses in the history conversation come from earlier versions of ChatGPT-3.5 and GPT-4, which may introduce bias. For example, the history conversation may contain factual errors caused by hallucinations from ChatGPT-3.5 and GPT-4, which could conflict with the prior knowledge of the evaluated models, leading to inaccurate evaluations of model capabilities. Have any measures been taken to filter out such issues?\n3. For multi-turn conversation filtering, besides limiting the number of turns and applying single-turn filtering rules, is there additional task theme consistency checking? Even with fewer turns, multiple themes might still be present, affecting task focus and coherence. There is also a potential issue where a task might consist of multiple relatively simple questions across different domains, making the overall difficulty appear higher and bypassing the difficulty filtering. However, the final question itself might be low in difficulty and easy to answer, affecting the overall benchmark difficulty.\n4. The provided anonymous link is not accessible.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. WildBench establishes a comprehensive evaluation pipeline, creating a task set from real user queries that is both well-distributed and diverse, enabling a more comprehensive evaluation. Additionally, WildBench can update over time to accommodate emerging task distributions, reducing some potential risks of data leakage.\n2. Experimental results show that WildBench has a high correlation with the Elo scores from Chatbot Arena, which is considered a fair evaluation. However, the human cost is lower than Chatbot Arena, and WildBench provides two evaluation metrics, offering researchers diverse options for assessing LLM performance.\n3. The evaluation incorporates task-specific checklists and considers length penalties, further reducing the negative impact of \"LLM as a judge.\"", "weaknesses": "1. Although multi-turn conversation evaluation is a highlight of WildBench, the authors did not conduct much analysis from this perspective, which is unfortunate.\n2. The task quality filtering method is not detailed enough or may have some inadequacies, which could lead to inaccurate evaluations of performance differences between specific models. Additionally, the analysis of the overall correlation with Chatbot Arena may not reflect the performance differences of specific models between the two leaderboards (e.g., the differences between Yi-Large and Yi-1.5-34B-Chat on WildBench and Chatbot Arena).\n3. The test set includes multi-turn data, and the publicly available history conversation, when used for training models, still provides some assistance in answering the final question, which means there is still a certain risk of data leakage in the dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)