MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures
Authors: Jinjie Ni
Fuzhao Xue
Xiang Yue
Yuntian Deng
Mahir Shah
Kabir Jain
Graham Neubig
Yang You
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM Evaluation
Approximating Human Preference
Dynamic Benchmarking
Benchmark Mixture
Web Query Detection
Primary area: evaluation
TL;DR: MixEval, a ground-truth-based dynamic benchmark which updates data points periodically, evaluates LLMs with a highly capable model ranking (\ie 0.96 correlation with Chatbot Arena) while running locally and quickly (1/15 time of running MMLU).

Abstract: Evaluating large language models (LLMs) is challenging. Traditional ground-truth- based benchmarks fail to capture the comprehensiveness and nuance of real-world queries, while LLM-as-judge benchmarks suffer from grading biases and limited query quantity. Both of them may also become contaminated over time. User- facing evaluation, such as Chatbot Arena, provides reliable signals but is costly and slow. In this work, we propose MixEval, a new paradigm for establishing efficient, gold-standard LLM evaluation by strategically mixing off-the-shelf bench- marks. It bridges (1) comprehensive and well-distributed real-world user queries and (2) efficient and fairly-graded ground-truth-based benchmarks, by matching queries mined from the web with similar queries from existing benchmarks. Based on MixEval, we further build MixEval-Hard, which offers more room for model improvement. Our benchmarks’ advantages lie in (1) a 0.96 model ranking correlation with Chatbot Arena arising from the highly impartial query distribution and grading mechanism, (2) fast, cheap, and reproducible execution (6% of the time and cost of MMLU), and (3) dynamic evaluation enabled by the rapid and stable data update pipeline. We provide extensive meta-evaluation and analysis for our and existing LLM benchmarks to deepen the community’s understanding of LLM evaluation and guide future research directions.

MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures
o Jinjie Ni Fuzhao Xue", Xiang Yue", Yuntian Deng. Mahir Shah, Kabir Jain, Graham Neubig, Yang You o National University of Singapore, Caanegie Mellon University, A Allen Institute for Al
https: ptmps://mixees meixxeal.gittubl io/
Abstract
Evaluating large language models (LL.Ms) challenging. Traditional ground-truth- based benchmarks fail to capture the comprehensiveness and nuance of real-world queries, while LLM-as-judge benchmarks suffer from grading biases and limited query quantity. Both of them may also become contaminated over time. User- facing evaluation, such as Chatbot Arena, provides reliable signals but is costly and slow. In this work, propose Mi xEval, efficient, gold-standard LLM evaluation by strategically mixing off- offethe-shelf bench- marks. It bridges (1) comprehensive and well-distributee real- world user queries and (2) efficient and fairly-graded ground-truth-be
paradigm for establishing
benchmarks, by matching queries mined from the web with similar queries from existing benchmarks. Based on Mi xEval, 1, we further build Mi xEval -Haraallaarra model improvement. Our benchmarks' advantages lie in (I) 0.96 model ranking correlation with Chatbot Arena arising from the highly impartial query distribution and grading mechanism, (2) fast, cheap, and reproducible execution (6% of the time and cost of MMLU), and (3) dynamic evaluation enabled by the rapid and stable data update pipeline. We provide extensive neta-evvluation and analysis for our and existing LLM benchmarks to deepen the community 's understanding of LLM evaluation and guide future research directions.
-Hard. rd, which offers more room for
Introduction
That Which is Measured, Improves. Evaluation is essential in the Al community for two main reasons: (1) benchmarks provide early signals to model developers, aiding in refining data and model design, and (2) benchmarks guide users in selecting suitable models for specific use cases. Therefore, benchmarks offer feedback to the entire community, facilitating model optimization. Consequently, the main concern of evaluating LLMs is impartiality- we need to optimize impartial objectives so that the community advances the right direction. In practical LLM evaluations, three primary biases contribute to lack of impartiality: (1) query bias-evaluation queries falling short of comprehensiveness or appropriate distribution (2) grading bias-the grading process involving significant bias or error (3) generalization bias moosll overfitting the evaluation data.
Large Scale User-facing Evaluation Provides More Impartial Signal. Practitioners generally adopt either automatic or user-facing approaches for L.M benchmarking. Automatic benchmarking typically employs traditional ground-truth-basee benchmarks, such as MMLU (17), which often fail to capture real- weal-world query comprehensiveness and nuance while involving a comparatively impartial grading process; or employs open-ended benchmarks using LLMs as graders, such as MTT-Bench [39), suffering from both grading bias and query incomprehensiveness due to the preference biases and high cost frontier LLM judges. Additionally, the static nature of automatic benchmarks results
"Core contributors. 'Comespondence to: Jinjie Ni <jinjii wwwnjjinn@eee eniiius. edu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Correlation with Chatbot Arena Elo
1.0
MixEval-Hard
Chatbot Arena
MixEval
WildBench
0.9
AppcaEval-22.0
Chatbot Arena (En)
ARC-c
MMLU
Arena-Hard
0.8
BBH
MT-Bench
GSMBk
0.7
HellaSwag
Benchmark
Correlation
- Cost ($)
CommonsenseQA
DROP
Arena
2936.9
NaturalQuestions
0.6
ARC-e
MixEval-Hard
0.96
WinoGrande
Arena (En)
0.96
2936.9
0.5
O MATH
TriviaQA
AlpacaEval-
0.95
24.4
AGIEval
OpenBookQA
WildBench
0.4
HumanEval
MixEval
0.93
2.3
BoolQ
Arena-Hard
TruthfulQA
PIQA
ARC-c
0.85
0.8
0.83
0.3
MBPP
MT-Bench
0.80
10.1
10-1
100
10t
102
103
Total Evaluation Cost ($)
Figure 1: Benchmark correlations (%) with Chatbot Arena Elo, against the total costs of evaluating a single GPT-3.5-Turbo--11 model. MixEval and MixEval-Hard show the highest correlations with Arena Elo and Arena Elo (En) among leading benchmarks. We reference the crowdsourcing price for Amazon Mechanical Turk ($0.05 per vote) when estimating the cost of evaluating a single model on Chatbot Arena (approximately $2,936). Chatbot Arena is prohibitively expensive, while MixEval and MixEval- are cheap and cost-effective alternatives. Details on the correlation and evaluation cost values are provided in Section B
in contamination over time, amplifying the generalization issue. Such biases lead to significant deviations from gold-standard evaluation, impeding model development. On the other hand, large- scale user-facing benchmarking, such as Chatbot Arena (6). offers more reliable objectives for model development and effectively mitigates the above-mentioned three biases because (1) it collects vast array of real-world user queries, thereby ensuring superior query comprehensiveness and distribution, (2) it judges diverse and complex model responses stably due to the "wisdom of the crowd" effect (33, where individual judgment noise is averaged out over large number of samples, mitigating the grading bias, and (3) it continuously receives fresh user queries, mitigating the benchmark contamination issue. Furthermore, it guides model optimization to meet user needs effectively in practical applications, which crucial goal of developing models. However, Chatbot Arena prohibitively expensive (Figure I slow, and irreproducible. Moreover, it is not directly accessible for public usage, hindering practitioners from conducting easy and fast model evaluations.
MixEyal: Towards Efficient Gold-Standard LLM Evaluations. In this work, we aim to establish highly impartial gold-standard benchmark without compromising efficiency. This be achieved by leveraging (1) the efficiency and grading impartiality of ground-truie the superior comprehensiveness and distribution of real-world user queries. To this end, we propose MixEval, a two-stage benchmark reconstruction pipeline consisting of (1) wild query mining and (2)
benchmarks and (2)
grounding existing benchmarks in the mined queries. We introduce accurate user query retrieval process, comprising query detection, filtering, and classification. In the detection phase, we train open-source LLMs on self-collected data to detect queries in Common Crawl splits. During filtering, we utilize GPT-4 Turbo to exclude non-query sentences. In classification, we categorize the filtered queries by input and output modalities, retaining text-in-text-out queries for LLM evaluation. To align benchmark queries with real-world queries, we match each crawled web user query with its most similar query the benchmark pool and the corresponding ground truth answer. We designate the resulting benchmark as MixEval. To improve the benchmark's 's ability distinguish strong models, we derive challenging subset from MixEval, termed Mi xEval Hard.. To mitigate the overfitting issue, we periodically update the data points in Mi xEval and Mi MixEval- aaaaaallaarr Hard using fast, stable pipeline, which performs benchmark mixture with a different batch wild queries from
'The Chatbot Arena leaderboard is not the sole indicator of real- world human preferences. but it currently serves as one of the gold standards within the community. Therefore, we utilize it as a reliable source of approximation.

---

the same distribution, showing low model score variance (0.36 Std. on 0-100 scale) and significant version difference (85% unique query ratio). We thereby effectively mitigate the above-mentioned three evaluation biases through the proposed benchmark mixture pipeline, while maintaining high efficiency. As shown Figure Mí xEval and xEva -- rd achieve similar model rankings as Chatbot Arena while being far less costly.
Why use MixEval?
MixEval offers five significant advantages for practitioners: (1) accurate model ranking, demonstrated by a 0.96 correlation with Chatbot Arena, (2) fasst cheap and reproducible execution, requiring only 6% the time and cost of MMLU and with no dependence on human input, (3) dynamic benchmarking enabled by low-effort and stable updating mechanism, (4) comprehensive and less biased query distribution, it bases queries on a large-scale web corpus, and (5) fair grading process without preference bias, ensured by its ground-truth--e truut--bthhhtd nature.
Research Contributions
We developed pipeline for detecting real-world instructions, capable of mining queries build benchmarks and providing scalable solution for collecting vast amounts of real-world instruction- following data.
We introduced way to utilize benchmarks, demonstrating that real-world query distributions and user preferences can be reconstructed by strategically mixing off-the-shelf benchmarks with web-mined queries.
To the best of knowledge, MixEval creates the first ground- terttttuuttthhhhete dynamic benchmark with general-domate queries, benefiting from a rapid and stable data updating mechanism.
The resulting dynamic benchmarks, i.e., MixEval and MxkEval-Hazee exhibit significant correlations (0.93 and 0.96) with real-world user preference leaderboard (i.e., Chatbot Arena) and showcase high impartiality and efficiency.
We provide meta-evaluation and extensive analysis for MixEval and other leading LLM bench- marks, delivering detailed insights that enhance the community' understanding of LLM evaluation and guide future research directions.
2 LLM Benchmarks are Biased from Realistic User Queries and Preferences
How Much Do Our Benchmarks Reflect Real- world User Queries and Preferences? The rapid advancement of LLMs has led to the in- troduction of numerous benchmarks. However, the community may still lack a comprehensive understanding of how well these benchmarks align with real-world use cases and human pref- erences. Without such understanding, the sig- nals derived from evaluations might be mislead- ing, thereby impeding model development. To investigate this issue, we (1) analyze the corre- lations between benchmarks (Figures and 10p and (2) visualize their query distributions in unified 2-D space (Figure 2). The setups for these analyses are detailed in Section
Technical
Coding, Malh, Education Analytics
Coding
Math
Physics, Chamistyy
Logal issues.
Technologyy Software Dany,
Science and Technology.
Chemiztry. Envinsemeentt Sclentow Physiis
Eodinn,
remonmmnntal Sc eioo. Fenggraphy
Logic Puzzles,
Acadomic Benwwarch
History, Sporta Demographies raphics.
Societal Issues
Culturaa and
Psychology,
Health. Hooblic,
Craitts
General toelc
Behavicr.
Animal
Grommar Language,
Mix Enterainment
Sports,
Fetional Nerreniee,
Gariing.
Enher Interninmeend
Intoractions,
Decision Makiing,
Comvental
Dynamics
Care
Non-technical
2.1 Important Takeaways
Current benchmarks show limited corre- lation with human preferences. Figures and TO reveal that many benchmarks exhibit low correlation with human preferences (Arena Elo). The highest correlations are Arena-Hard (0.86) for LLM-judged and ARC-c (0.85) for ground-truth-base benchmarks. Using the benchmark mixture technique, MixEval
Figure 3: Query topic summarization for Figure The plot aggregates all queries and divides them into 16 regions. From each region, 100 queries are uniformly sampled and analyzed by GPT-4 for topic summarization. A clear trend is observed, with topics transitioning from non-technical the bottom to technical at the top.
3

---

Web Queries
MixEval
Sh
WidChst
Cheibot Arena
LMSYS-Chet-1P
0.00:0.0
C-Dist: 2.81
C-Dist:3.2 3.42
C C-Dist 3.45
C- C-Die::3.49
Dist:3.06
MixEval-Hard
Natural
MMLU
TriviOA
C-Dist: 4.11
5.02
G-Dat 5.24
C-Diet 5.38
C-Diet: 6.60
C- Dist: 6.94
qsensees
PIOA
MT-Banch
(en)
Dist: 7.48
C-Diat: 10.53 10.53
C-Diat: 10.81
C-Die:: 10.65
Diatt 14.70
OaenBookiAA
ARC
ARC
GPQA
BBH
GSMBk
16.05
16.54
D.D.st::
Dist: 37.81
Distt:48.58
SIQA
MATH
MBPР
HumanEvel
WinoGrande
All Quenies Above
Techinlcal
Non-techhoical
CiDist:
Figure 2: Query Topic Distribution of the Benchmarks. Ground-truth-basee benchmarks repre- sented by orange dots, wild datasets by yellow dots, and LLM-judged benchmarks (MT-Bench and Arena-Hard) by yellow dots, all plotted against our detected web queries shown as blue dots. Query sentence embeddings were dimensionally reduced to map them onto a unified 2-D space, facilitating direct comparisons of topic distributions across benchmarks. As move from the bottom to the top of the figure, query topics transition from non con-ccchnical to technical. Topic summaries for each region detailed in Figure 3
and MixEval- Hard achieve the highest correlations with human preferences, 0.93 and 0.96, respectively.
Most benchmarks exhibit skewed topic distribution. Ground-truth truth-based and LLM-judged benchmarks show a skewed query distribution compared to detected web queries and wild datasets. Notably, BoolQ, Natural Questions, and MMLU align more closely with wild user queries due to their data collection methods. Specifically, questions in BoolQ and Natural Questions originate from Google Search, while MMLU designed to wide range of topies (57 topics, including Atari games (
Query comprehensiveness is crucial. General-domain benchmarks, which are not tailored for specific domains in their data collection pipelines, exhibit a stronger correlation with Arena Elo than domain-specifti ones. Notably, 10 out of 13 general-domain benchmarks have Arena Elo correlation score above 0.5, whereas only 1 out of 8 domain-specifie benchmarks achieves this. This underscores the significance of comprehensive query topics for achieving high correlation with human preferences.
Some general-domain benchmarks are actually domain-specific. Despite being labeled general- domain benchmarks, DROP and WinoGrande have limited scopes, often narrower than many domain- specific benchmarks. As depicted in Figure DROP queries mainly address History, Politics, Sports,

---

Demographics, and Societal Issues, whereas WinoGrande queries focus primarily on Grammar, Language, Decision Making, and Social Dynamics.
User population size affects the query distribution, Figure 2 shows distribution differences in wild queries across varying population sizes. ShareGPT, grounded in 100 million active users of ChatGPT by mid-2023, contrasts with WildChat (38). Chatbot Arena Conversations (6. and LMSYS- Chat-IM [39], which have user bases of 0.2 million, 0.13 million, and 0.21 million, respectively. The global internet user count was 5,4 billior f in 2023, an order of magnitude larger than all considered wild datasets. Consequently, user bases of the internet, ShareGPT, and other datasets span three distinct orders of magnitude. ShareGPT's larger user population (second order of magnitude) yields distribution most similar web queries from the global internet user base (third order of magnitude). both visually and in cluster distance (C-Dist).
Chathot Arena and Arena-Hard queries exhibit biases. Compared web queries and ShareGPT data, datasets from the Chatbot Arena wehsite-Chatbot Arena Conversations and LMSYS- S-Chat-
IM-have a higher proportion of technical queries (as presented in Figure 3 queries with higher position on the map are more technical). This indicates a user base skewed towards technical users, potentially affecting evaluation results, as effective LLM benchmark should mimic real-world use cases. Furthermore, a minor discrepancy exists between web and ShareGPT queries, suggesting that one or both of them may still slightly deviate from actual real- world query distributions. Moreover, Arena-Hard Hard queries exhibit pronounced bias towards technical topics. This likely stems from the design of their data pipeline for sampling hard prompts. We will demonstrate that employing carefully designed sampling technique is essential to preserve the query distribution while enhancing difficulty. This is supported by Mi xEva -a rd's similar distribution to the original web queries
and wild datasets (see Section 3.3
3 MixEval
MixEval
Train Test
Modality
Benchmark
LLM Classifier
Filter
Annotation
Mixture
MixEval-Hard
Dynamic Eval
Harre
Hand
Eval
Results
Sampling
Update
Mersion
fast stable
Version
Figure 4: MixEval, 1, a two-stage benchmark reconstruction pipeline, comprises (1) web query detection and (2) benchmark mixture. We further introduce Mi XEval-Hard to enhance model separability, alongside a dynamic updating mechanism to mitigate contamination risk.
In Section 2 we show that current ground-truth- and L.MM-ju benchmarks have skewed query distributions and limited correlation with human preferences. Additionally, LLM-judged benchmarks suffer from LLM preference bias and both of them become contaminated over time. In contrast, Chatbot Arena is less biased and more dynamic but requires slow and expensive human preference data collection, resulting in irreproducible outcomes.
To address these issues, introduce Mi MixEval (Figure 4. which aligns ground-truth tranndd-ttth-beeedd LLM benchmarks with real-world user queries. This method uses user queries mined from the web and matches them with similar queries from existing benchmarks, and involves two stages: (1) user query detection from the web and (2) benchmark mixture. To improve model separability and reduce contamination, also propose Mi xEva -H rd and dynamic updating mechanism.
3.1
Web User Query Detection
In this stage, we detect user queries from Common Crawl [12. Both recall and precision are crucial to ensure the query distribution reflects real-world scenarios. Therefore, we developed two benchmarks
http::/wwww.mmlyaaannnggggggggg http::/www..ttiii...mmsttttttits htwp:///eeettttooooooooooooeeene wnterrrseeeeeeeeesssssssowoeviie
5

---

to evaluate our query detector's performance. The first benchmark includes self-collected in-the-wii user queries as positive samples, with non-query datasets such as Wikipedia [T6] as negative samples. The second, higher-quality benchmark contains positive and negative samples hand-picked by our authors from in-the-wild query and non-query datasets. preliminary experiments, direct prompting of open-source language models performed poorly our benchmarks. Thus, we developed rectification pipeline to ensure high recall and precision cost-effectively. We started with a detection phase to gather training data. Testing various open-source LLMs, Vicuna 33B (7) achieved a high recall (>99%) on our test sets with careful prompt engineering, ensuring that very few positive samples were missed initially. In this phase, detected around 20k queries using Vicuna 33B over subset of Common Crawl. We then used GPT-4 to more accurately label these data as positive or negative samples, and used the resulting data to train Vicuna 33B. The trained Vicuna 33B achieved high recall (>99%) and precision (>98%) on our benchmarks and detected 2M user queries from the entire Common Crawl. Finally, we prompted GPT-4 Turbo to classify them, extracting text-in-text-out queries for LLM evaluation. Future work will address queries with other I/O modalities.
3.2 Benchmark Mixture
To bridge wild user queries Q and ground-truth LLM benchmarks, we create benchmark pool B (B,, ..,, B, where each B, = {b), b2, ..., b represents a distinct ground-truth LLM benchmark. We define mapping f q: + by. with qi E Q and bj E B. For each E Q. we rank similarities between each (g;, b,) pair and select the most similar by that satisfies bj = f(:)) = arg m S(4, by) s.t. 0. We use the dot-product between normalized sentence embeddings as the similarity score S(-). When retrieving the top-1 bj. 0 is a length constraint on the input (or context) field of each by. addressing the effect of long inputs the benchmark data mixture. model from [23). To ensure quality and comprehensive sample coverage, we selected the development and test splits of widely adopted benchmarks from diverse domains and topics. Details and distributions of these benchmarks are provided in Appendix [
The sentence embeddings of queries are computed using the alll-apnt-bbase
3.3 MixEval-Hard
Table 1: The key statistics of Mi xEval and Mi xEval xEval-Hard. -ard.. With dynamic benchmarking, the numbers may vary slightly while the number of queries will not change.
Avg. Toks
Avg. Inputs
Avg. # Toks
Min # Toks Max # Toks English Ratio
Eval Type
Queries
per Query
per Input
per Input
per Input
MixEval
4000
23
0.3
41.3
6
954
95.15%
Ground Truth
MixEval-Hard
1000
27.3
0.4
47.3
954
95.22%
Frontier LLMs are rapidly approaching human-level performance across diverse tasks. As these models progress, existing benchmarks will become saturated, hindering differentiation between models. Although MixEval reflects typical user queries, it is constrained by the benchmark pool B's overall difficulty. Our results in Table 3 indicate that top models, such as GPT-4 Turbo and Claude 3 Opus, have surpassed 88% accuracy on Mi xEval. To improve the benchmark's 's ability to discriminate between very strong models, we extract challenging subset from Mi MivEval to create Mi MixEval-Haze Hard.
Given MixEval denoted as B', we sample a hard subset 8 from B by computing a difficulty score S for each entry, prioritizing higher scores. Consider set of model prediction results A, where A a (-1 matrix of shape (Nmodet, Ng:) ), with 1 indicating an incorrect model response. Here, N Nmndet is the number of models, and Ng is the number of questions in B'. The difficulty score & for query B/ is computed by & = fI Ai, where each model's 's result on question i is weighted by its accuracy My on the dataset. Given {E1, E2...., ..., E..,. }. we sample from 13' with rejection:
R" bb B', p(b)) and (B" U (b,), B') <)
where (z y) denotes the cluster distance between x and y. The probability of drawing 6/. p(b() cbtii Enyeor e is based on & This rejection sampling ensures that Mi xEval 1 is difficulty-first

---

while maintaining a balanced query distribution. We obtain 1000 samples for MixEval-Hard. The statistics of MixEval and MexxxEvl-Hard are detailed Table
3.4 Dynamic Benchmarking
Table 2: Stability test for dynamic benchmarking. Five models tested across five updated versions of Mixaal show an average mean of 77.64 and Std. of 0.36, validating the stability of model scores over versions. The unique web query ratio, averaged across all version pairs, is 99.71 %, and the unique benchmark query ratio is 85.05%, indicating significant differences between versions.
Unique Web Unique
GPT-3.5-
GPT-3.5-
Claude
Mistral- Small
Reka Edge
Avg.
MixEval
Turbo-0125
Turbo-1106
3 Haiku
Query Ratio
Query Ratio
Mean
79.66
79.25
80.32
80.57
68.42
77.64
99.71%
85.05%
Std.
0.26
0.28
0.34
0.56
0.35
0.36
Static benchmarks risk contamination over time as models may overfit to the benchmark data 66 32 36), undermining evaluation reliability. To address this, we periodically update the data points in Mi xEval and MixEval- Hard using the automatic pipeline described above, i.e., performing benchmark mixtures based on the queries uniformly sampled from the massive web queries detected, which completes updates within minute. Table 2 shows score stability and version differences. We created five versions of Mi xEva by altering the random seed when sampling web queries and five models them. As shown, the average mean and standard deviation (Std.) for the models across the versions are 77.64 and 0.36, respectively, demonstrating high score stability. For each pair of versions, we compute the unique sample ratio for sampled web queries and benchmark data points. Given samples X {21,
Z2, { B, the unique sample ratio R is calculated R [X-Y++Y-XI
In from version A and Y = {V1, 92...., Un J from version
representing the unique ratio of the XUY set. The average unique web query ratio across all version pairs is 99.71 99.71%, and the unique ratio for MixEval versions is 85.05%, indicating significant differences between versions. This efficient updating mechanism, alongside stable model scores and significant data point variations, effectively mitigates benchmark contamination. Additionally, we plan dynamically expand our benchmark pool with newly released benchmarks to further enhance the mixed benchmark distribution.
XUY
To summarize, we update the data points of MixEval via (1) batch web query update (sampling different web queries batches from the crawled web queries). (2) source web query update (updating all the web queries with the latest Common Crawl) or (3) benchmark pool update (incorporating new ground-truie tit--tuuuthheeee benchmarks to the benchmark pool). Since the mechanism of MixEval is to match web queries with benchmark pool samples, the above three updating methods refreshes both the web queries (the first and the second method) and benchmark pool samples (the third method).
4 Results
4.1 Experiment Settings
We evaluate models on MixEval and MixEval -ixxvval-Hard using the Transformers library [3] for open-source models, adhering the official settings in their Hugging Face model card. Proprietary models are assessed via their official API endpoints, using the latest versions as of April 30, 2024. Chat models employ official chat templates or FastChat chat templates [39], and base models are evaluated in 5-shot setting. Both MixEval and MixEval-Haat Hardd comprising samples from various benchmarks, demonstrate the inadequacies of traditional rule-based parsing methods across all benchmarks and models. To improve parsing accuracy, we use GPTT-355-Turbo-1 parser to either score the response (free-form questions) or extract the model's I's choice (multiple-choice problems). The stability of the GPT-3.5 Turbo parser is evidenced in T this paper and Table 4 of (37). We will also provide an open-source model parser with its stability test to ensure long-term reproducibility. Section details the model parser prompts, and Section compares the model parser to the rule parser. Models are evaluated on or 100 GPUs. All correlations with Arena Elo are based on the Chatbot Arena Leaderboard of May 1, 2024.
as the model
7

---

4.2
Effectiveness of MixEval
Original Miked
Correlation with Arema EL (9)
100
92
MixEval-Hard MixEval
84
81
50
72
Sennhhark-leee Mix
Uniin BEMMI
60-
51
40
35
20
TriasA
MMLU
DROP
HallaSwag
Common-
BoolQ
GSMBK
AGIEval
PIQA
SenseQA
-17 SIQA
Figure 5: Our approach improves the correlation with Arena Elo and Arena Elo (En) (Figure for all the main splits of MixEval and outperforms benchmark-leve and uniform mixture.
MixEval and MixEval-Hard achieve the highest correlation with Arena Elo and Arena Elo (En) among all benchmarks. As shown in Figures I and [0] MixEval and MixEval-Han derived from the proposed MixEval pipeline to simulate diverse user queries, achieve significantly higher correlations (10% higher than the top SOTA benchmark) with human preferences (both Arena Elo and Arena Elo (En)), ranking second and first, respectively. Notably, Mi MixEval-HHas correlation with Arena Elo is even slightly higher than with Arena Elo (En). As discussed in Section 4.5 query difficulty impacts human preference correlation. Therefore, Mix -Hard r''s superior correlation may partially result from increased query difficulty. The high correlations of xEval Satval-Hal Ha rd with human preferences enable efficient, cost-effective, and reliable model
and Mi xEval ranking compared to noman-in-the-loop benchmarks.
MixEval improves the correlation with Arena Elo and Arena Elo (En) across all main bench- mark splits of MixEval. In Figure 5] we select the top-10 benchmarks from our pool with sufficient sample sizes (see sample number distribution in Figure For each benchmark, we present (1) the correlation between Arena Elo and the original benchmark, and (2) the correlation between Arena Elo and the MixEval dEval-men 1-mixed version. Remarkably, all benchmarks exhibit significant improvements in their correlations with Arena Elo after being processed by MixEval. The correlation increase is notably high (>40%) in benchmarks such BoolQ, AGIÉval,, SIQA, and PIQA. Mi xEval 1. and Mi xEval -Ha rd, which aggregate all benchmarks, consistently outperform any individual bench- mark mixture, underscoring the importance of large benchmark pool and query comprehensiveness.
MixEval outperforms both benchmark-level and uniform mixtures. Figure illustrates the correlations with Arena Elo for benchmark-level and uniform mixtures. The benchmark-leve mixture samples questions uniformly from each benchmark, proportional its split size in MixEval. The uniform mixture samples an equal number of questions from all benchmarks. Both methods yield sig- Furthermore, nificantly lower human preference correlations than Mi xEval and MixEval-Has the benchmark-leve mixture offers negligible improvement over the uniform mixture. These findings underscore the importance of an appropriate sample-level mixture, as implemented by Mi xEval.
MixEval effectively maps real-world user queries to ground-truth-basee benchmarks. Figure 2 shows the query distributions of leading benchmarks. Both Mi xEval and Mi xEva l-Hard closely resemble web queries and popular wild datasets, highlighting Mi xEval's efficacy in aligning benchmark query distributions with real-world data. The maps in Figure 2 are ordered by their cluster distances to our identified web queries, showing that wild datasets align more closely with our web queries than other LLM benchmarks. This underscores the robustness of our web query detection pipeline and the solid grounding of Mi xEval. Additionally, as discussed in Section 2 ShareGPT, with a larger user base (100M) compared other wild datasets (0. M0..MM-,,M), shows the highest similarity to our web queries, which are based on global internet user population (5.4B), further validating the accuracy of our web query detection.
4.3 Evaluation Results
Leaderboard Table (Section presents the detailed evaluation results on Mi MixEva MixEval-HHe -ardd, and their main subsets. Claude 3 Opus and GPT-4 Turbo consistently achieve

---

65
Performance (MixEval)
Performance (MixEval)
60
56
LLaMV-3
Command
Qwen1.5
YI
Mistral
35
Starlnng-M
Gomma
30
Viaune
LLaMA
Zephyr
40
GLMG
25
JetMoE
MPT
35
20
40
6D
80
100
60
Activated Parameters (Billions)
$ Cost Per Million Output Token
Figure 6: Activated parameters and API price per performance of open-source and proprietary models.
0.8
Chat Base
Spearman's Corretation: 0.95
0.6
0.4
0.2
LLaMA-3-700 9-708
Qwen1. 157728
YI-34B
Qwen1. 55-228
Mixtral-8x72 8x7B
L1aMA-2-706
Qwen1.5 M.MME -A2.78
Owen1 1.5-78
LLaMA 8B
Mistral-75 7B
Oemmaa-7
YI-6B
Qwen1.5 1.5 4B
JetMoE 8B
LLaMA-2-7B
Semma 28
MPT-7B
OLMo
Figure 7: The performance of chat and base models of the same model series in Table 3 Chat and
base model scores show a high correlation.
the highest performance across nearly all splits, except for BoolQ-Mixed. Gemini 1.5 Pro ranks third on both MixEval and MixEval-Hare followed closely by Claude 3 Sonnet, LLaMA- 3-70B-Instruct, and Reka Core, with similar scores. Notably, all these frontier models, except 1LAMA---77B-Instrree
support multi-modal input understanding. The LLaMA-3-8B-Img is the top-performing 7B model, outperforming some of the latest large models, such as Command R (35B) and Qwen1.5-328 2hCtt (32B). Proprietary models generally outperform open-source models.
model
Cost-effectiveness Figure compares the models in Table in terms of cost-effectiveness. Figure ealexminess the relationship between activated parameters and performance for open-source LLMSs, while Figure 6b) compares API price against performance for frontier proprietary L.LMs. Both figures exhibit roughly log-linear relationship between performance and the x-axis metric. In Figure 6al the Llama-3 series stands out as the most performant and parameter-efficient among open-source models. and JetMoE- 8B-Chat, demonstrate superior parameter efficiency. The proprietary data points reveal a clearer log-linear pattern. GPT-4 Turbo is more cost-effective than Claude 3 Opus, offering comparable performance at less than half the price. The Gemini series exhibits similar cost-effectiveness to the GPT series, while the Reka series parallels the cost-effectiveness of the Claude series. We conduct detailed error analysis in Section H to compare error rates of open-source and proprietary models on different Mi xEval splits. We also showcase error responses of frontier models in Section H.T to identify their potential weaknesses.
The MoE models, such Mixtral-8x7B- -nnt7r--tttttttoO... Qwen -wwel.5-MME-AA. A-AEE-A2.BB--hae

---

4.4 Can We Approximate the Human Preferences for Base Models?
The crowdsourced evaluation of LLMs, based on human preferences, assesses two main aspects: (1) model capability, optimized mainly during pre-training, and (2) non-capability attributes like toxicity and helpfulness, refined during post-training. We explore whether human preferences for models can be predicted before post-training, leading to the question: which stage, pre-training or post- training, more significantly influences human preferences in crowdsourced L.I.M evaluations? We evaluated the hase versions of the model series in Table Notably, the correlations in Figure ( a 0.95 correlation between base and chat models, indicating MixEval's potential to approximate human preferences treppost-traiing. This implies that pre-training may have greater impact on crowdsourced LLM evaluations, with post-training minimally altering human preference rankings. However, we also observe that the post-training has more impact on some smaller models, all of which went through heavy supervised post-training.
4.5 What Affects the Correlations with Human Preference?
Comprehensiveness and other features, such as difficulty, impact correlation. As shown in Figure TO, general-domaii benchmarks typically exhibit a higher correlation with human preference compared to domain-specifte benchmarks, highlighting the importance of query comprehensiveness. However, comprehensiveness is not the sole factor. Three observations support this: (1) Benchmarks like GSM8K, despite their skewed distributions [Figure[). achieve high correlation (0.78) with human preference, while others with high topic overlap with real-world queries, such as BoolQ. achieve a low correlation (0.37). (2) ARC-e and ARC-c, despite similar topic distributions, show significantly different correlations (Figure TO), likely due to varying difficulty levels. This indicates critical to correlation with human preference. (3) As shown in Figure 5 Mi Mival increases the correlation for each individual benchmark through benchmark mixture. For an individual benchmark, the topic becomes less comprehensive post-mixture since the mixed version represents a subset of the original: thus, the correlation gain is not due to a more comprehensive topic distribution. These observations suggest that correlation gains with human preference are influenced by factors beyond solely comprehensiveness, possibly including query difficulty and topic density. Section presents the full correlation matrix and analyzes other factors affecting benchmark correlations.
that other query features, such difficulty,
5
Conclusion
In this paper, we present Mi xEval. 1, an approach that bridges real-world queries and ground-trutte based evaluation by mining user queries from the web and matching them with similar benchmark queries. Mi xEval 1 and its hard variant can offer accurate evaluations that highly align with Chatbot Arena. MixEval operates locally and rapidly, eliminating the need for slow, costly human preference data collection or biased model judgment. MixEval's data points can be stably updated within one minute, mitigating benchmark contamination. We thereby effectively mitigate the query, grading, and generalization biases in LLM evaluation through the proposed benchmark mixture pipeline, while maintaining high efficiency. Our mnta-evaluuatio and extensive analysis of Mi xEval and other popular LLM benchmarks demonstrate Mi MixEval's 's effectiveness, providing insights to enhance the community's 's understanding of LLM evaluation.
Acknowledgement
We thank Yao Fu, Balázs Galambosi, Jason Phang, Jason Wei, Piotr Nawrot, Luca Soldaini, Guanzhi Wang, Deepanway Ghosal. Bo Li, Junhao Zhang, Yifan Song. Zangwei Zheng, Zian Zheng, Qinghong Lin, Wenhu Chen, Bill Yuchen Lin, and colleagues from CMU NeuLab for insightful discussions and pointers.
10

---

References
[1] J. Austin al. "Program synthesis with large language models," arXiv preprint
arXiv:2108.077 07733, 2021.
[2] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, "The arcade learning environment:
An evaluation platform for general agents," Journal of Artificial Intelligence Research, vol. 47,
pp. 253-279, 2013.
[3] E. M. Bender and B. Friedman, "Data statements for natural language processing: Toward mitigating system bias and enabling better science," Transactions of the Association for Computational Linguistics, vol. 6, pp. 587-604, 2018.
[4] Bisk, Zellers, Gao, Y. Choi, et al., "Piqa: Reasoning about physical commonsense in natural language." in Proceedings of the AAAI conference artificial intelligence, vol. 34, 2020, pp. 7432-7439.
[5] M. Chen et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107. 107.03307.0337,, 2021.
[6] W.-L. Chiang et al., "Chatbot arena: An open platform for evaluating Ilms by human preference," arXiv preprint 17Xiv:2403.04132 2024.
[7] W.-L. Chiang el al., "Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality," " See https://vicuna. Imsys. org (accessed April 2023), vol. 2, no. 3, p. 6, 2023. [8] Chicco and G. Jurman, "The advantages of the matthews correlation coefficient (mcc) over fl score and accuracy in binary classification evaluation," BMC genomics, vol. 21, pp. 1-13, 2020.
[9] C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, "Boolq: Ex- ploring the surprising difficulty of natural yes/no questions," arXiv preprint arXiv:1905. 10044, 2019.
[10] P. Clark et al., "Think you have solved question answering? try arc, the ai2 reasoning challenge," arXiv preprint arXiv:/803..om 05457, 2018.
[11] K. Cobbe et al., "Training verifiers to solve math word problems," arXiv preprint arXiv:2110 10.14668, 2021.
[12] T. Computer, Redpajama: An open dataset for training large language models, 2023. [Online]. Available: /
wewam//togeeeeeeccomtotttooo RedPa jama jamaaaaaaaaaae Data [13] O. Contributors, "Opencompass: A universal evaluation platform for foundation models," GitHub repository. 2023.
[14] D. Dua, Y. Wang. P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner, "Drop: A reading
comprehension benchmark requiring discrete reasoning over paragraphs," arXiv preprint arXiv:1903.0011
2019.
[15] H. Face, Open Ilm leaderboanl, 2023. [Online]. Available:
co/
spaces/Huggi har.gggi/nnaacee////po// 11m leaderboard
[16] W. Foundation, Wikimedia downloads, 2022. [Online]. Available: https
dumps
w wikimedia.oo org]
[17] D. Hendrycks et al., "Measuring massive multitask language understanding." arXiv preprint
arXiv:2009.03:3 03300, 2020.
[18] D. Hendrycks et al., "Measuring mathematical problem solving with the math dataset," NeurlPS, 2021.
[19] M. Joshi, Choi, D. S. Weld, and L. Zettlemoyer, "Triviaqa: large scale distantly supervised challenge dataset for reading comprehension," arXiv preprint arXiv:1705, 03551,,,,,,.....0 2017.
[20] T. Menzies and T. Zimmermann, "Software analytics: So what?" IEEE Software, vol. 30, no. 4,
pp. 31-37, 2013.
[21] T. Mihaylov, Clark, T. Khot, and A. Sabharwal, "Can a suit of armor conduct electricity? a new dataset for open book question answering." arXiv preprint arXiv: 02789, 2018.
[22] P. Padlewski el al., "Vibe-eval: hard evaluation suite for measuring progress of multimodal language models," arXiv preprint arXiv:2405. 240..08877 2024.
[23] N. Reimers and I. Gurevych, "Sentence-bert: Sentence embeddings using siamese bert- networks," arXiv preprint arXiv: 10084, 2019.
[24] D. Rein al., "Gpqa: graduate-level google-proof q&a benchmark," arXiv preprint
arXiv:2311.12022 2023.

---

[25] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi, "Winogrande: An adversarial winograd schema challenge at scale," Communications of the ACM, vol. 64, no. pp. 99-106. 2021. [26] M. Sap, H. Rashkin, D. Chen, R. LeBras, and Y. Choi, "Socialiqa: Commonsense reasoning about social interactions," arXiv preprint arXiv:1904.090
2019.
[27] T. ShareGPT, Sharegpt: Share your wildest chatgpt conversations with one click, 2023.
[28] M. Suzgun et al., "Challenging big-bench tasks and whether chain-of- thain-of-thought can solve them," arXiv preprint arXiv:2210. 10.061,,,, 2022.
[29] A. Talmor, J. Herzig, N. Lourie, and J. Berant, "Commonsensequ: A question answering challenge targeting commonsense knowledge," arXiv preprint / 1009977 2018.
[30] L. Tianle et al., "From live data to high-quality benchmarks: The arena-hare pipeline," See 19-area-aaaaee hard/, 2024.
[31] T. Wolf et al., "Huggingface's transformers: State-of-the-art natural language processing," "
arXiv preprint arXiv: 2019.
[32] S. Yang, W.-L.. Chiang, L. Zheng, E. Gonzalez, and I. Stoica, "Rethinking benchmark and contamination for language models with rephrased samples," arXiv preprint arXiv:231 1,008500 2023.
[33] S. K. M. Yi, M. Steyvers, M. D. Lee, and M. J. Dry, "The wisdom of the crowd in combinatorial problems," Cognitive science, vol. 36, no. 3, pp. 452-470, 2012.
[34] X. Yue, T. Zheng, G. Zhang, and W. Chen, "Mammoth2: Scaling instructions from the web," 2024.
arXiv preprint arXiv:2405.03355
[35] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, "Hellaswag: Can a machine really finish your sentence?" arXiv preprint arXivv:905.,0 078300 2019.
[36] Zhang et al., "A careful examination of large language model performance on grade school arithmetic," arXiv preprint arXiv:2405.00332. 2024.
[37] R. Zhang el al., "Direct preference optimization of video large multimodal models from language model reward," arXiv preprint arXiv:2404.00/558 2024.
[38] W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng, "Wildchat: Im chatgpt interaction logs in the wild." arXiv preprint arXiv:2405.0/44 014400 2024.
[39] L. Zheng et al., "Judging llm-as-a-judge with mt-bench and chatbot arena," Advances in Neural Information Processing Systems, vol. 36, 2024.
[40] W. Zhong el al., " "Agieval: A human-centric benchmark for evaluating foundation models," arXiv preprint arX1v:2300..0636/ 2023.
12

---

A Frequently Asked Questions
We list the potentially frequently asked questions and the point-to-point answers as follows:
A.1
Why are real-world human queries and preferences important?
One of the primary applications for Al model development is the automation of complex tasks traditionally performed by humans. As AI models coexist with humans, frequent interactions with humans are necessary to manage these tasks effectively. These interactions predominantly involve natural language queries, as the most common medium for human communication. Given the significance of human interaction in the use cases for general AI models, it is crucial to evaluate AI models- particulalyy large language models (LLMs) that rely on natural language- under conditions that mirror real-world scenarios, i.e., receiving human queries and assessing performance based on human preferences. Evaluating models based on their real-world use cases is well-supported across various research disciplines 20
A.2
Why do you use web queries real-world user queries?
Because web queries are grounded in the largest human population (5.4 billion internet users) among the accessible query sources. Figure illustrates the query distribution differences in wild queries across various user population sizes. ShareGPT, with 100 millior active users by mid-2023, contrasts with WildChat (38), Chatbot Arena Conversations (6. and LMSYS-Chat- 1M [39], which have user bases of 0.2 million, 0.13 million, and 0.21 million, respectively. The global internet user count was 5.4 billion in 2023, an order of magnitude larger than all considered wild datasets. Consequently, the user bases of the internet, ShareGPT, and other datasets span three distinct orders of magnitude. ShareGPT's larger user population (second order of magnitude) yields a distribution most similar to web queries frrm the global internet user base (third order of magnitude), both visually and in cluster distance (C-Dist), validating that user population size affects query distribution. Compared to web queries and ShareGPT data, datasets from the Chatbot Arena website-Chatbot Arena Conversations and LMSYS-Chat CaaaChha-aaaaee queries with higher position towards technical users, potentially affecting evaluation results, as an effective LLM benchmark should mimic real-world use cases.
IM-have a higher proportion of technical queries (as presented in Figure 3 the map are more technical). This indicates user base skewed
A.3
Why do you use benchmark mixtures instead of training judge models directly Arena Conversations to achieve a similar model ranking with Arena Elo?
The crucial difference lies the scoring methods: using ground truth answers versus LLM judges. Ground-truth-bosse evaluation is more interpretable, faster, and cost-effective compared to LLM-as- judges. Furthermore, training effective judge models is highly challenging because (1) LLMs possess inherent preference biases (39), and (2) to evaluate other models accurately without ground truth answers, the judge model must be either superior to or at least on par with the models assesses. Consequently, large model is required, complicating the training process.
A.4
Why isn't the cluster distribution ranking introduced in Figure consistent with the rankings in Figures] or 10
The key to human preference correlation, indicated in Figure 5 lies in ensuring that a benchmark' query distribution aligns with a subset of the wild queries, rather than encompassing the entire wild query distribution. This is evidenced by the high correlation between the Mi xEva deixvvllixee 1-mixed domain-specifie benchmarks and Arena Elo in Figure However, aligning with only a subset of wild queries significantly impacts the cluster distance metric shown in Figure Notably, covering all regions of wild queries enhances correlation scores, demonstrated by MixEval achieving higher correlation in Figure 5]
http::/wwww.allllaaannnngggggggu
13

---

A.5 How long does it take to dynamically update MixEval-Hard??
The update of MixEval-Hazd is somewhat slower than Mi MixEval, which can be updated within 1 minute. MixEval-Hare a subset of MixEval, is sampled based on the prediction results of several models. Thus, the update time depends on the models used to sample this subset. If only GPT-4 Turbo's prediction results are used to rank the question difficulties, the total update time is approximately minutes, which remains rapid. However, according to [22]. GPT-4 Turbo may yield lower score in this condition compared to using results from multiple models.
A.6
Arena-Hard's correlation with Arena Elo (En) is reported to be 0.94 in their blog while 0.83 in Figures [anrees[aannii 10
The number for Arena-Hard in Figures I and 10 is derived from the model scores reported on Arena-Hard's Ieaderboard as of May 01, 2024, using the Arena Elo (En) values from the same date. This discrepancy may arise because the figure reported in Arena-Hard's blog did not account for all models listed on their leaderboard. The number of models used for each pair of benchmarks shown in Figure 10 is reported in Figure 11
A.7 Why GPQA is not included in Figures I and T0P
Because we didn't find enough data points for GPQA that share enough common models with other benchmarks.
A.8 Is MixEval totally unbiased?
No, MixEval is not entirely unbiased. This is due to several factors: the detection pipeline is not perfectly accurate, Common Crawl data collection introduces biases, and there are inherent biases from web users in the real world. However, MixEval is relatively less biased because it draws from a broad internet user base. This is supported by: (1) The maps Figure 2 which are ordered by their cluster distances to our identified web queries, indicate that wild datasets align more closely with our web queries than with other LLM benchmarks. This demonstrates the robustness of our web query detection pipeline and the solid grounding of MixEval. (2) As discussed in Section 2 ShareGPT, with its extensive user base (100M) compared other wild datasets (0. IM-022M), shows the highest similarity to our web queries, which are based on the global internet user population (5.4B). This further validates accuracy of our web query detection. (3) The trained web query detector achieved high recall (>99%) and precision (>98%) benchmarks.
internal web query detection
A.9 Will the pipeline that creates MixEval-Hare introduce some noise that influences the result?
MixEval-Ha Hard sampling relies on the difficulty scores of benchmark questions, which inherently include some dataset annotation errors. During annotation issues. However, the number of annotation errors was minimal, rendering the noise negligible. Furthermore, the high correlation with Arena Elo demonstrates that the introduced noise does not significantly affect the model rankings.
case study (Section H identified several
B Related Work
LLM Benchmarking
Both frontier and open-source LLMs have made significant strides in recent years. Evaluation scores are a core objective in LLM development, necessitating an effective evaluation pipeline for successful model advancement. Current LLM evaluation methods can be categorized into three main types: (1) ground- rruth-based evaluation, (2) LLM-as-judge evaluation, and (3) crowdsourcing Human--refeereceebasseee evaluation.
Ground-truth-baat evaluation, or closed-ended evaluation, involves ranking the outputs of base and chat LLMs against predefined correct answers. Various benchmarks have been introduced by
14

---

These benchmarks facilitate rapid and straightforward LLM evaluation, providing clear and unbiased answer benchmarks often exhibit
the research community for this purpose 5,9
21 24 26
28 29
35
40).
judgments due to their closed-ended nature. However, ground-truth-base topic bias (as illustrated Figure 2 and may not accurately represent the diversity of real-world user queries, limiting their ability to assess the nuanced capabilities of LLMs.
On the other hand, two other categories of evaluation approaches primarily focus on the open-ended evaluations of chat LLMs. The LLM-as-judge evaluation uses frontier models to rank the responses set of open-endee queries without ground-truths. These queries are either manually designed [39 or sourced from crowdsourcing platforms [6]. However, due to the high cost of using frontier models as judges, such approaches are not scalable to a large number of user queries. This limitation hinders the ability to reflect the complexity and diversity of real-world queries and may deviate from the true distribution (see Figure 2. Additionally, previous research has identified several biases frontier model judges, including verbosity bias, position bias, and self-enhancement bias [39]. These biases can lead to unfair model rankings in practical evaluations. Additionally, the static nature of both ground-truth-be -rt-ttttth--hdd and LLM-as-judge benchmarks results in contamination over time, diminishing the reliability of evaluation outcomes.
As a comparison, Chatbot Arena (6) serves as robust benchmark for evaluating chat LLMs. It operates as a benchmarking platform where anonymous, randomized battles are conducted a crowdsourced environment. The platform's 's extensive real-world user queries provide comprehensive and less biased evaluations, ensuring the accuracy and stability of model rankings. Additionally, its real-time nature prevents models from overfitting the benchmark, thereby avoiding contamination issues. However, obtaining stable model score requires more than 5,000 human interactions and several days, making the process labor-intensive and slow. Furthermore, its open-ended format limits its ability to evaluate base models. Many open-ended queries, such as creating a travel plan, lack definitive standards for distinguishing good from bad answers (30], resulting in evaluations that are not purely ability-oriented.
Web Query Detection
Currently, real-world text-in-text-out user queries are primarily sourced from chat platforms (6]
27
Our concurrent work, MAmmoTH2 (34), also identifies real-world user queries from the web. However, MAmmoTH2 has fundamentally different objectives compared to Mix xEval. MAmmoTH2 focuses detecting large-scale -specifc query-answer answer pairs, while MixEval targets general-purpost user queries that accurately reflect the real-world user query distribution. This difference in objectives results in distinct web query detection pipelines.
38 39
C Considerations of Web User Query Crawling
Our user queries are not directly crawled from the web; instead, they are identified using Common Crawl, an openly available corpus of web crawl data widely used in the research community, Fur- thermore, we do not release the raw detected queries, while only releasing the final mixed version of MixEval for two reasons: (1) the raw detected queries may contain toxic content unexpected sen- sitive information, and (2) we update our benchmarks dynamically to avoid contamination. Releasing the detected raw queries would make the dynamic benchmarking process more predictable, reducing its effectiveness.
D Additional Benchmark Statistics
To ensure quality and comprehensive sample coverage, we selected the development and test splits of widely adopted benchmarks encompassing various domains and topics.
General-domain benchmarks: MMLU [17], BoolQ (9). HellaSwag 35
ARC [10), Com-
monSenseQA 29 AGlEval 40), OpenbookQA 21 GPQA 24
WinoGrande [25]
TriviaQA (99 DROP (14) and BBH (2
Domain-specific benchmarks: Math: GSM8K 24 and MATH [8]: Coding: MBPP 1 and
HumanEval 55: Physics: PIQA
4 and Social Interactions: SIQA (66]
15

---

0.30
MiXeval Original
0.25
0.20
0.15
0.05
TrviaQA
MMLU
DROP
HellsSwag
Cammon:
sense
BaoiQ
GSMak
ACIEVO
SIOA
PIQA
ARC
BBH
Open-
BookQA
MATH-
OPQA
Wine-
Grande
MBPP
Humen-
Eval.
Figure 8: The normalized number of queries in MixEval and the original benchmarks.
According to Figure 2 the mixed benchmark MixEval exhibits the highest overlap with Q among all benchmarks, suggesting that B adequately represents the wild query distribution. Let B' {Bi. By B,, denote the mixed benchmark. The distributions of B and are illustrated in Figure A positive correlation between the sizes of the mixed and original benchmarks is observed. Intuitively, larger benchmark is likely to be retrieved more frequently; however, this is not universally true. Benchmarks with skewed sample distributions, such HellaSwag. GSM8K, ARC, BBH, MATH, and GPQA, have a smaller relative size after mixing. This indicates that both quantity and distribution influence how frequently a benchmark is retrieved by Q. Overall, the retrieved benchmark splits exhibit a long-tail distribution.
E Implementation details for Benchmark Correlation Matrix, Query Distribution, and Evaluation Cost
Correlation Matrix Heatmap (Figures I and 10). We present the correlation matrix of prominent benchmarks, where warmer colors indicate higher correlations. Model scores are collected from various sources, including the Chatbot Arena Leaderboard [6). Open LLM Leaderboard [15] and OpenCompass Leaderboard [13]. Our data collection adheres to three principles: (1) We exclude scores reported by model authors, relying solely on evaluation leaderboards to ensure faireess. (2) For each benchmark, scores are sourced from a single platform to eliminate the influence of varying evaluation settings model rankings. (3) When multiple sources are available for benchmark, we
select the one with the highest number of models in common with other benchmarks. The number of common models for each pair of benchmarks is detailed in Figure
Query Distribution Map (Figure 2). We present the distribution of benchmark queries sorted by their distance to our detected web queries. Each benchmark (orange or yellow) is plotted against the detected wild queries (blue). We uniformly sampled 1000 queries from each LLM benchmark and wild dataset, with a sampling number of 200 for MT- Bench and trena-Hard due to their smaller sizes. We combined the query embeddings and reduced their dimensions the same 2-D space to facilitate direct comparisons of the benchmark query distributions. A detailed case study revealed that the reduced space primarily represents the topics of the queries, with queries on similar topics clustering in specific regions of the map. To better understand the topic distribution of different benchmarks, we divided the map into 16 patches based on location (Figure 3.. We then uniformly sampled 100 queries from each patch and used GPT-4 summarize the topics of the sampled queries. As illustrated in Figure 3 the 2-D query distribution exhibits a distinct regional trend: queries located higher on the map are more technical. The distribution transitions from non-technical topics, such as Social Interactions, at the bottom to technical ones, such as Programming and Mathematics, at the top.
Evaluation Cost Estimation. As illustrated in Figure 9 we consider two costs when evaluating the performance
on each benchmark: the inference cost and the judging (scoring) cost. The inference cost computation for straightforward, involving only the estimation of model input and output tokens for each benchmark. We estimate the model output tokens to be 20 for ground truth-based benchmarks and 329 for open-
and L---aa-j benchmarks are
16

---

Natural NuturalQuastion Quaations
Arerie Eto
MixEval MixEvat-Hard Hard
Arena Elo (En)
MixEval
Arena- Hard
AfC--c
HMLU
Ba-BBhhhh
6BH
OSHeK
CommersSemant
HelaSSnng
DHCUP
ARC
Trivag4
WineGrande
MATH
ActEval
OpenbrokKOA
HumanEsal
Boniq
TranhtigA
PIDA
HBPP
LAHBADA
Inter
58
0.078
0:27
0.089
0:99
6.085
1.5
0.058
0.15
0.39
0.022
0.016
0.039
0.085
0.38
2.0wm05
Judgo (8)
0.52
25
0.78
5.4
0.58
4.8
0.5
1.5
0.59
27
1.6
0.28
0.089
1.7
0.41
0.3
0.99
23
200
Total (S)
0.09
2.8
25
0.82
94
1u
6.89
5.6
6.4
0,96
1.8
9.6
0.28
0.11
0.45
0.92
1.1
3.2
Figure 9: Evaluation cost breakdown for the cost estimation in Figure
The total evaluation cost
broken down into the inference cost and judge cost.
ended benchmarks To compute the evaluation cost of EPT-3.5-Turbo--111 on the Chatbot Arena, we use the voting number of GPT-3.5-TTubb-- 25 on the Chatbot Arena leaderboard as its query count, with each query's token count estimated from the Chatbot Arena Conversations dataset. Since models on Chatbot Arena are evaluated pairwise, both input and output tokens are doubled. The c for boound-truth-bee output tokens of the model parser. However, estimating the human judgment cost for Chatbot Arena is more complex. We reference the crowdsourcing price for Amazon Mechanical Turk (MTurk), specifically the rate for Facebook Account Holde 10 ($0.05 per vote). Under this pricing scheme, evaluating a single model on Chatbot Arena costs approximately $2936, making it a highly expensive process.
judging cos
benchmarks is estimated similarly, accounting for the input and
F What Affects the Correlations with Human Preference and Other Benchmarks?
Both comprehensiveness and other features, such as difficulty, make an impact. As shown in Figure 10 general-domain benchmarks tend to achieve higher correlation with human preference compared to domain-specific benchmarks, underscoring the significance of benchmark query com- prehensiveness. However, comprehensiveness is not the only factor that matters. Three observations support this: (1) Some benchmarks with highly skewed distributions, such as GSM8K, achieve promising correlation (0.78) with human preference, while others with high topic overlap with real-world queries, like BoolQ, achieve a low correlation (0.37). (2) Despite their similar topic distributions, ARC-e and ARC-c show significantly different correlations with human preference (Figure 10), likely due to their differing difficulty levels, as they are known to have different levels of difficulty. This illustrates that other important query features, such as difficulty, are not apparent on the 2-D map that primarily features topic distribution but are crucial to the correlation with human preference. (3) As shown in Figure 5 Mi xEval increases the correlation with human preference for each individual benchmark through benchmark mixture. For an individual benchmark, the topic becomes less comprehensive after the mixture because the mixed version represents subset of the original on the topic map: thus, in this case, the correlation gain does not stem from more compre- hensive topic distribution. These three observations suggest that the human preference correlation gain is not solely related to topic comprehensiveness, other features, such as query difficulty and topic density, also play crucial role in improving correlation scores.
Benchmarks that are highly correlated with human preferences also tend to be correlated with each other, whereas those that are less correlated with human preferences are similarly less correlated with most other benchmarks. The heat map reveals a consistent red region in the lef- signifying high correlation, while the rest of the map is predominantly blue and inconsistent, indicating low correlation. This suggests that model rankings on benchmarks closely aligned with
According to the averaged output token for GPT-3.5-Turee Turbo-0125 as presented at: hutp:://mnsys.org//loo/22224 04- ao--rrna-harree
'The judge cost Arena-Hart and MT-Bench directly taken from http:://aay..oggbbb/222200040099 arena-hard/.
17

---

CommonnSenneed nA
HellaSwar (D-shot) eh--hhtt
HumanEvvl (2-shog)
LAMBADA shat))
Arena (En)
MixEval (0- (0-shot)
Arena-Hoa Hard
ARC (0. (o-ahot)
( s--hhot)
11-Benchh
89H shhtt)
3SMeK(4-ehoo
DROP (2 ahoth
ReturalQuestions
ARC. (0-00 (o-ahot)
Tri aQA (o shot)
WinuGrande
MATH (4-HH(4-shoo))
AGiEval (C-shot)
SpenbaakQA
Bool (0 shot)
TruthfuiQA
PIQA shot)
MBPP (1-ehot)
SIOA (0- (0-ehoot)
MixEval
(0-shot)
(8-eho0)
(0 (0-ahot)
(b-shot)
(0-shot
Arana
Arena Elo
100
96
93
86
85
78
78
72
63
38
51
55
50
50
44
44
36
35
28
-17
HarEval-Hard
[0-shot)
96
100
85
96
86
71
83
88
69
67
66
41
84
63
73
55
74
63
56
64
11
45
62
33
30
27
Arona Bo (En)
96
95
100
83
88
82
76
76
72
68
61
56
50
54
50
47
44
41
33
40
33
26
18
MixEval (0-shot)
96
7B
74
78
73
71
28
83
3Z
36
ARC-c (0-shot)
85
21
78
94
100
6-
62
58
47
84
41
56
62
74
31
35
66
44
36
MMLU (5-hoo))
63
83
82
82
88
78
100
78
66
63
30
70
39
45
69
52
48
73
67
67
43
51
31
57
50
36
37
MT-Bench
80
88
90
92
86
78
00
78
88
6B
53
84
2.3
60
28
83
78
44
61
47
111
39
-13
-54
(3-ahot)
69
(4-shot)
78
94
100
65
48
38
Ca
(8: shot)
72
69
72
77
94
68
39
36
36
100
60
35
64
25
30
47
28
49
34
4.
46
51
24
26
-2
HellsSwag (0-shot)
71
41
88
35
54
74
70
53
67
58
60
100
56
59
84
50
74
59
80
63
38
75
14
85
58
46
35
DROP shot)
69
84
89
94
58
39
62
71
41
100
51
68
28
54
35
41
38
44
52
39
26
NeturaiQuestions
(0-shot)
30
ARC-e (0-shot)
83
84
63
55
54
46
100
30
67
65
81
33
82
85
51
TriviaQA (0- (0-shos)
51
5:5
50
42
37
41
52
28
50
28
25
50
28
68
30
1:00
45
23
25
15
12
37
44
15
11
45
WinaGrande
(5-shot)
51
74
64
66
60
48
67
63
30
74
73
60
66
45
100
45
31
16
26
6-6
39
60
48
23
2.7
MATH (4-shot)
55
100
74
61
48
AGiEval (0 shot)
94
100
Openbook
(0-shot)
84
94
74
67
46
48
49
81
15
61
72
100
31
35
65
37
69
29
HumanEval (2-shot)
44
11
41
23
54
31
43
61
67
69
54
38
38
38
33
12
26
65
50
31
100
24
85
2.0
BoolQ (0-shot)
37
.4
14
70
51
51
40
45
42
72
37
32
40
57
100
19
78
33
63
TruthfulQA
(0-ahot)
35
100
-35
PIQA soot)))
35
100
MBPP (1-shot)
28
26
9
8
44
60
86
89
24
58
38
61
16
60
58
37
83
-3
41
100
28
27
SIQA (0 (0-shot)
-17
30
18
40
9
55
36
13
31
36
25
45
42
34
67
11
37
48
69
20
6.3
3
59
28
100
LAMBADA (0-shot)
-10
800
28
-54
32
41
30
29
-55
50
27
20
40
80
100
Figure 10: The correlation matrix for benchmarks. MixEval and sixEval-Haars achieve the highest correlations with Chatbot Arena Elo. Each value of the heatmap represents the Spearman's 's rank correlation (%) between the model rankings of the corresponding benchmark pairs, where warmer color indicates higher correlation and a cooler color indicates a lower correlation. The underlined numbers indicate the data for the corresponding benchmark pairs are insufficient (<15 models). The detailed statisties on the number models used for each pair of benchmarks are presented in Figure II]
human preferences are more stable and reflect a "True" ranking, whereas the remaining benchmarks exhibit greater variability in model rankings.
Benchmarks within the same domain exhibit higher correlations. Despite a low correlation with human preferences, some domain-specifte benchmarks demonstrate relatively high correlation with other benchmarks in the same domain. For instance, MBPP shows a correlation of only 0.28 with human preference but a substantial 0.83 with HumanEval. Similarly, MATH has a correlation of 0.50 with human preference yet presents 0.66 correlation with GSM8K. Furthermore, ARC-e has correlation of 0.58 with human preference while achieving notable 0.84 correlation with ARC-c.
18

---

HumanEval (2 shot)
MixEval Hard
Arena (En)
MixEval (D so-oho))
ARC ARC ( sh-shht)
MMLU (MLU (S-shot)
8aH shar)
OSM6K (4-66k(4-hho)))) (4-shoo)
Commer SennannSSSnSO (9-5 (9-shot)
HellaSwag (0-Swarg---hape (0-shot)
DROP (2 (2-ahott)
NeturalQueztione rosestooss
Triv (D shor)
WinoGrannd rande (5-shot)
AGiEval (C-avall (C-ahott
OpenboakCk
BoolQ t0 shott)
PIQA (0-shot)
MBPP (.chon
SIDA (00-ehottt
LAMBADA shot)
MT-Bench
(0-ahot)
ARC (0
(4
TruchfuiQk
Arana
Arena
AVG
22
29
22
29
32
17
31
31
31
31
31
31
31
33
31
31
31
31
31
31
31
17
17
17
17
MixEval-Hard (0-shor)
31
35
31
35
19
29
30
19
29
29
299
29
29
29
29
29
27
29
29
23
29
29
27
29
29
29
29
Arena Elo (En)
36
31
35
31
17
1/
1
17
1
17
36
17
1
17
17
17
Mixvval (0-shot)
31
35
35
299
30
13
29
29
2.9
26
29
2
29
27
29
29
29
28
29
29
29
Arena-Hard
34
19
34
19
34
6
1:
6
6
6
6
15
6
6
6
1.5
6
6
35
MMLU (S shos)
18
30
18
30
35
3G5
144
3G
35
35
36
3G
3G
35
36
36
3.5
355
35
36
35
3G5
35
35
36
35
MT-Bench
35
13
35
13
14
14
14
1
14
14
14
27
1.4
14
144
14
14
1.4
BBH (3-ahot)
17
29
17
29
35
355
35
35
35
35
36
35
35
35
35
35
GSM&K (4 shot)
17
29
17
29
36
35
14
3G
35
35
35
3G
3G
35
35
35
355
36
3G
35
35
35
36
35
35
3.5
CammonSense@A
(8-shot)
16
35
36
35
355
35
35
3.5
HellaSwag (0 shot)
17
29
17
29
35
36
144
35
36
35
36
3G
3G
35
36
36
3.5
36
3G
36
36
35
36
36
36
DROP (2-shot)
16
16
29
35
35
35
35
35
35
35
35
35
35
36
35
35
35
35
35
35
NaturalQuestions
shor
17
17
35
35
35
35
35
35
35
35
35
ARC-e (0-shot)
17
299
17
29
35
35
14
35
35
35
35
35
35
35
36
35
355
35
35
36
35
35
35
35
36
3.5
TriviaQA shosh
35
36
36
355
36
35
3/5
35
35
355
WineGrande (5-shot)
26
2.7
36
2/
35
355
2//
36
36b
35
36
315
36
36
36
36
316
3/
3/
36
35
36
36
35
MATH (4-shot)
17
29
29
35
35
35
35
35
35
35
35
35
35
36
35
35
35
35
35
35
35
AGiEval (0-shot)
OpenbookDA
(0-shot)
17
299
17
29
35
95
144
35
35
35
35
35
35
35
36
35
35
35
35
36
35
35
35
35
36
35
HumanEval shosh
35
35
36
315
36
355
35
35
35
BealQ (0-shot)
17
29
17
29
35
35
14
35
36
355
36
36
36
35
36
36
385
36
36
35
36
36
35
TruthfulQA
(0-shos)
36
277
35
27
35
35
27
36
35
355
35
3G
36
35
36
36
3.5
315
35
36
36
3.5
36
35
35
3.5
PIQA (0-shot)
MBPP (1-ahot)
17
299
17
29
35
95
14
35
35
35
35
35
35
35
36
35
355
35
35
36
35
35
35
35
36
355
SIQA shott
36
36
36
36
35
35
35
LAMBADA (0-shot)
17
17
29
35
35
35
35
35
35
Figure 11: The number of models used for each pair of benchmarks shown in Figure 10
G Detailed Evaluation Results
Table presents our evaluation results on MixEval, MixEval Sarival-Harae Hard, and their main subsets. Overall, Claude 3 Opus and GPT-4 Turbo consistently achieve the highest performance across nearly all splits, except for the BoolQ-Mixed split. Gemini 1.5 Pro ranks third on both Mi MixEval-HH -Hara and Mi xEval, followed closely by Claude 3 Sonnet, -aa-M----0---imme comparable scores. Notably, all these frontier models, except multi-modal input understanding. The LLaMA-3-8B -----srruttttt model is the top-performing 7B model, outperforming some of the latest large models, such as Command R (35B) and Qwen1.5-322 1-5-32B-hhat (32B). In general, proprietary models outperform open-source models.
and Reka Core, with support
19

---

Original
Correiation with Arena ELO (Em)
100
Mixed
MixEval-Herd MIXEva!
61
82
80
.62
72
Senchmerk-lewe Uniii Mix
60-
50
BB
33
33
TrisiQA
MMLU
DROP
HellaSwag
Common- ScnseQA
BoolQ
GSMBK
AGIEval
PIQA
-15
SIQA
Figure 12: Our approach improves the correlation with Arena Elo and Arena Elo (En) for all the main splits of MixEval and outperforms benchmark-level and uniform mixture.
Table 3: The Evaluation results of chat models Mi xEval, Mi MixEval- and their sub-splits.
Eval Harr
Mix-
TrivinQA- Mixed
MMLU-
Mixed
DROP-
Mixed
HellaSwag- Mixed
senseQA- Mixed
Common
BoolQ- Mixed
TriviaQA-
Hard- Mixed
MMLU.
Hard-
DROP-
Hard-
Mixed
Mixed
Progortion
100%
100%
31.2%
21.4%
12.4%
7.4%
5.3%
4.4%
26.6%
23.1%
16.7%
Claude Opus
63.5
88.1
90.4
83.2
91.5
93.3
87.7
85.1
71.4
55.0
75.2
GPT-4 Turbo
62.6
88.8
91.2
82.8
91.0
92.6
85.4
88.0
73.1
45.5
71.0
Gemini 1.5
38.7
85.3
79.2
89.2
84.4
87.4
44.6
64.8
LLaMA-3-700 1-3-7-B--nnstuug Instruct
55.9
84.0
83.1
80.5
90.1
81.8
83.0
88.6
60.5
46.1
74.5
Sonnet
40.7
Reka Core
52.9
83.3
82.8
79.3
88,1
88.6
81.6
85.7
51.6
46.3
66.6
Command
51.4
81.5
83.3
78.9
80.4
83.5
82.1
86.9
42.0
65.0
Mistral-Large
50.3
84.2
88.3
80.2
88.6
65.0
83.5
87.4
55.5
42.4
61.6
Qwen1. Breen1.--28-CCM 5-228-hhaa
48.3
84.1
83.9
80.
85.1
87.9
86.3
85.7
49.9
37.7
$6.5
Mistral-Medium
Gemini 1.0 Pro
46.4
78.9
$1.0
74.9
82.6
74.7
80.2
86.3
$8.2
35.5
$4.1
73.4
77.8
Reka Flash
46.2
79.8
76.4
75.4
86,7
90.6
80.7
85.7
42.9
34.6
65.0
LLaMA-3-8B-Lmss
45.6
75.0
71.7
71.9
86.4
65.7
78.3
86.9
40.2
40.7
67.6
Command R
45.2
77.0
80.9
72.0
75.8
77.4
39.0
42.0
Qwenl.5-328 --32B-hhhtt
43.3
81.0
75.7
78.0
82.9
85.9
88.2
85.1
39.1
29.9
$4.4
GPT-3.5-Turts
Claude 3 Haiku
42.8
79.7
79.9
76.1
85.0
75.8
78.8
86.9
42.4
30.7
51.5
Yi-34B-Chat
42.6
82.7
86.9
78.8
29.9
57.1
(istral-8x7B-Instteee
42.5
76.4
82.5
72.0
79.5
$4.2
77.4
82.9
48.5
37.2
47.7
Starling-LM--7 LB-BB--BB-- 7B-bta
41.8
74.8
75.1
69.0
86.4
48.5
84.9
84.6
33.4
34.2
62.9
Gemma-1.117B-I
39.1
64.3
66.3
73.6
39.0
55.1
Vicuna-331-vv11
38.7
66.3
79.2
$9.2
71.4
30.3
618
73.7
42.5
39.4
36.6
Chat
27.7
36.2
70.0
73.7
67.3
72.8
54.2
66.0
73.7
33.5
29.4
44.3
Qwen DwenI.5-7B-Chat
64.1
76.4
76.1
82.1
81.7
29.0
50.0
Reka Edge
32.2
68.5
60.0
63.6
80.0
74.7
80.7
77.1
18.6
26.4
56.9
Zephyr-7B-
31.6
69.1
74.7
64.9
77.3
39.1
69.3
78.3
30.2
24.2
45.3
LLaMA-2-9B-Chat
35.7
44.3
Yi-BB- Chat
30.1
65.6
66.1
65.4
70.5
$2.5
69.8
69.7
18.9
26.8
43.7
Qwen1.5. MoE.22.7
72.7
Demma--1.1220 1--2B-IT
28.4
51.9
53.7
51.5
$9.8
26.6
57.1
60.0
31.9
30.3
27.8
Vicuna-7B-v1.5
27.8
66.4
24.9
62.7
65.7
25.9
23,4
33.2
OLMo-7B-Insdee Insttruct
26.7
55.0
51.7
57.1
55.9
64.6
71.4
27.3
22.9
Qwen1.5 Chal
24.6
57.2
46.0
61.4
57.2
54,9
74.1
68.0
16.5
17.3
28.6
JetMoE-8B-CUn
25.5
MPT-7B.C Chat
23.8
43.8
50.2
37.8
50.0
25.6
36.3
60.0
17.5
24.7
31.0
H
Error Analysis
Figure illustrates the error rates of the models evaluated on the main splits of MixEla.. We separately compute the errоr rates for proprietary and open-source models to facilitate comparison. Both model types exhibit significant errors on the AGlEval split of Mi Mivala,, underscoring its difficulty. In contrast, performance on the PIQA split is generally saturated. Notably, there is substantial performance gap hetween proprietary and open-source models on the GSM8K split, with considerable gaps also observed on the HellaSwag, TriviaQA, and DROP splits.
20

---

Open-source Proprietary All
0.4
Eror Rate
0.3
0.2
0.0
AGiEvol
HelleSwag
SIQA
NMLU
TrivisA-
Common-
SeneeQA
GSMBK
DRO)
BoolQ
PIQA
Figure 13: Averaged error rates of open-source, proprictary, and models on MixEval splits.
made by frontier proprietary models. For each case, we present the incorrect responses from each model. We identify three primary causes of confusion for these models: strong domain knowledge, complex reasoning, and vague question definitions. Additionally, we identify several annotation issues within current benchmarks, though these negligible in number.
In Section H.1
conduct studies to examine the
H.1
Error Cases
Error Case
Question:
Letter the editor: You say that if the government were confiscate portion of the wages convicted burglars when they reenter the workforce, would a form stealing, hence an abuse power. Yet under the proposal now being considered, the govermment would confiscate such wages in order to fund account compensate burglary victims. So even if confiscating portion of burglars' wages form of stealing, it would still be justified.
Which one following principles, if valid, most helps to support argument in the letter to the editor?
A. Money stolen from burglar should be given to that burglar's victims.
B. Burglars are obligated to provide compensation the individuals they victimized. The motive prompting action determines whether not that action justificd.
A crime is justified only
means compensating people who deserve compensation.
E. Stealing is never justified even it benefits someone who has been a burglary victim.
Answer with the option letter from the given choices directly.
Answer:
Split: AGIEval
Main Causes of Errors: Hard Reasoning
Wrong Answers:
Claude 3 Haiku: The principle that most helps to support the argument in letter the editor is (B) Burglars are obligated to provide compensation to the same individuals they victimized.
The letter argues that if confiscating portion of burglars' wages is form of stealing. it would still justified because purpose fund an account compensate burglary victims. This suggests that the burglars have an obligation to provide compensation to the individuals they victimized, which aligns with principle (B).
Claude Sonnet: D
Claude 3 Opus:
Gemini 1.0 Pro: B
21

---

GPT-3. PPT-3.5--uB 5-Turbo: D. A crime is justified only if it is means of compensating people who deserve compensation.
GPT-4-Turbo:
Mistral Sattral-mmal:: The principle that most helps to support the argument in the letter to the editor B. Burglars obligated provide compensation the same individuals they vie- timized. This principle aligns with the argument that even if confiscating portion of burglars' wages were form of stealing, it would still be justified because serves to compensate burglary victims.
-estral-Mediin (B) Burglars are obligated to provide compensation to the same indi- viduals they victimized.
Explanation:
The principle that supports the argument in the letter to the editor is that burglars have obligation to provide compensation the individuals they have victimized. This principle justifies the government's confiscation of portion the burglars' wages fund an account to compensate their victims. While the letter acknowledges that confiscating wages could be considered form of stealing. it argues that this action would still he justified as a means of providing compensation to those who deserve it. Therefore, option (B) is the best choice among the given options it directly supports the argument in the letter. Options (A), (C), (D), and (F) do not directly address the obligation of burglars provide compensation their victims, which is erux of the argument.
Mastral-- -argee The principle that most helps to support the argument in the letter to the editor (B) "Burglars obligated provide compensation to the same individuals they victimized." This principle justifies the confiscation of portion burglars' wages means to compensate their victims, which is the main point of the argument in the letter.
Reka Edge: The principle that most helps to support the argument in the letter to the ed- itor is +*(D) A crime justified only if is a means of compensating people who deserve compensation. ** This principle suggests that a criminal act can only justified if results some form of compensation for the victims of the crime. In the scenario described in the letter, the government is proposing to confiscate portion of the wages of convicted burglars to fund an account to compensate burglary victims. The argument is that even though confiscating portion of the wages of hurglars may be form of stealing, is justified because it means of compensating people who deserve compensation. Thus, act of confiscating wages this case seen as a legitimate way achieving a just outcome (compensating victims).
Reka Core: The principle that most helps support the argument in the letter to the edi- tor
B. Burglars are obligated to provide compensation to the same individuals they victimized.
The argument in the letter acknowledges that confiscating portion of convicted burglar's contends
wages might considered stealing, the purpose of compensating burglary victims. This aligns with the principle that burglars have an obligation to provide compensation to those they have harmed. The letter suggests that even if the government's 's action considered stealing. it is still justified because it fulfills the burglar's obligation compensate their victims.
action still justified because serves
Error Case
Question:
which stage the planning process are the objectives decided on?
A. Defining the program
B. Planning the program
C. Taking action and implementing ideas Evaluation of the program
Answer option letter from the given choices directly.
22

---

Answer:
Split: MMLU
Main Causes of Errors: Strong Domain Knowledge
Wrong Answers:
Claude 3 Haiku: A. Defining the program
Claude Opus:
Gemin ni 1.0 Pro: A
Mistral- Small 1 A. Defining the program
At the defining stage of the planning process, the objectives are decided upon. This is where you outline the goals and desired outcomes for your program.
Mistral-Medimm
Defining the program
At the defining stage of the planning process, the objectives are decided upon. you outline the goals and desired outcomes for your program.
This is where
Reka Edge:
objectives are decided **A. Defining program**, During stage, the program's goals and desired outcomes are identified and clearly stated. These objectives serve as the foundation for the rest of the planning process and guide the development of the program's 's activities, timeline, and budget. Once the objectives established, they be further refined and prioritized during the planning and implementation stages, and evaluated the end of the program measure success.
Reka Flash: A. Defining the program. The objectives are typically decided upon during the defining conceptualization stage of the planning process, where the program' purpose, goals, and target audience are identified. This stage sets the foundation for the rest of the planning process. including the development of strategies, action plans, and evaluation criteria.
Reka Core: The objectives decided during the defining the program stage (Option
A). This the initial stage of the planning process where the program's purpose, scope, and goals are clearly defined. The objectives serve the foundation the rest the planning process, guiding the development of strategies, activities, and evaluation methods. Once the objectives are established, the planning process moves to the next stage. which is planning the program (Option B), followed by taking action and implementing ideas (Option C), and finally, evaluating the program (Option D).
Error Case
Question:
How many men has actress Flizabeth Taylor been divorced from?
A. five
B. six
seven
cight
Answer with the option letter from the given choices directly.
Answer: C
Split: MMLU
Main Causes of Errors: Strong Domain Knowledge, Hard Reasoning, Vague Definition of the
Question
(Elizabeth Taylor, the iconic actress, was married a total of eight times to seven different men.
She married Richard Burton twice. Here is a list of her husbands: Conrad Hilton (1950-195]) Michael Wilding (1952-1957), Michael Todd (1957-1958). Eddie Fisher (1959-1964), Richard Burton (1964-15 1974), Richard Burton (1975-1976), John Warner (1976-1982) Larry Fortensky (1991-1996). Here Michael Todd died a plane crash. consider death as a form divorce, then answer C,
23

---

otherwise B.)
Wrong Answers:
Claude Scnnet: D. eight
Gemini 1.0 Pro: A
Gemini 1.5 Pro:
SPP-3.5-Turbbo. D. eight
GPT-4-Turbo: A. five
Mistral-Medinme B. six
Elizabeth Taylor was married eight times, but she was divorced six times two of her marriages ended duc to the death of her spouses. The men she divoreed Conrad Hilton Jr. Michael Wilding, Mike Todd, Eddie Fisher, Richard Burton (twice), and John Warner.
Reka Edge: According to various sources and biographies, actress Elizabeth Taylor has been divorced from **six ** men. Therefore, the correct answer is **B, six**.
Reka Flash: Elizabeth Taylor was married eight times and divorced seven of those times. So, the correct answer D. eight.
Reka Core: Elizabeth Taylor married eight times to seven different men. So the cor-
D. eight.
I Rule Parser and Model Parser Comparison
As illustrated in Figure 14 the rule parser exhibits instability in both free-form and multiple-choie splits, with pronounced effect in the free-form case. Manual inspection reveals that multiple- choice scenarios, the discrepancies primarily arise from the rule parser's failure to capture the diverse output styles of different models. In contrast, the discrepancies in free-form scenarios stem from the expansive output space of free-form questions and the varying annotation comprehensiveness across different splits of Mi xEval.
24

---

0.8
Model Parser
Rule Parser
0.7
0.5
GPT-4-T O97-4-Turbu Turbu
Cloude S Opus
Qwen! Qwen1-s-72B- 1-111------0 Chas
Geminii Pro
Deka flash
Mistal- ango
Mounll-Hediun
Cloude Halku
Claude Sunnes
Mistral Small
Commund
(1-S4B. Che-
GPT-3.2 S. Turbo
Queen. 2-- Chet
LLaMA s 88 inttruc
Mistral Bex7B-lmes (b-imerust--O
Qwant Qwin1.5-Maa (5-MM Ao 78- Ch--
LLMM 708 Chas
Stariin UH 70- betb
Mistral- 78. Be-inntrucctoi No..
Y--i Chat
IntMo 8B chaz
qwant. 5-40: chat
Zeghoy Zeplhy--78-g -78--
OlMM 78-------o Ba-inatuues
LLaM 2-70. Chax
Vizune- --B--------3
Visune 70------11..
Gemman- 1-2--T-
MPT-7B- chat
Commanc
Palka
Raka
Gemma-
(a) Multiple-choice
0.9
Model Parser
Rule Parser
0.8
0.6
0.5
0.4
0.2
GPT-44-uurbo
Clauda s Opus-
MistralLLage Claude Somnet-
Mistal-Madin Maduum-
LLakA 3--00 707-7nn-n -on--nmmuee
Mistral Small
GPT GPT-3.S-Tude 3.S-Turbo
Qwwn1 15-728--5 Chat-
Reke Care-
Yi- 11-34BBB Chat-
Command Rt-
Cloude Heltu
Qwen1 S2B Chat
Starlingt MM-.gg--MM-------- 70 bets
LLaMA-2 2: 709 Chat-
Commandd I
LLMM-- 30----888 Ina---rrut
Vtcuna-SSB-e (aenn-----11-. 1.3
Zephyr- 70-p.
Instruct--..2 0..2
Gomma-11 1------8 -T-TT
Qwen1.. 5-78 Chat-
Reka Edge
Qwant. 55 MaE 42 7B Chat
Y1-B Chat.
LLaMW-2-7B- 22-77--hhu Chut
Vicuna-7B-r i-
Gemme-1, 1.1 1-2-111
Owant 5-1------00 Chat-
OLMe-78 inntuuu
JutMor an Chat-
Qovent 1.88 Chat-
1.0
Raka
MPT-70-00
(b) Free-form
Figure 14: The score differences computed by model parser and rule parser on Mixaval, The rule
parser is unstable on both free-form and multiple-choice splits, especially free-form.
J Model Parser Prompts
Free-form Parser Prompt
System:
In this task, I want act a judge.
User:
You be provided with a question, its golden answer(s), and model's answer, while the context
25

---

the question is not given here. Your task is to judge how correct the model's answer is based on the golden answer(s), without seeing the context of the question, and then give correctness score. The correctness score should be of the below numbers: 0.0 (totally wrong), 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0 (totally right). Your should first briefly give your reasoning process regarding how the model's answer conforms The correctness score must strictly follow this format: "[[score]]", e.g.. "The correctness score: [[O.5]]". Below are some examples.
contradicts golden answer(s), and then give correctness score.
Example
Question: Sandy bought million Safe Moon tokens. She has siblings. She wants keep half them to herself and divide the remaining tokens among her siblings. After splitting it up, how many more tokens will she have than any of her siblings?
Golden Answer(s): <answer 1> 375000
Model's Answer: Sandy will have more tokens than any sibling by 3/8 million.
Your Judgment: The golden answer states that Sandy will have 375,000 more tokens than any of her siblings, which is precise numerical value. The model'ss answer translates this scenario into fraction of the total, saying Sandy will have more tokens than any sibling by 3/8 million. 1 million tokens 3/8 = ,000 tokens. the model provided an answer fractional form that, when converted a numerical value, exactly matches the golden answer's quantity. The correctness score: [[1.0]].
Example 2:
Question: what used in the movie christine
Golden Answer: <answer vintage 1958 Plymouth Fury: <answer 1958 Plymouth Fury
Model's Answer: Christine.
Your Judgment: The golden answers specify the car used in the movie "Christine" as vintage 1958 Plymouth Fury, providing clear and detailed response including the make, model, and year of the car. The model's answer, though points the car's alias the context of the movic "Christine" is precise and specific enough. The correctness score: [[O.5[].
Example 3:
Question: 2015 Edgar Lungu became prime minister of?
Golden Answer: <answer 1> Zambia; <answer 2> Zamibia; <answer 3> People Zambia; <answer 4> Zambian cuisine; <answer 5> Zambians; <answer 6> Culture Zambia; <answer 7> Etymology of Zambia; <answer 8> Zambia; <answer 9> Health care in Zambia; <answer 10> ISO 3166-1 I:ZM; <answer 11> Republic Of Zambia; <answer 12> Cuisine of Zambia; <answer 13> Sport in Zambia; <answer 14> Republic of Zambia; <answer 15> Zamhian people; Model's Answer: Prime Minister
16> Name of Zambia
Your Judgment: The golden answers provide a detailed list of entities all relating to Zambia, indicating that Edgar Lungu became the leader (specifically, they mentioned "prime minister") of Zambia in 2015. The model's answer, "Prime Minister." merely repeats part of the question without answering it. The correctness score: [[0.0]].
Note that each one of the golden answers is considered correct. Thus if the model's answer matches any one of the golden answers, it should be considered correct. Judge the below case, give the brief reasoning process and the correctness score.
Question: <prompt>
Golden Answer(s): <golden answers>
Model's Answer: <model response>
Your Judgment:
Multiple-choice Parser Prompt
System:
In this task, I want act as an option extractor.
User:
will provided with multiple-choice question, its options, the model's answer, while the
26

---

context the question is not given here. Your task extract or judge which option is chosen by the model based on its response, without secing the context of the question. The extracted option should be of the provided option letters. Your should first briefly give your reasoning process, and then give the extracted option letter. The extracted option must strietly follow this format: "[[option letter]]" e.g.. "The option chosen by the model: [[A]]". Below are some examples.
Example 1:
Question: Which technology developed most recently?
Options:
cellular telephone
B. television
C. refrigerator
airplane
Model's Answer:
The technology that was developed most recently is airplane.
Your Judgment: The model's response directly identifies "D. airplane" as the technology that was developed most recently. This indicates that the chosen option is D. The option chosen by the model: [[D]].
Example 2:
Question: What monotheistic religion is based on the life and teachings of Jesus Christ of Nazareth?
Options:
Islam
Christianity
C. Hinduism
D. Buddhism
Model's Answer: B.
What the name of the first person be executed by the electric chair? John Wilkes Booth B. William Kemmler C. John Dillinger D. Bonnic and Clyde Answer with the option letter
Your Judgment: The model's response clearly identifies "B. Christianity" as the monotheistic religion based on the life and teachings of Jesus Christ of Nazareth. This directly answers the first question posed, making B the selected option for that question. The additional content appears introduce new, unrelated question without providing an answer The option chosen by the model: [[B]].
Example 3:
Question: Which solution is correct?
Options:
provide homes for people
B. provide homes for animals
Model's Answer: Neither A nor B is entirely correct because trees do not provide homesin the traditional sense. However. they do provide habitats and shelter for various organisms, including animals. you had choose between the options given, option (for animals) might be more accurate in the context trees being habitat. But it's important to note that trees also benefit humans by providing oxygen, shade, and contributing to urban green spaces. If you need to select one option, I would suggest: B. provide homes for animals
Your Judgment: The model's response indicates a preference for option mentioning that if one had choose between the given options, "B. provide homes for animals" would more accurate, especially in the context of trees serving as habitats. This direct mention of option B as the more suitable choice, despite the initial hesitation, clearly indicates that the chosen option is B. The option chosen by the model: [[]]].
Question: <prompt>
Options:
<options>
Model's Answer: <model response>
Your Judgment:
27

---

1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We carefully inspected the main claims we made in the abstract and introduc- tion, and ensure that of them are appropriate.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
Limitations are detailed as part of Section A in the form of QAs.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There is theoretical result made.
28

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All implementation details are provided (in Experiment settings and Appendix).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
29

---

We uploaded the code and data, and will release and periodically update the benchmark data and its related code.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (ht.:p:://iip:) cc/
public/ htt:/giiiess /CodeSumm ssi cesionnolicn for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
httpps.ccppubllce cwt.bbicccuiiieee Subee ss onPol oollice icy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We carefully reason our choices in our method and experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide significance test for the dynamic benchmarking by running 5 versions of random seeds. While for other experiments, it's too expensive to run multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
30

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify the hardware we use in the Experiment Settings.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/eeeurpp..c//u/bl//////tt
Answer: [Yes]
Justification: We carefully read and follow the NeurlPS Code of Ethics through out the whole research process.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This research focuses on an improved benchmarking method for LLMs, which does not directly involve significant societal impacts. Its primary influence is within the LLM research community, detailed in Section
Justification;
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Discussed in Section C
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite all the previous work, datasets, software use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper thcode com dataaetteett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We upload and release data and code with detailed instrutions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not available.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "YPRAprEFRe", "reviewer": "Reviewer_yGD1", "summary": "This paper introduces MixEval, a new approach/benchmark to evaluate LLMs effectively in real-world scenarios. Traditional benchmarks often miss the comprehensiveness and subtlety of actual user queries, while existing methods like LLM-as-judge benchmarks are difficult to scale up. MixEval addresses these issues by using user queries mined from the web and aligning them with similar queries from established benchmarks.\n\nThe key advantage of MixEval is its efficiency and dynamic nature. It achieves a high correlation with Chatbot Arena (0.96 Spearman correlation) but requires only 6% of the time and cost compared to mainstream benchmarks like MMLU. MixEval can also be updated quickly, significantly reducing the risk of benchmark contamination. The paper also introduces MixEval-Hard for better differentiation among strong models. Through comprehensive analysis, the authors demonstrate that MixEval offers accurate, less biased, and more dynamic evaluations, providing a potential scalable solution for real-world LLM assessment.", "questions": "I am a bit confused about Table 2. The authors briefly mention \"periodically update\" in the main text, but I did not fully understand the details of this update. Could you explain this further?", "limitations": "In the appendix, the authors address some limitations in a Q&A format, such as the potential biases in MixEval. My views on limitations are reflected in the weaknesses above.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Aligning web queries with mainstream benchmarks to simulate real-world user preferences is a novel approach. The authors cleverly transform the challenging task of evaluating open-ended queries into a benchmark mixture with groundtruth-based results, which is an interesting idea.\n2. MixEval is an effective alternative to ChatBot Arena as it can scale up, dynamically update, and has lower costs.\n3. The authors provide a comprehensive analysis of various LLMs on MixEval, demonstrating high correlations with ChatBot Arena.", "weaknesses": "**Major Issues**:\n1. MixEval dynamically updates by mixing popular benchmarks (e.g., MMLU, BoolQ, GSM8K), which may not mitigate contamination. Most of these benchmarks are saturated and suffer from contamination. Although the authors claim they will \"dynamically expand our benchmark pool with newly released benchmarks to further enhance the mixed benchmark distribution,\" this does not address the root issue of contamination.\n\n**Minor Issues**:\n1. While the topic distributions in Figure 2 are impressive, most of these benchmarks were not designed to simulate human preferences, hence the skewed topic distributions may be intentional for evaluating specific tasks, domains, or capabilities. Therefore, the section title \"LLM Benchmarks are Biased from Realistic User Queries and Preferences\" seems biased, implying these benchmarks are meant to measure human preferences.\n2. Another potential issue is the assumption that ChatBot Arena serves as the groundtruth for calculating correlations. As noted by the authors in lines 127-133, ChatBot Arena itself is biased. Therefore, even though MixEval has a larger user base, it might overfit to ChatBot Arena to some extent, making it more of an alternative rather than a more accurate evaluation of user preferences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding this work novel, effective, comprehensive, and solid! Below are our responses to the concerns:\n\n# Major Issue:\n\n## Concern 1\n> MixEval dynamically updates by mixing popular benchmarks (e.g., MMLU, BoolQ, GSM8K), which may not mitigate contamination. Most of these benchmarks are saturated and suffer from contamination. Although the authors claim they will \"dynamically expand our benchmark pool with newly released benchmarks to further enhance the mixed benchmark distribution,\" this does not address the root issue of contamination.\n\nWe thank the reviewer for this insightful comment!\n\n**We analyzed the dynamism and contamination of MixEval in detail in the general response to all reviewers and AC (on the top of this page). Hope that will solve this concern!**\n\n\n# Minor Issues:\n\n## Concern 2\n> While the topic distributions in Figure 2 are impressive, most of these benchmarks were not designed to simulate human preferences, hence the skewed topic distributions may be intentional for evaluating specific tasks, domains, or capabilities. Therefore, the section title \"LLM Benchmarks are Biased from Realistic User Queries and Preferences\" seems biased, implying these benchmarks are meant to measure human preferences.\n\nThanks for pointing that out, this is an interesting topic to discuss!\n\nFirst of all, as indicated in the section title, what we care about are **real-world use cases** and **user preferences**, instead of solely the user preferences. \"Real-world use cases\" depicts the distribution of tasks, while \"user preferences\" depicts the grading process.\n\n**We believe the core principle of doing model evaluation is to evaluate them as they will be used in the real world.** As a result, the evaluations should be designed based on this principle. (We also illustrated this in Section A.1 \"Why are real-world human queries and preferences important?\")\n\n**However, most of the existing evaluations are developed in a way that is based on the interest of its creators instead of the real-world users**, e.g., GSM8K was created to measure models' mathematical reasoning abilities. Is mathematical reasoning something that we should measure? Yes, but we are not sure how important it is. Hence, Section 2 aims to measure the significance of existing evaluations by (1) compare their distributions with real-world use case distribution and (2) show their correlations with large-scale human preferences. \"LLM Benchmarks are Biased from Realistic User Queries and Preferences\" is the conclusion we got after rigorous analysis.\n\nIt is important to note that Section 2 did not aim to completely criticize the existing evaluations, because they are still a good measure for specific tasks, domains, or capabilities. Section 2 is only serving as a meta-evaluation to the community: (1) it tells people that the results obtained from these evaluations may not generalize to real-world use cases, as they show a deviated task distribution and limited correlation with real-world use cases, (2) it visualizes the task distributions of different evaluations to help people select the correct benchmarks for the abilities they want to measure–e.g., it shows that you shouldn't take WinoGrande or DROP as a general-purpose benchmark, as their tasks only focus on a very small range of topics.\n\n## Concern 3\n> Another potential issue is the assumption that ChatBot Arena serves as the groundtruth for calculating correlations. As noted by the authors in lines 127-133, ChatBot Arena itself is biased. Therefore, even though MixEval has a larger user base, it might overfit to ChatBot Arena to some extent, making it more of an alternative rather than a more accurate evaluation of user preferences.\n\nWe thank the reviewer for this insightful comment! As illustrated by the footnote of the first page, the Chatbot Arena leaderboard is indeed not the sole indicator of human preference, but it currently serves as the only gold standard large-scale human preference benchmark within the community. Therefore, we could not find a better proxy of real-world user preferences than Chatbot Arena to compared with. Because of this, we didn't claim that MixEval provides a better approximation of real-world human preference than Chatbot Arena (it might be better or not); instead, MixEval is providing an efficient and low-biased evaluation that can reflect real-world use cases.\n\n## Concern 4\n> I am a bit confused about Table 2. The authors briefly mention \"periodically update\" in the main text, but I did not fully understand the details of this update. Could you explain this further?\n\nSure! As illustrated in lines 51-53 and Section 3.4 of the submitted paper, we update the data points of MixEval via (1) **batch web query update** (sampling different web queries batches from the crawled web queries), (2) **source web query update** (updating all the web queries with the latest Common Crawl) or (3) **benchmark pool update** (incorporating new ground-truth-based benchmarks to the benchmark pool). Since the mechanism of MixEval is to match web queries with benchmark pool samples, the above three updating methods refreshes both the web queries (the first and the second method) and benchmark pool samples (the third method). We will specify it more clearly in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "In75WEUcTi", "reviewer": "Reviewer_4BB9", "summary": "The paper introduces MixEval, a new benchmarking framework designed to overcome the limitations of traditional benchmarks and LLM-as-judge methods for evaluating LLMs. By leveraging web-mined user queries and matching them with existing benchmark queries, MixEval aims to offer a fast, efficient, and dynamic evaluation method that aligns closely with real-world human preferences. This framework promises significant cost and time savings and shows a high correlation with human preference leaderboards like Chatbot Arena.", "questions": "1. Could you provide detailed qualitative and quantitive analysis of module Web User Query Detection and Benchmark Mixture? \n2. It seems that the query of MixEval-Hard is longer than MixEval, is there any distribution difference between them?", "limitations": "See weaknesses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper tackles the important evaluation problem timely. MixEval introduces a fresh approach to bridging the gap between real-world user queries and efficient evaluation. By using web-mined queries, it aims to better reflect actual user interactions.\n2. The framework is efficient, could do dynamic updates and has high Correlation with Human Preferences.\n3. The paper goes above and beyond by providing extensive analysis and comparison with other popular LLM benchmarks, giving valuable insights into the strengths and weaknesses of different evaluation methods.\n4.  The paper is nicely written, with figures and tables well-organized. In particular, Figure 2 stands out for its clarity and effectiveness in presenting complex information.", "weaknesses": "1. Pipeline Brittleness: a. Web User Query Detection. The web user query detection phase has an overabundance of negative examples, which does little to help distinguish positive examples. b. Benchmark Mixture. Additionally, the sentence transformer used in the benchmark mixture phase has limited performance. Have you tried other sentence embedding models? Also it's unclear how many ground-truth LLM benchmarks can be accurately matched to mined queries.  c. The error rates for each module and the accumulated error rate are not provided.\n2. Single-Language Focus: The all-mpnet-base-v2 model used in the framework is designed for English only, raising concerns about its adaptability to different linguistic and cultural contexts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing MixEval as a timely work to the community! We are also grateful for your acknowledgment of the novelty, efficiency, thoroughness, and clarity of our work. Below are our responses to the concerns:\n\n## Concern 1 & 3\n> Pipeline Brittleness: a. Web User Query Detection. The web user query detection phase has an overabundance of negative examples, which does little to help distinguish positive examples. b. Benchmark Mixture. Additionally, the sentence transformer used in the benchmark mixture phase has limited performance. Have you tried other sentence embedding models? Also it's unclear how many ground-truth LLM benchmarks can be accurately matched to mined queries. c. The error rates for each module and the accumulated error rate are not provided. Could you provide detailed qualitative and quantitive analysis of module Web User Query Detection and Benchmark Mixture?\n\n**Overabundance of negative examples** We controlled a balanced positive and negative samples for training by down-sampling the negative samples. The original specification may be a bit ambiguous, we will carefully fix this in the revised version later.\n\n**Weak retriever** It's worth noting that in our retrieval task, both the query and key are short sentences, as the queries are real-world user prompts and the keys are benchmark questions, therefore it does not require state-of-the-art embedding models. We tested with both all-mpnet-base-v2 and openai's text-embedding-ada-002, they showed negligible difference in our task (only 6 cases are significantly different over 500 retrievals). Besides, as shown in Figure 2, MixEval data points exhibit a perfect mapping from the original web queries (the lowest C-Dist), further illustrating its effectiveness.\n\n**Quantitative analysis of web query detection** We report the quantitative results of the whole detection pipeline below (on our devised benchmarks). Before being trained, a language model achieving high recall was chosen (Vicuna-33B), with 99.12% recall and 46.21% precision; The looped training (as described in Section 3.1) significantly improves the precision while maintaining the recall, illustrating the low error accumulation rate of the devised web query detection pipeline.\n\nTable 2: The breakdown metrics of the web query detection pipeline.\n| Model           | Param | Pipeline Recall | Pipeline Precision | Pipeline F1 |\n|-----------------|-----------------|--------------|-----------|-----------|\n| Web Detector (initial) | 33B            | 99.12          | 46.21      | 63.03      |\n| Web Detector (trained)        | 33B            | 99.55          | 98.61      | 99.07      |\n\n## Concern 2\n> Single-Language Focus: The all-mpnet-base-v2 model used in the framework is designed for English only, raising concerns about its adaptability to different linguistic and cultural contexts.\n\nWe thank the reviewer for pointing this out! This is indeed a good point. Though MixEval and MixEval-Hard are English-dominant (>95% English), keeping the whole pipeline as multi-lingual is beneficial. **Note that MixEval is dynamic and can be updated with time. We will replace the current retriever with a capable multi-lingual one in the future MixEval releases to further improve the distribution.**\n\n## Concern 4\n> It seems that the query of MixEval-Hard is longer than MixEval, is there any distribution difference between them?\n\nYes, they have some distribution differences, as they correspond to different difficulty levels. However, we have successfully controlled the distribution shift of MixEval-Hard with the rejection sampling mechanism introduced in Section 3.3. As shown in Figure 2, MixEval-Hard achieves a low C-Dist with the original web queries. The average length difference might arise from the fact that harder tasks tend to be longer in its average length, which aligns well with our commonsense.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z6fBffik28", "reviewer": "Reviewer_u4BL", "summary": "The paper reconstructs a new benchmark named MixEval by matching queries collected from the internet with existing benchmarks. This new benchmark aligns with the distribution of human preferences, reflecting the real distribution of queries on the internet. Additionally, considering the overlap and difficulty among different benchmarks, the paper introduces MixEval-Hard by recalculating model scores across various benchmarks. Both MixEval and MixEval-Hard can be updated quickly. Finally, the paper demonstrates that MixEval aligns better with Arena Elo compared to other benchmarks, and the mixing operation enhances the alignment of other benchmarks with Arena Elo.", "questions": "Please see above.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The experiments are comprehensive, providing substantial results that validate the proposed method's ability to align well with the Arena Elo benchmark.\n2. By utilizing queries from the internet to shift the distribution of existing benchmarks to reflect real human preferences, the approach helps prevent large models from overfitting on existing benchmarks. Additionally, the paper offers insights into the consistency between benchmarks, indicating that alignment results are influenced not only by query distribution but also by question difficulty and density.\n3. An interesting visualization method is proposed, effectively reflecting the main aspects evaluated by the existing benchmarks.", "weaknesses": "1. Given that alignment with Arena Elo is used to measure the degree of alignment with human preferences, why not use data directly from Arena Elo when constructing MixEval, or even create a subset sampled from Arena Elo (considering Arena Elo directly reflects human preferences)?\n2. To avoid models overfitting on existing benchmarks, the paper updates the benchmarks by drawing new queries from the internet. However, I believe overfitting occurs due to models overfitting on a fixed benchmark. Without changing the benchmark, the overfitting issue will not be resolved. Therefore, a method to determine if a benchmark is overfitted should be designed to add LLM benchmarks to the pool. Additionally, Table 2 does not demonstrate that this method alleviates overfitting.\n3. Queries drawn from the internet might have already been seen by LLMs, so the retrieved samples from existing benchmarks might be simpler. The authors could exclude such factors during query extraction.\n4. Some symbols are not explained, such as \"j\" in line 187, \"tau\" in the formula between lines 187 and 188, and \"lambda\" in line 189.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the comprehensive experiments, analysis, and insights of this work. Meanwhile, we understand the reviewer's concerns, which are also very important to us. Below we clarify:\n\n## Concern 1\n> Given that alignment with Arena Elo is used to measure the degree of alignment with human preferences, why not use data directly from Arena Elo when constructing MixEval, or even create a subset sampled from Arena Elo (considering Arena Elo directly reflects human preferences)?\n\nWe thank the reviewer for raising this concern. We think there may exist some misunderstandings here. \n\nAs a warm reminder, we illustrated the reasons of using web queries in Section A.2 (FAQs). Below we reorganize Section A.2 as a response to this concern:\n\nFirst of all, it's worth noting that we are just taking Chatbot Arena as a measure of the capable model ranking of MixEval, **while our approach and data have nothing to do with Chatbot Arena**–our real-world user queries are solely detected from the web and the benchmark pool consists of the existing ground-truth-based benchmarks.\n\nMore importantly, **our goal isn't to fit Chatbot Arena's distribution; we are trying to fit real-world task distributions instead**. As shown in Section 2.2, line 127, **Chatbot Arena queries are slightly biased towards technical users**. However, web queries are grounded in the largest human population (5.4 billion internet users) among the accessible query sources and thus being more representative for the real-world users (illustrated Section 2.2).\n\nMoreover, **the web queries are multi-modal**, meaning that the queries on the web do not only comprise the text-to-text user queries, they also feature other modalities, such as image-to-text, text-to-audio, etc. Such queries are not available on Chatbot Arena or other chat platforms if they do not support models with the corresponding inputs/outputs. Our detected queries of other modalities provide a proxy of building an upgraded MixEval with any-to-any modalities, which facilitates a better evaluation suite for the whole AI community. However, without web queries, it's impractical to get well-distributed any-to-any real-world queries within the community. We will release the multi-modal version soon.\n\n## Concern 2\n> To avoid models overfitting on existing benchmarks, the paper updates the benchmarks by drawing new queries from the internet. However, I believe overfitting occurs due to models overfitting on a fixed benchmark. Without changing the benchmark, the overfitting issue will not be resolved. Therefore, a method to determine if a benchmark is overfitted should be designed to add LLM benchmarks to the pool. Additionally, Table 2 does not demonstrate that this method alleviates overfitting.\n\nWe thank the reviewer for this insightful comment!\n\n**We analyzed the dynamism and contamination of MixEval in detail in the general response to all reviewers and AC (on the top of this page). Hope that will solve this concern!**\n\nAs mentioned, although the contamination ratio of MixEval is comparatively low, we will indeed do benchmark contamination detection before putting it to the benchmark pool in the future versions of MixEval to further reduce the contamination. In addition, we will perform benchmark pool update to incorporate newly released ground-truth-based benchmarks to the benchmark pool, e.g., replacing MMLU with MMLU-pro, to mitigate contamination.\n\n## Concern 3\n> Queries drawn from the internet might have already been seen by LLMs, so the retrieved samples from existing benchmarks might be simpler. The authors could exclude such factors during query extraction.\n\nThanks for pointing that out! We will carefully add a pre-processing step accordingly in the future MixEval releases. And at the same time, we illustrate below that the described situation has **negligible impact** to the model evaluation.\n\nThis concern can be converted to \"We should exclude the benchmark queries leaked to the web from the extracted web queries to avoid introducing bias in the later benchmark mixture stage\" (This conversion requires some reasoning. Due to the character limitation, we do not expand here and would be happy to discuss later.) **We argue that it will have negligible impact to the model evaluation**. Below we do a rough estimation about the ratio of the leaked benchmark queries in our extracted web queries.\n\nThe benchmark pool has a size of 10^5-10^6. Suppose the web sentences has a size of 10^12-10^13 entries [1] and it has a query ratio of 6%, i.e., in 100 web sentences there are 6 valid web queries, then the web query quantity would be around 10^11-10^12. Thus, the ratio of benchmark entries in the web queries will only be around 10^-6. **Containing such a low proportion of leaked benchmark queries in our web queries will introduce negligible bias to the later benchmark mixture stage, because the amount of web queries we finally sample for benchmark mixture is only around 10^3-10^4, meaning it’s highly possible that there won’t be any leaked benchmark query in the final sampled web queries**.\n\nBesides, in the previous illustration of contamination and dynamism (Table 1 of this rebuttal, in the general response), it is shown that MixEval is relatively less contaminated (~10%) compared with other popular benchmarks, which can also be interpreted as \"MixEval samples are less likely to have already been seen by LLMs\".\n\n[1] Xue, Fuzhao, et al. \"To repeat or not to repeat: Insights from scaling llm under token-crisis.\" Advances in Neural Information Processing Systems 36 (2024).\n\n## Concern 4\n> Some symbols are not explained, such as \"j\" in line 187, \"tau\" in the formula between lines 187 and 188, and \"lambda\" in line 189.\n\nThanks for pointing that out! These symbols are indeed under-specified, and we will carefully fix it in the next version. Here the j denotes the model's index; τ denotes the rejection threshold; λ denotes the scaling hyperparameter.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U3r64aU5M8", "reviewer": "Reviewer_7ypd", "summary": "The authors propose the MixEval to match the real-world human queries with existed benchmarks. MixEval is a two-stage benchmark reconstruction pipeline consisting of (1) wild query detection, and (2) grounding existing benchmarks in the mined queries. The authors match each crawled web user query with its most similar query in the benchmark pool and the corresponding ground truth answer to align benchmark queries with real-world queries. In order to improve the benchmark’s ability to distinguish strong models, the authors derive a challenging subset from MixEval, which is called MixEval-Hard.", "questions": "1. Why can queries from different batches of the same distribution mitigate the contamination problem, instead of sampling queries from different distributions? \n2. Why the performance of the Benchmark-level Mix in Figure 5 is lower than the average performance of the Mixed benchmarks?", "limitations": "The authors have discussed detailed limitations in the Appendix A.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors match each crawled web user query with its most similar query in the benchmark pool and the corresponding ground truth answer to align with the human perferences. \n2. The experimental results demonstrate that the MixEval and MixEval-Hard are highly aligned with Chatbot Arena, outperform singular benchmarks.", "weaknesses": "1. In Section 3.2, the authors use dot product to match the query with the original benchmark, but do not explain how to use new queries in the test process and whether the queries are rewritten adaptively to match the ground truth answers. Why the answers from the original benchmark can be considered as the answers for the new queries?\n2. In Figure 5, the description of the experimental setup is missing. It is not clear 0-shot or 5-shot is used in Figure 5, and how the inputs of \"Mixed\" and \"Original\" are formated for the models. \n3. It is better to demonstrate the \"User query\" and \"Ground truth-benchmark\" processes are effective for constructing benchmarks through the ablation studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding this work solid and effective! Below are our responses to the concerns:\n\n## Concern 1\n> In Section 3.2, the authors use dot product to match the query with the original benchmark, but do not explain how to use new queries in the test process and whether the queries are rewritten adaptively to match the ground truth answers. Why the answers from the original benchmark can be considered as the answers for the new queries?\n\nWe thank the reviewer for the insightful comments! To clarify, after matching the web query with the original benchmark, **the original query will be dropped and we only use the questions and answers of the matched benchmark samples**. The benchmark mixture aims to map the real-world task distributions to the ground-truth-based benchmarks.\n\nWe specified this in line 47-48, Figure 4 (graphic illustration), and Section 3.2 (notation illustration). We will carefully improve the related specifications in the later versions.\n\n## Concern 2\n> In Figure 5, the description of the experimental setup is missing. It is not clear 0-shot or 5-shot is used in Figure 5, and how the inputs of \"Mixed\" and \"Original\" are formatted for the models.\n\nWe thank the reviewer for pointing that out! **The setting for Figure 5 is the same as other experiments**: we use 0-shot and official, unified model input formatting; the correlation numbers of the \"original\" benchmarks are the same as Figure 1 and 10, whose settings are specified in Section E. We will make it clearer in the next version.\n\n## Concern 3\n> It is better to demonstrate the \"User query\" and \"Ground truth-benchmark\" processes are effective for constructing benchmarks through the ablation studies.\n\n**We wish to highlight that we did the ablation study in Section 4.2–\"MixEval outperforms both benchmark-level and uniform mixtures.\"** This ablation study aims to illustrate the effectiveness of the core step: benchmark mixture of MixEval. We compare the MixEval benchmark mixture with two other mixture schemes: Benchmark-level Mixture (ablating the sample-level mixture of MixEval) and Uniform Mixture (ablating both the sample-level and benchmark-level mixture of MixEval). Benchmark-level Mixture samples data points uniformly from each benchmark of the benchmark pool, with the benchmark size proportional to its split size in MixEval. In other words, its benchmark size distribution is the same as MixEval, while it's uniform inside each benchmark. The Uniform Mixture simply uniformly samples an equal number of data points from all benchmarks of the benchmark pool. Both methods yield significantly lower Arena Elo correlations than MixEval mixture, illustrating the effectiveness of our method.\n\n**Beyond that, we report the quantitative results of the whole query detection pipeline below (on our devised benchmarks) to illustrate the importance of the looped training.** Before being trained, a language model achieving high recall was chosen (Vicuna-33B), with 99.12% recall and 46.21% precision; The looped training (as described in Section 3.1) significantly improves the precision while maintaining the recall, illustrating the accuracy of the devised web query detection pipeline and the importance of the looped training.\n\nTable 2: The breakdown metrics of the web query detection pipeline.\n| Model           | Param | Pipeline Recall | Pipeline Precision | Pipeline F1 |\n|-----------------|-----------------|--------------|-----------|-----------|\n| Web Detector (initial) | 33B            | 99.12          | 46.21      | 63.03      |\n| Web Detector (trained)        | 33B            | 99.55          | 98.61      | 99.07      |\n\nBesides the above two ablations, the whole Section 4.2 has demonstrated the effectiveness of MixEval based on experimental results from multiple perspectives, which we believe has adequately illustrated the effectiveness.\n\n## Concern 4\n> Why can queries from different batches of the same distribution mitigate the contamination problem, instead of sampling queries from different distributions?\n\nWe thank the reviewer for this insightful comment!\n\nThe main reason is that there is a significant difference between every two batches, as illustrated in the Table 2 of the submitted paper. Note that beyond **batch web query update**, we will also perform **source web query update**, which updates all the web user queries with the latest Common Crawl splits and it can be interpreted as updating queries from different distributions.\n\n**We analyzed the dynamism and contamination of MixEval in detail in the general response to all reviewers and AC (on the top of this page). Hope that will help solve this concern!**\n\n## Concern 5\n> Why the performance of the Benchmark-level Mix in Figure 5 is lower than the average performance of the Mixed benchmarks?\n\nBecause **benchmark-level mixture (Benchmark-level Mix)** is supposed to be worse than the **sample level mixture (Mixed)**.\nAs we mentioned earlier, the \"Benchmark-level Mix\" is part of the ablation study, which samples data points uniformly from each benchmark of the benchmark pool, with the benchmark size proportional to its split size in MixEval. In other words, its benchmark size distribution is the same as MixEval, while it's uniform inside each benchmark. So such mixture scheme is supposed to perform worse than the \"Mixed\" benchmarks in Figure 5, which, as specified in line 238-239, performs the same sample-level mixture as MixEval. This performance gap arises from the fact that MixEval reconstructs real-world use cases more effectively than \"Benchmark-level Mix\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable feedback. We identify that the main concern among reviewers is about the contamination of MixEval. **Therefore, we conduct additional contamination analysis and provide the general response here.**

# High-Level Takeaways

1. **Low Natural Contamination**: MixEval and MixEval-Hard demonstrate a low natural contamination ratio, as detailed in the Table 1 of this rebuttal (below).
2. **Purpose of Model Evaluations**: Model evaluations typically serve two main purposes: **developer self-validation** and **leaderboard competition**. For self-validation, contamination of MixEval is not a concern. For leaderboard competition, MixEval effectively mitigates contamination through **batch web query updates**, **source web query updates**, and **benchmark pool updates**, demonstrating its high dynamism compared with traditional ground-truth benchmarks.
3. **Broader Contributions**: While dynamism is a key feature of MixEval, the contributions of this work extend significantly beyond this aspect, as elaborated in lines 60-73 of the submitted paper.

# Detailed Justification

When does contamination affect model evaluations? Generally, model evaluations serve two purposes: **developer self-validation** and **leaderboard competition** [1].

## Self-Validation
For self-validation, MixEval is effective regardless of contamination levels, as **developers aim to exclude evaluation data from training data to ensure accurate testing**. At this stage, they will not deliberately overfit the evals. In addition, they usually employ contamination detection to remove leaked evaluation data from the training data, enhancing the generalizability of the internal evaluation pipeline. MixEval, being efficient and low-biased, accurately mirrors real-world use cases, making it ideal for rapid model iteration.

## Leaderboard Competition
Conversely, in leaderboard competitions, some contamination is inevitable due to the static nature of ground-truth-based benchmarks. **Developers may not rigorously exclude evaluation data from training, or may even include it intentionally to improve leaderboard rankings**. Below, we demonstrate that MixEval can mitigate both natural and deliberate contamination..

**We wish to highlight that MixEval is only mitigating the contamination instead of solving it completely, as mentioned on line 50, 207, and Figure 4 of the submitted paper.**

### Natural Contamination
**For natural contamination, MixEval mitigates it via benchmark mixture.** According to Table 1 of [2], contamination levels of existing web benchmarks range from 1.1% to 40.6%. Generally, more popular benchmarks exhibit higher contamination. For example, MMLU shows a relatively high contamination ratio (24.3%), yet remains crucial to the community and our benchmark pool. MixEval addresses this by mixing popular benchmarks with less contaminated ones smartly (e.g., CommonsenseQA), thus reducing the natural contamination ratio. Note that MixEval can be updated with time, and we will include contamination detection in future releases to further minimize contamination.

We use the contamination detector in [2] to detect natural contaminations of benchmarks by searching the ratio of benchmark data existing on the web. As shown in the table below, MixEval and MixEval-Hard exhibit a lower contamination ratio compared with popular benchmarks such as MMLU due to the smoothing effect mentioned above. Overall, MixEval achieves the highest correlation with real-world use cases while staying at a low natural contamination ratio and high efficiency.

Table 1: Contamination ratio of different benchmarks.
| Dataset        | Split | #Total | #Input-only Contamination | #Input-and-label Contamination |
|----------------|-------|--------|---------------------------|-------------------------------|
| ARC_c          | Test  | 1172   | 53 (4.5%)                 | 283 (24.1%)                   |
| CommonsenseQA  | Dev   | 1221   | 3 (0.2%)                  | 17 (1.4%)                     |
| Winogrande     | Dev   | 1267   | 0 (0.0%)                  | 14 (1.1%)                     |
| C-Eval         | Dev   | 1346   | 69 (5.1%)                 | 547 (40.6%)                   |
| Hellaswag      | Dev   | 10042  | 46 (0.5%)                 | 1201 (12.0%)                  |
| MMLU           | Test  | 13987  | 678 (4.8%)                | 3399 (24.3%)                  |
| **MixEval**        | Test  | 4000   | 129 (3.2%)                | 414 (10.4%)                   |
| **MixEval-Hard**   | Test  | 1000   | 39 (3.9%)                 | 106 (10.6%)                   |

### Deliberate Contamination
**For deliberate contamination, MixEval mitigates it by dynamically updating web user queries and the benchmark pool.** If model developers deliberately overfit evals, contamination is nearly impossible to fully eliminate. Even with dynamic systems like Chatbot Arena, evaluations can still be hacked, e.g., fitting on LMSys user data or hiring biased workers. Developers may hack MixEval by **(1) directly fitting on MixEval data**, or **(2) fitting the benchmark pool**. We address method (1) by periodically updating MixEval data points through "**batch web query update**" (sampling new web query batches from the crawled web query pool) or "**source web query update**" (updating the whole web query pool with the latest Common Crawl), and then perform benchmark mixture. Table 2 of the submitted paper shows it effectiveness, demonstrating significant differences between MixEval versions. Method (2) is tackled by "**benchmark pool update**", incorporating new ground-truth benchmarks in the community, e.g., replacing MMLU with MMLU-pro. 

We will add this discussion to Section A as another FAQ to strengthen people's understanding regarding the dynamism of MixEval.

Reference:

[1] Fourrier, Clémentine. "Let's Talk About LLM Evaluation."

[2] Li, Yucheng. "An open source data contamination report for llama series models.".

Meta-review: This paper proposes a new benchmark designed to better align LLM evaluations with real-world user queries called MixEval. MixEval matches web-mined queries to existing benchmarks, aiming to offer a fast, efficient, and dynamic evaluation method.

The reviewers agreed that the proposed approach was novel and efficient and that the paper was well-written and provided a comprehensive analysis of the experimental results.

The reviewers also identified a number of areas where the paper could be improved.  Some of their suggestions were related to the robustness of the approach, questions around generalizability beyond English, and concerns about things like contamination, bias, and overfitting. The authors are strongly encouraged to carefully consider this feedback and address it to the extent possible in future versions of the paper.
Decision: Accept (poster)