MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures

Authors: jinjie  ni
fuzhao  xue
xiang  yue
yuntian  deng
mahir  shah
kabir  jain
graham  neubig
yang  you
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:25:32

Abstract: Evaluating large language models (LLMs) is challenging. Traditional ground-truth- based benchmarks fail to capture the comprehensiveness and nuance of real-world queries, while LLM-as-judge benchmarks suffer from grading biases and limited query quantity. Both of them may also become contaminated over time. User- facing evaluation, such as Chatbot Arena, provides reliable signals but is costly and slow. In this work, we propose MixEval, a new paradigm for establishing efficient, gold-standard LLM evaluation by strategically mixing off-the-shelf bench- marks. It bridges (1) comprehensive and well-distributed real-world user queries and (2) efficient and fairly-graded ground-truth-based benchmarks, by matching queries mined from the web with similar queries from existing benchmarks. Based on MixEval, we further build MixEval-Hard, which offers more room for model improvement. Our benchmarks’ advantages lie in (1) a 0.96 model ranking correlation with Chatbot Arena arising from the highly impartial query distribution and grading mechanism, (2) fast, cheap, and reproducible execution (6% of the time and cost of MMLU), and (3) dynamic evaluation enabled by the rapid and stable data update pipeline. We provide extensive meta-evaluation and analysis for our and existing LLM benchmarks to deepen the community’s understanding of LLM evaluation and guide future research directions.

Main paper: 
 introduction: 
 That Which is Measured, Improves. Evaluation is essential in the AI community for two main reasons: (1) benchmarks provide early signals to model developers, aiding in refining data and model design, and (2) benchmarks guide users in selecting suitable models for specific use cases. Therefore, benchmarks offer feedback to the entire community, facilitating model optimization. Consequently, the main concern of evaluating LLMs is impartiality-we need to optimize impartial objectives so that the community advances in the right direction. In practical LLM evaluations, three primary biases contribute to a lack of impartiality: (1) query bias-evaluation queries falling short of comprehensiveness or appropriate distribution (2) grading bias-the grading process involving significant bias or error (3) generalization bias-models overfitting the evaluation data.Large Scale User-facing Evaluation Provides a More Impartial Signal. Practitioners generally adopt either automatic or user-facing approaches for LLM benchmarking. Automatic benchmarking typically employs traditional ground-truth-based benchmarks, such as MMLU [17] , which often fail to capture real-world query comprehensiveness and nuance while involving a comparatively impartial grading process; or employs open-ended benchmarks using LLMs as graders, such as MT-Bench [39] , suffering from both grading bias and query incomprehensiveness due to the preference biases and high cost of frontier LLM judges. Additionally, the static nature of automatic benchmarks results Figure 1 : Benchmark correlations (%) with Chatbot Arena Elo, against the total costs of evaluating a single GPT-3.5-Turbo-0125 model. MixEval and MixEval-Hard show the highest correlations with Arena Elo and Arena Elo (En) among leading benchmarks. We reference the crowdsourcing price for Amazon Mechanical Turk ($0.05 per vote) when estimating the cost of evaluating a single model on Chatbot Arena (approximately $2,936). Chatbot Arena is prohibitively expensive, while MixEval and MixEval-Hard are cheap and cost-effective alternatives. Details on the correlation and evaluation cost values are provided in Section E. in contamination over time, amplifying the generalization issue. Such biases lead to significant deviations from gold-standard evaluation, impeding model development. On the other hand, largescale user-facing benchmarking, such as Chatbot Arena3 [6] , offers more reliable objectives for model development and effectively mitigates the above-mentioned three biases because (1) it collects a vast array of real-world user queries, thereby ensuring superior query comprehensiveness and distribution, (2) it judges diverse and complex model responses stably due to the "wisdom of the crowd" effect [33] , where individual judgment noise is averaged out over a large number of samples, mitigating the grading bias, and (3) it continuously receives fresh user queries, mitigating the benchmark contamination issue. Furthermore, it guides model optimization to meet user needs effectively in practical applications, which is a crucial goal of developing models. However, Chatbot Arena is prohibitively expensive (Figure 1 ), slow, and irreproducible. Moreover, it is not directly accessible for public usage, hindering practitioners from conducting easy and fast model evaluations.MixEval: Towards Efficient Gold-Standard LLM Evaluations. In this work, we aim to establish a highly impartial gold-standard benchmark without compromising efficiency. This can be achieved by leveraging (1) the efficiency and grading impartiality of ground-truth-based benchmarks and (2) the superior comprehensiveness and distribution of real-world user queries. To this end, we propose MixEval, a two-stage benchmark reconstruction pipeline consisting of (1) wild query mining and (2) grounding existing benchmarks in the mined queries. We introduce an accurate user query retrieval process, comprising query detection, filtering, and classification. In the detection phase, we train open-source LLMs on self-collected data to detect queries in Common Crawl splits. During filtering, we utilize GPT-4 Turbo to exclude non-query sentences. In classification, we categorize the filtered queries by input and output modalities, retaining text-in-text-out queries for LLM evaluation. To align benchmark queries with real-world queries, we match each crawled web user query with its most similar query in the benchmark pool and the corresponding ground truth answer. We designate the resulting benchmark as MixEval. To improve the benchmark's ability to distinguish strong models, we derive a challenging subset from MixEval, termed MixEval-Hard. To mitigate the overfitting issue, we periodically update the data points in MixEval and MixEval-Hard using our fast, stable pipeline, which performs benchmark mixture with a different batch of wild queries from The plot aggregates all queries and divides them into 16 regions. From each region, 100 queries are uniformly sampled and analyzed by GPT-4 for topic summarization. A clear trend is observed, with topics transitioning from non-technical at the bottom to technical at the top.How Much Do Our Benchmarks Reflect Realworld User Queries and Preferences? The rapid advancement of LLMs has led to the introduction of numerous benchmarks. However, the community may still lack a comprehensive understanding of how well these benchmarks align with real-world use cases and human preferences. Without such understanding, the signals derived from evaluations might be misleading, thereby impeding model development. To investigate this issue, we (1) analyze the correlations between benchmarks (Figures 1 and 10 ) and (2) visualize their query distributions in a unified 2-D space (Figure 2 ). The setups for these analyses are detailed in Section E.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We carefully inspected the main claims we made in the abstract and introduction, and ensure that all of them are appropriate.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
important takeaways: 
 Current benchmarks show a limited correlation with human preferences. and MixEval-Hard achieve the highest correlations with human preferences, at 0.93 and 0.96, respectively.Most benchmarks exhibit a skewed topic distribution. Ground-truth-based and LLM-judged benchmarks show a skewed query distribution compared to detected web queries and wild datasets. Notably, BoolQ, Natural Questions, and MMLU align more closely with wild user queries due to their data collection methods. Specifically, questions in BoolQ and Natural Questions originate from Google Search, while MMLU is designed to cover a wide range of topics (57 topics, including Atari games [2] ).Query comprehensiveness is crucial. General-domain benchmarks, which are not tailored for specific domains in their data collection pipelines, exhibit a stronger correlation with Arena Elo than domain-specific ones. Notably, 10 out of 13 general-domain benchmarks have an Arena Elo correlation score above 0.5, whereas only 1 out of 8 domain-specific benchmarks achieves this. This underscores the significance of comprehensive query topics for achieving high correlation with human preferences. Some general-domain benchmarks are actually domain-specific. Despite being labeled as generaldomain benchmarks, DROP and WinoGrande have limited scopes, often narrower than many domainspecific benchmarks. As depicted in Figure 3 , DROP queries mainly address History, Politics, Sports, Demographics, and Societal Issues, whereas WinoGrande queries focus primarily on Grammar, Language, Decision Making, and Social Dynamics.User population size affects the query distribution. Figure 2 shows distribution differences in wild queries across varying user population sizes. ShareGPT, grounded in 100 million 4 active users of ChatGPT by mid-2023, contrasts with WildChat [38] , Chatbot Arena Conversations [6] , and LMSYS-Chat-1M [39] , which have user bases of 0.2 million, 0.13 million, and 0.21 million, respectively. The global internet user count was 5.4 billion5 in 2023, an order of magnitude larger than all considered wild datasets. Consequently, user bases of the internet, ShareGPT, and other datasets span three distinct orders of magnitude. ShareGPT's larger user population (second order of magnitude) yields a distribution most similar to web queries from the global internet user base (third order of magnitude), both visually and in cluster distance (C-Dist).Chatbot Arena and Arena-Hard queries exhibit biases. Compared to web queries and ShareGPT data, datasets from the Chatbot Arena website-Chatbot Arena Conversations and LMSYS-Chat-1M-have a higher proportion of technical queries (as presented in Figure 3 , queries with higher position on the map are more technical). This indicates a user base skewed towards technical users, potentially affecting evaluation results, as an effective LLM benchmark should mimic real-world use cases. Furthermore, a minor discrepancy exists between web and ShareGPT queries, suggesting that one or both of them may still slightly deviate from actual real-world query distributions. Moreover, Arena-Hard queries exhibit a pronounced bias towards technical topics. This likely stems from the design of their data pipeline for sampling hard prompts. We will demonstrate that employing a carefully designed sampling technique is essential to preserve the query distribution while enhancing difficulty. This is supported by MixEval-Hard's similar distribution to the original web queries and wild datasets (see Section 3.3). In Section 2, we show that current ground-truth-based and LLM-judged benchmarks have skewed query distributions and limited correlation with human preferences. Additionally, LLM-judged benchmarks suffer from LLM preference bias and both of them become contaminated over time. In contrast, Chatbot Arena is less biased and more dynamic but requires slow and expensive human preference data collection, resulting in irreproducible outcomes.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Limitations are detailed as part of Section A in the form of QAs.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
mixeval: 
 To address these issues, we introduce MixEval (Figure 4 ), which aligns ground-truth-based LLM benchmarks with real-world user queries. This method uses user queries mined from the web and matches them with similar queries from existing benchmarks, and involves two stages: (1) user query detection from the web and (2) benchmark mixture. To improve model separability and reduce contamination, we also propose MixEval-Hard and a dynamic updating mechanism.In this stage, we detect user queries from Common Crawl [12] . Both recall and precision are crucial to ensure the query distribution reflects real-world scenarios. Therefore, we developed two benchmarks to evaluate our query detector's performance. The first benchmark includes self-collected in-the-wild user queries as positive samples, with non-query datasets such as Wikipedia [16] as negative samples. The second, higher-quality benchmark contains positive and negative samples hand-picked by our authors from in-the-wild query and non-query datasets. In preliminary experiments, direct prompting of open-source language models performed poorly on our benchmarks. Thus, we developed a rectification pipeline to ensure high recall and precision cost-effectively. We started with a detection phase to gather training data. Testing various open-source LLMs, Vicuna 33B [7] achieved a high recall (>99%) on our test sets with careful prompt engineering, ensuring that very few positive samples were missed initially. In this phase, we detected around 20k queries using Vicuna 33B over a subset of Common Crawl. We then used GPT-4 to more accurately label these data as positive or negative samples, and used the resulting data to train Vicuna 33B. The trained Vicuna 33B achieved high recall (>99%) and precision (>98%) on our benchmarks and detected 2M user queries from the entire Common Crawl. Finally, we prompted GPT-4 Turbo to classify them, extracting text-in-text-out queries for LLM evaluation. Future work will address queries with other I/O modalities.To bridge wild user queries Q and ground-truth LLM benchmarks, we create a benchmark poolB = {B 1 , B 2 , ..., B n }, where each B n = {b 1 , b 2 , ..., b k } represents a distinct ground-truth LLM benchmark.We define a mapping f : q i → b j , with q i ∈ Q and b j ∈ B. For each q i ∈ Q, we rank similarities between each (q i , b j ) pair and select the most similar b j that satisfies θ: b j = f (q i ) = arg max bj ∈B S(q i , b j ) s.t. θ. We use the dot-product between normalized sentence embeddings as the similarity score S(•). When retrieving the top-1 b j , θ is a length constraint on the input (or context) field of each b j , addressing the effect of long inputs in the benchmark data mixture. The sentence embeddings of queries are computed using the all-mpnet-base-v2 model from SentenceTransformers [23] . To ensure quality and comprehensive sample coverage, we selected the development and test splits of widely adopted benchmarks from diverse domains and topics. Details and distributions of these benchmarks are provided in Appendix D. i = ⃗ µ • ⃗ A i ,where each model's result on question i is weighted by its accuracy µ j on the dataset. Given ξ = {ξ 1 , ξ 2 , ..., ξ N B ′ }, we sample from B ′ with rejection:B ′′ = {b ′ i ∈ B ′ : p(b ′ i ) and α(B ′′ ∪ {b ′ i }, B ′ ) ≤ τ }, where α(x, y) denotes the cluster distance between x and y. The probability of drawing b′ i , p(b ′ i ) = e λξ i b ′ k∈B ′ e λξ k , is based on ξ i . This rejection sampling ensures that MixEval-Hard is difficulty-first while maintaining a balanced query distribution. We obtain 1000 samples for MixEval-Hard. The statistics of MixEval and MixEval-Hard are detailed in Table 1 . Static benchmarks risk contamination over time as models may overfit to the benchmark data [6, 32, 36] , undermining evaluation reliability. To address this, we periodically update the data points in MixEval and MixEval-Hard using the automatic pipeline described above, i.e., performing benchmark mixtures based on the queries uniformly sampled from the massive web queries detected, which completes updates within one minute. Table 2 shows score stability and version differences. We created five versions of MixEval by altering the random seed when sampling web queries and ran five models on them. As shown, the average mean and standard deviation (Std.) for the models across the versions are 77.64 and 0.36, respectively, demonstrating high score stability. For each pair of versions, we compute the unique sample ratio for sampled web queries and benchmark data points. Given samples X = {x 1 , x 2 , ..., x n } from version A and Y = {y 1 , y 2 , ..., y n } from version B, the unique sample ratio R is calculated asR = |X-Y |+|Y -X|, representing the unique ratio of the X ∪Y set. The average unique web query ratio across all version pairs is 99.71%, and the unique ratio for MixEval versions is 85.05%, indicating significant differences between versions. This efficient updating mechanism, alongside stable model scores and significant data point variations, effectively mitigates benchmark contamination. Additionally, we plan to dynamically expand our benchmark pool with newly released benchmarks to further enhance the mixed benchmark distribution.To summarize, we update the data points of MixEval via (1) batch web query update (sampling different web queries batches from the crawled web queries), (2) source web query update (updating all the web queries with the latest Common Crawl) or (3) benchmark pool update (incorporating new ground-truth-based benchmarks to the benchmark pool). Since the mechanism of MixEval is to match web queries with benchmark pool samples, the above three updating methods refreshes both the web queries (the first and the second method) and benchmark pool samples (the third method).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 
experiment settings: 
 We evaluate models on MixEval and MixEval-Hard using the Transformers library [31] for open-source models, adhering to the official settings in their Hugging Face model card. Proprietary models are assessed via their official API endpoints, using the latest versions as of April 30, 2024. Chat models employ official chat templates or FastChat chat templates [39] , and base models are evaluated in a 5-shot setting. Both MixEval and MixEval-Hard, comprising samples from various benchmarks, demonstrate the inadequacies of traditional rule-based parsing methods across all benchmarks and models. To improve parsing accuracy, we use GPT-3.5-Turbo-0125 as the model parser to either score the response (free-form questions) or extract the model's choice (multiple-choice problems). The stability of the GPT-3.5 Turbo parser is evidenced in Table 2 of this paper and Table 4 of [37] . We will also provide an open-source model parser with its stability test to ensure long-term reproducibility. Section J details the model parser prompts, and Section I compares the model parser to the rule parser. Models are evaluated on 4 or 8 A100 GPUs. All correlations with Arena Elo are based on the Chatbot Arena Leaderboard as of May 1, 2024. MixEval improves the correlation with Arena Elo and Arena Elo (En) across all main benchmark splits of MixEval. In Figure 5 , we select the top-10 benchmarks from our pool with sufficient sample sizes (see sample number distribution in Figure 8 ). For each benchmark, we present (1) the correlation between Arena Elo and the original benchmark, and (2) the correlation between Arena Elo and the MixEval-mixed version. Remarkably, all benchmarks exhibit significant improvements in their correlations with Arena Elo after being processed by MixEval. The correlation increase is notably high (>40%) in benchmarks such as BoolQ, AGIEval, SIQA, and PIQA. MixEval and MixEval-Hard, which aggregate all benchmarks, consistently outperform any individual benchmark mixture, underscoring the importance of a large benchmark pool and query comprehensiveness.MixEval outperforms both benchmark-level and uniform mixtures. Figure 5 illustrates the correlations with Arena Elo for benchmark-level and uniform mixtures. The benchmark-level mixture samples questions uniformly from each benchmark, proportional to its split size in MixEval. The uniform mixture samples an equal number of questions from all benchmarks. Both methods yield significantly lower human preference correlations than MixEval and MixEval-Hard. Furthermore, the benchmark-level mixture offers negligible improvement over the uniform mixture. These findings underscore the importance of an appropriate sample-level mixture, as implemented by MixEval.MixEval effectively maps real-world user queries to ground-truth-based benchmarks. Figure 2 shows the query distributions of leading benchmarks. Both MixEval and MixEval-Hard closely resemble web queries and popular wild datasets, highlighting MixEval's efficacy in aligning benchmark query distributions with real-world data. The maps in Figure 2 are ordered by their cluster distances to our identified web queries, showing that wild datasets align more closely with our web queries than other LLM benchmarks. This underscores the robustness of our web query detection pipeline and the solid grounding of MixEval. Additionally, as discussed in Section 2, ShareGPT, with a larger user base (100M) compared to other wild datasets (0.1M-0.2M), shows the highest similarity to our web queries, which are based on a global internet user population (5.4B), further validating the accuracy of our web query detection.Leaderboard 3 . Chat and base model scores show a high correlation.the highest performance across nearly all splits, except for BoolQ-Mixed. Gemini 1.5 Pro ranks third on both MixEval and MixEval-Hard, followed closely by Claude 3 Sonnet, LLaMA-3-70B-Instruct, and Reka Core, with similar scores. Notably, all these frontier models, except LLaMA-3-70B-Instruct, support multi-modal input understanding. The LLaMA-3-8B-Instruct model is the top-performing 7B model, outperforming some of the latest large models, such as Command R (35B) and Qwen1.5-32B-Chat (32B). Proprietary models generally outperform open-source models.Cost-effectiveness Figure 6 compares the models in Table 3 in terms of cost-effectiveness. Figure 6a examines the relationship between activated parameters and performance for open-source LLMs, while Figure 6b compares API price against performance for frontier proprietary LLMs. Both figures exhibit a roughly log-linear relationship between performance and the x-axis metric. In Figure 6a , the Llama-3 series stands out as the most performant and parameter-efficient among open-source models.The MoE models, such as Mixtral-8x7B-Instruct-v0.1, Qwen1.5-MoE-A2.7B-Chat, and JetMoE-8B-Chat, demonstrate superior parameter efficiency. The proprietary data points reveal a clearer log-linear pattern. GPT-4 Turbo is more cost-effective than Claude 3 Opus, offering comparable performance at less than half the price. The Gemini series exhibits similar cost-effectiveness to the GPT series, while the Reka series parallels the cost-effectiveness of the Claude series. We conduct detailed error analysis in Section H to compare error rates of open-source and proprietary models on different MixEval splits. We also showcase error responses of frontier models in Section H.1 to identify their potential weaknesses.The crowdsourced evaluation of LLMs, based on human preferences, assesses two main aspects: (1) model capability, optimized mainly during pre-training, and (2) non-capability attributes like toxicity and helpfulness, refined during post-training. We explore whether human preferences for models can be predicted before post-training, leading to the question: which stage, pre-training or posttraining, more significantly influences human preferences in crowdsourced LLM evaluations? We evaluated the base versions of the model series in Table 3 . Notably, the correlations in Figure 7 show a 0.95 correlation between base and chat models, indicating MixEval's potential to approximate human preferences pre-post-training. This implies that pre-training may have a greater impact on crowdsourced LLM evaluations, with post-training minimally altering human preference rankings. However, we also observe that the post-training has more impact on some smaller models, all of which went through heavy supervised post-training.Comprehensiveness and other features, such as difficulty, impact correlation. As shown in Figure 10 , general-domain benchmarks typically exhibit a higher correlation with human preference compared to domain-specific benchmarks, highlighting the importance of query comprehensiveness. However, comprehensiveness is not the sole factor. Three observations support this: (1) Benchmarks like GSM8K, despite their skewed distributions (Figure 2 ), achieve a high correlation (0.78) with human preference, while others with high topic overlap with real-world queries, such as BoolQ, achieve a low correlation (0.37). ( 2) ARC-e and ARC-c, despite similar topic distributions, show significantly different correlations (Figure 10 ), likely due to varying difficulty levels. This indicates that other query features, such as difficulty, are critical to correlation with human preference. (3) As shown in Figure 5 , MixEval increases the correlation for each individual benchmark through benchmark mixture. For an individual benchmark, the topic becomes less comprehensive post-mixture since the mixed version represents a subset of the original; thus, the correlation gain is not due to a more comprehensive topic distribution. These observations suggest that correlation gains with human preference are influenced by factors beyond solely comprehensiveness, possibly including query difficulty and topic density. Section F presents the full correlation matrix and analyzes other factors affecting benchmark correlations. 
conclusion: 
 In this paper, we present MixEval, an approach that bridges real-world queries and ground-truthbased evaluation by mining user queries from the web and matching them with similar benchmark queries. MixEval and its hard variant can offer accurate evaluations that highly align with Chatbot Arena. MixEval operates locally and rapidly, eliminating the need for slow, costly human preference data collection or biased model judgment. MixEval's data points can be stably updated within one minute, mitigating benchmark contamination. We thereby effectively mitigate the query, grading, and generalization biases in LLM evaluation through the proposed benchmark mixture pipeline, while maintaining high efficiency. Our meta-evaluation and extensive analysis of MixEval and other popular LLM benchmarks demonstrate MixEval's effectiveness, providing insights to enhance the community's understanding of LLM evaluation. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: Discussed in Section C.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We properly cite all the previous work, datasets, or software we use.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We list the potentially frequently asked questions and the point-to-point answers as follows:A.1 Why are real-world human queries and preferences important?One of the primary applications for AI model development is the automation of complex tasks traditionally performed by humans. As AI models coexist with humans, frequent interactions with humans are necessary to manage these tasks effectively. These interactions predominantly involve natural language queries, as it is the most common medium for human communication. Given the significance of human interaction in the use cases for general AI models, it is crucial to evaluate AI models-particularly large language models (LLMs) that rely on natural language-under conditions that mirror real-world scenarios, i.e., receiving human queries and assessing performance based on human preferences. Evaluating models based on their real-world use cases is well-supported across various research disciplines [3, 8, 20] .A.2 Why do you use web queries as real-world user queries?Because web queries are grounded in the largest human population (5.4 billion internet users) among the accessible query sources. Figure 2 illustrates the query distribution differences in wild queries across various user population sizes. ShareGPT, with 100 million6 active users by mid-2023, contrasts with WildChat [38] , Chatbot Arena Conversations [6] , and LMSYS-Chat-1M [39] , which have user bases of 0.2 million, 0.13 million, and 0.21 million, respectively. The global internet user count was 5.4 billion7 in 2023, an order of magnitude larger than all considered wild datasets. Consequently, the user bases of the internet, ShareGPT, and other datasets span three distinct orders of magnitude. ShareGPT's larger user population (second order of magnitude) yields a distribution most similar to web queries from the global internet user base (third order of magnitude), both visually and in cluster distance (C-Dist), validating that user population size affects query distribution. Compared to web queries and ShareGPT data, datasets from the Chatbot Arena website-Chatbot Arena Conversations and LMSYS-Chat-1M-have a higher proportion of technical queries (as presented in Figure 3 , queries with higher position on the map are more technical). This indicates a user base skewed towards technical users, potentially affecting evaluation results, as an effective LLM benchmark should mimic real-world use cases.Conversations to achieve a similar model ranking with Arena Elo?The crucial difference lies in the scoring methods: using ground truth answers versus LLM judges. Ground-truth-based evaluation is more interpretable, faster, and cost-effective compared to LLM-asjudges. Furthermore, training effective judge models is highly challenging because (1) LLMs possess inherent preference biases [39] , and (2) to evaluate other models accurately without ground truth answers, the judge model must be either superior to or at least on par with the models it assesses. Consequently, a large model is required, complicating the training process.A.4 Why isn't the cluster distribution ranking introduced in Figure 2 consistent with the rankings in Figures 1 or 10?The key to human preference correlation, as indicated in Figure 5 , lies in ensuring that a benchmark's query distribution aligns with a subset of the wild queries, rather than encompassing the entire wild query distribution. This is evidenced by the high correlation between the MixEval-mixed domain-specific benchmarks and Arena Elo in Figure 5 . However, aligning with only a subset of wild queries significantly impacts the cluster distance metric shown in Figure 2 . Notably, covering all regions of wild queries enhances correlation scores, as demonstrated by MixEval achieving higher correlation in Figure 5 .A.5 How long does it take to dynamically update MixEval-Hard?The update of MixEval-Hard is somewhat slower than MixEval, which can be updated within 1 minute. MixEval-Hard, a subset of MixEval, is sampled based on the prediction results of several models. Thus, the update time depends on the models used to sample this subset. If only GPT-4 Turbo's prediction results are used to rank the question difficulties, the total update time is approximately 2 minutes, which remains rapid. However, according to [22] , GPT-4 Turbo may yield a lower score in this condition compared to using results from multiple models.A.6 Arena-Hard's correlation with Arena Elo (En) is reported to be 0.94 in their blog while 0.83 in Figures 1 and 10The number for Arena-Hard in Figures 1 and 10 is derived from the model scores reported on Arena-Hard's leaderboard as of May 01, 2024, using the Arena Elo (En) values from the same date. This discrepancy may arise because the figure reported in Arena-Hard's blog did not account for all models listed on their leaderboard. The number of models used for each pair of benchmarks shown in Figure 10 is reported in Figure 11 A.7 Why GPQA is not included in Figures 1 and 10 ?Because we didn't find enough data points for GPQA that share enough common models with other benchmarks.No, MixEval is not entirely unbiased. This is due to several factors: the detection pipeline is not perfectly accurate, Common Crawl data collection introduces biases, and there are inherent biases from web users in the real world. However, MixEval is relatively less biased because it draws from a broad internet user base. This is supported by: (1) The maps in Figure 2 , which are ordered by their cluster distances to our identified web queries, indicate that wild datasets align more closely with our web queries than with other LLM benchmarks. This demonstrates the robustness of our web query detection pipeline and the solid grounding of MixEval.(2) As discussed in Section 2, ShareGPT, with its extensive user base (100M) compared to other wild datasets (0.1M-0.2M), shows the highest similarity to our web queries, which are based on the global internet user population (5.4B). This further validates the accuracy of our web query detection. (3) The trained web query detector achieved high recall (>99%) and precision (>98%) on our internal web query detection benchmarks.A.9 Will the pipeline that creates MixEval-Hard introduce some noise that influences the result?MixEval-Hard sampling relies on the difficulty scores of benchmark questions, which inherently include some dataset annotation errors. During our error case study (Section H), we identified several annotation issues. However, the number of annotation errors was minimal, rendering the noise negligible. Furthermore, the high correlation with Arena Elo demonstrates that the introduced noise does not significantly affect the model rankings.Both frontier and open-source LLMs have made significant strides in recent years. Evaluation scores are a core objective in LLM development, necessitating an effective evaluation pipeline for successful model advancement. Current LLM evaluation methods can be categorized into three main types: (1) ground-truth-based evaluation, (2) LLM-as-judge evaluation, and (3) crowdsourcing human-preference-based evaluation.Ground-truth-based evaluation, or closed-ended evaluation, involves ranking the outputs of base and chat LLMs against predefined correct answers. Various benchmarks have been introduced by the research community for this purpose [1, 4, 5, 9-11, 14, 17, 21, 24-26, 28, 29, 35, 40] . These benchmarks facilitate rapid and straightforward LLM evaluation, providing clear and unbiased answer judgments due to their closed-ended nature. However, ground-truth-based benchmarks often exhibit topic bias (as illustrated in Figure 2 ) and may not accurately represent the diversity of real-world user queries, limiting their ability to assess the nuanced capabilities of LLMs.On the other hand, two other categories of evaluation approaches primarily focus on the open-ended evaluations of chat LLMs. The LLM-as-judge evaluation uses frontier models to rank the responses to a set of open-ended queries without ground-truths. These queries are either manually designed [39] or sourced from crowdsourcing platforms [6] . However, due to the high cost of using frontier models as judges, such approaches are not scalable to a large number of user queries. This limitation hinders the ability to reflect the complexity and diversity of real-world queries and may deviate from the true distribution (see Figure 2 ). Additionally, previous research has identified several biases in frontier model judges, including verbosity bias, position bias, and self-enhancement bias [39] . These biases can lead to unfair model rankings in practical evaluations. Additionally, the static nature of both ground-truth-based and LLM-as-judge benchmarks results in contamination over time, diminishing the reliability of evaluation outcomes.As a comparison, Chatbot Arena [6] serves as a robust benchmark for evaluating chat LLMs. It operates as a benchmarking platform where anonymous, randomized battles are conducted in a crowdsourced environment. The platform's extensive real-world user queries provide comprehensive and less biased evaluations, ensuring the accuracy and stability of model rankings. Additionally, its real-time nature prevents models from overfitting the benchmark, thereby avoiding contamination issues. However, obtaining a stable model score requires more than 5,000 human interactions and several days, making the process labor-intensive and slow. Furthermore, its open-ended format limits its ability to evaluate base models. Many open-ended queries, such as creating a travel plan, lack definitive standards for distinguishing good from bad answers [30] , resulting in evaluations that are not purely ability-oriented.Currently, real-world text-in-text-out user queries are primarily sourced from chat platforms [6, 27, 38, 39] . Our concurrent work, MAmmoTH2 [34] , also identifies real-world user queries from the web. However, MAmmoTH2 has fundamentally different objectives compared to MixEval. MAmmoTH2 focuses on detecting large-scale domain-specific query-answer pairs, while MixEval targets general-purpose user queries that accurately reflect the real-world user query distribution. This difference in objectives results in distinct web query detection pipelines.Our user queries are not directly crawled from the web; instead, they are identified using Common Crawl, an openly available corpus of web crawl data widely used in the research community. Furthermore, we do not release the raw detected queries, while only releasing the final mixed version of MixEval for two reasons: (1) the raw detected queries may contain toxic content or unexpected sensitive information, and (2) we update our benchmarks dynamically to avoid contamination. Releasing the detected raw queries would make the dynamic benchmarking process more predictable, reducing its effectiveness.To ensure quality and comprehensive sample coverage, we selected the development and test splits of widely adopted benchmarks encompassing various domains and topics.• General-domain benchmarks: MMLU [17] , BoolQ [9] , HellaSwag [35] , ARC [10] , Com-monSenseQA [29] , AGIEval [40] , OpenbookQA [21] , GPQA [24] , WinoGrande [25] ,TriviaQA [19] , DROP [14] , and BBH [28] .• Domain-specific benchmarks: Math: GSM8K [24] and MATH [18] ; Coding: MBPP [1] and HumanEval [5] ; Physics: PIQA [4] ; and Social Interactions: SIQA [26] . According to Figure 2 , the mixed benchmark MixEval exhibits the highest overlap with Q among all benchmarks, suggesting that B adequately represents the wild query distribution. Let B ′ = {B ′ 1 , B ′ 2 , ..., B ′ n } denote the mixed benchmark. The distributions of B and B ′ are illustrated in Figure 8 . A positive correlation between the sizes of the mixed and original benchmarks is observed. Intuitively, a larger benchmark is likely to be retrieved more frequently; however, this is not universally true. Benchmarks with skewed sample distributions, such as HellaSwag, GSM8k, ARC, BBH, MATH, and GPQA, have a smaller relative size after mixing. This indicates that both quantity and distribution influence how frequently a benchmark is retrieved by Q. Overall, the retrieved benchmark splits exhibit a long-tail distribution.Correlation Matrix Heatmap (Figures 1 and 10 ). We present the correlation matrix of prominent benchmarks, where warmer colors indicate higher correlations. Model scores are collected from various sources, including the Chatbot Arena Leaderboard [6] , Open LLM Leaderboard [15] , and OpenCompass Leaderboard [13] . Our data collection adheres to three principles: (1) We exclude scores reported by model authors, relying solely on evaluation leaderboards to ensure fairness. (2) For each benchmark, scores are sourced from a single platform to eliminate the influence of varying evaluation settings on model rankings. (3) When multiple sources are available for a benchmark, we select the one with the highest number of models in common with other benchmarks. The number of common models for each pair of benchmarks is detailed in Figure 11 .Query Distribution Map (Figure 2 ). We present the distribution of benchmark queries sorted by their distance to our detected web queries. Each benchmark (orange or yellow) is plotted against the detected wild queries (blue). We uniformly sampled 1000 queries from each LLM benchmark and wild dataset, with a sampling number of 200 for MT-Bench and Arena-Hard due to their smaller sizes. We combined the query embeddings and reduced their dimensions to the same 2-D space to facilitate direct comparisons of the benchmark query distributions. A detailed case study revealed that the reduced space primarily represents the topics of the queries, with queries on similar topics clustering in specific regions of the map. To better understand the topic distribution of different benchmarks, we divided the map into 16 patches based on location (Figure 3 ). We then uniformly sampled 100 queries from each patch and used GPT-4 to summarize the topics of the sampled queries. As illustrated in Figure 3 , the 2-D query distribution exhibits a distinct regional trend: queries located higher on the map are more technical. The distribution transitions from non-technical topics, such as Social Interactions, at the bottom to technical ones, such as Programming and Mathematics, at the top.Evaluation Cost Estimation. As illustrated in Figure 9 , we consider two costs when evaluating the performance of GPT-3.5-Turbo-0125 on each benchmark: the inference cost and the judging (scoring) cost. The inference cost computation for ground-truth-based and LLM-as-judge benchmarks are straightforward, involving only the estimation of model input and output tokens for each benchmark. We estimate the model output tokens to be 20 for ground-truth-based benchmarks and 329 for open-Infer Cost ($) Judge Cost ($)Total Cost ($)Figure 9 : Evaluation cost breakdown for the cost estimation in Figure 1 . The total evaluation cost is broken down into the inference cost and judge cost.ended benchmarks8 . To compute the evaluation cost of GPT-3.5-Turbo-0125 on the Chatbot Arena, we use the voting number of GPT-3.5-Turbo-0125 on the Chatbot Arena leaderboard as its query count, with each query's token count estimated from the Chatbot Arena Conversations dataset. Since models on Chatbot Arena are evaluated pairwise, both input and output tokens are doubled. The judging cost 9 for ground-truth-based benchmarks is estimated similarly, accounting for the input and output tokens of the model parser. However, estimating the human judgment cost for Chatbot Arena is more complex. We reference the crowdsourcing price for Amazon Mechanical Turk (MTurk), specifically the rate for a Facebook Account Holder10 ($0.05 per vote). Under this pricing scheme, evaluating a single model on Chatbot Arena costs approximately $2936, making it a highly expensive process.Benchmarks?Both comprehensiveness and other features, such as difficulty, make an impact. As shown in Figure 10 , general-domain benchmarks tend to achieve a higher correlation with human preference compared to domain-specific benchmarks, underscoring the significance of benchmark query comprehensiveness. However, comprehensiveness is not the only factor that matters. Three observations support this: (1) Some benchmarks with highly skewed distributions, such as GSM8K, achieve a promising correlation (0.78) with human preference, while others with high topic overlap with real-world queries, like BoolQ, achieve a low correlation (0.37). ( 2) Despite their similar topic distributions, ARC-e and ARC-c show significantly different correlations with human preference (Figure 10 ), likely due to their differing difficulty levels, as they are known to have different levels of difficulty. This illustrates that other important query features, such as difficulty, are not apparent on the 2-D map that primarily features topic distribution but are crucial to the correlation with human preference. (3) As shown in Figure 5 , MixEval increases the correlation with human preference for each individual benchmark through benchmark mixture. For an individual benchmark, the topic becomes less comprehensive after the mixture because the mixed version represents a subset of the original on the topic map; thus, in this case, the correlation gain does not stem from a more comprehensive topic distribution. These three observations suggest that the human preference correlation gain is not solely related to topic comprehensiveness; other features, such as query difficulty and topic density, also play a crucial role in improving correlation scores.Benchmarks that are highly correlated with human preferences also tend to be correlated with each other, whereas those that are less correlated with human preferences are similarly less correlated with most other benchmarks. The heat map reveals a consistent red region in the top-left, signifying high correlation, while the rest of the map is predominantly blue and inconsistent, indicating low correlation. This suggests that model rankings on benchmarks closely aligned with human preferences are more stable and reflect a "True" ranking, whereas the remaining benchmarks exhibit greater variability in model rankings.Benchmarks within the same domain exhibit higher correlations. Despite a low correlation with human preferences, some domain-specific benchmarks demonstrate a relatively high correlation with other benchmarks in the same domain. For instance, MBPP shows a correlation of only 0.28 with human preference but a substantial 0.83 with HumanEval. Similarly, MATH has a correlation of 0.50 with human preference yet presents a 0.66 correlation with GSM8K. Furthermore, ARC-e has a correlation of 0.58 with human preference while achieving a notable 0.84 correlation with ARC-c. Table 3 presents our evaluation results on MixEval, MixEval-Hard, and their main subsets. Overall, Claude 3 Opus and GPT-4 Turbo consistently achieve the highest performance across nearly all splits, except for the BoolQ-Mixed split. Figure 13 illustrates the error rates of the models evaluated on the main splits of MixEval. We separately compute the error rates for proprietary and open-source models to facilitate comparison. Both model types exhibit significant errors on the AGIEval split of MixEval, underscoring its difficulty. In contrast, performance on the PIQA split is generally saturated. Notably, there is a substantial performance gap between proprietary and open-source models on the GSM8K split, with considerable gaps also observed on the HellaSwag, TriviaQA, and DROP splits. In Section H.1, we conduct case studies to examine the error cases made by frontier proprietary models. For each case, we present the incorrect responses from each model. We identify three primary causes of confusion for these models: strong domain knowledge, complex reasoning, and vague question definitions. Additionally, we identify several annotation issues within current benchmarks, though these are negligible in number.Error Case 1 Question: Letter to the editor: You say that if the government were to confiscate a portion of the wages of convicted burglars when they reenter the workforce, it would be a form of stealing, hence an abuse of power. Yet under the proposal now being considered, the government would confiscate such wages in order to fund an account to compensate burglary victims. So even if confiscating a portion of burglars' wages were a form of stealing, it would still be justified. Which one of the following principles, if valid, most helps to support the argument in the letter to the editor? A. Money stolen from a burglar should be given to that burglar's victims. B. Burglars are obligated to provide compensation to the same individuals they victimized. C. The motive prompting an action determines whether or not that action is justified. D. A crime is justified only if it is a means of compensating people who deserve compensation. E. Stealing is never justified even if it benefits someone who has been a burglary victim. Answer with the option letter from the given choices directly. Answer: C Split: AGIEval Main Causes of Errors: Hard Reasoning Wrong Answers: Claude 3 Haiku: The principle that most helps to support the argument in the letter to the editor is (B) Burglars are obligated to provide compensation to the same individuals they victimized.The letter argues that even if confiscating a portion of burglars' wages is a form of stealing, it would still be justified because the purpose is to fund an account to compensate burglary victims. This suggests that the burglars have an obligation to provide compensation to the individuals they victimized, which aligns with principle (B). Mistral-Small: The principle that most helps to support the argument in the letter to the editor is B. Burglars are obligated to provide compensation to the same individuals they victimized. This principle aligns with the argument that even if confiscating a portion of burglars' wages were a form of stealing, it would still be justified because it serves to compensate burglary victims.Mistral-Medium: (B) Burglars are obligated to provide compensation to the same individuals they victimized.The principle that supports the argument in the letter to the editor is that burglars have an obligation to provide compensation to the individuals they have victimized. This principle justifies the government's confiscation of a portion of the burglars' wages to fund an account to compensate their victims. While the letter acknowledges that confiscating wages could be considered a form of stealing, it argues that this action would still be justified as a means of providing compensation to those who deserve it. Therefore, option (B) is the best choice among the given options as it directly supports the argument in the letter. Options (A), (C), (D), and (E) do not directly address the obligation of burglars to provide compensation to their victims, which is the crux of the argument.The principle that most helps to support the argument in the letter to the editor is (B) "Burglars are obligated to provide compensation to the same individuals they victimized." This principle justifies the confiscation of a portion of burglars' wages as a means to compensate their victims, which is the main point of the argument in the letter.The principle that most helps to support the argument in the letter to the editor is **(D) A crime is justified only if it is a means of compensating people who deserve compensation.** This principle suggests that a criminal act can only be justified if it results in some form of compensation for the victims of the crime. In the scenario described in the letter, the government is proposing to confiscate a portion of the wages of convicted burglars to fund an account to compensate burglary victims. The argument is that even though confiscating a portion of the wages of burglars may be seen as a form of stealing, it is justified because it is a means of compensating people who deserve compensation. Thus, the act of confiscating wages in this case is seen as a legitimate way of achieving a just outcome (compensating victims).The principle that most helps to support the argument in the letter to the editor is: B. Burglars are obligated to provide compensation to the same individuals they victimized.The argument in the letter acknowledges that confiscating a portion of a convicted burglar's wages might be considered stealing, but it contends that this action is still justified because it serves the purpose of compensating burglary victims. This aligns with the principle that burglars have an obligation to provide compensation to those they have harmed. The letter suggests that even if the government's action is considered stealing, it is still justified because it fulfills the burglar's obligation to compensate their victims. Figure 14 : The score differences computed by model parser and rule parser on MixEval. The rule is unstable on both free-form and multiple-choice splits, especially free-form.Free-form Parser Prompt System:In this task, I want you to act as a judge.You will be provided with a question, its golden answer(s), and the model's answer, while the context of the question is not given here. Your task is to judge how correct the model's answer is based on the golden answer(s), without seeing the context of the question, and then give a correctness score. The correctness score should be one of the below numbers: 0.0 (totally wrong), 0. System:In this task, I want you to act as an option extractor.You will be provided with a multiple-choice question, its options, and the model's answer, while theJustification: There is no theoretical result made.Justification: We uploaded the code and data, and will release and periodically update the benchmark data and its related code. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: This research focuses on an improved benchmarking method for LLMs, which does not directly involve significant societal impacts. Its primary influence is within the LLM research community, as detailed in Section 1.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).The Chatbot Arena leaderboard is not the sole indicator of real-world human preferences, but it currently serves as one of the gold standards within the community. Therefore, we utilize it as a reliable source of approximation.https://www.mlyearning.org/chatgpt-statisticshttps://www.statista.com/statistics/273018/number-of-internet-users-worldwide/https://www.mlyearning.org/chatgpt-statisticshttps://www.statista.com/statistics/273018/number-of-internet-users-worldwide/According to the averaged output token for GPT-3.5-Turbo-0125 as presented at: https://lmsys.org/blog/2024-04-19-arena-hard/.9 The judge cost for Arena-Hard and MT-Bench is directly taken from https://lmsys.org/blog/2024-04-19arena-hard/. 10 https://requester.mturk.com/pricing 


Decision: This paper proposes a new benchmark designed to better align LLM evaluations with real-world user queries called MixEval. MixEval matches web-mined queries to existing benchmarks, aiming to offer a fast, efficient, and dynamic evaluation method.

The reviewers agreed that the proposed approach was novel and efficient and that the paper was well-written and provided a comprehensive analysis of the experimental results.

The reviewers also identified a number of areas where the paper could be improved.  Some of their suggestions were related to the robustness of the approach, questions around generalizability beyond English, and concerns about things like contamination, bias, and overfitting. The authors are strongly encouraged to carefully consider this feedback and address it to the extent possible in future versions of the paper.

Reviews: review: paper_summary: paper_summary: This paper introduces MixEval, a new approach/benchmark to evaluate LLMs effectively in real-world scenarios. Traditional benchmarks often miss the comprehensiveness and subtlety of actual user queries, while existing methods like LLM-as-judge benchmarks are difficult to scale up. MixEval addresses these issues by using user queries mined from the web and aligning them with similar queries from established benchmarks.

The key advantage of MixEval is its efficiency and dynamic nature. It achieves a high correlation with Chatbot Arena (0.96 Spearman correlation) but requires only 6% of the time and cost compared to mainstream benchmarks like MMLU. MixEval can also be updated quickly, significantly reducing the risk of benchmark contamination. The paper also introduces MixEval-Hard for better differentiation among strong models. Through comprehensive analysis, the authors demonstrate that MixEval offers accurate, less biased, and more dynamic evaluations, providing a potential scalable solution for real-world LLM assessment.
strength_weakness: strength_weakness: **Major Issues**:
1. MixEval dynamically updates by mixing popular benchmarks (e.g., MMLU, BoolQ, GSM8K), which may not mitigate contamination. Most of these benchmarks are saturated and suffer from contamination. Although the authors claim they will "dynamically expand our benchmark pool with newly released benchmarks to further enhance the mixed benchmark distribution," this does not address the root issue of contamination.

**Minor Issues**:
1. While the topic distributions in Figure 2 are impressive, most of these benchmarks were not designed to simulate human preferences, hence the skewed topic distributions may be intentional for evaluating specific tasks, domains, or capabilities. Therefore, the section title "LLM Benchmarks are Biased from Realistic User Queries and Preferences" seems biased, implying these benchmarks are meant to measure human preferences.
2. Another potential issue is the assumption that ChatBot Arena serves as the groundtruth for calculating correlations. As noted by the authors in lines 127-133, ChatBot Arena itself is biased. Therefore, even though MixEval has a larger user base, it might overfit to ChatBot Arena to some extent, making it more of an alternative rather than a more accurate evaluation of user preferences.
questions: questions: I am a bit confused about Table 2. The authors briefly mention "periodically update" in the main text, but I did not fully understand the details of this update. Could you explain this further?
limitations: limitations: In the appendix, the authors address some limitations in a Q&A format, such as the potential biases in MixEval. My views on limitations are reflected in the weaknesses above.
paper_summary: paper_summary: The paper introduces MixEval, a new benchmarking framework designed to overcome the limitations of traditional benchmarks and LLM-as-judge methods for evaluating LLMs. By leveraging web-mined user queries and matching them with existing benchmark queries, MixEval aims to offer a fast, efficient, and dynamic evaluation method that aligns closely with real-world human preferences. This framework promises significant cost and time savings and shows a high correlation with human preference leaderboards like Chatbot Arena.
strength_weakness: strength_weakness: 1. Pipeline Brittleness: a. Web User Query Detection. The web user query detection phase has an overabundance of negative examples, which does little to help distinguish positive examples. b. Benchmark Mixture. Additionally, the sentence transformer used in the benchmark mixture phase has limited performance. Have you tried other sentence embedding models? Also it's unclear how many ground-truth LLM benchmarks can be accurately matched to mined queries.  c. The error rates for each module and the accumulated error rate are not provided.
2. Single-Language Focus: The all-mpnet-base-v2 model used in the framework is designed for English only, raising concerns about its adaptability to different linguistic and cultural contexts.
questions: questions: 1. Could you provide detailed qualitative and quantitive analysis of module Web User Query Detection and Benchmark Mixture? 
2. It seems that the query of MixEval-Hard is longer than MixEval, is there any distribution difference between them?
limitations: limitations: See weaknesses.
paper_summary: paper_summary: The paper reconstructs a new benchmark named MixEval by matching queries collected from the internet with existing benchmarks. This new benchmark aligns with the distribution of human preferences, reflecting the real distribution of queries on the internet. Additionally, considering the overlap and difficulty among different benchmarks, the paper introduces MixEval-Hard by recalculating model scores across various benchmarks. Both MixEval and MixEval-Hard can be updated quickly. Finally, the paper demonstrates that MixEval aligns better with Arena Elo compared to other benchmarks, and the mixing operation enhances the alignment of other benchmarks with Arena Elo.
strength_weakness: strength_weakness: 1. Given that alignment with Arena Elo is used to measure the degree of alignment with human preferences, why not use data directly from Arena Elo when constructing MixEval, or even create a subset sampled from Arena Elo (considering Arena Elo directly reflects human preferences)?
2. To avoid models overfitting on existing benchmarks, the paper updates the benchmarks by drawing new queries from the internet. However, I believe overfitting occurs due to models overfitting on a fixed benchmark. Without changing the benchmark, the overfitting issue will not be resolved. Therefore, a method to determine if a benchmark is overfitted should be designed to add LLM benchmarks to the pool. Additionally, Table 2 does not demonstrate that this method alleviates overfitting.
3. Queries drawn from the internet might have already been seen by LLMs, so the retrieved samples from existing benchmarks might be simpler. The authors could exclude such factors during query extraction.
4. Some symbols are not explained, such as "j" in line 187, "tau" in the formula between lines 187 and 188, and "lambda" in line 189.
questions: questions: Please see above.
limitations: limitations: Yes.
paper_summary: paper_summary: The authors propose the MixEval to match the real-world human queries with existed benchmarks. MixEval is a two-stage benchmark reconstruction pipeline consisting of (1) wild query detection, and (2) grounding existing benchmarks in the mined queries. The authors match each crawled web user query with its most similar query in the benchmark pool and the corresponding ground truth answer to align benchmark queries with real-world queries. In order to improve the benchmark’s ability to distinguish strong models, the authors derive a challenging subset from MixEval, which is called MixEval-Hard.
strength_weakness: strength_weakness: 1. In Section 3.2, the authors use dot product to match the query with the original benchmark, but do not explain how to use new queries in the test process and whether the queries are rewritten adaptively to match the ground truth answers. Why the answers from the original benchmark can be considered as the answers for the new queries?
2. In Figure 5, the description of the experimental setup is missing. It is not clear 0-shot or 5-shot is used in Figure 5, and how the inputs of "Mixed" and "Original" are formated for the models. 
3. It is better to demonstrate the "User query" and "Ground truth-benchmark" processes are effective for constructing benchmarks through the ablation studies.
questions: questions: 1. Why can queries from different batches of the same distribution mitigate the contamination problem, instead of sampling queries from different distributions? 
2. Why the performance of the Benchmark-level Mix in Figure 5 is lower than the average performance of the Mixed benchmarks?
limitations: limitations: The authors have discussed detailed limitations in the Appendix A.

Comments: title: Thank you for your response!
Thank you for raising score!
Thanks for the response!
Thank you for your appreciation!
Thank You for the Response!
comment: Thanks for the response. Looking forward to your final version.
Thank you for your insightful comments and appreciation. Your support is very important to MixEval!
Thank you for your response. The user's query is multimodal, so it is reasonable to use a mixed benchmark. My concern regarding the leakage issue has been partially resolved. Additionally, concerning Concern 3, although the queries in the benchmark are relatively rare on the entire web, it's possible that all queries on the web have been used as training data for the query-matching model. Therefore, I wonder if using these queries to match those in the benchmark might be more likely to match the polluted queries rather than with queries that the model has never seen. However, based on Table 1 in the final rebuttal response, it seems that the query matching does not introduce this bias. Thank you again for your response, and I will raise my score.
Thank you once again for recognizing MixEval and for your insightful comments. We will incorporate the feedback from this rebuttal to enhance future releases and revisions.
Thank you for your response and the additional experiments, which have led me to increase the score from 6 to 7. I agree that MixEval can mitigate contamination, but I still do not believe it fundamentally solves the problem, as it still heavily relies on existing benchmarks. I disagree with the authors' explanation for Concern 2, as many benchmarks were not originally designed to measure human preferences, and human preferences are just one aspect of the many evaluations for LLMs (although a very important one). Technical reports for all LLMs release results from numerous benchmarks to give the community a comprehensive understanding of their capabilities. Calculating correlations between all these benchmarks and human preferences introduces some potential biases, such as assuming they should align with human preferences, which is not necessarily the case.

Despite these issues, which I consider minor, I personally recommend the acceptance of this paper. Good luck!
Dear Reviewers,

Thank you for your dedication and time spent reviewing our paper.

As the discussion period draws to a close, we wish to remind you of the approaching deadline.

Please feel free to raise any points or questions about the paper that may need additional clarification.

We deeply appreciate your valuable feedback.

Thank you once again.