Bench-O-Matic: Automating Benchmark Curation from Crowdsourced Data

Authors: Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, Ion Stoica
Primary area: datasets and benchmarks
Keywords: LLM, Evaluation

Abstract:
The rapid evolution of Large Language Models (LLMs) has outpaced the development of model evaluation, highlighting the need for continuous curation of new,
challenging benchmarks. However, manual curation of high-quality, human-aligned
benchmarks is expensive and time-consuming. To address this, we introduce Bench-O-Matic, an automated pipeline that leverages LLMs to curate high-quality, open-
ended prompts from large, crowd-sourced datasets, enabling continuous benchmark
updates without human in the loop. We apply Bench-O-Matic to datasets such as
Chatbot Arena and WildChat-1M, extracting challenging prompts and utilizing
LLM-as-a-Judge for automatic model evaluation. To validate benchmark quality,
we propose new metrics to measure a benchmark’s alignment with human preferences and ability to separate models. We release Eval-O-Matic, a benchmark
consisting 500 challenging prompts curated by Bench-O-Matic. Eval-O-Matic
provides 3x higher separation of model performances compared to MT-Bench and
achieves 98.6% correlation with human preference rankings, all at a cost of $20.
Our work sets a new framework for the scalable curation of automated benchmarks
from extensive data.

Decision: reject
Reviewer scores: 5, 8, 5

Reviews and discussion:
<thread>
creation date:
1729530085496

thread_type:
official_review

<thread_object>
creator: reviewer_5rny

<content>
summary:
The paper proposes approaches to automate the benchmark generation process via the prompting of LLMs. The proposals for different characteristics to establish the baselines are fair and the contributions are around the different scoring mechanisms to 1) evaluate the quality of prompts 2) LLM-based judging of prompt outputs to generate 1-5 score instead of binary preferences and 3) combining them with statistical aggregators to differentiate end evaluate different LLM outputs.

strengths:
The promise of the paper is excellent if delivered -- Reconfigurable automated benchmarks without humans in the loop and via crowd sourced data. With a series of prompting techniques in the pipeline, the approach is fair and well studied. Key innovations are in the design of metrics to separate various models and crux of thesis on generating evaluation data that is of high quality and can be separable.

weaknesses:
The key weaknesses around this paper are the claims that the proposed approach is human-free and easily configurable as shown in the Table comparing the multiple methods. Given that the approach leverages use of ChatBotArena supplied queries  and even though the quality filter will remove the specific poor quality queries, it is not free from the input i.e., humans prompting the different LLMs on the arena and easily being configured to a use case that end users may have in mind. Discussing results on adapting the evaluation framework to beyond what is available in Chat bot Arena would be needed to support the claims of the paper.  Also it would be good to discuss potential biases introduced by using ChatBotArena queries as a starting point. The paper could be strengthened by providing concrete examples or experiments showing how their approach could be adapted to different domains or use cases beyond ChatBot Arena data



An additional area of concern is that almost every step of the pipeline involves a prompt engineering exercise including scoring the final models on a scale of 1-5. This is standard but the question emerges on the fidelity of the LLMs themselves and when they hallucinate themselves. As evidenced by the score sorted by topic cluster, the data does show that for exact answer situations like Python game coding versus loose open ended questions the LLM-judges are not very good.  To strengthen the paper - discuss potential failure modes or biases introduced by relying heavily on LLMs, provide more detailed analysis of how performance varies across different types of questions or topics and suggest ways to mitigate or detect potential hallucinations or errors introduced by LLMs in the pipeline


The details of human annotation were very unclear. See questions below.

questions:
- Is the approach really adaptable/configurable ? Restate the claims if not. 
- Can the approach work irrespective of humans in the loop ? i.e., crowd-sourcer providing initial prompts. 
- human Annotation study; 
    How many human annotators were involved?
    What was the inter-annotator agreement?
    How were discrepancies between annotators resolved?
    Were the human annotators experts in any particular domains?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698041661

thread_type:
official_review

<thread_object>
creator: reviewer_3lvj

<content>
summary:
The paper introduces an automated pipeline, Bench-O-Matic, designed to curate prompts and create benchmarks for evaluating large language models (LLMs). The authors propose new metrics to assess benchmark quality, ensuring a clear separability of confidence scores and alignment with human preferences. The prompts are organized into topic clusters to ensure diversity, and an "LLM-as-a-Judge" approach is used to evaluate responses from various LLMs, fully automating the evaluation process. Additionally, the paper presents two novel benchmarks generated using this pipeline: Eval-O-Matic, based on Chatbot Arena, and Wild-O-Matic, derived from WildChat-1M.

strengths:
- The problem statement is clearly defined.
- The paper addresses a significant challenge highly relevant to the current state of AI and places well in the current literature.
- The pipeline is flexible and open-ended, allowing for continuous improvements over time.
- The experiments are comprehensive, demonstrating that the pipeline effectively creates benchmarks based on the metrics defined in the paper, with multiple LLMs evaluated on Eval-O-Matic.
- The paper presents new ideas to evaluate benchmarks to overcome previous issues.

weaknesses:
- Using an LLM to evaluate other LLMs’ responses may limit the complexity of the benchmark prompts. While employing an ensemble of judges partially mitigates this issue, there is still an inherent limitation. However, the advantages of an automated pipeline outweigh this concern, and the authors have implemented techniques to reduce evaluation biases.

I have a hard time finding weaknesses for the paper. It is a well-executed and solid paper, though not necessarily groundbreaking.

**Minor Comments** 
- On line 80, "achieve 98.6% correlation" should be "achieve**s** 98.6% correlation".
- On line 82, "Our work**s** makes" should be "Our work makes".
- On lines 206 and 352, "Section C" should probably be changed for "Appendix C" for clarity.
- On line 464, "an regression based approach" should be corrected to "**a** regression-based approach."

questions:
- Previous studies have shown that fine-tuning the LLM-as-a-Judge can significantly improve evaluation robustness. Has this been considered in the current work? This could help improve the quality of the judges, the main limitation of this benchmark.
- In Section 4.2, it states, "We also ensure the final dataset is free from personally identifiable information or offensive content." Could the authors elaborate on how this is achieved? Was this done manually or automatically with the help of an LLM?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730785996997

thread_type:
official_review

<thread_object>
creator: reviewer_ewc5

<content>
summary:
The work proposes Bench-O-Matic, a system for automatically curating high-quality, open-ended LLM benchmarks by using large-scale, crowd-sourced data. This tool addresses the need for evolving benchmarks that adapt to the rapid development of LLMs without requiring human intervention.

strengths:
- Bench-O-Matic efficiently creates high-quality benchmarks from crowd-sourced data without human input, the work addressed the scalability issue in benchmark curation.
- The work Introduces novel metrics like Separability with Confidence and Pair Rank Brier Score, enhancing the robustness and reliability of benchmark assessments.
- Eval-O-Matic achieves strong performance alignment with human preferences for only $20 per evaluation, and provides a cost-effective alternative to static benchmarks.

weaknesses:
- Quality insurance. The seven quality criteria may not fully encompass the diversity of user tasks, potentially favoring specific types of prompts over others.
- The synthesis of data is reliant on on LLMs as Judges. The LLM-as-a-Judge framework may introduce stylistic or self-bias, even with adjustments, which could influence benchmark objectivity in certain cases.

questions:
Is there any estimation on the error/quality of the data generated? Or using some metrics to evaluate the similarity of the generated data with the real-world data?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents an automated pipeline, Bench-O-Matic, which curates prompts and creates benchmarks from large, crowd-sourced datasets for evaluating LLMs. Additionally, new metrics are proposed to assess benchmark quality. Experiments demonstrate that its performance aligns with human preferences. However, the reliability of the new benchmark relies heavily on LLMs’ judgments, and the current version does not address the generalizability of the pipeline. While I acknowledge that the pipeline can be applied to other data sources, generalizability is crucial if this work focuses on LLM evaluation.