Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions

Authors: Ruochen Zhao, Wenxuan Zhang, Yew Ken Chia, Weiwen Xu, Deli Zhao, Lidong Bing
Primary area: datasets and benchmarks
Keywords: LLM Evaluation, LLM Agents

Abstract:
As LLMs continuously evolve, there is an urgent need for a reliable evaluation method that delivers trustworthy results promptly. Currently, static benchmarks suffer from inflexibility and unreliability, leading users to prefer human voting platforms like Chatbot Arena. However, human evaluations require significant manual effort. To address this, we propose the Auto-Arena, an innovative framework that automates the entire evaluation process using LLM-powered agents. Firstly, an LLM examiner generates questions. Then, two LLM candidates engage in a multi-round peer battle based on individual questions, aiming at revealing their true performance differences. Finally, a committee of LLM judges collaboratively discusses and decides the winner, reducing bias and enhancing fairness. During the peer battles, we observe intriguing scenarios where the LLM candidates display competitive behaviors and even learn from the opponents. In our extensive experiments involving 15 recent LLMs, Auto-Arena shows a 92.14% correlation with human preferences, surpassing all previous expert-annotated benchmarks without any manual efforts. As a result, Auto-Arena offers a promising alternative to current human evaluation platforms for evaluating LLMs automatically.

Decision: reject
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1729667900094

thread_type:
official_review

<thread_object>
creator: reviewer_j7qn

<content>
summary:
The paper proposes Auto-Arena, an evaluation framework that automates the evaluation of large language models (LLMs) through a multi-stage process involving question generation, peer battles, and committee discussions. This approach mimics human-like evaluation while addressing limitations of static datasets and manual assessments. Auto-Arena achieves high correlation with human preferences, enhancing the reliability of LLM evaluation. It also proves adaptable to different languages and domains, making it suitable for diverse LLMs and scalable leaderboard creation.

strengths:
- Auto-Arena provides a scalable, automated evaluation of LLMs, reducing reliance on manual efforts.
- Works across different languages and tasks, making it versatile.
- By using the peer-battle mechanism, the framework offers a competitive context that reveals subtle model strengths and weaknesses.

weaknesses:
- The code provided for review has not been anonymized properly and reveals the author's name: https://anonymous.4open.science/r/Auto-Arena-Code/analysis_scripts/result_analysis.ipynb
- The paper only involves baselines experiments for automated LLM evaluations using the Auto-Arena framework. While the study introduces a modified evaluation method, the results primarily showcase this framework's effectiveness compared to existing evaluation methods, rather than demonstrating significant new findings about model capabilities, which makes the contributions seem incremental. For example, it would be nice to see the individual model performances on each domain according to Auto-Arena, and some analysis on it.
- It seems like Auto-Arena involves a lot more agents interacting (used in both peer debate and committee discussions), making it more resource-intensive than existing evaluation methods. Does the increase in accuracy justify the added computational costs? How does Auto-Arena compare in cost-efficiency to existing evaluation methods? It would be nice to see a more detailed cost breakdown and comparison, including computational resources used, to better assess the cost-efficiency trade-off between Auto-Arena and existing evaluation methods.

questions:
See above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730130174780

thread_type:
official_review

<thread_object>
creator: reviewer_pycx

<content>
summary:
Auto-Arena uses LLM-driven agents to handle the entire evaluation process through debates and discussions, cutting down the need for manually labeled data. This framework closely aligns with human preferences and is flexible enough to adapt across different domains and languages, showing robust performance in both English and Chinese evaluations. The process has three main stages:
1. An LLM examiner dynamically creates varied questions to prevent data contamination and simulate real-world use cases.
2. Two LLM candidates debate across multiple rounds on each question, showcasing their abilities in reasoning, interaction, and strategy.
3. A panel of top-performing LLMs jointly assesses the debates to determine the winner, enhancing fairness and reducing biases in the evaluation process.

strengths:
1. The paper provides solid experimental validation for Auto-Arena, showing it aligns with human preferences at a correlation of 92.14%. In terms of experimental design, the authors included multiple benchmarks and ablation studies, which effectively demonstrate how multi-round battles and committee discussions enhance evaluation accuracy and consistency.

2. In the appendix, the authors mention that their method is cheaper than comparable approaches (though still more expensive than dataset-based evaluation), estimating around $5 to evaluate a single model. However, they don't break down this calculation, which seems like a pretty relevant detail. Could you elaborate on that? I think it would really strengthen the paper to clarify the cost calculation.

weaknesses:
1. Using LLMs to evaluate other LLMs has always been considered somewhat unreliable, as their effectiveness as judges is closely tied to the foundational model’s performance. In classic tasks where these models perform well as judges, it’s often due to data leakage (i.e., the model has likely seen the answers before). The authors claim that Auto-Arena can be “easily extended to other domains and languages,” but there’s a lack of experimental support for this. Testing on newer tasks and data—ones that no LLM agent has seen—would better demonstrate if it can still perform well in unfamiliar contexts.

2. Although Auto-Arena aligns closely with human preferences, the paper could improve by analyzing instances where LLM judgments differ from human judges, especially in nuanced or high-stakes cases. Examples or further analysis in these areas could highlight where LLM judges may still need refinement, helping the framework align more closely with human evaluation standards.

questions:
1. As I mentioned earlier, the paper states that Auto-Arena is cheaper than other methods but still costs more than traditional dataset-based evaluations, estimating around $5 per model evaluation. Could you provide a more detailed breakdown of this cost? Specifically, how much goes to computational resources, model inference, multi-round battles, and committee discussions? In addition to the computational cost, what about the time required for these evaluations?

2. Why did you choose stronger models as judges instead of the initial participants in the debate? Wouldn’t using the original debaters as judges improve the quality of candidate answers?

3. Auto-Arena is presented as a dynamic, automated alternative to static benchmarks, but it still relies heavily on the strength of the foundational model itself. Have you considered incorporating external information sources, similar to how Bing does, to enhance performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730577423806

thread_type:
official_review

<thread_object>
creator: reviewer_muxt

<content>
summary:
The paper proposes a new framework name "Auto-Arena", the core idea is make evaluations dynamic and rely fully on static datasets for evaluating LLMs as new models and datasets gets introduced. The authors make use of different set models in agent framework to make the models act like peers in a debate and then use LLM as jusge to judge the conversation or the debate the previously occurred between the models. The results are good when compared to the existing baselines and detailed analysis is done in terms of contaminations, ablation study etc.

strengths:
-> The architecture of the new framework is novel

-> Using questioning as part of the framework is something new and would help in coming works, as questioning is an important aspect in terms of reasoning

-> The results are well above the existing baseline results

-> Detailed analysis of contamination results

-> Evaluation is very big problem in the LLM research field and evaluations like this help in making it easier for evaluating LLMs as more 
and more models are released

-> The paper is very well written

weaknesses:
-> The Limitations section is missing, while the work is good there are limitation of this work as well

-> The results would have been more robust if there was human in loop evaluations rather than comparing with static baselines

-> The evaluation for the questions is limited to only 30 questions, there should have more questions taken into consideration for the evaluation of the question as there questions are the most important part for this framework, they lay the foundation. Also more detailed analysis of these questions in the main paper would be better, given the importance of this part.

-> There is need for exploring the models for unseen data and how the models can debate or judge on the topic they have not seen before, you would need all the models to have equal time-stamp cutoffs to consider for peer-review or judge the evaluations

questions:
What would happen if you use one weak model or a small model in the framework? how would the results change if that model is in peer-battle and when it is acting as judge and when it is creating questions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730670644506

thread_type:
official_review

<thread_object>
creator: reviewer_y6px

<content>
summary:
This paper proposes a novel framework for automatically evaluating LLMs using LLM-powered agents. The framework consists of three main components: question generation, multi-round peer battles, and committee discussions. The authors validate their approach through extensive experiments with 15 recent LLMs, achieving a 92.14% correlation with human preferences without manual effort. The paper also reveals some of the intriguing LLM behaviors in competitive peer battles.

strengths:
1. The motivation of the paper is good and clear: current LLM evaluation methods face major limitations. Static benchmarks lack flexibility and may suffer from data contamination, human evaluation requires extensive manual effort, and single-judge evaluations risk bias. This highlights the need for an automated, reliable evaluation method that aligns well with human preferences.

2. The proposed method is validated by comprehensive experiments on 15 modern LLMs, achieves a 92.14% correlation with human preferences, and demonstrates extensibility across languages and domains.

3. The paper is well-structured, well-written, and clearly explained; it is a pleasure to read.

weaknesses:
The application of multi-agent debate in automatic evaluation is not particularly novel, and this paper lacks discussion of a closely related work, ChatEval [1].

There are no formal guarantees on the reliability of the evaluation.

I cannot identify further weaknesses in the paper, but there are several questions I would like the authors to address to enhance the work:

(1) How sensitive are the results to the choice and number of initial committee members and their ranking initialization? Would including low-capability LLMs in the judge committees reduce overall performance?

(2) Can you provide examples of cases where Auto-Arena's evaluation significantly diverged from human preferences? What insights did these cases provide?

(3) How do you ensure that the questions generated by LLMs are sufficiently challenging to evaluate a variety of LLMs as they continue to evolve?

[1] ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

questions:
See the weakness section above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732346667516

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Please let us know if there are lingering concerns

comment:
Dear reviewers,

We have addressed the review comments in the rebuttal. Once again, thank you so much for the valuable inputs. If there are lingering concerns, do you mind raising them in the discussion? We'd be happy to discuss them further. We look forward to your replies.

Best regards,
Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733207848944

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Conclusion

comment:
We'd like to sincerely thank all the reviewers for their thoughtful reviews and instructive feedbacks. Here, we summarize some of the strengths noted by the reviewers, changes we have made in the rebuttal revision, and our key takeaways once again. 

> Strengths
> 1. **Motivation**: "good and clear" (y6Px); important to the field of LLM evaluation (MUxT, y6Px).
> 2. **Method**: "Novel" (MUxT), "Versatile across different languages and tasks" (j7qN, y6Px); "Cheaper" costs compared to "comparable approaches" (PycX).
> 3. **Experiments**: "Solid experimental validation" (PycX); "Comprehensive" (y6Px); "Results well above existing baselines" (MUxT).
> 4. **Paper writing**: "well-structured, well-written, and clearly explained" (y6Px, MUxT)

Corresponding to the feedbacks provided by reviewers, we have made the following changes in the rebuttal revision. No further concerns have been raised.

> Rebuttal Revisions
> 1. For a more in-depth analysis on the evaluation results, we have included a radar chart of model performances in 4 representative categories (writing, reasoning, math, and coding) along with a detailed analysis in Appendix F
> 2. For computational costs, besides the current overall analysis on costs, we have included a detailed breakdown of computational resources in Appendix I. It now includes a table of detailed breakdown costs for each framework component in Auto-Arena and an analysis on the costs.
> 3. To improve the paper, we added references to two more papers: ChatEval and Peer-reviewer-in-LLMs in Related Work L509-510. We have also added discussion on potential limitations in L535-539.

To summarize, our paper has made the following contributions:

> Key Takeaways
> 1. We proposed Auto-Arena, which tackles the LLM evaluation task with full automation in an end-to-end manner. It provides a fast, cheap, and scalable way to evaluate LLMs in the fast-paced industry today.
> 2. As the method is fully automated, it can be easily scaled to other domains and languages. This is especially helpful for usage scenarios where evaluation datasets are scarce.
> 3. We conduct comprehensive experiments and ablations to show the robustness of our method, which demonstrates the highest alignment with human preferences despite zero manual efforts and cheaper costs.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Auto-Arena, an automated evaluation framework for large language models (LLMs) that utilizes a multi-stage process comprising question generation, peer battles, and committee discussions. By emulating human-like evaluation, Auto-Arena overcomes the limitations of static datasets and manual assessments. The framework demonstrates a strong correlation with human preferences, improving the reliability of LLM evaluation. Furthermore, Auto-Arena is highly adaptable to various languages and domains, making it a versatile tool for evaluating diverse LLMs and facilitating scalable leaderboard creation.

Positive points:
+ The motivation of the paper is interesting and seems good.
+ The paper is well written

Negative points:
-  The selected topic can be not particularly novel
-  The alignment between LLM and real humans should be further studied
-  The baselines in the experiments can be more comprehensive