A Statistical Framework for Ranking LLM-based Chatbots

Authors: Siavash Ameli, Siyuan Zhuang, Ion Stoica, Michael W. Mahoney
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models (LLMs), Paired Comparison, Statistical Ranking, Human Preferences, Chatbot Arena, Logistic Regression

Abstract:
Large language models (LLMs) have transformed natural language processing, with frameworks like Chatbot Arena providing pioneering platforms for evaluating these models. By facilitating millions of pairwise comparisons based on human judgments, Chatbot Arena has become a cornerstone in LLM evaluation, offering rich datasets for ranking models in open-ended conversational tasks. Building upon this foundation, we propose a statistical framework that incorporates key advancements to address specific challenges in pairwise comparison analysis. First, we introduce a factored tie model that enhances the ability to handle ties—an integral aspect of human-judged comparisons—significantly improving the model's fit to observed data. Second, we extend the framework to model covariance between competitors, enabling deeper insights into performance relationships and facilitating intuitive groupings into performance tiers. Third, we resolve optimization challenges arising from parameter non-uniqueness by introducing novel constraints, ensuring stable and interpretable parameter estimation. Through rigorous evaluation and extensive experimentation, our framework demonstrates substantial improvements over existing methods in modeling pairwise comparison data. To support reproducibility and practical adoption, we release leaderbot, an open-source Python package implementing our models and analyses.

Decision: accept (poster)
Reviewer scores: 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730381333583

thread_type:
official_review

<thread_object>
creator: reviewer_cyaq

<content>
summary:
This paper addresses a novel problem in the evaluation of large language model (LLM)-based chatbots by proposing an advanced statistical framework that builds on existing methods used in the Chatbot Arena setting. The framework integrates well-established models, such as Rao-Kupper and Davidson, to account for ties in a rigorous axiomatic framework, and introduces Thurstonian representations to model covariance structures between competitors. These additions allow for more nuanced insights into chatbot rankings and performance consistency. A Python package, leaderbot, is also provided for reproducibility and ease of experimentation.

strengths:
1. **Pioneering Approach**: This paper tackles a novel and important problem in LLM evaluation, presenting a unique perspective on using advanced statistical models to refine the ranking process in chatbot comparisons. Its approach to systematically modeling ties and latent structures sets a new precedent for evaluating LLM-based chatbots and offers fresh insights that go beyond traditional ranking methods.
2. **Innovative Use of Thurstonian Representations**: By integrating Thurstonian models and introducing covariance structures, the paper offers a groundbreaking method for capturing relationships among LLMs. This enables a deeper analysis of model consistency and performance that extends beyond simple rankings, marking a notable advancement in chatbot evaluation techniques.

weaknesses:
1. While the paper introduces an alternative framework, it does not clearly discuss why Arena’s Elo-based approach is insufficient beyond the issue of ties. Additional insight into Arena’s limitations in capturing competitive dynamics or certain statistical shortcomings would strengthen the argument for this new model.
2. The authors mention consistency in high-ranking models and variability in lower-ranking models, but do not explore further distinctions within these groups. For example, identifying specific characteristics (e.g., model size) associated with high consistency could provide more actionable insights.

questions:
1. The paper mentions that high-ranking models show greater consistency than lower-ranked ones. Could you expand on this observation? For example, are high-ranking models generally larger or more sophisticated in architecture? And at what ranking position does this shift in consistency typically occur?
2. How is the effectiveness of handling ties quantified across different statistical models? Is there an analysis of which models are more favorable to certain LLMs, and if so, what might explain these differences?

If these questions can be addressed, the paper could potentially be rated one point higher.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730482604602

thread_type:
official_review

<thread_object>
creator: reviewer_fnun

<content>
summary:
The authors propose an improved statistical model for ranking large language models (LLMs) on the Chatbot Arena dataset to enhance the traditional Elo rating system. Paper points out that the Elo system has limitations in handling ties and capturing relationships between models. To address these issues, paper introduces the Rao & Kupper and Davidson models, as well as a novel factor model to better capture the complexity of ties between different models, thereby improving prediction performance. Finally, they provide a Python package, “Leaderbot,” to reproduce the statistical model and support further experiments.

strengths:
1. The authors identify symmetry issues in the likelihood function of traditional models, which could lead to instability in parameter estimation. To address this, they propose symmetry constraints that ensure stable parameter estimation, thereby enhancing the model’s optimization performance and interpretability;
2. They effectively address the issue of ties, which is a limitation of the existing Elo system, and make optimizations to handle this;
3. They provide a Python package that allows for the reproduction of the paper's results and supports further research.

weaknesses:
1. The authors use multiple models to rank models, showing that high-ranking models exhibit greater consistency than lower-ranking ones. However, they do not provide further analysis, such as examining the specific characteristics of models that initially show ranking inconsistencies;
2. The authors’ work focuses on optimizing the ranking model but lacks subsequent analysis. Additional insights, such as a more in-depth examination of correlations between LLMs or an analysis of the differences between Leaderbot rankings and Chatbot Arena Elo rankings in relation to model characteristics, would enhance this work.

questions:
1. In the treatment of pairs and unique pairs, have the authors considered introducing semantic clustering to their model, as opposed to solely using statistical techniques? Additionally, could they clarify the reasoning behind not exploring semantic relationships within pairs?
2. Have the authors considered methods other than PCA to further analyze the correlations between models in the results? The current approach does not sufficiently reveal the underlying correlations between LLMs in an intuitive way.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730618827357

thread_type:
official_review

<thread_object>
creator: reviewer_ztf7

<content>
summary:
This paper presents a statistical framework for ranking LLM-based chatbots. It addresses the limitations of existing methods like the Elo rating system by incorporating ties and covariance structures. The proposed apply well-established statistical models to properly account for ties within an axiomatic framework and introducing factor analysis. Experiments on the Chatbot Arena dataset show improved accuracy and insights into chatbot rankings and relationships.

strengths:
1. The proposed method can account for ties in an axiomatic framework. This improves not only tie prediction but also enhances win-loss inference.
2. The mathematical analysis is comprehensive and thorough.
3. This work has an open source python package which is easy to use.

weaknesses:
1. For model ranking, it has no "ground truth". Therefore, it is hard to convince others under this method, the ranking is more accurate.
2. The evaluation is highly dependent on the Chatbot Arena dataset which makes this work a slight improvement on chatbot arena. Therefore, the impact of this work is limited.
3. As for chatbot arena, a simple enough ranking rule is more important if the users are common users. This work will make the rule too complicated for them to understand and then reduce the impact of chatbot arena.

questions:
1. Is this method only useful to chatbot arena or it is useful to all Elo ratings?
2. We need another figure to see the ranking difference of original chatbot arena and this methods. Also the figure 1 showing the result of PCA analysis is confusing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732621417996

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thank you for your constructive feedback. Some comments highlighted areas where our explanations could be clearer, while others raised insightful questions that inspired us to expand our analyses and discussions. These suggestions have significantly enhanced the clarity and presentation of our manuscript. Below, we summarize the key enhancements and additions.

---

## Enhanced Analysis of Ranking Variability Across Models

To provide a more systematic and comprehensive examination of ranking variability, we expanded the analysis in **Appendix Section E: Comparative Analysis of Ranking Variability**, building on the visualization provided by the bump chart in the previous submission. Key additions include:

- **E.3: Quantifying Ranking Similarity Across Models.** Using Kendall's ranking correlation matrix, we systematically quantified the alignment of rankings across all 30 models. This approach offers a robust mathematical framework for comparing ranking similarities, going beyond the visual insights provided by the bump chart.

- **E.4: Identifying Ranking Similarities via Hierarchical Clustering.** By applying hierarchical clustering to the Kendall correlation matrix, we identified distinct clusters of ranking behaviors. This analysis revealed an interesting result: the inclusion or exclusion of the Thurstonian covariance model is the main factor driving these clusters. These findings illustrate how covariance modeling shapes ranking consistency and complements earlier results on the impact of generalized tie modeling on inference accuracy.

---

## Incorporation of Complementary Methods for Correlation Analysis

We expanded our correlation analysis by introducing **multidimensional scaling (MDS)** in **Appendix Section C.2: Interpretation and Visualization of Covariance**. This method complements kernel PCA from the original submission by offering an intuitive spatial representation of chatbot relationships. MDS arranges competitors in a configuration that mirrors their rankings—achieved without direct access to absolute scores. Instead, this structure is derived solely from score differences and covariance-based uncertainties, effectively uncovering the ranking order.

In **C.3: Hierarchical Clustering of Competitor Performance**, we extended this analysis to identify performance tiers among competitors. The hierarchical clustering approach, with optimal leaf ordering, not only grouped competitors into meaningful tiers but also retrieved their ranking order purely from the dissimilarity matrix derived from our generalized model. This result highlights the strength of the dissimilarity matrix in capturing both relative performance and structural relationships among competitors.

---

## Exploration of Model Characteristics and Rankings

In response to comments about the relationship between ranking and model characteristics, we introduced **Appendix Section F: Relationship Between LLM Characteristics and Scores**. This section explores the connection between performance scores and attributes such as the number of parameters, computational budget (FLOPs), and dataset size. Our analysis highlights strong correlations between performance scores and computational budget (FLOPs) and dataset size, consistent with trends in scaling laws.

---

## Clarifying Covariance and Identifiability

We added **Appendix Section B: Unidentifiability of Parameters in Paired Comparison Models** to provide a foundational understanding of the unidentifiability of covariance in pairwise comparison frameworks. This section sets the stage for the subsequent analysis in **Section C**, ensuring that the covariance-based methods employed in our work are well-justified.

---

In addition to these substantive analyses, we made a few adjustments to improve clarity and readability. For instance:
- Table 1 from the previous submission has been moved to **Appendix Section D.1**.
- Figure 2, previously in the appendix, has been moved to the main body as **Figure 1** with slight reordering of its panels for better interpretability.

We hope these revisions address your comments and enhance the overall quality of the manuscript. If there are any remaining concerns or areas where further clarification would be helpful, please let us know. We truly appreciate your feedback and the opportunity to improve our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an improved version of ELo rating in Chatbot Arena by considering ties and covariances by using Rao & Kupper and Davidson models. Experiments found that the proposed improvement is better at modeling empirical data.

Strengths:
1. The method improves the ELo rating by considering ties and covariances.
2. The authors also released an open-source Python package.

Weaknesses:
1. It is not clear how much benefits we get by considering ties compared to the existing ELo ratings.
2. As reviewers pointed out, the proposed metric is harder to be understood by normal users of Chatbot Arena.

Since reviewers are overall positive about this paper, I'm recommending acceptance, but I wouldn't mind if the paper gets rejected.