League of Lexicons: Considerations for Ranking LLMs in Head-to-Head Combat
Authors: Roland Daynauth, Christopher Clarke, Krisztian Flautner, Jason Mars, Lingjia Tang


Reviewer paper summaries: This long paper explores human pairwise ranking of LLM outputs, and introduces some principles and recommendations on how to produce an accurate global ranking of models based on these head-to-head "matchups". Three desirable properties of ranking methods are outlined: transitivity, high predictive accuracy, and an appropriate degree of sensitivity to their own hyperparameters and to the battle conditions, i.e. arena-style (as in ChatBot Arena) or controlled style (as in SLAM, where all models have similar numbers of matchups). Then, several methods of obtaining the global ranking (i.e. Elo, Bradley-Terry model, Glicko, Markov chain) are compared in terms of the desirable properties, using pairwise preference data from Chatbot Arena and SLAM. The impact of the hyperparameters of the ranking methods is then analyzed, and the optimal ranges are identified. Finally, the authors make several general recommendations on how to effectively conduct LLM model ranking.
Summary of strengths: - The motivation is clearly laid out in the intro.
- The analysis of the desirable properties of LLM ranking systems is somewhat novel (as far as I know), and may be of some use.
- The experimental methodology is sound as far as I can tell, and there are some interesting findings.
- The paper is generally clear and very well written.
Summary of weaknesses: - The paper does not specify which implementations of the four ranking methods was used, or if the authors developed their own implementations.
- The paper does not specify whether any experimental code will be made available to reproduce the results. 
- Some of the equations seem to contain slight errors or terms that are not defined (see below).
Comments, suggestions and typos: - Subsections 2.2, 2.4, and 5.3: "system" should be capitalized, since title case is used for all the other subsections. 
- Eq. 6 and 7: what is $d$?
- Lines 219 and 522: maybe replace "for" with "with".
- Eq. 10 and 11: what is $N_i$?
- Line 240: what does $\pi T = \pi$ mean? Is there a typo here?
- Line 402: add "Figure" before "2".
- Line 419: did you use HuggingFace?
- Lines 443-444: I think you mean: "In contrast to Chatbot Arena, the distribution of matchups in SLAM is..."
- Footnote 2: the marker should be after the full stop.
- "Hyperparameter is sometimes spelled with a hyphen, sometimes without.
- I wonder why the title contains the term "lexicons". I get that "league of lexicons" probably refers to the game "league of legends", but "lexicons" is not an important concept in this work specifically...
Overall assessment scores: 3