Re-evaluating Open-ended Evaluation of Large Language Models
Authors: Siqi Liu
Ian Gemp
Luke Marris
Georgios Piliouras
Nicolas Heess
Marc Lanctot
Venue: ICLR 2025 Poster (2025)
Keywords: Evaluation
Game Theory
Large Language Model
Equilibrium
Open-Ended
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We study open-ended LLM evaluation and propose a scalable equilibrium rating framework that provides robust and interpretable ratings of models and prompts.

Abstract: Evaluation has traditionally focused on ranking candidates for a specific skill. Modern generalist models, such as Large Language Models (LLMs), decidedly outpace this paradigm. Open-ended evaluation systems, where candidate models are compared on user-submitted prompts, have emerged as a popular solution. Despite their many advantages, we show that the current Elo-based rating systems can be susceptible to and even reinforce biases in data, intentional or accidental, due to their sensitivity to redundancies. To address this issue, we propose evaluation as a 3-player game, and introduce novel game-theoretic solution concepts to ensure robustness to redundancy. We show that our method leads to intuitive ratings and provide insights into the competitive landscape of LLM development.

Published as a conference paper at ICLR 2025
RE-EVALUATING OPEN-ENDED EVALUATION OF LARGE LANGUAGE MODELS
Siqi Liu", Ian Gemp", Luke Marris, Georgios Piliouras, Nicolas Heess, Mare Lanctot Google DeepMind London, UK
{liusiq qi, imgemp, marris, gpil, heess, lanctot google com
ABSTRACT
Evaluation has traditionally focused on ranking candidates for a specific skill. Modern generalist models, such Large Language Models (LLMs), decidedly outpace this paradigm. Open-ended evaluation systems, where candidate mod- els are compared on user-submitted prompts, have emerged popular solution. Despite their many advantages, we show that the current Elo-based rating systems can be susceptible to and even reinforce biases in data, intentional or accidental, due to their sensitivity to redundancies. To address this issue, propose eval- uation as 3-player game, and introduce novel game-theoretic solution concepts ensure robustness to redundancy. We show that method leads to intuitive ratings and provide insights into the competitive landscape of LLM development.
INTRODUCTION
We can only improve what we measure, yet measuring the performance of Large Language Models (LLMs) has become an clusive endeavor owing to their breadth and depth of capabilities. Real- world benchmarks are costly curate, increasingly requiring feedback from human domain experts (Hendrycks et al.) 202I, Rein et al.] 2023). Synthetic benchmarks can help, but their relevance to real-world performance is less clear (Zhang et al., 2024 Hsieh et al. 2024). An vexing challenge of static benchmarks is that of test set contamination, a phenomenon difficult to prevent despite efforts (Golchin & Surdeanu 2024 Balloccu al., 20244]] Palavalli al. 2024). Enumerating skills of interests with narrowly defined static benchmarks seems to be an uphill battle from the outset, as frontier models become generally capable.
more
An emerging trend in LLM evaluation is therefore to rely on open-ended evaluation systems, no- (022)). In such system, users submit
table example being the LMSYS Chatbot Arena (Chiang prompts of interest, with each model assigned an Elo score (Elo 1978) based on how they compare to each other on all prompts. In contrast static benchmarks, this open-ended approach enjoys liveness, diversity and scale, lending itself to become important reference in LLM development. Despite an intuitive sense of progress, issues around redundancy, bias and quality of crowdsourced data have been raised (Chiang et al.) 2024 Ahuja et al. 2023 Li et al. 2024b). Several recent stud- ies reverted back to centralized curation for quality (Taori et al. 2023 Lee et al. 2024 White et al., 2024). Increasing commercial efforts have been invested in private and proprictary evaluation too.
Perhaps this tension between quality and open- open-endedness to be expected in LLM evaluation. Bi- ases, redundancies and quality issues in the prompt distribution affect Elo ratings, as they reflect performance on average. This along with other identified deficiencies of the Elo system (Balduzzi)
al. 2018 Bertrand al. [2033] Lanctot al., 2023) raise crucial questions for LLM develop- ment: how does an Elo-based open-ended evaluation system affect model development today, and how can we mitigate its drawbacks, if any, the future? In this paper, we provide an empirical simulation-base investigation of the former and lean on game theory for a solution to the latter.
The connection between evaluation and game theory needs unpacking. Consider set of agents and a set of tasks, a naive approach to evaluation would rank agents by their average performance over tasks, propagating biases and redundancies in the task set. rame-theoree approach (Balduzzi
Equual contribution.

---

Published as a conference paper at ICLR 2025
et al. 2018). would be to consider evaluation as agent-vs-task game where the agent (task) player chooses one of its agents (tasks) and is rewarded (penalized) by the agent's 's performance on the task. This teeneetheeettcccco perspective accomplishes two goals simultancously. First, it lets the evaluation system designers express their goals in players' objectives: here, [Balduzzi et al.] (2018) evaluates agents under adversarial task selection. Second, game-theoretic equilibrium decides which actions are played during evaluation: quality and redundancies in players' action sets do not matter. It is in this sense that game theory is well suited for evaluation systems that are open-ended.
Applying game theory to LLM evaluation however has its own challenges. Indeed, the decision of Balduzzi et al. (2018) in comparing agents under adversarial task selection was guided by theoretical benefits. In 2-player zero-sum games, approximating a Nash equilibrium (NE, Nash et al. (1950)) is computationally tractable. NEs are also interchangeable in this setting as playing any NE guarantees zero exploitability. Beyond this setting, both benefits are lost: approximating NEs is computationally hard in the worst case (Daskalakis et al. 2006) and despite recent progress important challenges remain (Gemp et al. 2022 (2024). Equilibrium selection in this generalised setting remains a long- standing challenge too (Harsanyi & Selten 1988 Rinott & Scarsini 2000). For instance, driving on either side of the road is an equilibrium, but it is unclear which equilibrium should be used for evaluation. Past attempts at game- theoretic evaluation have therefore been restricted to the 2-player zero-sum settings when LLM evaluation calls for at least 3 players (e.g., model-vs s--model-vx-prompe
In this paper, we make several contributions that lead up to our equilibrium rating framework:
1. We show, via a simulated example (Section I.1 the risk of models specializing in a few skills, at the expense of others, as they maximise their Elo ratings. Similarly, popular practice in prompt selection further reinforces this trend;
2. We introduce novel equilibrium solution concepts for N-player general-sum games that are unique and dlone-invariant, pre-requisite for our equilibrium rating method (Section B):
3. We show our method scales to a real-world LLM evaluation dataset (Section 4.2 and pro- vide ratings that invariant redundancy and correspond to intuition in the sense of risk-dominance (Harsanyi & Selten 1988), with empirical evidence (Appendix R:4);
4. We provide examples of analyzing these equilibrium structures of the game, drawing in- sights into the competitive landscape of LLM evaluation (Section 4.3)
1.1 ELO RATING IMPROVEMENT PATH: A SIMULATED EXAMPLE
With models continually improving their Elo ratings in systems such as LMSYS Chatbot Arena (Li et al. 2024a), it is worth asking if higher Elo scores translate to meaningful progress across skills of interest. This is difficult answer from real-world data: we cannot replicate LLM development at scale nor can we disentangle factors driving model development besides maximizing leaderboard ratings. A synthetic example can provide insights in a controlled setting.
Consider S orthogonal skills of interests, M models and P prompts with each prompt a probability vector p E A over the skills and each model vector m E TR. representing its competencies in each skill. We can then define the utility of selecting model m: when compared to model m, on prompt Pk, as Um ( (k. my; my) P% (m; m2) with i, E [M] and k E [K]. A less common but equally valid question is what should be the utility, any, for selecting a prompt. We follow a similar definition as Li et al. (2024b) and define the utility in choosing prompt ps as (p m., m,j) Imm (Pk. m,, my)). The separability of a prompt is then 1 the prompt selection criterion used in offline benchmarks such as arena-hard-vo.1
3/2 Σig up m,, m,). consistent with
We now observe how this system evolves with rating- maximizing players. Consider two settings: a) the "initial prompts" setting where the set of prompts is fixed but the set of models expands; and b) the "additional prompts" setting where prompt and model players alternate to introduce new prompts and models. We use simple evolutionary process for our simulation (see Appendix F. for pseudocode). Let P, and M, be the number of prompts and models at iteration and Po. Mo the number of initial prompts and models sampled from Dirichlet(1,.5 iteration which is a sum of improvement vectors sampled from Dirichlet (1.ss), such that the new addition receives the highest rating according the rating method used (i.e. Elo or our equilibrium- based method). In the "additional prompts" setting, a best-of-64 prompt is added at each iteration, selected by their separability when Elo ratings are used, and by their equilibrium ratings otherwise.
We introduce a model at each
2

---

Published as a conference paper at ICLR 2025
CO
Elo models
Model sidll entropy
models
Submit Rating
1.4
1.3
Equilibrium models
Models
-10
5
10
Skill
Iteration
Open-ended Evaluation System
10
Prompt skill entropy
Elo prompts
Intiaal promptt
1.4
Prompts
Equilibrium prompts
Rating
Submit
Iteration
Skill 10
Additional prompts
O C?
Equilbrium (Ours)
Inttial prompts
Iteration
Figure (Left) We simulate the effect of the rating method on model development with users submitting highly rated models (and prompts) iteratively. (Center) We show how model and prompt skill entropy evolves under different rating methods over 32 trials. (Right) show an example sequence of models and prompts maximising their respective ratings. Darker indicates higher value.
Figure (Center) shows our findings. Let HP:)). H(m;) be the prompt and model skill entropy at iteration t with p. = P. P.Ph p: and m. M. M. m. and the Shannon entropy. The Elo rating method leads to a consistent decline in skill entropy: the sequence of models improve along specific skill dimensions that are over-represented in the fixed set of initial prompts (dashed). Adding prompts with high separability further reinforces this trend in both model and prompt skill entropy (solid). We offer intuitive explanation. Improvement reflects improvements against the average. At iteration f, the expected utility to model m, is given by Um (P., m,, m))) with its gradient defined by P. improving on the most prevalent skill in p. therefore leads to the steepest ascent in model utility. Similarly, the gradient for prompt vector is defined by the absolute deviation of the model vectors along each skill dimension M. Σ |m; - m Prompts that target the skill dimension with the highest "spread" averaged across all model pairs are therefore the most highly rated. Figure [ (Right) illustrates this phenomenon from single trial. The Elo-maximising sequence of models specialises in skill due to prompt redundancy while the sequence of prompts remains focused on skill the expense of others.
the Elo ratings the separability metric
The underlying challenge, one that we address, is to propose practical rating method that compares models and prompts a way that is intuitive and robust to redundancies. Figure (Center) suggests that maximising equilibrium ratings preserves skill entropy. Indeed, Figure (Right) shows models focusing on different skills across iterations. As prompts are no longer measured against the average model pairs, they also remain diverse. In both cases, ratings computed at game-theorete equilibrium distribution, instead of the uniform. We now present our equilibrium rating framework.
(Right)
2 BACKGROUND
Normal-form game normal-form game is a tuple (N, A, u) where is a finite set of players N' = {1...., _) indexed by i, a tuple of strategy (action) sets A (Ai...., An), and tuple of utility functions u = (u....., u,,) with A, and u, A - R being player i's strategy set and utility respectively. Let A = (aI...., an) with a, A, for all denote a strategy profile. We allow strategy profiles to be selected randomly according to a distribution æ E A(A) over joint actions. Let æ; denote the marginal distribution player i's strategy set A;, i.e., with all players j y
3

---

Published as a conference paper at ICLR 2025
marginalized out. Likewise, let x- denote the distribution with player marginalized out. We call x a pure strategy it places all mass on a single action profile and mixed otherwise. Each player's utility function is naturally extended to randomized strategy profiles by considering its expected value (x) = (aa=((((a)) Similarly, let (z( x-)) aisaz [u:[a)].
Coarse Correlated Equilibrium (CCE) and Nash Equilibrium (NE) An equilibrium is a strat- egy profile æ from which no player has an incentive unilaterally deviate. Define player i's incen- tive to deviate to x ((A,) unilaterally as regret, (2(,x)) x) = us (22, (r-)) ux(x), where A(A,) is the simplex over A,. Then, player i's maximum regret for deviating from æ, defined as:
regret, (2), m)]
max afe
max *(EA(A))
The profile a is an approximate Coarse Correlated Equilibrium (Aumann 1974 1987) (c-CCE) iff Vi, (aAA)) regret, (x),,, at) If be factorized into player marginals such that players cannot correlate, i.c., x= X i-1 Zi, then æ is also an e-NE. NEs are a subset of CCEs.
Equilibrium Selection Games can have many equilibria. Additional criteria are often introduced to make their selection unique. The set of CCEs is always convex, and any strictly convex objective function such as negative Shannon entropy can used to select a unique equilibrium.
In contrast, the set of NEs need not be convex, however, several solutions have been proposed to
solve for unique Nash equilibria in general-sum games (Harsanyi & Selten 1988). The LLE was originally defined by McKelvey & Palfrey (1995) along with their introduction of quantal response (logit) equilibria (QREs) which satisfy the following fixed point equation for all players i E N:
24 softmax | Va; u:)
where Vz, Uy is the gradient of d, w.r.t. Zt- QREs are defined by a temperature parameter т and can be interpreted as the Nash equilibria of a game with payoffs perturbed by Gumbel(0,7 r) noise.
Computing the LLE involves tracing a continuum QREs, starting at temperature T = 00 (corre-
sponding to the uniform strategy profile) and ending at the LLE in the limit of r 0. The LLE is unique in all games except a O-measure set (McKelvey & Palfrey 1995; Goeree et al., 2003).
Another reason to solve for an LLE is that it falls into the family of homotopy methods (Herings & Peeters 2010), which were shown to select risk donmiomnnant equilibria in some general settings, a Nobel prize winning result of Harsanyi & Selten| (1988). Empirically, LLEs have also been shown
to approximate human play in games (McKelvey & Palfrey 1995 Goeree et al. 2003).
3 METHOD
We now describe our rating method in terms of gamification, equilibrium solving and its selection. In gamification, we endow prompt and model players with utility functions, partly inspired by prior works, such that actions played at an equilibrium reflect our intuition. We note that our specific gam- ification defines an N-player general-sum game where equilibrium solving and selection requires more careful consideration. For equilibrium solving, we build cxisting methods for approxi- mating NEs and CCEs, reformulated to accommodate entropy-based techniques that select unique equilibria and explain why ratings derived from these equilibria remain vulnerable to manipulation in the face of redundant actions. We then propose a family of algorithms based on a novel kemne-- ized entropy that select unique equilibria yet are also robust to redundant actions. Finally, for given equilibrium solution æ, we define the rating of an action a, to be regret, (at, x)).
3.1 GAMIFICATION: EVALUATION VIA A GAME BETWEEN MODELS AND PROMPTS
We study a 3-player general- sum game in our experiments. Consider a prompt player with og Ap the set of prompts. king player and rebel player each with actions am Am the set of models. Let ue (ap: am, a'm) (-1,- -1/2, 0, + +1/2, +1} be the utility function to the king player representing preference towards king model response am over the rebel model response d' on prompt Op-
4

---

Published as a conference paper at ICLR 2025
The prompt player is rewarded for separating the models, with up(ap am, d'y) |ukkapp am dm)l The rebel player receives u, (p Om, dmm = -uy((p Om, am,a', d'm) except for when am = am in which case u, (ap, am d'm)) = -1. This asymmetry discourages the same model being played by both model players deterministically with a prompt player indifferent over its actions. We refer to this game k cing-of-the-hi 11 as it favours the king player, leaving the rebel player to mount its best resistance without relying on some of the best models that the king player may choose. We refer the king player ratings the model ratings in our results.
Given a collection of prompts and models, the utility function can be tabulated with |Ap x |Am 1 pairwise preference ratings. We query a gemini Beminii-i 5 5-pro-api--0-55
judge for preference ratings similar to ZZheng et al.] (2023); Verga et al. (2024); Dubois et al. (2024a]b); CChiag & Lee (2023; ;
Liu et al. (2023). ). caveat that our results could therefore suffer from self-preference
Paniikssery et al.
2024 and should not be viewed as an objective assessment of frontier L.Mss.
3.2
EQUILIBRIUM SOLVING
For an instance of the evaluation game, we can compute different equilibrium solutions æ which then define ratings. Here we present options they are unique, scalable and lead to intuitive, invariant ratings when combined with selection criteria that we describe in Section 3.3
Nash Equilibrium (NE) While LLE computation (Turocy 2005) typically formulated as solv- ing a differential equation that evolves the temperature J towards O while obeying the logit constraint in Equation 2 i.e., 2; softmax V., u4) for all i, this also equivalent to satisfying the con- straint x; arg max (x; x=-i) + (3;))) where S(rr;) is the Shannon entropy of z;. In this work, we choose another condition
T2 arg max {ui (2,, (2,, x-1) def (((2,,=))) -DL--DDKKaaa (KL(2)lltt)
zEA
which is equivalent in the case where the target strategy t; is set to player i's uniform strategy- Using this definition of u. (&,, x)),, ,, we can define a loss function as in (Gemp et al. 2022) such that arg min L (ar) is QRE at temperature
(x)) (xx
L
Σ uf (BR - uu (1:, x)))
(4)
where player i's best response BR; = softmax( ( ur + log(t.)). By annealing т from a high value and successively re-solving for the global minimum of L we can approximately trace the QRE continuum to the LLE. In Section 3.3 we explore non-uniform tę to achieve clone-invariance.
Coarse Correlated Equilibrium (CCE) Solving for unique CCE is computationally easier than NE as the problem is convex (Equation (I) Therefore any strictly convex function can be used to uniquely select an equilibrium. For example, maximum entropy would be a suitable default criterion following the principle of maximum entropy. However,
show in Section 3.3 different target As such, we opt for maximum relative entropy to to allow for non-uniform target joint distributions. A number of off-
formulation is necessary for clone-invariance
target joint /. X the-shelf solvers (Domahidi et al., (013) and frameworks (Diamond & Boyd (016)) can be used to compute solutions to this problem. We used a particularly efficient dual space gradient based algorithm described in Appendix A for scaling.
3.3 INVARIANT EQUILIBRIUM SELECTION
There may be many NEs and CCEs McLennan Park, 1999 Sturmfels]] 2002
Mel ennan 2005
Some equilibria exhibit sparse or heavily skewed strategy profiles (see examples in Appendix F.4 Intuitively, these equilibria are risky the sense of risk dominance: playing one such equilibrium when other players do not would be costly mistake. Our goal is propose selection procedure that along with our equilibrium solving algorithms, approximates a ddooneinvvarinn equilibrium.
Shannon entropy plays key role in several equilibrium selection approaches, however, its definition vulnerable to redundancy in games. Consider a game with 2 distinct actions A and B per player and introduce b-11 clones of B into player I's action set. The maximum entropy strategy for player
5

---

Published as a conference paper at ICLR 2025
in the new game is uniform across their actions with mass 1 each, but this induces distribution that places A entropy profile defines the precise starting point for tracing the path of QREs towards the LLE. This starting point is sensitive to clones. Hence, if we compute the LLE using the uniform distribution in this new game, we will effectively start from the (A, B) mixed-strategy (145 148) rather than the desired mixed-strategy (1/2, 1/2); hence, will not necessarily arrive at the LLE of the original game.
T+8 cumulative mass on the cloned action B. From Section 3.2, the maximum Shannon
Desired properties. A clone-invariane invariant entropy definition should be:
Pl. Real-valued, finite, and non-negative for any distribution
P2. Have a well-defined gradient for any x in the interior of the simplex;
P3. Its maximizers should form a convex set. In the case of duplicate strategies (clones), the max- imizers should form precisely the set of distributions which arbitrarily distribute a mass of addition, they should achieve an entropy value
I across each of the c sets of clones. which is equal to the entropy of the system with clones removed;
P4. Amenable to efficient estimation and flexible to re-interpretation of redundancy.
Note P3. resolves the issue with Shannon entropy that we highlighted above. P1 is necessary for a reasonable measure of information content. P2 is necessary for gradient-based optimization, and P4 is practically helpful for efficient implementation and adaptation to bespoke game settings. We now introduce affinity entropy Hg: A IR. generalized Tsallis entropy (T'allis 1988) that recognises similar or redundant strategies. Its derivation from the above axioms can be found in Appendix B
Definition (Affinity Entropy H
1 (U(0))) Hg(x) [[ p
(5)
with entropic-index parameter p e (0, 17. Ub) KAJ and K a similarity kernel with entries in [o, 1] with indicating two strategies are clones, and Ap a diagonal matrix containing the (p 1)-
norms of the columns of K on its diagonal.
Theorem 1. Affinity entropy HE satisfies all desiderata P1-P4.
In experiments, we define a similarity kernel K(( for each player with entries K-(I) with
( a. Ea Ea-uts[(u((aaa_) - nu(3,a-:))
(6)
K (() 0.3 exp(-D exp(-D)((200)) (200))
(7)
where D measures the strategic dis-similarity between player i's strategies and B and K is simply radial basis function (RBF) kernel under the metric D. Note ( achieve exactly the same utility for player irrespective of the actions chosen by other players in the game. It should also be clear from the definition how one might Monte-Carlo estimate D. To select
0. is zero iff two strategies and
for NE CCE, set & arg max 1/p=1 (x) Equation (3) and Equation ( respectively.
RESULTS
We use the same hyper-parameters for equilibrium solving in all results (see Appendix F.2 For evaluation on real-world prompts, we consider the arena arena-hard-aa www-aaarrr.... dataset with 500 prompts, selected to separate frontier LLMs, as well as responses from many candidate LLMs. We consider responses from 17 LLMs in particular and queried 5aa.iiii----r--p----000- preference ratings on each prompt for each model pair. See Appendix F3 for more details.
for 8 pairwise
4.1 EQUILIBRIUM RATING IMPROVEMENT PATH: SIMULATED EXAMPLE
Recall from Figure that contrary to the Elo improvement path, maximizing equilibrium ratings led to models (and prompts) improving across skills. We inspect the equilibrium improvement path and offer our interpretation. Figure 2 (Right) shows that the shifts in focus between skills by the model

---

Published as a conference paper at ICLR 2025
Prompts
Prompt Nash equilibria
NE prompts
Models
2
3
3
4
5-
5-
8
nerrtton
Initial prompts
5
10
10
12
12 15 14 15
12
12
13
15
13
14
14
18
Prompt (Initial and additional)
Figure We inspect the model improvement path induced by NE ratings as shown in Fig- ure [ Right)) (Left) shows the sequence of additional prompts added at each iteration. Each prompt is the best-of-64 samples according to their NE ratings. (Center) shows the sequence of prompt prompts. (Right) shows the equilbrrunn-weighted
player NEs. Each defines distribution prompt skills and the sequence of king player models. Recall prompts and models are non-negative vectors over skills, darker indicates higher focus or capability each skill.
player coincides with transitions in the NE prompts, prompts weighted by their NE strategies (shown in (igure[](enttrr)) Similarly, to gain support under an NE, new prompts must highlight skill dimension along which equilibrium models are better differentiated (Figure (Left)) In sum, equilibrium prompts separate equilibrium models. This dynamic encourages exploration of new skill dimensions and incentivises models to well-rounded across skills.
4.2 INVARIANT EVALUATION
We now turn to arena-ha har and show that candidate LLMs' equilibrium ratings are in- variant redundancies when their Elo ratings are not. this experiment, we will introduce prompts targeted at bringing down the rating of certain action (in this case, model). Specifically, let ū [A.. Σ., ux ux(:, akg a,) be the vector of expected king player payoffss when playing action aş against a randomly chosen rebel model on each prompt. We can then sample prompts adversarial aj from softmax( >ttx (ax)) and add them to the prompt set. Figure 3 reports the king model rankings under different methods with ag = gemini ni -1. aa-prr--pii--- -55 and = 10.
Our first observation is that without redundant adversarial prompts, proposed equilibrium rank- ings of LLMs are fairly consistent with their Elo rankings, with a few models moving up or down two positions. This deserves attention. Out of multiplicity of equilibria, the NE and CCE selected led to rankings that correspond to our intuition. Indeed, we show in Appendix F44 that the NE we select is risk-dominant among 128 mixed-stratess NEs of this game. Second, the Elo ratings can be arbitrarily influenced by redundancy, with the top-ranked model falling through the ranks. Equilibrium rankings remain invariant. In fact, while we lose the invariance guarantee with near redundant prompts, we show models' equilibrium rankings to degrade gracefully in Appendix F.5) Third, the CCE ratings show the top-3 models to tie for the first place: correlating models with prompts affects the competitive landscape which inspect in Section 4.3 Lastly, solving for unique equilibrium is not sufficient for invariant ratings. We show in Figure (Right) that using Shannon's is entropy for tracing the QRE continuum for selecting max-entropy CCE would not lead to invariant ratings. For completeness, we provide a detailed breakdown our equilibrium ratings in terms of action ratings and marginals for each player in Appendix F.5

---

Published as a conference paper at ICLR 2025
Ratings
CCE Ratings
Elo Ratings
NE(-a)
CCE(-a)
gemini-1.5prro-pp46611
gp1-4/tubbo22244-0449
gpt-4o-2024-05-13
gemini-1.
gpt-4-015-previeew
0
0
0
gemma-2270--1-0625
7-
0
0
0
gemma-2-90-110625
0
0
X
2audde-3opus-22220222
0
0
claude
10
0
hama-3-7bb-chat-hf
command-r-plus
Owen1.5-72B-Chal
13
gpt-4-0314
14
mistral-large-2402
15
Ilama3--b-chat-hff
16
gpt-3.5-turbo-0125
17
290
500
Number of redundant prompts adversarial gemini-1-5-pro-api--544
Figure 3: We introduce an increasing number of redundant copies of prompts adversarial to gemini-1.. ---pr--ppp------ 4 and show model rankings under each method. Models at the same rank are grouped in grey and ordered alphabetically. (Right) We show equilibrium rank- ings under NE(-a) and CCE(-a) selected using Shannon's entropy instead of the affinity entropy. Dotted lines connecting different rating panels indicate continuity in the labeling. For instance, consistently ranks first under our NE and CCE ratings, despite the introduction of up to 500 redundant adversarial prompts. However, its ranking suffered signifi- cantly under the Elo ratings as soon as 250 adversarial prompts have been introduced.
gemini-1. -ap.---p----------5
4.3 INTERPRETING EQUILIBRIUM SOLUTIONS
Besides rankings, the equilibrium solutions can surface interpretable insights. We share two exam- ples using NE and CCE solutions respectively from ratings shown in Section 4.2
Nash Equilibrium Prompts We have shown that equilibrium ratings are intuitive and invariant to redundancy. follow-up question is which actions are highly-rated and which actions affect other players' ratings (i.e., with positive support at the NE).
Recall that the prompt player utility up(ap, ag a,) = lux (ap, ags, a.)) reflects the extent to which prompt separates the pair of responses from models ag and ar. The prompt player's equilibrium rating is then regret(a, (ppe x) Eag.-ay Upppp 4k, a,) with Tk. 2y the NE strategies of the king and u. rebel player respectively. By definition, prompts that are highly rated under NE ratings separate models played at the NE. In other words, while the Elo ratings reflect the strength of an action on average, equilibrium ratings reflect the strength of actions the selected equilibrium.
We can now illustrate these phenomena using the same game investigated in the second columns of Figure 3 with 250 redundant prompts added the game. First, show in Figure (Top) the king-vs-rebel payoff matrices induced by 6 sample prompts, with increasing equilibrium prompt ratings. Prompts with low ratings tend fail to differentiate performant models (i.e. top-left block of each heatmap). Second, we can ask which prompts should we expect to have support at an equilibrium. Figure A (Botoom) shows that empirically, highly rated prompts are played more often at the equilibrium we select. This implies that the model ratings are heavily influenced by small subset of prompts that separate frontier models. We note that this correlation is not guaranteed, following our discussion in Section 4.2 on redundant actions. Indeed, our final observation is that prompts that are clones with other prompts tend to receive lower probability mass than their ratings would have required. In fact, since have introduced 250 redundant prompts explicitly, highlight in gray prompts that are indeed redundant - many of these prompts enjoy high ratings, but significantly lower mass. In other words, equilibrium ratings reflect quality of an action in isolation
8

---

Published as a conference paper at ICLR 2025
King-va-rebel payofts example prompts of increasing NE ratings (left right) and columns ordered by king model ratings Figure
0.1
0.0
0.01
0.001
NE rallin
-0.1-
of mrn uru
0.00001
3N
-0.2-
0.000001
-0.3
1e-7
1e.8
-0.4
-0.5
1e-11
Prompt (ordered by NE ratings)
Figure 4: Highly rated prompts generally have high support under the NE. Redundant prompts (gray bands) receive identical ratings but notably lower support. In sum, equilibrium ratings reflect sepa- rability of each prompt with respect to the model equilibrium strategies isolation, whereas equi- librium support of each prompt further accounts for its redundancy with respect to other prompts. (Top) We show the king-vs-rebar payoffs induced by example prompts. Green indicates king-player winning and red losing. Highly rated prompts tend to discriminate between strong models (top-left corners). (Bottom) We show the NE supports and ratings of all prompts, ordered by their NE ratings.
while equilibrium mass further takes into account redundancy of action with respect other actions. This observation is even clearer in games studied Appendix DE
Marginal rating contribution by co-player action With ratings derived from underlying equilibria, we can decompose the rating each action into a sum of marginal contributions from each co-player's actions. Recall from Equation ① that the rating of an action af is its regret, x) = Σ, x(a) (a, a-;) -u(()) i's action af into weighted Σ,,, x(aa) [u, ag. aaa.a.ai,,, a-i,-))- u,(ay, a-ş)l the marginal contribution of aj to af's equilibrium rating. Note that regret(a(,, x) = Ex, 8(d/, x)) The marginal contribution 8(a), aj, x) there- fore explains a, contribution to player i's decision not deviate.
can decompose the rating player of each of player j's contributions, with 8(a,, aj, æ))
Recall from Figure where several models tied for the first place under the CCE profile but are fully differentiated under NE. We mechanism underlying this phenomenon. Figure 5 shows the CCE king model ratings decomposed from the perspective of the rebel player. In other words, we ask which rebel models contribute most positively or negatively to each king model's CCE rating. For clarity of presentation, we focus on the top 5 models and we group rebel models into families of models if they share the same naming prefix. The contribution of each family of model is therefore the sum of the contribution by models within each family F or Σa.er 5(dk, a,, æ) with d'. king model and a,- rebel model.
leverage the marginal contribution analysis understand the
We make several remarks. First, all top-ranked king models benefit the most when compared against rebel models in their own model family: the GPT family (Achiam et al. 2023) of models con- tribute positively to the ratings of gpt-40-2024 20000022222-5-3 and 2ob---0-0bb--222---0 Similarly, gemini-11 6-flash api api- 0514, the only other model in the Gemini family (Team) al. 2023), improves gemini-1- 5-pro api 0510 4's rating the most. We speculate that this can a result of model developers selecting models to release based on favourable comparisons to their earlier smaller models. Second, all top-ranked models remain vulnerable to open- ppen-weight models such as the Mistral (Jiang et al. 2023) and Llama (Dubey et al. 2024) families of models. More fine-ge -ne-ngained analysis may shed light on the prompts which these losses tend to occur.

---

Published as a conference paper at ICLR 2025
king ratiigs
Contribution by family king ratings
-3.04 -0.02
0,00
-0045
033
-0.025
-0.015
-0.005
0.005
0,015
Family of models
gomini-155 pro-aii15p
gal-oo2024405-11 13
deede
command
gt-44omm--2224-0--00
gemini
gemma
clsude sormet-22044000
apo-4-0125
mistral
The CCE joint distribution can surface insights in the comparison data. Each bar rep- resents model family F and its width corresponds to Ea,.er((a', dy with d' king player model choice and a, a rebel model belonging to the family F. A model's family is determined by its model name prefix. For brevity, show the king model rating breakdown for the top models.
Figure 5:
We caveat that our results are in part derived from the preference ratings of a gemini-1.S-pro-ppi. 5-pro-api-051 model and may reflect the true dynamics of real-world LLM development. Nevertheless, the interpretability offered by the game-theer theoretic equilibria further dis- tinguishes game-theoretic evaluation from prior works be discussed in the Section
RELATED WORKS
There is rich body of literature studying rating methods with applications in Chess, Go, Tennis and video games. One family of probabilistic methods follows the Bradley-Terry model and predicts pairwise win probabilities from ratings. A widely used example is Elo (Elo 1978) with extensions Bayes-Elo, mīlo and Elo-MMR (Coulom 2008; Balduzzi et al. 2018 Ebtekar & Liu 202 Vadori Savani 2024) capturing temporal variation, eyclicality and ordinal ranks data. Elo ratings can typically be efficiently solved as regression problems, although their ratings are vulnerable to re- dundancy. separate line of work draws from Social Choice (or Voting) Theory (SCT, [Sen (977);; Lanctot et al. (2023), which also studies independence of clones: rankings should be invariant to redundant candidates (e.g., I.IM models) being added. However, invariance redundancy votes (e.g., prompts) is in direct opposition the spirit of social choice theory. In this sense, SCT provides partial (one-sided) clone invariance, which we argue is insufficient for open-ended, LMSYS-style evaluation. Finally, game-theoretic evaluation has been previously studied in Balduzzi et al. (2018) and Marris et al. (2022b) where full clone-invariance is guaranteed the 2-player zero-sum set- ting. Our method generalises the approaches in these works N-player general-sum settings, with practical equilibrium solving and selection algorithms based on our novel affinity entropy definition. Other approaches have been concurrently developed that avoid the equilibrium selection dilemma, and hence obviate the use of entropy (Marris et al. 2025).
CONCLUSIONS
We studied the effect of maximizing Elo ratings in the context of open-ended evaluation and showed that its sensitivity to redundancy could bias model (and prompt) selection. We then proposed an equilibrium rating framework, with practical equilibrium solving and selection algorithms that can scale to real-world LLM evaluation. We show our method provide intuitive and robust rankings of models (and prompts), with interpretable structures.
We see several exciting future directions. First, although our methods can scale to tens thousands of prompts and tens of models on commodity hardware, scaling further would be challenging. Tab- ulating the evaluation payoff tensor with pairwise preference ratings can costly too. Research into alternative solution concepts, or how could leverage their equilibrium structure for analysis (e.g. prompt and model pruning) also promising. Finally, while we target LLM evaluation in par- ticular, methodology methods could evaluate multi-modal model generation capabilities (Jiang et al. 2024) or analysing game dynamics for video game development (Pendurkar & Chow 2023
be applied more generally to other domains. For instance, rating
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Ak- shay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, et al. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303. /2528, 2023.
Robert Aumann. Subjectivity and correlation in randomized strategies. Journal of mathematical Economics, 1(1):67-96, 1974.
Robert J Aumann. Correlated equilibrium expression of bayesian rationality. Econometrica: Journal of the Econometric Society, pp. 1-18, 1987.
David Balduzzi, Karl Tuyls, Julien Perolat, and Thore Graepel. Re-evaluating evaluation. Advances in Neural Information Processing Systems, 31, 2018.
Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek. Leak, cheat, re- peat: Data contamination and evaluation malpractices in closed-source 11ms. arXiv preprint arXiv:2402.03927, 2024.
Quentin Bertrand, Wojciech Marian Czarnecki, and Gauthier Gidel. On the limitations of the elo, real-world games are transitive, not additive. In International Conference on Artificial Intelligence and Statistics, pp. 1921. PMLR. 2023.
Cheng-Han Chiang and Hung-Yt Yi Lee. Can large language models be an alternative human eval- uations? In Proceedings of the 6lst Anmual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 15607-15631, 2023.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena: An open platform for evaluating lims by human preference, 2024.
Rémi Coulom. Whole-history rating: A bayesian rating system for players of time- varying strength. In International conference on computers and games, pp. 113-124. Springer, 2008.
Constantinos Daskalakis, Aranyak Mehta, and Christos Papadimitriou. A note on approximate nash equilibria. In International Workshop Internet and Network Economics, pp. 297-306. Springer, 2006.
Steven Diamond and Stephen Boyd. CVXPY: A Python-emhedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1-5, 2016.
A. Domahidi, E. Chu, and S. Boyd. ECOS: An SOCP solver for embedded systems. In European Control Conference (ECC), pp. 3071-3076, 2013.
Konstantinos Drakakis, UCD CASL, and Barak Pearlmutter. On the calculation of the 12- 11 induced matrix norm. International Journal Algebra, 3(5:2331-240, 2009.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra. Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan

---

Published as a conference paper at ICLR 2025
Mísra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid EI-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Hontala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh. Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang. Pengwei Li, Petar Vasic, Peter Weng. Prajiwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileamu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang. Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou. Thomas Scialom, Tobias Speckbacher, Todor Mihaylov. Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao. Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg. Alex Vaughan, Alexei Buevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadch Yazdan, Beau James, Ben Maurer, Benjamin Leonbardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Bran- don Spence, Brani Stojkovic. Brian Gamido. Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai,, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Lc, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanaa, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk. Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov. Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean -aanistt Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthia Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael I.. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli,
12

---

Arpad E. Elo. The Rating of Chessplayers, Past and Present. Arco Pub., New York, 1978. ISBN 0668047216 9780668047210. URL http:/ www. amazon. com/ /ptp://///0/666712111 Published as a conference paper at ICLR 2025
Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shugiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li. Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang. Xiaojian Wu, Xiaolan Wang. Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vitooo, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The Ilama 3 herd of models, 2024.
URL
org/ abs/ 2407 21783
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators, arXiv preprint arXiv:2404.04475, 2024a.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024b.
Aram Ebtekar and Paul Liu. Elo-mmr: A rating system for massive multiplayer competitions. In Proceedings of the Web Conference 2021, pp. 1772-1784, 2021.
Rat Baatigg-oi Chess- Players
Ian Gemp, Rahul Savani. Mare Lanctot, Yoram Bachrach, Thomas Anthony, Richard Everett, An- drea Tacchetti, Tom Eccles, and János Kramár. Sample-based approximation of nash in large many-player games via gradient descent. In Proceedings of the 2/st International Conference Autonomous Agents and Multiagent Systems, pp. 507-515, 2022.
Ian Gemp, Luke Marris, and Georgios Piliouras. Approximating nash equilibria in normal-form games via stochastic optimization. In The Twelfth International Conference on Learning Repre- sentations, 2024. URL httpp:://ppereviiiiee wwt.:/wwwwweeeeeeeeeee.eeeeeeeee
Jacob K Goeree, Charles A Holt, and Thomas R Palfrey. Risk averse behavior in generalized match- ing pennies games. Games and Economic Behavior, 45(1):97-113, 2003.
Shahriar Golchin and Mihai Surdeanu. Time travel in LLMs: Tracing data contamination in large language models. In The Twelfth International Conference on Learning Representations, 2024.
URL https: /tttps://opeerevvtt
net www.nneefforrrn forum? i 2wg6633vv
John Harsanyi and Reinhard Selten. A general theory of equilibrium selection games. MIT Press Books, 1, 1988.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021. URL httpps::/ppeneeeiew..tt netmmfrumiid
d7KB jmI 3mm.
Jean-Jacques Herings and Ronald Peeters. Homotopy methods to compute equilibria in game theory. Economic Theory, 42(1):119-155 2010.
13

---

Published as a conference paper at ICLR 2025
Cheng-Ping Hsich, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's s the real context size of your long-contens context language models? arXiv preprint ar arXhv:2404.06654. 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Dongfu Jiang, Max Ku, Tianle Li, Yuansheng Ni, Shizhuo Sun, Rongqi Fan, and Wenhu Chen. Genai arena: An open evaluation platform for generative models, 2024. URL.httss://arxiv.co
abs/240c 04485
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6:41,,,9900 2014.
Marc Lanctot, Kate Larson, Yoram Bachrach, Luke Marris, Zun Li, Avishkar Bhoopchand, Thomas Anthony, Brian Tanner, and Anna Koop. Evaluating agents using social choice theory. arXiv preprint arXiv:2312.03121, 2023.
Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta. Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. Holistic evaluation of text- to-image models. Advances Neural Information Processing Systems, 36, 2024.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gon- zalez, and Ion Stoica. From crowdsourced data high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024a.
Tianle Li, Wei-Lin Chiang. Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024b. URL
Stttps//Immyss ottp:///////yooooooo//////200--2
Siqi Liu, Luke Marris, Georgios Piliouras, Ian Gemp, and Nicolas Heess. Nigtransformer: Equiv- ariant representation learning for normal-form games. In The Twelfth International Confer- ence on Learning Representations, 2024. URL 4YESQqIys7]
net net/forum?id
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu, G-eval: Nlg evaluation using gpt-4 with better human alignment. Pruceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511 1-5122522, 2023.
Luke Marris, Ian Gemp, Thomas Anthony, Andrea Tacchetti, Siqi Liu, and Karl Tuyls. Turbocharg- ing solution concepts: Solving NEs, CEs and CCEs with neural equilibrium solvers. CoRR, abs/2210.09257,, 2022a.
doi: 1048550/ARXX::210009533 URL /attps://axxiv.ort
abs/2210..0925
Luke Marris, Mare Lanctot, Ian Gemp, Shayegan Omidshafiei, Stephen McAleer, Jerome Connor, Karl Tuyls, and Thore Graepel. Game theoretic rating in n-player general-sum games with equi- libria, 2022b. URL
2210 02205
Luke Marris, Siqi Liu, Ian Gemp, Georgios Piliouras, and Mare Lanctot. Deviation ratings: A gen- eral, clone-invariant rating method. 2025.
645
Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for normal form games. Games and Economic Behavior, 10(1):6-38, 1995.
Andrew McLennan. The expected number of nash equilibria of normal form game. Econometrica, 73(1):141-174, 2005.
Andrew McLennan and In-Uck Park. Generic 4x 4 two person games have most 15 nash equi- libria. Games and Economic Behavior, 26(1):111-33,, 1999.
John F Nash al. Non-cooperative games. Annals of Mathematics, 1950.
14

---

Published as a conference paper at ICLR 2025
Medha Palavalli, Amanda Bertsch, and Matthew R. Gormley. A taxonomy for data contamination in large language models, 2024. URL httpp:///aaxxx
2407. 08.07.08720 08716
Arjun Panickssery, Samuel R Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. arXiv preprint arXiv:2404. 13076, 2024.
Sumedh Pendurkar and Chris Chow. Bilevel entropy based mechanism design for balancing meta in video games. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems. Proceedings of the 2023 International Conference on Autonomous Agents and ...., 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark, 2023.
Yosef Rinott and Marco Scarsini. On the number of pure strategy nash equilibria in random games. Games and Economic Behavior, 33(2):774-293, 2000.
Amartya Sen. Social choice theory: A re-examination. Econometrica: journal of the Econometric Sociely, pp. 53-89, 1977.
Bernd Sturmfels. Solving systems of polynomial equations. Number 97. American Mathematical Soc., 2002.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. battps://gize /ithub. cow/tatsu- saab/stanfort stafordd alpaca, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, rean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11880 2023.
Constantino Tsallis. Possible generalization of boltzmann- statisties. Journal of statistical physics, 52:479-499 487, 1988.
Theodore Turocy. A dynamic homotopy interpretation of the logistic quantal response equilibrium correspondence. Games and Economic Behavior, 51(2):243-25 263, 2005.
Nelson Vadori and Rahul Savani. Ordinal potential-based player rating. In International Conference Artificial Intelligence and Statistics, pp. 118-126. PMLR, 2024.
Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus. Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating I1m generations with panel of diverse models. arXiv preprint arXiv:2404. 18796, 2024.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination- free I1m benchmark. arXiv preprint arXiv:2406. 19314, 2024.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang. Zhiyuan Liu, and Maosong Sun. coBench: Extending long context evaluation beyond 100K tokens. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15262 15277, Bangkok, Thailand, August 2024. Association for Computational Lin- guistics. URL htllhhtps://eee anthology. orr.orgg///00 2024. acl aal.2on.cllloognoggo..l long. 814
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and lon Sto- ica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Informa- tion Processing Systems, volume 36. pp. 06695-46622. Curran Associates, Inc., 2023. URL https://proceedinsss..uuu..uuuau www.:iipp...ppppp.eeeeeeeppppppe ftt.e//pppe:/222///1.. DII oag. Dattt-aaastts and Benchmarks.
pdf
15

---

Published as a conference paper at ICLR 2025
A COMPUTING THE MAXIMUM RELATIVE ENTROPY CCE
A maximum relative entropy CCE, that minimises the distance of the log-joint, log(z(a)), to target log-joint, t(a) e R can be computed using gradient descent. We formulate the problem in dual space (Maris et al. 2022a) with dual parameters, at aa(a IR |R| Vi, defined functions, ay(a)) softplus( 0; (a/)) Vi. of learned parameters, {(a,) RAAi Vi. Let lo(a) be logit term used to construct the loss function.
la - ΣΣ apla)) [ui(a,, u((a)] t(a)
(8)
Minimizing loss function, ming Lo. converges optimal dual variables, (a softplus( 8t (a.)) Vi with Lg = log IΣ. exp [l([a]]]- The loss is convex, deterministic, and un- constrained. Therefore many optimization algorithms are suitable. The primal joint can be simply recovered from the optimal logit term s0(a) softmax [Lge (a)].
B AFFINITY ENTROPY
Consider defining a modified Tsallis entropy Hg with temperature parameter p E (0, 1] as:
_ - Tz p [
(g(x)
where = = (U) _ Note that this definition recovers the standard definition of Tsallis entropy when (T6) is the identity matrix.
Remark. U(P) > for all entries for Hy to be real-valued.
U(()) must be non non-negative for every i.j. otherwise, there exists x ee where e, is standard- -baris vector such that U/!) x < O and (Ul²) x is not real for p (0,
Remark. The (p + 1)-norm of each column of U(t) must be less than or equal to 1 for Hg to be non-negative for any x A.
We need TTz 1 for p e (0, and any x A. Equivalently, we require (x & for E (0, 1].
Note (xTz) 2) p+ = (U x)) = (10(0) 2l1t1. Therefore, we require
1 > sup |1000 x11pp1
(10)
sup ==
||060)/01--11 for U(r 0
(11)
||0|(0) 11,11
(12)
max
|10 I1-+1 by Drakakis al. 2009
(13)
Remark, Among all admissible UM),, defining U such that its columns have exactly unit (p (p++) 1)-
norm achieves mingre min (x)).
This follows from the previous remark and is desireable for the sake of defining a "tight" definition
of entropy. Intuitively, by the conditions set thus far, (p) = 0 is admissible. Yet, this gives a loose definition of entropy where Hy 1/p. It turns out that this intuition is required in the limit p 0.
Remark. (()) must be precisely column stochastic for HB to remain finite in the limit of p O.
In the limit p - 0, the denominator of HP goes to zero, therefore, by L L'Hôpital's rule, the numerator must well. The numerator goes = Σ, U x 1 (())).. Therefore,
AreA dd11 1 1 T((O) 0.
(14)
16

---

Published as a conference paper at ICLR 2025
Finite distributions only obey single equality constraint, that is x 1 1, therefore it must be the case that (P(P) i.e., U(P) is column stochastic.
i...,
Remark, Hi concave in x.
Let y; Uli x. Then each element of the sum, v/--1 linear transformation on æ. Therefore, (U,), x)²-1 1 is convex in æ. Hence H is concave in æ.
convex function in yi, which itself is
Remark. The gradients V Ha are well defl-defined.
Recall (9). then:
HHE p+1 E x)PU θzz
(15)
+ (W))))
V V
(16)
p
which is well-defined for any choice U ij > 0 for all i,j.
Remark. H well-defined in the limit as p 0, ie., Shannon affinity entropy is well-defined.
It is known that Shannon entropy be recovered from Tsallis entropy in the limit p 0. We repeat that derivation here and use L L'Hôpital's 's rule. The derivative of the denominator is 1, hence find the limit is given by the finite derivative of the numerator:
dpHHl dp
d
Σ
yi
p-1
(17)
dp
d
Σ e (0+1) log(ue)
(18)
dp
- (log(u)) (o 1)
1 dyi
c(o+1))oR(())
(19)
Y dp
In the limit - O, the derivative evaluates to
d[pHE]
Σ elo+1) log(s)) log(ys)
- -(p (p 1) Σ
1 dyi
c(++ log(ve)
(20)
dp
p-0
ys dp
p-
Σ ye log(y:) -
Σ
dy
(21)
dp
p=0
-(())
Σ dp p=0
dy
(22)
Remark. Let K be similarity matrix between actions with non-negative entries with positive column- sums. Then U(r) (ddaag1/(1T (1tt/11111)) ()++1))))))) satisfies the conditions stated above for ((o)
Remark. Under the above choice of Uf)), Shannon affinity entropy S. = - can be derived as:
= = ((U x)) Σ log Σ Ki) - Σ U log( Ki)
(23)
The necessary Y: term can be rewritten and its derivative (evaluated at = 0) can derived as follows:
17

---

Published as a conference paper at ICLR 2025
y/ U.00 x
Kij
Dj
(24)
[2 v K {j )=1l
Kija
Kptly )= v+l
(25)
Σ Kijazee tti log(En kg!1)
(26)
> Kijuze otr loR Y (i (01
(27)
dy: dp
= Σ Kijaze भी - log (2ip (2ig (2tteltt)))))
(0+ 1)2 log
Σ
e c (ou()))))
(28)
p 1 Eg elot1)log(KK Σ
log( (ez)ellot) (110(((5)))))
(9))
Σ Kijay
K pp+1
1
log( Σ K2
(30)
(1+ + 1)2
log( (gg(Kzj) K
(31)
p Σp Ktt
= -> (p + 1)2
log kett p+1
E(ul) Log(k)ss)) uu
(32)
dy: dp p-0
[[ log Σ Krz)
Ug log( Kv;) J
(33)
where we define K, log(K;)) 0 Kij 0 (which implies (U log(K,)) if K., = O. Plugging this back into the second term in the formula for Shannon affinity entropy, we find
Σ
dy: dp p-0
=
ΣΣ log Kej) Σ
UU9 log( log(K(;) 11000 xj
(34)
Σ log' Kv;) - Σ
U log( Kv;)) 133 _0!0
(35)
[[
[log() log Σ Kry) Σ ΣV log Kvy)
(36)
completing the claim.
Remark. In the case of duplicate strategies (clones), the maximizers of H form precisely the set of distributions which arbitrarity distribute mass of to across each of the C sets of clones.
Consider the case of exact clones, i.e., K is block diagonal (w.l.o.g.) with blocks of ones. Let there be clone groups each of size ne for c E {1...., C). Let c(1) map an action i to its clone set. In this
case, it can be shown that U:() Te(i)) p+1 if c(i) = e(j), otherwise US = O. Note that the gradient of entropy w.r.t. æ must be proportional to the ones vector for to be maximizer in the interior of the simplex. Let x [ 1
C xc] with each x, E R w.l.o.g. We will show that the set of maximizers of HP is necessarily the set of x where each x, E - For x to be a maximizer, the gradient must be equal to the ones vector multiplied by a scalar dd I::
18

---

Published as a conference paper at ICLR 2025
HHg()) dzz
-
(()))))))))
(37)
P
p+1 1
(P) Ik.
PU
(38)
P
k
p+1
c()
PTI 1 xe())) )U
(39)
p+l
ptf 1 xc(i)) ) デे.
nd))
(40)
P
p+1
1 x())))
net))
m
dii
pt)
C
(41)
p p+1 1
C
[a())))== = =dd.
(42)
We also require x E A, which implies
Ij P = Øn(j) 7
(43)
1
Σ 23 - 1
_ 11 TC
(44)
c dp 1/8 d.r0 P
1/0 )
(-p(P+)
1p+1 1
p+1
d= C-P
(45)
p+1
p
Finally, we know from 42
dp
( T )o))) p+1 = C
(46)
1'x0)=1 =1
(47)
proving the claim.
Remark. In the case of duplicate strategies (clones), the maximizers of H. achieve an entropy value which is equal to the Tsallis entropy of the system with clones removed.
If we evaluate the max entropy distribution we find
H (x:
1 - Σ
(PP)
(48)
p
1
Σ
11 c(1)
1 (Teess)) )01l
(49)
p
1
1
no
r.r 1 xe) )tt
(50)
p
 p
n
(51)
p
1 1
Σ mome
1 ( |tt]]
(52)
p
1 Σ 2
)
(53)
p
which is precisely the Tsallis entropy of the uniform distribution over C distinct clones.
C INTEGRALS OVER SIMPLEX
is possible to derive a closed-form result for the dim-siiily kernel in 6 by appealing to known results of integrals of polynomial functions over the simplex.
19

---

Published as a conference paper at ICLR 2025
24 2 0, Σ 2, & 1] be the standard simplex in R Let vs > 0 for all Let T xa) then
JT " 2 (1 x1 x2) _0 ((4))
(54)
r( v))
Proposition C.1. From player i' 's perspective, the expected dis-similarity between two actions p and q under uniform distribution over all opponent joint strategy profiles x- is equal to
D Pq
(d, + (): + 2) [110 1/
1
(1 -1)))
(55)
where U is [A, x |Asi matrix where each entry Uaia_ is the expected utility for player i playing action a, against the background joint action a_i. U!!) indicates an entire row of the matrix. The integer d; = -11 (Ag).
Proof. Recall (54 and (n) (n 1)! for n N. Let -n. Upo be the rating for the pth action under opponent strategy profile --..
Then we want to compute E, DDDraiir - 3.. Recall the volume of the simplex is a Then
Ex Dir())t - rrg = Sra(rp - de Sra dæ_i
(56)
d! Td
(rp - -ra)²² 2
(57)
d!
U(i) pm De
Σ
qu0 zw) dd
(58)
d!
U Upy FwTty) +
(59)
Ja
myy
ur
Uit )rottt)) dr-
(60)
au
dl
( pru py
(61)
M
( wwm clse TT-
+ (U(0)
Fn 2 2U qu
J In
(62)
(w -0(122 - (() gmi)
(63)
(d + 2)
Σ
((i)(())) ((
py
+
qw ay
-2 u() ]
(64)
1
Σ
E pu
(z(i)) +
T76i) Pro
(
(65)
(d + 1)(d + 2)
1
[100%] - 11 + (U -((1))
(66)
(d+ l)(d +
Proposition C.2. From player i's perspective, the expected dis-similarity between two actions p and q under a uniform distribution over all factorize-able opponent strategy profiles x = 11 2z is equal to
(
11 j+i (d, + 1)(d, + 2)
(67)
Σ Σ
-u(p.a-=)-weeeeeaa - (u) (al0)--lggg, ull.a' ol)
(68)
.EA
20

---

Published as a conference paper at ICLR 2025
where the integer d, = [Ail and "#a=a'" Exzt 1aa d, indicates the number of action matches between two opponent profiles.
Proof. Let Ya led ui(p,
a-1) ( *Jj be the rating for the pth action under an opponent jzz Zy. Let dx be shorthand for dr_s. Likewise, let Sp be shorthand for Sp ... Spen-
profile r-; = i1i
ST ... Spmc-
Then want compute SennDDr(ivvssll -r Recall the volume of a simplex is 4. Then
Be,~ ((r=))(()))) - []
(69)
Spe (rr dr
(70)
Sp dt
(I[dj) dj))
(r'-r')'dz-
(71)
jti
(I[d)))
Σ
ui(p, am-)
11
Tjja; Σ
a((ga-)) I1
22,0) dx
(72)
jFi 1 a_
a_,Ed_,
3F1
(IId)))
Σ
II Bj (ui(p, uilq dr_
(73)
J+1 -
a-1EA-1
(IId))) 1
(74)
Σ Σ
Q ITaa, [ - ula. a-)) (uilp, u(q.d==i)) dz_
EEA a²_ E.A aL
jtz 341
(75)
= (IT f T (
(76)
iti
Σ Σ
(ui(p, (w(.))))-((. us(g ax(g,
(77)
EAA D'A
(II&)))
(78)
iti
Σ Σ
(ui(p,a.) (u: ul(g
D. Ji d
Tn
G.A dL
123
a '
else va t2)
(79)
= ([]d))) (I[(d +2)1
(80)
jti
341
Σ Σ
(ugaa-.)-ulgaa:))0ddddd a'i))a'i)) (2#t=dd)
(81)
EAA EA=
II
(82)
Jti (d, 1)(d, 2)
Σ Σ
(ui(,aa.) ullg (wl.:))))(la..i))uu(udd.. (ui(p,d's) ulla (2ftadd))
(83)
a d' EAA.
21

---

Published as a conference paper at ICLR 2025
P1 ratings
Contribution by action to P1 ratings
-0.25
0.05
0.25
Action
P-
Paper
Rock-1
Rock
R2
Scissors
Figure 6: We visualise the marginal NE rating contributions each player 2 action to each player action. We show that a) all actions receive zero ratings and b) the rating of each action is interpretable and corresponds to our intuition.
D WARMUP:
RANKING OF rock pack-pape--scisoors
We provide demonstration of thee-the ranking on the classic 2-player, 3-action zero-sum Rock Bock-Paper-Soe Paper-Scissors game. Balduzzi et al. (2018) proposed rating actions under the max-entropy Nash equilibrium of the game. In that case, each action receives rating of zero. If we duplicate the Rock action, for example, the ratings remain zero under the max-entropy NE. Our proposed LLE based approach returns the same ratings.
Rock1
Rock2
Paper
Scissors
Rock
Paper
Scissors
Rock1
0.0
0,0
1, +1
+1, -1
Rock Paper
0.0
1, +1
+1.-
Rock2
0.0
0,0
- 1, +1
+1,
+ +1,-1
0,0
1.
Paper
+1, -1
+ +1,
0,0
1,
Scissors
+1
+1,-1
0.0
Scissors
-1, 1, +1
- +1
+1,- -1
0,
Figure Rock-Paper-Scissor (RPS) Game and RPS Game with Duplicate Rock Action.
In Figure 6 we show that the equilibrium underlying the scalar ratings reflects incentive structure of the game player does not wish to deviate the Paper action precisely because doing so would lead to losses against the Scissors action despite wins against the two Rock actions.
E VULNERABILITY OF STANDARD SHANNON ENTROPY
Prior work has shown max-entropy Nash equilibrium (cquivalently max-entropy (C)CE) to be in- variant clones in 2-player zero-sum games (Balduzzi et al., 2018). We include simple experi- ment here to illustrate why max max-entropy Nash equilibrium becomes vulnerable to redundancy the N-player general-sum setting.
Chicken Game Consider the 2-player 2-action general-sum Chicken game. Let players receive 0 if they both swerve. If one player swerves while the other goes straight, the one who swerves receives -1 and the other +1. If both go straight, then they both receive -12. This game has three NEs. Two are pure in which one player goes straight and the other swerves. The third is symmetric and the max-entropy NE of this game; each player swerves with probability 11/12. Both straight and swerve have an expected payoff of 11/12 under this NE. If we duplicate the straight action, the original max-entropy NE becomes the min- min-entropy NE! The other two NEs representing each player swerving while the other goes straight now have higher entropy. The player that swerves rates their swerve and straight actions -1 and 12 respectively. The player that goes straight rates their swerve and straight actions as 0 and 1 respectively, demonstrating that the max- max-entropy NE solution concept is not invariant clones in the general-sum setting.
The story in the max max-entropp entropy CCE setting is more nuanced. We find that although the CCE ratings change under the addition of clones, the ratio of the ratings of the two actions remains stable. Further investigation is necessary to understand whether max-entropy CCE ratings are equivariant (robust up affine transformations of the ratings) to cloned actions.
By contrast, we show Figure 9 that all actions would receive zero ratings under our proposed equilibrium ratings. In other words, equilibrium selection procedure continues select the
22

---

Published as a conference paper at ICLR 2025
Swerve
Straight
Straight
Swerve 0.0
Straight
Swerve Straight
0.0
-1, +1
-1.
Swerve Straight
-1,+1
+1
- -12 - 12
-12, - 12
+1,-1
-12, 1
Straight
+1 +1,
- 12, 12
-12, - 12
Figure 8: Chicken Game and Chicken Game with Duplicate Straight Actions.
P1 ratings
Contribution by action to P1 ratings
-0.5
0.0
06
-10
-0.0
-06
-0.4
-0.2
00
1.0
Action
Staihtt
Sraght-11
Sraihht2
Staigh-22
Figure 9: We visualise the marginal NE rating contributions of each player 2 action to each player action. show that a) all actions receive zero ratings and b) the rating of each action interpretable and corresponds to our intuition.
mixed-strategy NE in the original game, unaffected by the additional redundant "straight" action. Further, the widths of the bars are interpretable: suggesting that deviating to the Swerve action is safe option without major risk or reward. Deviating one of the Straight actions however, can lead to high rewards but also catastrophic losses.
F EXPERIMENTS
F.1 SIMULATED MODEL AND PROMPT IMPROVEMENT PATH
Algorithm describes our simulated model and prompt improvement procedure. each iteration, we add new prompt and model following evolutionary procedure. We require all prompts to be probability distributions over skill dimensions. We model for transitive dimension for models by representing each model vector the set of models Am if and only it becomes top-ranked according to the rating function r. A new prompt is added long it is the best-of- P' sampled prompts and does not have to be top-ranked.
sum of probability vectors skills. A new model is added to
F.2 EQUILIBRIUM
S.PER-PARAMETERS
We use the same set of hyper-parameters for all our experiments. For affinity-entropy H²))),, we use and set kernel variance to 1e-6. To solve for max affinity-enroppy distribution we use gradient descent. The max affinity-entropy distribution is then used in NE and CCE solving.
which annealed exponentially with decay rate of 0.95 every 250 gradient updates if and only if the exploitability the annealed game (3))) (Equation ( is at most le-5. We set the terminal temperature to
For NE solving using LLE approximation, we initialize temperature т
1e-2. 2. We early terminate the equilibrium solving if we have found e-NE with 1e-3. For CCE solving, the optimization problem is convex and we minimize Equation g directly. For gradient descent, use Adam optimizer Kingma (2014) with fixed learning rate 1e-2 2 for all steps (maximizing affinity-entropy and equilibrium solving).
F.3 THE
EVALUATION DATA
We evaluate our method on the arena-hard-vo.1 dataset (Li et al., 2024b) with 500 prompts and 17 competing models. The set of prompts as well as model responses are downloaded from LMSYS data repository (https://huggingffcce. http:faaacccc/ppppeeeeeeeee harsss/aaeeaaaaad--aa browserr,, with the exception of gemini mommini-1.35 p1w.----pppppp--0111111---- pero--ppi-00 As we need to tabulate the payoff tensor for all model pairs, sampled preference ratings using gemini-1 mariiii..-pr---- --pp--api-052 for each model pair, with samples for each permutation to account for potential position bias of the LLM rater. Pairwise model utility is averaged over all ratings samples.
and
23

---

Published as a conference paper at ICLR 2025
Algorithm Evolutionary model and prompt selection procedure
Let K be the number orthogonal skill dimensions.
2: Let r : Ap x Am Tp. I'm be rating function assigning scalar rating to each action.
3: Let Po. Mg be the number of initial prompts and models.
4: Let P', .M' be the number of sampled candidate prompts and models at each iteration.
5:
6: A Dirichlet( 11:K, Po)
Po sampled initial prompts.
7: A Dirichlet( 11:K, Mo)
My sampled initial models.
8:
9:
for [l....Ja do if additional prompts then
10:
t- If adding new prompts.
Ar Dirichlet(11.k: P!)
Sampling P' candidate prompts.
Tp rAp u Ap, Am)
A Ap u (A max 1 P']]]
Add best-of-P" prompt.
14:
end if
A'm
16:
while true do
Am - Dirichlet((1:55 I: M')
Sampling M' model improvement vectors. t- Evaluate improved candidate models.
18:
Tm r(p,, (A + Am3 Am)
19:
A'm - A' mm larg max Tm M']l
[arg
20:
if arg Tml M'] arg rrgmax,,, m. then
21:
Am = (.4mm Jarg max mm]) U Am
t Add a new top-ranked model.
22:
break
23: end
24:
end while
25:
end for
Table 1: Prompt and king actions that each define 16 pure-strategy Nash equilibria any rebel action except the model played by the king player is pure-strategy NE.
Prompt
King
"Can you implement python tool that is intended to ru..." "Hi. have this URL which can paste in my Microsoft
1.-pni-155prr-app-055
semini-1.5-pro-api--85
"Please provide simple RESPONSE to the following PROM.." "Take on the rol cof an Gherkin expert. Can you improve..."
Blaube-3-5-oomet-22240000
"Write small python function that get all the links o..."
1omini-1,5-fuah--pi--011
F,4 RISK-DOMINANT EQUILIBRIA
Our king-of-the-hil evaluation game admits multitude of Nash equilibria, among them 80 are pure-strategy NEs (see Table I. Additionally, we computed 128 mixed-strategy NEs with exploitability most 1e-2 that each derives distinct of ratings. In particular, of the 128 mixed-strategy NEs is pre-computed by our NE solving and selection procedure by tracing the QRE continuum, which we refer
the 0-th equilibrium, x
A longstanding challenge in game theory is that of equilibrium selection. Suppose that every player knows that there are many equilibria in the game, each player must confront the following ques- tion during play: out of all equilibria, which equilibrium strategy should I play and relatedly, which equilibrium would each of my co-players play? This critical, as miscoordinating could lead to arbitrarily bad outcome, despite each player playing one of its equilibrium strategies. For instance, everyone driving on the right or left hand side of the road are two valid equilibria, but miscoordinat- ing would be devastating.
is for this reason that the notion of risk-dominance of Harsanyi & Selten (1988) critically important: the Nobel-prize winning theorem suggests that players would each iterate on their prior beliefs over which equilibria its co-players would play and choose the one that the least risky when players miscoordinate under such priors. Here, show empirically that our solution concept leads to risk- dos-idomnant equilibria as suggested by Herings & Peeters (2010). To do so, we simultancously minimize the exploitability of several profiles in parallel with regularizer that maximizes the L
24

---

Published as a conference paper at ICLR 2025
rating differences between any two profiles by gradient descent as in Liu et al. (2024). This yields an additional 127 NEs with exploitability at most 1e-2 that we analyze in Figure 10
Figure 10 (Tpp) shows the 128 mixed-strategy NEs with distinct model ratings. Figure 10 (Center) shows the expected payoffs to player i when plays its p-th equilibrium strategy x when other players uniformly choose one of theirs, or E [u,["", x²,)] with Tu uniform distribution over 128 equilibria. In yellow, NEs are indeed risky, as their stability relies heavily on all players coordinating on the same equilib- rium. Figure [0] (Bottom) takes things step further and follows the intuition of risk dominance more closely. Starting from a uniform prior helief over player i's choice of equilibria, T. Tu. each player iterate their believes over other players' choices of equilibrium based on the expected payoff of them playing each equilibrium.
show the of per-player expected payoffs. We confirm that many
Specifically, we let
++1 softmax log TH 2
nE vjvi {(})~
uui(
1(i-1) ((++)) (u. -i. 2441 .....)]
(84)
with 7 = 1c-2 2 the step-size and we compute the expected payoff' to player when playing its k-th equilibrium T 00,
E Vifi uz( us(...,
c(--1) h e(++)) Fy J 2441 2.--))
(85)
<(j)~m)
Ordered by the sum of expected payoff' for all players, observe that the Nash equilibrium procedure selects (equilibrium x²) is the least risky among 128 mixed-strategy NEs of the game, without any player being particularly worse off than others even when players miscoordinate.
pays
F,5 INVARIANT EVALUATION
We show in Figure I1 the effect of introducing near redundant adversarial prompts on the equi- librium ratings. While our invariant property is limited exact clones, our results show that our approach results in rankings that degrade gracefully in this approximate case, even with 1,000 ad- versarial prompts. The Elo rating system suffers from such bias in data similarly as the exact case
Figure 3
In Figure 12 we provide detailed breakdown of our NE and CCE ratings results (without redundant
adversarial prompts). We show the actions of each player ranked by their equilibrium ratings and by their support under the equilibrium marginal distribution.
25

---

Published as a conference paper at ICLR 2025
Figure 10: From top to bottom: a) we show the distinct king player action ratings derived from 128 mixed mnixed-stratege strattrtegy NEs of the king - of otethe---eeeee hll.h game. All NĖS have exploitability at most & le 2; b) we show the expected payoff' to each player under uniform priors over their 128 equilibria; yellow circles show the sum of expected per-player payoff's c) we show the same analysis as in b) but the expectation is taken under optimized equilibrium priors. Equilibrium 0 (rightmost) is the LLE our NE vilning procedure select.
Rating
plaype
1000
player Hkg
20
201
15
20
10
21
214
11
Expected player under unfoom coordiinntoonn pafors 128 Neeh livrie
Nash equilibra
King model ratingg umber 120 exammple mix wwweeed--tttiiggppooo eeatligiee NE (exploitabiilyy <c te-3)
00
20
2011
aquitbuu
coullbbuum
payott
50
26
50
10
101
60
211
aadon 8o000
26

---

Published as a conference paper at ICLR 2025
NE Ratings
CCE Ratings
Elo Ratings
gemini-1 1-110p-1.5-
1-
gpt+4/uubbo22244044999
2-
dludde3--cconne2222440000
gp1-40-2024-05-13
gemini-1.5-flasie
15-mhi-1.5-llashh-p--05444
X
gpt-4-0125-preview
gemma-2-276-t-06225
gemme-2-86-1t-0225
X
claude-3-ous-222242229
claude33sonnee-224402999
llama-3-70b-chathtf
command-r-plus
Swenn.5-77B--hat
gpl-4-0314
14
mistral--arge-2402
llama--8b-chat-hf
gpt-3.5-turbo-0155
250
500
730
1000
250
900
730
1000
250
300
722
1000
Figure 11: We introduce increasing number of redundant copies of prompts adversarial to gemini-1... 211.5-pro-ppi-05- 4 with noise sampled from Uniform(-0.01, 0.01) applied their payoffs. Equilibrium ratings with clone invariant selection procedure degrades gracefully noisy redundancy while the Elo ratings become incrementally skewed. Models at the same rank (with an absolute rating difference most 1c-4) grouped in grey and ordered alphabetically. We caveat that the specific rankings reported are subject to the LLM preference model used which in this case may exhibit self- eefferreeerenc to the Gemini family of models.
27

---

Published as a conference paper at ICLR 2025
Max affinity-entropy NE ratings and marginale
sperin 1.3man-1.3-pp--aaee 14-10---2200 12pt
tani-io 28-12
me aohee the tollowing on, "iheee
Hi have this UFI, which can paste
ackaal
9 502408207 ppt 40
ppc 4
5-acmnet 20040620
foou are The coordinator notwol's
Igonma 0025
(pernn- E-taatheiittt
1pal-4-0
make ins tipp hoow usingg au key foor...
tap--- 1o0ppreeee
Viite Kollin JNl oode that and ieve...
Thl ratre-teme champpanenig
sphn
SCIM
choude 100-00000 10pog
'gem -00005
'hha ase the stops, onder ao beco...
Ideude www.eee-eeeeeeeooooooooo
(Dypiias how to Implemment model parn
command plus"
ton 8 nurbro 0125
tradtaa 3410
SSc,, achurna wfeer....
Dwent 7-728 Chat
"Rama 3 7-000--- hox-1"
ane an Shelleed programms.
"ppt--00314"
1wen1 1-798-Chha
How con use ladaa no fiold zoth..
RRod peoffs
Blana 9 80 chat H 'pp 8.5 turbo 0129
Inibbul large p400
Yurr 3 char- 10"
4 8516
"Comadder the fawvrne the ingredient...
bominii 1.5 opi 0514
00202 0.008 ratiigss
-0.4 ratings
ratings
which paste
genan-1 Seproeretts
"pn-
teep me sohvo the lollowiing on, Ploce...
oo
"oo ses the convriintie of networc
Epp 20--006-05--
'gemmn
Ipeminn-
(gal-d-0195-
make ms fuszer using aulley fuzz...
lap 0685
code simulall
Wiite Kollin code that ievn.
IA rable tenn's champlonetee 58hn
actiona
1go 0829
'oloude 00240229
20040029
0665
"What are the siace, onder, to beco....
Shaudde--onn aal---ommee 202440220
Idaudde 2 sormat 200402201
"Expiian how implement model para
What dows the aharach cornes.
Naara 756 chal-if
Yihen 5-73B-Ther
coomard glus
last 3 turbo 0122
triutral argo 2412"
You and an expert Byollikk programm.
1pt---mm1
Owent 1--255-hhal
Con
Howe can Lse cockance beibe for zath...
Censider the flavara figrodiant...
gpe (.p-3.--tutooo0100
Nima---tb-chhe
bemiini- Eppro-pp--551
1an-6-0314
0.00
0:08
0.04
marginals
marginate
marginals
Max affinity-entropy CCE ratings and marginals
Vitto code simulate ballisio
175-to-2005-3-117
bwitnn- Sppro-pp--05-
am Plyton progra rigramm net Nou ln.... ideen this upre sythoneemn mocet implomant sython
2pt-4-tu
20940600
tyn--lo 1gal-1-01 preeeeee
Velle Python function corvert coc...
What in the moot suaoosalld go ma
(Redd pastiiy book
we riga chatbel
"garnnn- 4-
cloude 8 oous 20240029
'pon 27t 0665
'olludo 8 opus 20840220
Igomina 2 276 0625
for oi "yne" i yyee mo
1/
gemmar 2 00-1-0000
'genmna: 2 otes
herrove deeat cadte from tollowing have clack teotbench
sowured--orreee 20740220
Idludde O aconst--to002200
white
Taam 3 720 chat
"Mama 5 70p chall
with sends
Owent 728 Chat
bommrand glus
bente of re chen is cartee
ppt 0314
'Owen1. Chat
loommardeoiiua
12n-6-0314
Thl mo how to implement BCMM cor... give
Frovide code solation this... Tlesse rode (poth
lama 3 Aib-hhal-ii
triutral- arge-24022
Yurr 3
1pt 3.5 tutoo
Igon 9 turo 0129
ratings
ratings
ratings
Bello instruction
"gennn-- E-po--si-0t1
Thok world you eccape thes shhieh
Pleaas provade sampra VESSPCNN rable wiil byttonn oodo wob corage rmps
ustuacate thes fursion muceine ennkuntt....
ludddo 002406207
idlude 6 sornet S0040500
provkow
tgai--i 122--preenn
garmens 1 s-taath-p---d1-
Igaman- 1.5 E--lish--ackie 14"
Viluue cop.-ops-----a--o ac2402299
have function funce steesh
nodieps in thene way to impamennt
creates
Viuude 8 sonnet 202402209
claude cous 202402207
beeme Siwi--0000
aallons
idaude & sornec 20043220
serrin 1.5 pro opi 2514
"penms- 248---------000
Virte an tade
tog....
bemmm--b277---00e 5-2025
Owent Lo-2tt-hhal
will a program to play comn oft
Qhent 5-738- Cher
Igemmre 2om.me-22-----00
gon-4-054
"As part of eerreling stree and intor
gge 40034
Interaa arge-2400
would "ind the Plyton program "ribed Teats...
"Whorts
a compleet Python posgmim
command glus"
bommiand plus
The Ackusian larguage only had thre...
Ipe 015
boo 8
0.10 marginals
0.04
0.08
marginals
marginals
Figure 12: We show actions of each player ranked by their rating and equilibrium support under NE (Top) and CCE (Bottom) profiles respectively.
28


Reviews:
[{"review_id": "BOXWhPcIZo", "reviewer": "Reviewer_v6QS", "summary": "This paper proposes a methodology for rating-based evaluation of Large Language Models (LLMs). The motivation arises from the shortcomings of the current Elo rating system, which may suffer from redundant or invariant prompts, leading to biased or less robust evaluation results. The proposed solution extends the evaluation method from a common 2-player game scenario to a 3-player game, incorporating a measure of separability based on game outcome distributions to favor prompts that better differentiate between models. The quality of the proposed method is assessed using Shannon entropy over model or prompt strategies, and the strategy derivation is based on Coarse Correlated Equilibrium (CCE), which is more accessible compared to the commonly used Nash Equilibrium (NE). Experimental results show that the proposed evaluation method effectively resists the impact of redundant prompts while maintaining similar performance in standard scenarios.", "questions": "* How can we identify redundant prompts or models? Would it be necessary to include human verification of redundancy?\n\n* Scalar rating evaluations still suffer from intransitivity issues, as seen in Rock-Paper-Scissors scenarios. Does LLM evaluation face this problem? If so, does your method have the ability to handle it?\n\n* I found some recent game balance research related to this problem. Is LLM evaluation similar to game balance analysis? For example:\n    * Bilevel Entropy-based Mechanism Design for Balancing Meta in Video Games. Adaptive Agents and Multi-Agent Systems (AAMAS), 2023.\n        * Discusses game design aimed at maximizing strategy entropy to achieve balance.\n    * Identifying and Clustering Counter Relationships of Team Compositions in PvP Games for Efficient Balance Analysis. Transactions on Machine Learning Research (TMLR), 2024.\n        * Discusses redundant settings and the problem of using strategy entropy, proposing conditional analysis with evaluation ratings.\n\n* What are the trade-offs of the proposed evaluation method? It is clearly stated that it can alleviate redundancy, but what are the potential drawbacks? Are there risks in excluding some common evaluation prompts, even if they seem redundant? It would be valuable to know if the proposed method has any specific limitations when applied, especially in cases where common prompts might still hold evaluative value despite redundancy concerns.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The study of LLM evaluation addresses a promising and urgent need. The proposed method appears to be a reasonable approach for focusing on separability among LLMs, especially for those fine-tuned from the same foundational model.\n* Including the condition of prompts in the rating evaluation is a straightforward approach, and I believe this direction will soon gain more popularity.\n* The overall narrative is promising, supported by sufficient experiments that demonstrate the robustness of the evaluation method against redundancy.\n* Although I did not go into the details in the appendix to verify all theoretical aspects, my rough review suggests that they are likely correct.", "weaknesses": "* The main idea is based on the fact that the importance of prompts in average cases may not be balanced; however, the proposed solution may lead to exploitation by niche prompts if LLMs share similar general abilities and some major prompts are marked as redundant. This approach lacks a mechanism to verify prompt redundancy. For a more formal evaluation, it would be better to provide some information on redundancy rather than directly applying the inferred results.\n* I found the use case of the proposed method highly related to game balance analysis. It would be beneficial to discuss the strengths and weaknesses of Elo ratings in greater detail, such as their scalar strength representation, which is better than vanilla win values but cannot handle scenarios like Rock-Paper-Scissors. The discussion mainly focused on Elo’s inability to handle redundancy, but it inherently has some resistance to redundancy, at least compared to vanilla win values.\n* I have concerns about the focus on the utility of separability. Is high separability always beneficial for evaluation? In my understanding, it can sometimes lead to biased positioning, and it requires more careful assessment rather than direct application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F3tqKezLGR", "reviewer": "Reviewer_rQkm", "summary": "This work studies an equilibrium rating framework that can scale to real-world LLM evaluation and move beyond traditional Elo ratings. The authors start by displaying that Elo ratings are sensitive to redundancy. Then, the authors introduce their specific method, describing the proposed rating method in terms of gamification. The authors then provide several results, finding that the proposed equilibrium ranking framework is fairly consistent with Elo rankings and that the proposed system is invariant to redundancy.", "questions": "Please address the weaknesses noted above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n+ This paper has an abundance of citations and explains how the proposed work relates to those prior well. \n+ There are several interesting, important results. Figure 2, specifically, provides a very clear indication of the utility of the proposed equilibrium ranking approach.", "weaknesses": "Weaknesses\n- I found the paper a bit difficult to read. As the paper is a bit outside my exact expertise, this difficulty may be attributed to that. However, if other reviewers also found difficulty in understanding the exact details in the methodology section, I would recommend expanding this section with additional intuitive explanations and also expanding some of the descriptions in the appendix.\n- Figure captions could be improved by concluding with important takeaways.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o2uP5ftIvB", "reviewer": "Reviewer_itPT", "summary": "This paper explores a new approach to evaluating LLMs in open systems to address the bias introduced by Elo, proposing an evaluation framework based on game theory that views the evaluation process as a three-player game involving the \" promptplayer,\" \"king player,\" and \"rebelplayer.\" By introducing novel solutions applicable to both general and game settings, the proposed method is more robust in handling data redundancy and excels in balancing model skills. Experimental results demonstrate the effectiveness of this approach.", "questions": "In Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper is good. Firstly, from a technical perspective, it introduces a 3-player game model based on game theory, providing equilibrium solutions applicable to N-player games. This is highly innovative in my opinion. Moreover, the paper conducts an analysis based on this paradigm, utilizing Affinity Entropy to meet assumptions and thereby addressing the limitations of traditional Elo evaluation methods in handling redundant data and biases. \n\nSecondly, there is a very close integration between the main text and the experimental section of the paper. The theoretical design of the game theory model and its experimental validation complement each other, showcasing the effectiveness of the method on real-world datasets. The authors demonstrate the advantages of the Affinity Entropy method through theoretical analysis, which is further supported by experimental data, ensuring a coherent and consistent research logic.\n\nLastly, in terms of experimental design, the authors conducted thorough validation on real open LLM evaluation datasets, demonstrating the robustness and wide applicability of the method. The introduction of various comparison methods and key evaluation metrics adds to the robustness of the study.", "weaknesses": "There are a few points that confuse me:\n1. In the background section, the introduction to NE and CCE seems somewhat limited, although this is a focal point of the paper. Insufficient explanation might lead to misunderstandings upon re-reading. LLE is also an important concept, yet it is only mentioned in section 3.2, causing some logical confusion.\n2. In the paper, what specific meaning does \"p\" represent in the context of Affinity Entropy? Is it reasonable to directly set p=1 in Theorem 1? The subsequent appearance of p in D_{pq} further adds to the confusion.\n3. Could it be beneficial to consider including more benchmark comparison methods, such as other evaluation systems based on game theory or related equilibrium solutions, to clearly demonstrate the relative performance of this method across different evaluation frameworks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "el4IEFisxB", "reviewer": "Reviewer_56qC", "summary": "The current evaluation focuses on assessing a specific ability, but the emergence of technologies like LLMs has made the evaluation of open-ended tasks more important. Existing methods use Elo ratings for evaluation, but this approach's sensitivity to redundancy may lead to biased assessments. The authors propose a game theory-based evaluation method to address this issue. Experimental results indicate that this method can effectively identify prompts that can be distinctly differentiated and is less sensitive to redundancy compared to Elo-based methods.", "questions": "Around line 60, why is this setting a necessity? I understand that using game-theoretic methods might have certain advantages, but why must this paper choose this type of method?\n\nThis paper uses gemini-1.5-pro as the judger. What considerations went into this choice? Why not use other more powerful models?\n\nThe text repeatedly mentions skill, skill entropy. What exactly is the definition of skill in this paper? Is it capability? If so, why can't we use targeted evaluations, or LLM-eval (in fact, I think this paper is also a type of LLM-eval method), and must instead use Elo-based methods?\n\nFor other questions see the **Weaknesses** section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The motivation of this paper is relatively meaningful, and studying the impact of evaluation on the development of LLMs is indeed an important issue, especially for open-ended evaluation.\n\nUsing game theoretical methods to assess the open-ended response capabilities of LLMs seems to be a relatively novel approach.", "weaknesses": "The article severely lacks background information, leaving readers quite confused about its position in the current research landscape. The Background section is too general; at least, it should introduce what 'king-of-the-hill' used in the actual modeling of this paper is. The Related Works section also fails to fulfill its intended purpose and function in the whole paper. I believe there are far more rating methods than those mentioned (such as [1][2] mentioned later). The differences between the method in this paper and other rating methods, other Elo-based methods, and other game-theoretic methods are not explained in detail. Moreover, the contribution of this paper is unclear. What specific shortcomings of the Elo-based method does it aim to address? Is it the robustness and sensitivity to redundancy mentioned in the abstract, or the poor skill assessment mentioned in section 1.1? Given the unclear statements in this paper, it is difficult for me to evaluate its contribution. If there are any misunderstandings on my part, I welcome further discussion at a later stage.\n\nThe paper emphasizes that this method can be applied to fully open skill assessments, but I believe its openness and generality come from the scoring model based on LLM (such as the gemini-1.5-pro mentioned in the paper). This is not a satisfactory solution, and what are the essential advantages of this over traditional LLM-eval-based methods (such as AlpacaEval [1][2])?\n\nFigure 3 is confusing and lacks explanations for the connections. Figure 5 is even more perplexing, as it is unclear what the purpose of displaying these color blocks is and what it is trying to convey.\n\nThe experimental section lacks a comparative baseline. How much advantage does this evaluation method have over existing methods (such as various Elo-based methods and previous game-theoretic methods mentioned in the related works section)? Additionally, the dataset used in the experiments of this paper consists of only 500 questions, which is not a sufficiently convincing amount of data.\n\n\n**Reference**\n\n[1] AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (https://arxiv.org/abs/2305.14387)\n\n[2] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (https://arxiv.org/abs/2404.04475)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)