Investigating Non-Transitivity in LLM-as-a-Judge
Authors: Yi Xu
Laura Ruis
Tim Rocktäschel
Robert Kirk
Venue: ICML 2025 spotlightposter (2025)
Keywords: LLM-as-a-Judge
transitivity
pairwise comparison
Primary area: deep_learning->large_language_models
TL;DR: We investigate non-transitivity in LLM-based evaluation frameworks, showing that LLM judges exhibit non-transitive preferences, leading to ranking instability.

Abstract: Automatic evaluation methods based on large language models (LLMs) are emerging as the standard tool for assessing the instruction-following abilities of LLM-based agents. The most common method in this paradigm, pairwise comparisons with a baseline model, critically depends on the assumption of transitive preferences. However, the validity of this assumption remains largely unexplored. In this study, we investigate the presence of non-transitivity within the AlpacaEval framework and analyze its effects on model rankings. We find that LLM judges exhibit non-transitive preferences, leading to rankings that are sensitive to the choice of the baseline model. To mitigate this issue, we show that round-robin tournaments combined with Bradley-Terry models of preference can produce more reliable rankings. Notably, our method increases both the Spearman correlation and the Kendall correlation with Chatbot Arena (95.0\% $\rightarrow$ 96.4\% and 82.1\% $\rightarrow$ 86.3\% respectively). To address the computational cost of round-robin tournaments, we propose Swiss-Wise Iterative Matchmaking (Swim) tournaments, using a dynamic matching strategy to capture the benefits of round-robin tournaments while maintaining computational efficiency.

Investigating Non-Transitivin in LLM-as-a-Judge
Yi
Laura Ruis' Tim Rocktäschel'
Robert Kirk
Abstract
Automatic evaluation methods based on large lan- guage models (LLMs) are emerging the stan- dard tool for assessing the cnttructionfoollviiii abilities of LLMM-baed agents. The most common method in this paradigm, pairwise comparisons with a baseline model, critically depends on the as- sumption of transitive preferences. However, the validity of this assumption remains largely unex- plored. In this study, we investigate the presence of non-transitivity within the Al AlpacaEval frame- work and analyze its effects model rankings. We find that LLM judges exhibit non-transion tr-nssiniitvee preferences, leading to rankings that are sensitive to the choice of the baseline model. To mitigate this issue, we show that round-robin tournaments combined with Bradley-
Aajaad 5g0 1 S-ddt
Cilmool 200 invinect heateeeee
qp-4
chando Sosaeee-ooooo---0e00 bakkac
+100 v
sgoi cois aearhhg gh
sk-ange 0.50
0.58
0.57
0.69
6.73
0.74
0.76
0:78
0.27
0.61
0.50
0.90
0.55
0.62
8.67
0.72
0.75
0.75
Liama-3-01---
0.31
0.37
0:43
0.50
6.50
0.50
0 0.25
0.27
0.29
0.33
0:43
0:45
0.47
0.30
0.51
G2-4-014
models of prefer- ence produce more reliable rankings. Notably, our method increases both the Spearman corre- lation and the Kendall correlation with Chatbot Arena (95.0% - 96.4% and 82.1% - 86.3% re- spectively). To address the computational cost of round-robin tournaments, we propose Twiss-Wise Iterative Matchmaking (SWIM) tournaments, us- ing a dynamic matching strategy to capture the benefits of round-robin tournaments while main- taining computational efficiency.
Figure 1. Rankings from baseline-fixed frameworks show high sensitivity to the choice of baseline. Each entry (a:, y) represents win rate model m. against my. where each column reflects ranking with the column model as the baseline. Inconsistency emerges when Llama-3-70B and Claude-3-Opus are used as base- lines. Appendix C.I provides the detailed matrix comparing 20 models.
aspect existing automatic metrics cannot adequately assess. However, human evaluation is costly and lacks scalability (Karpinska et al., 2021). As a result, are now widely used to automate the process, with pairwise comparisons proving particularly effective in aligning with human ratings (Liusie et al., 2024; Liu al., 2024; Chiang et al., 2023; Li et al., 2023; Lin et al., 2025; Zheng et al., 2023; Samvelyan et al., 2024: Khan et al., 2024).
1. Introduction
evaluators
The growing adoption of large language models (LLMs) generalist systems for complex, open-ended tasks (Ope- nAI et al., 2023; Meta AI, 2024b) presents critical chal- lenge: the lack of a universally accepted gold-standard eval- uation. In many cases, multiple valid responses exist for a given task, complicating the establishment of effective benchmarks. Consequently, a new paradigm for evaluating open-ended tasks focuses on quantifying the alignment of LLMs with human preferences (Ouyang et al., 2022)
The typical pipeline for LLM-based automatic evaluation frameworks is using pairwise comparisons between a target model and fixed baseline model, where an oracle model serves the judge. By calculating the relative win rate against the baseline model, such comparisons enable rank- ing target models. However, it is unclear whether using fixed baseline provides consistent results. If the judge ex- hibits non-transitive preferences, such as favoring A over B, B over C, but C over A, the resulting rankings can become sensitive to the choice of the baseline model (Figure 2).
AA Centre, UCL FUK Al Security Institute. Correspondence to: Yi Xu 23 2.x..23uucl.c.uk. ukk.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Investigating Non- Non-Transitivity in LLM-assa-Judge
Anseline-Fioed Foomework
Bound tourooment
Bradley Terry
User Instruction
User Instruction
User instruction
O @other respanses diferentmeeel
Conduct yuirwdae oraiiioon through round tournament.
Responaee A
of
Responae A
moddll
Responseed
oesponse
Besponse of
Jadge Evahattee
Judge Evaluation
Relichle
Which mede perforus better overthe Inscrnctione?
boseline model:
Mg
Model
Scare
baseline modet:
Compuut Elo scores with the Bradley Cereenyy-erraaaan moddd
6.1
6.8
Mg
Inconsistent Mnnellable
eoch trsttiction.
Figure 2. (Left) Inconsistent rankings observed in baselinc-fixx: frameworks based pairwise comparisons due to non contrransttivity in the judge's evaluations. Different choices of baselines can lead to varying rankings, undermining the reliability and robustness of this approach. (Right) We propose round-robin tournament framework where all models compared pairwise. The results used capture non- transttaiit in the judge's evaluations and score models using the Bradley-Terry model. This method produces rankings that robust and better aligned with human evaluation.
In this work, we investigate the existence and impact of non-transitivity within AlpacaEval (Li et al., 2023), which has been largely overlooked in previous work. AlpacaEval popular pairwise comparison framework that employs GPT-4-Turbo the fixed baseline model. We introduce Soft Non-Transitivity Deviation (SNTD) as metric to measure the degree of soft non-tramse con-transitivity in the judge's continuous preferences and find that LLMS exhibit both hard and soft non- con-transitive preferences. Additionally, previous studies have demonstrated that LLMS often exhibit various biases (Gallegos et al., 2024) such as position bias (Zheng et al., 2023; Wang et al., 2024; Zhou et al., 2024b), which can lead to spurious correlations in the judge's preferences. We show that the occurrence of non-transitivity is jointly in- fluenced by position bias and the judge model's inherent non non-transitive reasoning abilities.
culminates in model-level non-transitivity (Figure 1). We demonstrate that such non-transitivity makes the ranking highly sensitive to the choice of the baseline model. Chang- ing the bascline model makes the rank order inconsistent and unstable, highlighting the importance of proposing new ranking methods. 2) We find that while position bias sig- nificantly contributes to non-transitivity, it not the sole cause. Our experiments confirm that position switching out- performs random assignment in mitigating position bias for stronger judges when using continuous values for judge's preferences, with reductions ranging from 17% to 44%. 3) We demonstrate that applying round-robin tourna- ments combined with the Bradley-Terry model reduces the impact of non-transitivity, resulting in more robust rankings. This method also aligns better with human eval- uations of model rankings in Chatbot Arena. Finally, we introduce SWIM, efficient method for adding models with nearly identical performance compared to naive round-robin tournaments.
To address the above, we propose the of round-robin tournaments in the pairwise comparison setting, overcom- ing the need for a fixed baseline model. We subsequently apply the Bradley-Terry model (Bradley Terry, 1952) to score models based on tournament outcomes, yielding a more consistent ranking compared to baseline-fixed rank- ing. To address the computational cost in the round-robin tournament, propose Swiss-Wirs Iterative Matchmaking (SWIM) tournaments to improve efficiency while preserving the robustness of model comparisons.
2. Related Work
The I.LMaas-a-Jude (Zheng et al., 2023) evaluation method leverages frontier models rank responses to open-ended queries without explicit ground- truths. A common approach involves using fixed baseline model for pairwise comparisons assess the performance of the target model, as seen in frameworks such as VicunaE- val (Chiang et al., 2023), AlpacaEval (Li et al., 2023), and Arena-Hard (Li al., 2024). The target models are then ranked on the basis of their win rates against the baseline.
I.LM-as-a-Judge
Our contributions are as follows: 1) We show that LLMs exhibit non-transitive preferences when performing pair- wise comparisons. Additionally, we observe that the ag- gregation of instruction-level non-transitive relationships
2

---

Investigating Non- Von-Transitivity in LLM-as-aaJudge
However, an implicit assumption in these frameworks is that transitivity holds in preference judgments, which has not been empirically verified. Transitivity requires that if an LLM judge prefers model MA over mg and m over mec it must consequently prefer m over mc- Violations of transitivity can result in unstable rankings that undermine the evaluation framework's 's reliability (Figure 2). To address this gap. we examine the robustness of current LLM ranking methodologies by extending the AlpacaEval framework to investigate the existence of non-transitivity, aiming es- tablish more rigorous foundation for the LLM evaluation system.
pute the Percentage of Non-Transitive cases (PNT) over the instruction set I, defined as:
PNT- I poon-trnns (m, MB, mC mJ, 1á), (2)
where the indicator function non-trans. returns when the judge's preferences violate logical transitivity, and 0 other- wise. See Appendix B. for the complete set of conditions.
However, this metric demonstrates a limitation in sensitivity: given J(mA > m. []] = 1 and JmmB > mc | Τ) = 1, it classifies both (m X mc I) = and J(m > mc | I) = 0.49 as non-transitive, despite the latter exhibiting sub- stantially stronger transitivity tendency as it closer the transitive threshold. Such insensitivity transitional values near the decision boundary undermines the metric's capacity to capture nuanced deviations from ideal transitivity.
Non-Transitivity in Zero-sum Games Prior work has ex-
plored -ransitiiiiit two-player zero-sum games within multi-agent reinforcement learning. Balduzzi al. (2019) characterize agent interactions through convex polytopes, us- ing their dimensionality to decompose transitive and cyclic components. Czamecki et al. (2020) demonstrate that real- world strategy spaces exhibit a spinning top distribution, where non non-transinee transitivity peaks at middling performance levels but diminishes at either lower or higher levels. Given the presence of non-transitivity, evaluating a strategy based on its performance against a single opponent does not reliably reflect its true capability. Therefore, previous achievements in complex games such as StarCraft (Vinyals et al., 2019) and Dota 2 (O et al., 2019) employ population-based self-play training and evaluate agents through tournament- style competitions against diverse opponents. Mirroring the population- based evaluation paradigm that succeeded in non-transitive games, we adopt tournament-based compar- isons in LLM-as-a--Jude frameworks to mitigate ranking instability induced by non-transitivity.
Soft Transitivity Deviation. To address this limitation, we propose Soft Non- con-Transitivity Deviation (SNTD) to measure the degree of non-transitivity for a single instruction with a triplet of models, defined as:
SNTD(mA: m. mc [.)
3
1 x
E
JSD 0000
(
OB
(
my. 1))l oo I.)
(
(4)
( mJ It) my
+ JSD
(000:000 30
+ JSD (60
60
|m
(0) o8 mJ, Ii)
where the discrepancy between observed win rates e and estimated win rates é under transitivity assumptions, as defined below.
divergence (JSD) quantifies the
3. Methods
Estimated Win Rate. We denote the latent quality of the outputs from models m.a. mB, and mc on instruction T; (6) (8)
3.1. Measuring Non-Transitivity in Pairwise Comparisons
as 7 7 and 76 respectively. Given empirical obser- vations Φ, Bradley-Terry model estimate the quality gap as:
We employ an LLM, denoted as my, to conduct pairwise comparisons between models mA and mg- The objective (i)
is to determine which of the two outputs, (f) or og better follows given instruction T,. To facilitate the comparison, each model output assigned a unique token identifier. The
$lOA 0(0
mJ, T:) ) (4) mj.4.)
(2) AB 7 7
(
In
OB
1
antisymmetric judge function di OB |m., T4) evalu- ates pairs of outputs from models and determines the proba-
(c) (i)
Based on that, we can estimate the expected win rate under transitivity between any two models from a triplet (m mB, me) by utilizing the observed win rates between the other two pairs (See Appendix B.4 for the derivation):
bility of favoring 0'A as the win rate by applying softmax operation to the log probabilities of the corresponding model tokens. The preference of a over my is then quantified by taking the expected value of the judge function:
8(00i A
mJ 4))
(5)
122 63
1 e (0 BC)
J J(mA > mg L)) E
A oB (
mJ. (1)
3.2. Measuring Model Performance
Hard -on-Transitive Cases. To quantify non non-transytivity among a triplet of models (mA, mB, mc), we first com-
In this section, we define metrics to quantify and rank model performance given a model pool M, instruction dataset I,
3

---

Investigating Non- Non-Transitivity in LLM-as-a-Judge
and judge my.
model to comparison outcomes assign scores, which are then converted into Elo scores for the final ranking.
Win Rate Against Baseline. Through currying the judge function with fixed baseline model Mhase we define the win rate against the baseline model as rating function:
Swiss-Wiss Iterative Matchmaking Tournament. While round-robin evaluation yields reliable rankings, it presents significant computational challenges at scale. Incorporat- ing a new model into a leaderboard of size M necessitates M model-level comparisons compared to single compari- son in Paseline-tixet frameworks. To address this compu- tational bottleneck, propose the Swiss- Swiss-Wise Iterative Matchmaking (SWIM) tournament (Algorithm 1), drawing inspiration from binary search and Swiss-system tourna- ments. Our approach dynamically adjusts matchmaking based on Bradley- Terry coefficients, focusing comparisons near model capability boundaries in a logarithmic manner, thereby reducing the number of comparisons to (logg(M))
1 Z
Roase (.)
E
of
(1) mbase
mm 1i)
(6)
LEE
Bradley-Terry Coefficients. Given series of pairwise comparisons, we employ the Bradley- Terry (BT) model to convert comparison outcomes into coefficients B, E IR that quantify the strength of model m, The optimal BT coefficients 3 are estimated by maximizing the likelihood:
1 ΣΣ Wiy In (1+ cl 3.) )] (7)
B arg max
3.4. Evaluation Setup
where Wi, represents the number of times model i wins against model j. Rather than using discrete labels {0, to count victories, we utilize the judge's preferences as soft labels, defining W., = Ei [m, > mj Tx), which yields more accurate estimations (See Appendix D).
Datasets. We use the AlpacaEval dataset (Li et al., 2023), which includes a wide variety of instruction types, such as information search tasks and coding problems.
Participating models. We evaluate 20 models that appear on both the AlpacaEval and Chatbot Arena' I leaderboards (see Appendix A.1 for details).
To establish a standardized measure of model performance, convert Bradley-Tery coefficients Elo ratings (Elo, 1966) by setting & = 400 log 3, Under this system, the probability of model mn; winning against model my is expressed as:
Elo Rating.
Scenarios. We denote significant performance advan- tages with >> and marginal advantages with ss. Based relative model performance, we classify model triplets (m mB, mc) into four categories:
P(m; > m2) + 10(5, 400/400
1. Lead & Lead (LL): mA
> mB and mB >> mc-
3.3. Yournament-Based Ranking
2. Lead & Margin (LM): m, > mg and mg = mc.
We formalize the LLM-as-a-Judge evaluation a multi- player game framework, where evaluated language models act as players. Each player's s strategy space is defined by its response generation approach under given instructions. When the judge exhibits non-transitive evaluation behavior, model assessment through fixed-opponent comparisons can- not provide reliable rankings, leading us to characterize this evaluation framework non-transitive game.
3. Margin & Lead (ML): m, R mB and mB >> mc.
4. Margin & Margin (MM): m. & ma and mn mc.
For each scenario, we select representative model triplets based on the win rates of participating models from the AlpacaEval leaderboard (see Appendix A.2 for details).
Judge models. For consistency with AlpacaEval, we main- tain the judge configuration and prompt templates. We examine non-transitivity in judgments using two models: GPT-4-Turbo2 and GPT-3.5-Turbo (OpenAl al., 2023), both with the temperature set to O. The detailed prompt is provided in Appendix G.I.
Round-Robin Tournament. Tournament-basee competi- tion with diverse opponents has been established as an effec- tive approach for performance evaluation in -anttaansitive games (OpenAl et al., 2019; Vinyals et al., 2019), as it enables robust assessment of relative capabilities while mit- igating the impact of non- transitivity. Based on this insight, we propose round-robin tournament structure where each model engages in pairwise evaluation against every other model in the pool, with evaluations conducted by judge m. over instruction set I. This method enables comprehensive model evaluation through comparisons against a diverse pop- ulation of models rather than relying on a fixed perspective for assessment, We subsequently apply the Bradley-Terry
Position Switching. LLMs are known to exhibit biases and inconsistencies based on the order of outputs presented in the prompt (Zheng et al., 2023; Pezeshkpour & Hruschka, 2024; Raina al., 2024). To mitigate this bias, we employ
'Refer to Fully Style-Controlled Chatbot Arena Leaderboard (2024/09/15)
*GPT-4-Turbo refers to gpt-4-1106 -previeew in the win rate matrix avoid ambiguity.
4

---

Investigating Non-Transitivity in LLM--as-aJJdge
Table 1. We measure non- non-transitivity in four scenarios, evaluated by GPT-4-Turbo and GPT-3.5-Turbo. Orange cells indicate maximum PNT/SNTDD values (highest -anttiaiit;:;::): blue cells indicate minimum PNT/SNTD values (highest transitivity). When using GPT- 4-Turbo as the judge, more con-transitivity can be observed as evaluated model performance becomes more similar and the highest non-transitivity occurs when the performances of all three models are similar; however, GPT-3.55-Turbo does not exhibit this pattern.
GPT-4-Turbo
GPT-3.5-Turb
Scenarios
Models
PNT
SNTD
PNT
SNTD
LL
T 4pt-4o-2024-05-13
m >>
n Qwen1.5-72B-Cher
3,98
0.1121
21.37
0.2654
my
mc
MC Mistral-7B-Innsrrut---00
LM
A gpt-4o-2024-05-12
A mB
mg (wen1.5-72B-Chtt
5.96
0.1336
22.48
0.2586
mg & mc
ML
M 11-343-Char
MA R mB
mg Qwen 5-722-Chhat
3.98
0.1215
22.86
0.2625
mB 2 mc
Mc Mistral-788 Iinttt-tt--ttt
MM
A Qwen Chat
MA R mB
mg c --snne--- 00-sonett-22202222 20240229
8.45
0.1431
20.87
0.2629
me
Mc gpt-4-0314
position switching, where each comparison is evaluated with responses in both orders. The final preference score is calculated the mean of these balanced evaluations. To reduce the impact of API randomness, we invoke the judge function twice for each order configuration.
sesses stronger reasoning capabilities and exhibits signifi- cantly less bias compared GPT-3.5-Turbo (Zheng et al., 2023). We hypothesize that the strong non-transitivity ob- served with GPT-3.5-Turbo stems from its inability to dis- tinguish the quality differences among outputs, as it is gen- erally considered to have weaker eestruction-oolowinn abil- ities than most participating models (Chiang et al., 2024; Lin al., 2025; Li et al., 2023; White al., 2025). This inability leads to preferences driven by bias predominantly, which is empirically validated in Section 4.3.
4.
Judge Preferences
In this section, we investigate the judge's non- non-transitive behaviors and analyze their underlying mechanisms.
4.1. Increased Non-Transitivity with Similar Model
4.2. Aggregate Non-Transitivity
As shown in Table 1, crantiiantt emerges across all four scenarios when GPT-4-Turbo serves the judge. Both PNT and SNTD generally increase the performance gap between model pairs (M., m)) B or (mB, mc) narrows. No- tably, while scenarios LL and ML have identical PNT scores, scenario ML exhibits a higher SNTD value, indi- cating more non-transitivity. This discrepancy highlights the limitation of the PNT-it fails to capture the continu- nature of judge preferences in assessing non- transsiiittivtt. Notably, we observe similar trends across other judges and datasets, confirming the generality of the finding (See Ap- pendix B.2).
We use J[m mB) + Σn.e J(mA mB Tt) to denote the averaged pairwise preference, representing the model-level win rate between mA and m. We subse- quently perform pairwise comparisons across all models and present the win rate matrix in Figure with GPT-4-Turbo as the judge to assess whether instance-level non-transitivity extends to the model-level.
at Model Level is Mild. Surpris- ingly, we detect no instances of hard non-transitivity (e.g.. ma > ma, my My, and ma < m.) at the model level, which we partially attribute to the effectiveness of calibra- tion and randomness mitigation techniques. When imple- menting a more aggressive approach-whine where positions are randomly assigned for each evaluation, reducing the pro- cess a single call-we observe occurrences of hard non- transitivity (see Appendix C.2). Nevertheless, model-level non-tran non-transitive cases remain notably rare. We hypothesize that this scarcity stems primarily from the low proportion of non-transitive evaluations when using GPT-4-Turbo as the judge. Given the sparsity of non-trannen transitive comparisons,
Hard
Weaker Judge is More Non-Transitie Replicating our evaluation with GPT-3.5-Turbo as the judge reveals an in- triguing pattern (Table 1): both PNT and SNTD values are consistently higher than those observed with GPT-4-Turbo and remain relatively stable across all scenarios, suggesting persistent and substantial level of tonntransitivity.
Previous studies have demonstrated that GPT-4-Turbo pos-
5

---

Investigating Non- Von-titasitvvity in LLMM-ss---Jdde
Consistent (GPT-44
Ambiguous (GPT-4)
Consistent (GPT:3.5)
Ambiguous (GPT-335)
39
0.8
Propurtion
0.4
0.2
MA So My
Mu Me
Ma 2o Me
Ma Ma Me
M, 20 Me
Ma Ma Margin & Lead (ML)
Ma 1r Me
Ma
My Me Ma si Mc
Lead & Lead (LL)
Lead & Margin (LM)
Margin & Margin (MM)
Figure 3. Larger performance gaps lead to more consistent preferences. We quantify the proportion of consistent preferences of GPT-44-Turb and GPP-335-TTubo across four scenarios differentiated by relative model performance, where denotes substantial performance advantages and z indicates marginal diffcrences.
Namber of Noo--vvrstt
Nocnoizel Degree Noe-Truskiivity
Influence of Model Performance Difference. We further investigate the relationship between non-transitivityy and the performance gap among model triplets within all partici- pating models. For each triplet. we define the x-axis as the win rate difference between models m, and ma from the AlpacaEval leaderboard and the y-axis as the difference between m and mc- The computed PNT and SNTD val- ues, visualized in Figure 4, demonstrate that non- con-transitivity intensifies as the win rate differences between both model pairs decrease. Both metrics peak near the origin, indicating that non-transitivity is most pronounced when comparing models of similar capabilities (See Appendix B.5 for imple- mentation details).
Win Rate Cateee-app between. Mr and Mc
20
-0-
-20 Gup
Gap
Figure 4. trantiianiyyivity becomes pronounced the model performance gap approaches the origin. We find that both PNT and SNTD peak near the origin when GPT-4-Turbo serves as the judge.
4.3. Non- Non-Transitivity is Jointly Influenced by Position Bias and Judge's Inherent Reasoning Abilities
their aggregated effect likely overwhelmed by the pre- dominance of transitive evaluations, thus preventing the emergence of observable non-transitivity the model level.
Position Bias in Judge Preferences. During the evaluation, we observe that both judges exhibit position bias. Specifi- cally, when evaluating two models on given instruction, we define preference as consistent if the judge's 's preference maintains its relationship to 0.5 (either consistently above or below) with position switching. We report the proportion of consistent preferences in each scenario, using GPT-4-Turbo and GPT-3.5-Turbo as judges (Figure 3).
Despite this, notable instances of soft non- -ontraansitivity re- main evident, leading to inconsistent ranking as shown by an example in Figure! 1. Specifically, while GPT-4-Turbo achieves win rate 0.50 against GPT-4o, and GPT-4o wins against Staude-3-Opus with rate of 0.68, transitivity would predict win rate of 0.68 for GPT-4-Turbo against Claude-3-Opus. However, the observed rate of 0.72 reveals subtle violation of transitivity at the model level.
In all scenarios except MM, both judges show the highest preference consistency when comparing m and mc. at- tributable to the substantial performance gap. A potential explanation is that AlpacaEval may have limited discrim- inative ability when evaluating models with similar capa- bilities, meaning the presumed performance gap does not hold. Moreover, GPT-3.5-Turee shows a markedly lower preference consistency than GPT-4-Turbe indicating that its evaluations primarily driven by position bias rather than comparing output qualities.
Limitations of the Baseline-Fixed Framework. We further quantify the sensitivity of baseline-fixed frameworks. For each participating model m, we apply the rating function R.)()) to generate rankings, resulting 20 distinct ranking lists. We find that only 20% of models maintain consistent rank positions across all rankings. Moreover, when compar- ing any pair of ranking lists, only 61% % of models preserve their rank positions on average. These findings demonstrate that rankings are highly sensitive to the choice of baseline, indicating that baseline- fixed frameworks produce inconsis- tent and unreliable model evaluations.
Factors of Non- Non-Transitivity. We further categorize instruc- tions into two groups: ambiguous and consistent. An in- struction is considered consistent only when the preferences
6

---

Investigating Non- Ton-Transitivity in LLM-assaaJudge
Transttive (Consistent)
Non-transttive (Consistent)
Transitive (Ambiguuus)
Non-transitive (Ambiguous)
32
32
67
44
64
57
81
172
181
184
157
15k
Proportion
468
493
409
468
0.6
521
307
0.4
605
603
611
614
614
614
0.2
352
289
280
215
352
289
280
215
2
3/4
LL
LM
ML
MM
LL
LM
ML
MM
LC
LM
ML
MM
L.M
ML
MM
GPT-
GPT- (Random Choios)
GPT-3.5
GPT-3.5 (Random Choice)
Figure 5. Proportion of -oon-)transitive instructions across all scenarios, as evaluated by GPT-4-Turbo and GPT-3.5-Ture When evaluating model triplets with GPT-3.5-Turbo judge, over 96% of instructions exhibit position bias effects. In contrast, GPr-4-Turbo demonstrates substantially higher evaluation consistency. Our analysis reveals that position switching provides more effective bias miligation than random assignment for less position-biased judges.
between (ma; mB). (m,; mc), and (m,; mc) are all con- sistent, implying that all comparisons are not influenced by position bias. Otherwise, the instruction is categorized as ambiguous, as at least one of the comparisons is affected by position bias. We report the proportion of non-transitive cases in Figure 5. We find that ambiguous instruction ex- hibits significantly higher non- -onnssaanttiv rates compared to consistent instructions, suggesting position bias is indeed contributing factor. Furthermore, when using GPT-3.5- Turbo as the judge, the proportion of ambiguous instructions exceeds 96%, validating that it exhibits much stronger po- sition bias than GPT-4-Turbo
Usefulness of Position Switching. Instead of using position switching, we repeat the experiment by randomly assigning the positions of the outputs in the prompt (Figure 5). Since all preferences in the consistent instruction are consistent, the proportion of non-transitive cases remains unchanged. However, for ambiguous instructions, we observe divergent effects: GPT-4-Turie exhibits a significant increase in non- transitivity, while GPT-3.5 shows slight decrease.
The distributions of judge preference (see Appendix B.6) show distinct evaluation patterss between judges. When mitigating GPT-3.5-Turb) position bias through position switching. the model tends to generate more uncertain out- comes (averaged preference ~ 0.5). In contrast, GPT- 4-Turbo exhibits different characteristics: while position switching occasionally introduces uncertainty, its debiased preferences generally maintain clear output distinctions. This finding suggests that position switching can reduce non-transitivity for stronger judges that are less affected by position bias, with reductions ranging from 17% to 44%. However, for weaker judges that are more susceptible to position bias, it may have the opposite effect.
Interestingly, we find non-transitivity still occurs within consistent instructions, with GPT-4-Turbo serving as the judge, indicating that position bias not the sole cause of non-transitivity. Therefore, we argue that non-transion con-transitivity arises from two primary factors. The first is the inherent rea- soning capability of the model, which is non-transitive due to the judge's 's latent comparison criteria. When the quality of the outputs is similar, the judge may display preferences dynamic. The second factor the position bias, which affects the judge's preferences. These two factors interact and compound the occurrence of non-transitivity.
akin to a lock-paper-sciseee
Prompting Strategies Mitigate Non-transitiiiity. We ex- plore various prompting strategies address con-taansiyyvity in model judgments.. Our analysis focuses on Scenario MM, where the capabilities of the compared models are closely matched, making it casier to observe both non-transitive behaviors and the effects of different prompts. Our find- ings show that providing judges with a structured evaluation checklist (Cook et al., 2024) would marginally reduce non- transitive cases. Interestingly, while incorporating Chain-of- Thought reasoning (Wei et al., 2022) helps mitigate position bias, it also leads to higher incidence of non-transitive preferences. Moreover, allowing the judge to declare ties not only increases position bias but also further amplifies non- non-transitivin -unttransitivity. See Appendix C.3 for detailed results.
Stronger Position Bias Increases Non- Non-Transitivity. To investigate the impact of position bias, we introduce Po- sition Difference (PD). Given an instruction 1 and a model triplet (m m., mc), we define this measure as PD(MA, m. T,) + PD(mB, mc: T²) + PD(m me; It), ranging from 3, where PD(m m, 44) is defined as (i) (i)
, m., T+)] Using GPT-4-Turbo as the judge, we evaluate all triplet permuta- tions and partition PD values into bins. As shown in Figure 6-Left, the proportion of non-transitive cases increases with PD, demonstrating a strong positive correlation.
E[ 0g (m,, I,)] Eiá(o (c) (8)

---

Investigating Non- Non-Transitivity in L.M-as-a-Judge
lation and a 4.2% increase in Kendall correlation compared to length-controlled AlpacaEval.
Table Correlation comparison between the round-robin-basee framework and AlpacaEval. with and without length control (LC).
Performance of SWIM. We demonstrate that both round- robin-based ranking and SWIM- based ranking outperform AlpacaEval, as shown in Figure 6-Right. We do not com- pare performance under length control, as the generalized linear model an empirical approach that may be less inter- pretable, potentially affecting fairness.
Method
Spearman Conrelation
Kendall Correlation
w/o, LC
w, LC
A
W/.LCC
w. LC
A
AlpocaEval 2.0
81.4%
95.0%
+13.6%
63.2%
82, 1%
+18.9%
Round-Robin
85.4%
96.4%
+10.0%
68.4%
86.3%
+17.9%
A
+4.0%
+1.4%
+5.2%
+4.2%
6. Limitations and Future Work
0121
Our study has several limitations. While AlpacaEval pro- vides diverse instructions, it may not fully capture real-world open-ended tasks, necessitating validation of our method across broader domains. Additionally, extending our find- ings judge models beyond GPT-4-Turbo and GPT-3.5- Turbo is important direction for future work. Further- more, while our benchmark relies on human rankings from Chatbot Arena, inherent human biases (Chen et al., 2024) may introduce non- -on-trannitivity in human preferences, fun- damentally limiting the achievable alignment between auto- mated and human evaluations.
Cormlaatee Coutiocen
Proference Dife
Figure 6. (Left) Non-transitivity strongly correlates with position difference. (Right) Both round-robin and SWIM tournaments achieve nearly identical performance, consistently outperform- ing AlpacaEval in all cases. We compare the performance between tournament- based ranking and AlpacaEval leaderboard across different numbers of participating models. For each model count, we randomly sample models and conduct 2000 trials, re- porting the mean correlation with 95% confidence interval.
Secondly, our foeus on pairwise comparisons leaves open questions about non-transitivity in pointwise evaluations. While pointwise methods inherently avoid position bias caused by output ordering, converting these scores to pair- wise comparison (A > B if score(A) > score(B)) may intro- duce new forms of non-transitivity, depending on the granu- larity and consistency of rating criteria. Future work should investigate whether such conversions preserve transitivity and identify conditions that modulate cyclic preferences.
5. Results of Tournament-Basen Ranking
We conduct a round-robin tournament to obtain pairwise comparisons and apply the Bradley-Terry model to compute ratings, which are then converted Elo scores. The result- ing Elo scores and rankings for all 20 evaluated models are presented in Table 9 in the Appendix.
Finally. our analysis relies on the Bradley-Terry model, which assumes transitive model-level preferences by assign- ing each model a global scalar score. While we do observe instance-level non-transitivity in our pairwise comparisons, these cases are relatively rare, and hard non- transtttniiiittt in the aggregated model-level preferences is mild. Therefore, we find the Bradley-Terry model sufficient for our ranking purposes. Nevertheless, we acknowledge that this imple- mentation may not fully capture the nuanced capabilities of models. We leave this as a direction for future work, focus- ing on more expressive alternatives that parameterize model capabilities in a nulti-dimensional space (Duan al., 2017). which remains promising and under-explored approach for improving the robustness of LLM-as-a-judee evaluations.
To assess the effectiveness of our framework, we consider the human preference ranking from the Chatbot Arena as the reference. We compute the Spearman and Kendall cor- relations between our round-robin manddroobi--baee based ranking and the Chatbot Arena. We also compare these correlations with those between the AlpacaEval and the Chatbot Arena. As shown in Table 2, our method achieves higher correlations, with a 4% increase in Spearman correlation and 5.2% increase in Kendall correlation.
Length-Controlled Winrate. To mitigate verbosity bias and ensure a fair comparison, we adopt the generalized linear model with the same weights as Length-Controoled AlpacaEval (Dubois et al., 2024) to derive length-controlled preferences. Using these preferences, we compute the length-controlle Bradley-Terry coefficients, which are then converted to controlldd Elo scores. Table 2 shows that our coottollode cound-robin ranking further im- proves correlations, with a 1.4% increase in Spearman corre-
7. Conclusion
In this paper, we comprehensively study the impact of non- transitivity in the current based framework with pair- wise settings, filling a gap in this area of research. Our find- ings show that -on-traniitivity can be observed at the instruc-
8

---

Investigating Non- Von-Transitivity in LLM-as-a-Judge
tion level during judgment and is related to the reasoning capability of the judge. The aggregation of instruction-level non-transitivity further leads to model-level ton-transitivity, revealing the limitations of the baseline-fixee framework, as the rankings in this setting depend on the choice of the hase- line model. Our analysis also demonstrates that position bias a key factor in non-transitivity, with systematic position switching proving more effective than random assignment in reducing non-transitivity for stronger judges.
H. When benchmarks are targets: Revealing the sen- sitivity of large language model leaderboards. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceed- ings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pp. 13787-13805, Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10.186533w1/7002:eeeloorrg
Anthropic. Model card and evaluations for claude models, 2023.
To address the above, we propose a baseline-free frame- work utilizing round-robin tournaments with Bradley-Tery model, which captures non- non-trans transiiiniiiit patterns and demon- strates better alignment with human. Recognizing the com- putational constraints of round-robin tournaments, which require O(nm²) instruction-level comparisons for rank- ing models across instructions, we propose SWIM tournaments. This approach achieves (nem log m) com- plexity through dynamic matching, substantially reducing computational cost while maintaining nearly identical per- formance. The code and data are available at ht tps: //github. ctttuuu.com/e/ com/cm////////o fomm/yix//// x 11m nontransitiiiiiiiiiin.iii
Anthropic, A. The claude 3 model family: Opus, sonnet, haiku, 2024. Claude-3 Model Card.
Balduzzi, D., Garnelo, M.. Bachrach, Y., Czarnecki, W.
Perolat, J., Jaderberg, M., and Graepel, T. Open-ended learning in symmetric zero-sum games. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th In- ternational Conference on Machine Learning, volume 97 Proceedings of Machine Learning Research, pp. 434- 443. PMLR, 09-15 Jun 2019.
Acknowledgements
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952. ISSN 00063444, 14643510.
We thank to the reviewers and the area chair for their con- structive suggestions. We also thank to the OpenAl re- searcher access program for providing the OpenAl API credits used in this project. Finally, we are grateful to Akbir Khan for early comments, suggestions and advice on the project. LR is supported by the EPSRC Grant FP/S021566/1 and UCL International Scholar Award for Doctoral Training Centres.
Chen, G. H., Chen, S., Liu, Z., Jiang, F., and Wang, B. Humans or LLMs as the judge? a study on judgement bias. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pp. 8301- 8327, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18833/v1/2222 emnip-main.474.
Impact Statement
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chiang, W.-L., Li. Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L.,, Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An oeeoopooooees chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
References
Chiang. W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating LLMs by human preference. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A.. Oliver, N., Scarlett,
OL.AL Yi-large Ilm launch, 2024.
0LAAI, Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang. J., Yu, K., Liu, P., Liu, Q., Yue, S., Yang, S., Yang, S., Yu, T., Xie, W., Huang, W. Hu, X., Ren, X., Niu, X., Nie, P. Xu, Y., Liu, Y., Wang, Y., Cai, Y., Gu, Z., Liu, Z., and Dai, Z. Yi: Open foundation models by 01.ai, 2024.
J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 8359-8388. PMLR, 21-27 Jul 2024.
Alzahrani, N., Alyahya, H., Alnumay, Y., AlRashed, S.,
Cook, J., Rocktäschel, T., Foerster, J., Aumiller, D., and Wang, A. Ticking all the boxes: Generated checklists improve Ilm evaluation and generation, 2024.
Alsubaie, S., Almushayqih, Y. Mirza, F., Alotaibi, N., AI- Al-Twairesh, N., Alowisheq. A., Bari, M. S., and Khan,

---

Investigating Non- Ton-Transitivity in LLM-as-a-Judge
Czarnecki, W. M., Gidel, G., Tracey, B., Tuyls, K., Omid- shafiei, S., Balduzzi, D., and Jaderberg, M. Real world games look like spinning tops. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 17443-17454. Curran Associates, Inc., 2020.
Li, T., Chiang, W.-L., Frick, E., Dunlap, L... Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From crowdsourced data high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin. C., Liang, P., and Hashimoto, B. Alpacaeval: An automatic evaluator of instruction-followinn models, 2023.
Duan, J., Li. J., Baba, Y., and Kashima, H. A generalized model for multidimensional intransitivity. In Kim, J., Shim, K., Cao, L., Lee, J.. Lin, X., and Moon, Y. (eds.), Advances in Knowledge Discovery and Data Mining 21st Pacific-Asia Conference, PAKDD 2017, Jeju, South Korea, May 23-26, 2017, Proceedings, Part II, volume 10235 of Lecture Notes in Computer Science, pp. 840- 852, 2017. doi:
Lin, B. Y., Deng, Y., Chandu, K., Ravichander, A., Pyatkin, V., Dziri, N., Bras, R. L.., and Choi, Y. Wildbench: Bench- marking LLMs with challenging tasks from real users in the wild. In The Thirteenth International Conference on Learning Representations, 2025.
3-119/778--11-772223- 65.
Dubois, Y., Liang, P., and Hashimoto, T. Length-controlle alpacaeval: simple debiasing of automatic evaluators. In First Conference on Language Modeling, 2024.
Liu, Y., Zhou, H., Guo,, Z., Shareghi, E., Vulić, I., Korhonen, A., and Collier, N. Aligning with human judgement: The role of pairwise preference in large language model evaluators. In First Conference on Language Modeling, 2024.
Elo, A. The USCF Rating System: Its Development, Theory, and Applications. United States Chess Federation, 1966.
Liusie, A., Manakul, P., and Gales, M. J. F. LLM com- parative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models. In Graham, Y. and Purver, M. (eds.), Proceedings of the 18th Conference of the European Chapter of the Associa- tion for Computational Linguistics, EACL 2024 Volume 1: Long Papers, St. Julian's 's. Malta, March I 17-22, 2024, pp. 139-151. Association for Computational Linguistics, 2024.
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., and Ahmed, N. K. Bias and fairness in large language models: A survey. Computational Linguistics, 1097)19977 September 2024. doi: 10.1162/coli a 00524.
Gemini Team Google. Gemini: A family of highly capable multimodal models. arXiv preprint 1rX1v:2312.11855, 2023.
Jang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C.,, Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A,, Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. F. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023.
Meta AI. Introducing meta llama 3: The most capable openly available Ilm to date, 2024a.
Meta AI. Introducing llama 3.1: Our most capable models to date, 2024b.
OpenAI, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Demnison, C., Farhi, D.,, Fischer, Q., Hashme, S., Hesse, C., Józefowicz, R., Gray, S., Olsson, C., Pa- chocki, J., Petrov, M., d. O. Pinto, H. P., Raiman, J., Sali- mans, T., Schlatter, J., Schneider, J., Sidor, S., Sutskever, I., Tang, J., Wolski, F., and Zhang, S. Dota with large scale deep reinforcement learning, 2019.
Karpinska, M., Akoury, N., and lyyer, M. The perils of using Mechanical Turk to evaluate open-ended text gen- eration. In Moens, M.-F,, Huang, X., Specia, L., and Yih, S. W.tt (eds.), Proceedings of the 2021 Conference on Empiriaal Methods in Natural Language Processing, pp. 1265-1285, Online and Punta Cana, Dominican Re- public, November 2021. Association for Computational Linguistics. doi: 10.18333/11/02/1 1. www.leenmaaaa..... main.97.
OpenAL, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., and othres. Gptt-4 technical report, 2023.
Khan, A., Hughes, J., Valentine, D., Ruis, L.. Sachan, K., Radhakrishnan, A., Grefenstette, E.,, Bowman, S. R., Rocktäschel, T., and Perez, Debating with more persua- sive LLMs leads more truthful answers. In Salakhut- dinov, R., Kolter, Z.,, Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings the 41st International Conference on Machine Learning, vol- ume 235 of Proceedings of Machine Learning Research, pp. 13662-23733, PMLR, 21-27 Jul 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instruc- tions with human feedback. In Koyejo, S., Mohamed, S.,
10

---

Investigating Non- Von-Transitivity in LLM-as-a-Judge
Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, vol- ume 35, pp. 27730-27744. Curran Associates, Inc., 2022.
White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Dey, S., Shubh-Agrawal, Sandha, S. S., Naidu, S. V., Hegde, C., LeCun, Y., Goldstein, T., Neiswanger, W., and Goldblum, M. Livebench: A challenging, contamination LLM benchmark. In The Thirteenth International Con- ference on Learning Representations, 2025.
Pezeshkpour, P. and Hruschka, E. Large language models sensitivity to the order of options in multiple-choice ques- tions. In Duh, K., Gomez, H., and Bethard, S. (eds.), Findings of the Association for Computational Linguis- tics: NAACI 2024, pp. 2006-2017, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.186653 winniinnnnnnall......
Xu, C., Sun, Q. Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. WizardL.M Empowering large pre-trained language models to follow complex in- structions. The Twelfth International Conference on Learning Representations, 2024.
Qwen Team. Introducing qwenl.5, 2024.
Raina, V., Liusie, A., and Gales, M. Is LLM-es-a-juge robust? investigating universal adversarial attacks on zero-shot LLM assessment. In Al-Onaizan, Y., Bansal, M., and Chen, Y.NN. (eds.), Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Pro- cessing. pp. 7499-7517, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10. 10.18655//1/2024emmlppp
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing. E.,, Zhang, H., Gonzalez, J. E., and Stoica, Judging LLM-as-a-judd with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Zhou, H., Wan, X., Liu, Y., Collier, N., Vulić, I., and Ko- rhonen, A. Fairer preferences elicit improved human- aligned large language model judgments. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1241-1252, Miami, Florida, USA, November 2024a. Association for Computational Linguistics.
Saito, K., Wachi, A., Wataoka, K., and Akimoto, Y. Ver- bosity bias in preference labeling by large language mod- cls, 2023.
Samvelyan, M., Raparthy, S. C., Lupu, A., Hambro, E., Markosyan, A. H., Bhatt, M., Mao, Y., Jiang, M., Parker- Holder, J., Foerster, N., Rocktäschel, T., and Raileanu, R. Rainbow teaming: Open-ended generation of diverse adversarial prompts. In The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024.
Zhou, H., Wan, X., Proleev, I.., Mincu, D., Chen, J., Heller, K. A., and Roy, S. Batch calibration: Rethinking cali- bration for in-context learning and prompt engineering. In The Twelfth International Conference on Learning Representations, 2024b.
Vinyals, O.,. Babuschkin, I., Czarnecki, W., Mathieu, M., Dudzik, A., Chung. J., Choi, D., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I, Huang, A., Sifre, I.., Cai, T., Agapiou, J., Jaderberg, M., and Silver, D. Grandmaster level in starcraft using aulti-agent reinforcement learning. Nature, 575:350- 354, 10 2019. doi: 10. 10.0038845556-01--77222
Zhu, B., Frick, E., Wu, T., Zhu, H., Ganesan, K., Chiang, W.-L., Zhang, J., and Jiao, J. Starling-7b: Improving helpfulness and harmlessness with RLAIF. In First Con- ference on Language Modeling, 2024.
Wang, P., Li, L.,, Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., and Sui, Z. Large language models are not fair evaluators. In Ku, I..W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis- tics (Volume I: Long Papers). pp. 9440-9450, Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10. 105653//1/2222aal--looo
Wei, J.. Wang, X., Schuurmans, D., Bosma, M., ichter, b., Xia, F. Chi, Le, Q. V., and Zhou, D. Chain-of of-thought prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Informa- tion Processing Systems, volume 35, pp. 24824-24837. Curran Associates, Inc., 2022.

---

Investigating Non- Von-Transitivity in LLM-as-a-Judge
A. LLM Details.
In this section, we provide detailed information about all models participating in the ranking evaluation for our experiments.
A.1. Participating LLMs.
The experimental model set consists of 20 LLMs encompassing a range of top proprietary models, large open-source models, and small open-source models. All models are concurrently presented on the AlpacaEval leaderboard and the Fully Style-Controlles Chatbot Arena (2024/09/15). The AlpacaEval leaderboard supplies pre- rre-generated outputs for each model on the AlpacaEval dataset, allowing us to avoid the computational costs associated with output generation and focus solely on the costs involved in the evaluation process. The Fully Style-Controlled Chatbot Arena provides human preference rankings, which we use reference for calculating the agreement. A detailed list of participating LLMs is presented below:
Proprietary models includes four OpenAl models:
gpt-4o-2024-05-13.
gpt4_ opt-4-turbo-20222 cloude-3-opus-2224022 claude-3-sonned claude-S-somet-2222222 (Anthropic, 2023; 2024); two Mistral mod- mi
(OpenAI et al., 2023); three Anthropic models: claude
(Jian et al., 2023); one Google model: gemini-pro (Gemini Team Google, 2023); and one Yi model: yil-arrge-reeiieee.. (OLAA, 2024).
els: mnistral-large--e
Large open-source models includes yi Yi-34B-Chat (01.Al al., 2024), L.lama-3, (Meta AI, 2024b), 1-ma--37055 17CB-IInntruc (Meta AI, 2024a), Qwenl 5-72B-Cha (Qwen Team, 2024),
wizardlm-70b (Xu et al., 2024).
Small open-source models includes Meta- ama 3-8B-Irstruct (Meta AI, 2024a), vicuna 13b (Chiang et al., 2023), Starling-LM-
ama (Zhu et al., 2024), Mistral-78-Instzutt 2 (Jiang et al., 2023).
A.2. Selection of Representative Model Triplets across Scenarios.
For each scenario, we select representative model triplets based on the win rates of participating models (shown in parentheses) from the AlpacaEval leaderboard:
1. LL: GPT-44-2022405--13 (51.3%) as V.A QWENI.5-72B-OO 5-722BCHAT (26.5%) as mn. and MISTRAL-7B-INSTRUCT v0.2 (14.7%) as mc.
2. LM: GPT-40-2024-05-13 (51.3%) as m.. QWENI.5-72B--- (26.5%) as mn, and CLAUDE-3-SONNET 20240229 (25.6%) as me.
3. ML: Y1-34B-CHAY (29.7%) as mA, QWENI 5-72BBCHAT (26.5%) mB, and MISTRAI--7B-NNSTRICTVVO (14.7%) mc.
4. MM: QWEN1 5-22B-CHT (26.5%) as mA: SOO-RR---SNNNE-224020000 (25.6%) as mB, and GPT-4-0314 (22.1%) mc.
B. Non-Transitin in Preference
B.1. Conditions for Non-Trorne Von-Transitivity
In this section, we define the conditions under which non-transitivity arises in pairwise model comparisons. Consider a
triplet of models, (m mB, me), and the corresponding pairwise comparisons on instruction I,:
J(mA > mp I)), J(mB > me Ic), J(m > mg I:)
where (m, > my I,) denotes the preference of the judge that model m,, outperforms model my under instruction I,-
Non-transitivity occurs if the results of these comparisons form any of the following patterns:
m. > mB, mB mc; MA mc
12

---

Investigating Non- Non-Transitivity in LLM-as-a-Judge
m m. mB mc, Ma me
mA > mB. mg mc: MA mc
m > m. mB mc: A < mc
Ma ~ m.g mB > mc: M ~ Mc
MA ~ m.g mB > mc, A < me
MA ~ mB, mB mc, MA - Mc
MA ~ m, MB mc: MA X MC
MA m., MB < TTC, MA > MC
MA m. mg < Mc, MA Mc
MA < M. mB MC, MA > MC
MA < M.. mg MC, MA Mc
MA < m, mB < MCC M. > MC
mA < m. mg < mC, M ~ mc
where means the left side wins against the right, < means the left side loses the right, and represents a tie between the two sides.
Threshold Setting. In practice, given the continuous nature of probability estimates, ties where Jm, > my T,) 0.5 occur with negligible frequency. Therefore, we introduce the following thresholds to determine the outcome of pairwise comparisons:
1. If0.475 S Jm, my 4) S 0.525, the outcome is treated as a tic (~)).
2. If J(mz > my T.) > 0.525, the outcome classified as win for M₂ (>)).
3. If J(m, > my 1,) < 0.475, the outcome classified as loss for M2 (-).
Notably, even without threshold settings, the non-transitivity patterns observed across all four scenarios remain consistent with Section 4, which is shown in Appendix B.3.
B.2. Results Under Varying Judges and Datasets
To further assess the robustness of our findings, we evaluate the same four scenario settings on the AlpacaEval dataset using GPT--4-mini the judge. As shown in Table 3, the results align closely with those obtained using GPT-4-Turbo: the SNTD metric confirms that non-transitivity increases the performance gap between model pairs narrows. In addition, based on the Chatbot Arena rankings (Chiang et al., 2024), GPT-4o-mini is ranked higher than GPT-4-Turbe suggesting that it serves as a stronger judge. Across almost all scenarios, GPT--4-mini exhibits lower SNTD and PNT values than GPT-4-Turbo, indicating more transitive judgments. These results provide further empirical support for our claim that stronger judges tend to exhibit less non-transitivity.
To evaluate whether this pattern holds across datasets, we also conduct experiments on the Arena-Hard-Aun (Li al., 2024) dataset, which consists of 500 high-quality prompts curated from Chatbot Arena. Due to computational constraints, we sample 200 prompts for evaluation. We utilize GPT-4-Turbo, GPT-3.5-Turbo, and GPT-4o-mini judges under the four-scenario framework, selecting models based on their rankings in the Arena-Hard-aa leaderboard. As shown in the Table 4, the results remain consistent with those observed on AlpacaEval: the SNTD metric confirms that non-transn -onttrannitivity intensifies as the performance gap narrows, particularly for stronger judges. contrast, GPT-3.5-Turbo exhibits high non-transitivity across all scenarios, due its inability to reliably distinguish quality differences among the outputs. This consistency suggests that the non-transitive behavior of LLM judges is robust across datasets.
"Specifically, gpt-4o 1ott-4omiime fidommnn---- 2022 4-2-18 used for evaluation.
13

---

Investigating Non- Ton-Transitivity in LLM-as-a-Judge
Table We measure transitivity the AlpacaEval dataset across four scenarios, evaluated by GPT-4o-mini. Orange cells indicate
maximum PNT/SNTT values (highest non -n-traasitiivity;; blue cells indicate minimum PNT/SNTD values (highest transitivity). Consistently, non-transitivity be observed evaluated model performance becomes similar and the highest non-transitivity occurs when the performances all three models are similar.
GPT-4o-min mini
Scenarios
Models
PNT
SNTD
LL m.A > mB
mA 1pt-41-2024-05-13
mg Qwen ---21--ha7---
3.35
0.1006
mB > mc
#Lsttaa-/BB-Instrutt-V
LM
gpt-44--0244-05-3
Sweni.5-72tt-Che.
3.60
0.1070
mB mg
mC claude-3-sonset2222222
ML
MA Yi-34B-Chat
MA mB
mB Qwen1 Sven1.5-72B-Chat
3.98
0.1036
mB 2 mc
Mc
MM
MA Qwen1. 5-72B-hhat
MA R mg
mg claude--3ssonee-
3.60
0.1173
mB R mc
Mc 9pt-4-0314
Table 4. We measure non-transitivity on the Seena-HardAutte dataset across four scenarios, evaluated by GPT-4-Turbo, GPT-3.5- Turbb,
GPT-4o-mini, Orange cells indicate maximum PNT/SNTD values (highest tonttraniitiiity); blue cells indicate minimum PNT/SNTD values (highest transitivity). We observe similar pattem on the AlpacaEval dataset.
GPT-4-Turbo
GPT-3.5-Turbo
GPT-4o-mini
Scenarios
Models
PNT
SNTD
PNT
SNTD
PNT
SNTD
LL
TA gpt-44o-0024-05-12 5-
> mB me
2.00
0.0820
17.00
0.2071
1.00
0.0813
mc #1stral-7B-Instrueet
LM
MA gpt-44--0024-5-113
MA mB
mB Mistral-Large-2444
3.00
0.1083
17.50
0.2002
1.50
0.0880
me mc
MC Swen1.5-72B-Cha..
ML
MA misttallll.rrge
TIA me
mı Qwen1.3-72BB-hatt
2.50
0.0945
24.50
0.2370
0.1085
mB > me
MC Mistral-7B-IIntrrun
MM
MA gpt-4-0613
TRA A mB
T&B Mistral-Larqe-220
5.00
0.1270
28.00
0.2294
5.00
0.1181
= Qwen1 Sxen1.5-728-Chat
14

---

Investigating Non- Ton-Tiansityvity in LLMM-ss---Jdde
Transttive (Consistent)
Non-transtiive (Consistent)
Transitive (Ambiguuus)
Non-transitive (Ambiguous)
22
57
73
16
156
0.8
Proportion
451
413
566
521
0.6
515
0.4
620
615
617
352
289
280
289
280
215
215
2,8
3/
LL
LM
ML
MM
LM ML
MM
LC
LM
ML
MM
LMM
ML
MM
GPT-
GPT-4 (Random Choloc)
GPT-3.5
GPT-3.5 (Random Choice)
Figure 7. Proportion of (non-)transitiv instructions across all scenarios (without the threshold of ties), evaluated by GPT-4-Turbo and GPT-3.5-Turbo. When evaluating model triplcts with GPT-335- Turubo judge, over 96% of instructions exhibit position bias effects. In contrast, GPT-4-Turbo demonstrates substantially higher evaluation consistency.
B.3. Results with Preferences without the Threshold of Ties
Table We measure non-transitivity (without the threshold of ties) on the AlpacaEyal dataset across four scenarios, evaluated by GPT-4-Turbo and GPT-3.5-Turbo. Orange cells indicate maximum PNT/SNTD values (highest non-transitivity); blue cells indicate minimum PNT/SNTD values (highest transitivity). When using GPT-4-Turbe as the judge, more non-transitivity can be observed as evaluated model performance bocomes more similar and the highest non-transitivity occurs when the performances of three models are similar: however, GPT-3.5-Turke does not exhibit this pattern.
GPT-4-Turbo
GPT-3.5-Turbo
Scenarios
Models
PNT
SNTD
PNT
SNTD
LL
mAA = =gppt------------ gpt-44--004405-11
MA >
Swen1.5-72B-2
0.25
0.1121
1.12
0.2654
mg > mc
mc Mistral-1B--nstruct--
LM
gpt-44--0224055-11
TIA > me
m8 Qwen1.5-72B-CCn
1.24
0.1336
0.25
0.2586
mB R mc
TCC clause-3-someet22024000
ML
174 Yi-34B-Char
A & mB
m = Qwen1.5-72Br
5-72B-Chat
0.99
0.1215
0.2625
mg 2 mc
mc
MM
1 @wenl airem.5572255
mp = cl 5-72B-Chat
A & mB
2.86
0.1431
1.99
0.2629
&
gpt 003144
We observe the pattern from the table 5 as in the main text, which is with the threshold for ties. When GPT-4-Turbo as the judge, both PNT and SNTD increase the performance gap between any pair of models, (m mB) (m,, decreases. In cases where all three models exhibit similar performance, such as in scenario MM, the incidence of non-transitivity rises significantly. We attribute this to the increased uncertainty judges face when assessing quality differences between similar outputs. When the comparisons between m, and m, my and mc, and m. and me are all uncertain, non non-transitivity reaches its highest level. Replicating our evaluation with GPT-3.5-Turbe as judge reveals an intriguing pattern: while the PNT remains minimal across scenarios, the consistently high SNTD values indicate substantial non-transitivity. This observation motivates us define the tie threshold, ties can serve as an indicator of model uncertainty.
To explain the low number of hard non-transitive cases when using --Trbb as the judge with position switching in Figure 5, we hypothesize that GPT-3.5-Turie is also affected by other biases (Zhou et al., 2024a), such as verbosity bias (Saito et al., 2023) and token bias (Alzahrani et al., 2024). Since GPT-3.5-Turke struggles to accurately assess the quality of outputs, these combined biases influence the judge's preferences. As a result, even though position switching mitigates the position bias, the averaged preference is still not determined by the actual quality of the outputs but rather by other fixed
15

---

Investigating Non- Von-Transitivity in LLM-assa-Judge
biases in the prompt, leading to transitive preferences. This observation also motivates to define the threshold, it can be used reduce the impact of other biases.
B.4. Derivation of Expected Win Rate
The Bradley-Terry model (Bradley & Terry, 1952) provides probabílistic framework for estimating pairwise win rates based on these latent quality scores. Specifically,, the probability that model m, outperforms model my instruction 1; is given by:
0 110 my 4,) + e (1 - = o AR).
(6) (8) where we denote SAB = VA 7B observations Φ as:
(6)
as the quality gap. Conversely, this quality gap can be calculated from empirical
(0) 8 AB = In
(
mJ. 1i)
(10)
(8)
my. mJ,
Based that, we can estimate the expected win rate à under transitivity between any models from triplet A,,, m, (,,,, mc) by utilizing the observed win rates between the other two pairs. For instance, to estimate the win rate for model m, beating model my on instruction I, without direct observations, we assume that the observed win rates for the remaining pairs reflect true performance differences and compute the estimated win rate as:
Ho .0 mJ. It)
(11)
1+ - -1 - - -)) -100)))) 1te 1620-
B.5. Heatmap Implementation
In this experiment, we aim to investigate the relationship between non- non-transitivity and the performance gap between two models being compared. From the pool of 20 models, we generate all possible tuples (m mp, mc) by computing 3) 20! 840 permutations. For each tuple, we calculate the number of hard non-transitive cases and the 17!
degree of soft non-transitivity. The results are visualized between model m and model ma, measured by their win-rate difference AlpacaEval. Similarly, the y-axis represents the win-rate difference between model mg and model mc. A positive win-rate difference indicates that the former model performs better, whereas a negative difference suggests that the latter outperforms the former.
2D heatmap, where the x-axis represents the performance gap
According to the AlpacaEval leaderboard, yi yi-laag large preview achieves the highest relative win rate of 57.5%, while vicuna-13b 13 records the lowest at 5.8%. This establishes win rate differential range of [-51.7%, +51.7%]. which we partition into 35 x 35 grid. For each grid cell, we compute the mean number of PNT and SNTD across all possible model triplet permutations. We apply Gaussian filter (a 1) to reduce noise in the resulting data, and then perform quadratic interpolation to generate the final heatmap.
B.6. Preference Distributions of Judge
All scenario assumes that m outperforms mp, my outperforms mc, and mA outperforms mc- Consequently, expect the judge's preference distribution to exhibit heavy-tailed pattern concentrated around 1. In scenario LL, because the models differ significantly in performance, the judge should tend to select the superior output. However, under the random assignment setting, GPTT-355-Turhe exhibits a U-shaped distribution across all scenarios (Figure 8), validating that it fails to distinguish response quality and is instead primarily driven by position bias. As a result, after applying position switching, its preference distribution changes significantly, forming sharp peak at 0.5 while rapidly decaying away from it, leading to a large number of ties.
By contrast, 4PT--Turbb's distributions vary across scenarios (Figure 9). In scenario LL, where m., MB, and mc have large performance gaps, the distribution precisely follows a heavy-tailed pattern concentrated at 1, indicating that when GPT-4-Turbo perceives a substantial quality difference, it strongly favors the superior response. In LM and ML scenarios, where one model pair has a clear performance gap while the other is closer in quality, increased uncertainty arises when evaluating the latter, causing the tail to shift towards In MM, -PT-4-Turbo also exhibits U-shaped distribution. However, unlike GPT-3.5-Turbe retains 38% of its preferences distributed across the full range from
1, demonstrating that its
16

---

Investigating Non- Non-Transitivity in LLM-as-a-Judge
Prefennce Valuo
Profermoo Value
Pereentagg (10)
Prefosaue
Percenngh (75
(S
Prefevncc
Peneetinge (10
Figure 8. Preference distribution of G.T-3.55-Tufi across scenarios (from top bottom: LL, LM. LM, MM). (Left) Distribution with
random assignment. (Right) Distribution with position switching.
17

---

Investigating Non- Ton-Transitivity in LLM-as-a-Judge
Porernus
Prefennce Valuo
Profermnc Volue
Percentage.co (10
Porccatue
Prefosaee
Prdfezzas Vilac
Pertringh (75)
Ponentue (S))
Preferncc
Penentinge (10
Parentage (%)
64
Figure 9. Preference distribution of GPT-4-Turbb across scenarios (from top to bottom: LL, LM. LMM MM). (Left) Distribution with random assignment. (Right) Distribution with position switching.
18

---

Investigating Non-Transitivie in LLM-as-a-Judge
preferences are guided by reasoning rather than solely by position bias. Thus, position switching smooths its preference distribution while preserving considerable proportion of decisive judgments (non-ties), reflecting that Gur-o-Turbo still distinguishes quality differences
This also explains why position switching is least effective in Scenario MM, reducing non- contrrnnittivity by only 17%.
C. Additional Experimental Results
C.1. Full Pairwise Comparison Matrix (Position Switching and Two API Calls per Order)
Model B (Ma)
622000000 -ndd-- sadd-----ap -pppp
67z0rroz tamner spore
arussi di carry Moo
Z0 monuuusseeee EL manu
R2- WT suppes
oud- nunuu
ye-lrgg--rexttew-eeet
Bxd
677
834
0.9
gpt to 2024 45 13 sual
6:42
gpt-4 4 curbo 2024 oa-de
0.0
0.4
0.8
Mate--lama-3 1-2258- Turbbon
instruct
0.7
cleade 9-0pm-0000-000000
YI-34B Chatioo
0.0
XX5
0.02
8.5
67
0.6
Model A (M.)
Qweet 5-720-hhm Mo
A.77
0.33
RA4
622
810
0.7
0.5
2F-4-081441
miszal large 2410
0.4
mistral medium
8.10
141
0:17
0.3
geminiegu
3.52
0.23
380
0.4
claude 2
014
3.58
6.28
0,2
wiaaddnn-700 700
1.17
C35
Strring-LL--77--alhh M--71--llhh
0.1
Misual-BB- 70-
vicuma-1:6
XG C12
Figure 10. Win rate matrix for 20 models using default settings (Position Switching and Two API Calls per Order).
C.2. Position Switching and Multiple API Calls Reduce the Occurrence of Non-transitivity at the Model Level.
We hypothesize that the absence of observed hard non-transitivity in Figure 10 is due to the of position switching and two API calls per order, which help ensure the consistency of judgments. To validate this hypothesis, we adopt more aggressive approach by randomly assigning positions for each evaluation, reducing the process to single API call to mitigate position bias. However, since the preference between each model pair for given instruction is determined by log probability rather than a binary label (0 or 1), we argue that random assignment may not fully eliminate position bias. As a result, this setup is expected to perform worse than position switching, leading to lower judgment consistency compared to the original setting.
To reduce computational costs, the judge's s new preference can be interpreted as a random sample from the four API calls made in the original experiment. In other words, in this ablation experiment, the judge's 's preference is equivalent to selecting one random sample from the pre-computed preferences in Section 4.2.
Figure 1l presents the corresponding win rate matrix from this ablation. In contrast to Figure 10, we now observe the occur- rence of hard non- non-transitive case at the model level. Specifically, Qwen1. 5- 72B-Chat outperforms Yi-34B-Chat, and -1-3aB- outperforms claude- 3- opus-20240225 However, claude- Laaude---opuse -o-oopuu2220000 outper- forms Qwen1. S-72B-Chats thus exhibiting a clear case of -on-transitivity.
19

---

Investigating Non- Non-Transitivity in LLM-as-a-Judge
Model B (Mg)
cqpiy soape cit WARTT TWW
pegpaa wag aad dlbeeck
ocan to wow.eeoooggggg.o casi---paa sppaa
Ftx pount du prssie
att
oGam coc Casntar
organa pooro
Esprap
SP10 prectee
0.5
do 45-445-11
00
Mete-lammdd 1-0000 Trat--ttrr----erre
Mesa Llans 3.708 Iistrcc Ki5
A32
ABC
XBC
550
Clode 20240220
M1-301-Che
Model A (MG)
QweaLS 728 Cluu K.2.
6.54
6.5
6.0
XG
OAT
675
dasde 20240220
0.5
gpr c0
Wete-lanee-----mmeere K11
321
632
36
6/68
X54
0225
RC
6.3
344
560
andiann
0.5
gentri-p Ki5
3.55
300
caode2 412
02
wiurdas770
Sodiag M-77happh L1
812
024
327
3/
500
01
vicana-130
Figure 11. Win matrix for 20 models using ablated settings (random assignment and single API call per order). Hard non con-transitivity is observed compared to Figure 1. For instance, Qwen1.5-7225 -Chhtt outperforms Yi-34B-CaB -hatt, and yi el claude-33 saude-3-opus Sippuu22024025 However, aude-3 --oees---------- -op-ppp2220222 outperforms Qwen1 1eeml.5-720 -5-728-Ches highlighting the presence of non-transitive relationships among the models.
outperforms
To further verify that the observation of non-transitivity in the ablated setting is not merely due to randomness, we repeat this ablation experiment 50 times. We quantify the degree of soft non-transitivity in the win rate matrix in manner similar to Equation 3, but applied at the model level. Specifically, for a set of 20 models, we first compute all possible permutations of triples Am m,, mc). For each triplet, we sequentially select two pairs of models and extract their corresponding values from the win rate matrix as ground truth. We then calculate the expected win rate for the remaining model pair and measure the associated SNTD the model level. Finally, we average the results across all permutations to assess the overall non-transitivity in the win-rate matrix.
Table Comparison of the degree of soft non-transitivity between the original and random assignment settings. The values represent the mean SNTD, with the standard deviation reported for the random assignment setting based on 50 independent trials.
Experiment Setting
SNTD
Position Switching and Two API Calls per Order
4.00 x 10-4
Random Assignment and One API Call (50 times)
(5.38 + 0.04) x 10
As shown Table 6, the degree of non non-transitivity in the ablated experiment is significantly higher than in the original experiment. This finding demonstrates that by employing position switching and multiple API calls, we can improve the consistency of the judge's evaluations and thereby reduce the occurrence of non-transitivity at the model level.
20

---

Investigating Non- Von-Transitivity in LLM-as-a-Judge
C.3. More Prompting Strategies
We evaluate six prompting strategies Scenario MM encourage the judge exhibit more transitive preferences from prompting perspective (R=²). The prompt templates are provided in Appendix G.I.
1. Direct Comparison: Standard binary choice comparison identical to our previous experimental setup, serving the baseline.
2. CoT Comparison: Requires the judge to output its reasoning through Chain-of-Thought (Wei et al., 2022) before making a decision.
3. Direct Comparison with Checklist: Provides a detailed evaluation checklist (Cook et al., 2024) for the judgment without explicit reasoning.
4. CoT Comparison with Checklist: Combines detailed evaluation checklist with Chain-of-Thouugh reasoning before judgment.
5. CoT Comparison (Tie Allowed): Extends the binary choice three options by introducing the possibility of ties, while maintaining the Chain-of-Thought reasoning process.
6. CoT Comparison with Checklist (Tie Allowed): Incorporates both the three-choice option and evaluation checklist while preserving Chain-of-Thoughe reasoning.
Table Comparison different prompting strategies, judged by GPT-4-Turbo. Red cells indicate the lowest consistency (most affected by position bias); green cells represent the highest consistency (least affected by position bias). Orange cells denote the highest number of non-transitive cases (greatest non- non-tramsinn transitiviitt), while blue cells indicate the lowest number of non-transitive cases (greatest transitivity). The values in parentheses represent the number of non-transitive cases in consistent instructions (left) and ambiguous instructions (right).
Method
A vs B (consist.)
B vs C (consist.)
vs C (consist.)
# of Consistent of Non-trans. Instr.
# of Non-trans. (w/o. threshold)
(w. threshold)
Direct
473
496
476
217
(1, 67)
(1, 22)
Direct w. Chk
478
506
440
227
(0, 64)
(0, 23) 23)
CoT
572
577
560
301
(1, 152)
(1, 46)
CoT Chk
548
571
535
268
(5, 172)
(5,
CoT Tie
474
496
493
210
(5, 139)
(5, 87)
CoT w. Tik&
466
479
456
181
(10, 183)
(10, 129)
For the checklist-based method, we first use GPT-4-Turbo to generate a checklist- a set of YES/NO questions assessing different aspects of the given instruction. The corresponding prompt is provided in Appendix G.2.
As shown Table 7, providing the judge with a checklist slightly reduces non-transitivity. This aligns with our earlier assertion that the judge's latent comparison criteria are inherently non-transitive for closely matched models. While introducing explicit criteria helps guide the judge toward more transitive preferences, the effect remains limited, likely because the automatically generated checklists lack the granularity capture subtle output differences.
Meanwhile, although Chain-of-Thoughh prompting reduces position bias and improves overall preference consistency, it increases trantiinsty for ambiguous instructions and can introduce additional non-transitiv cases even in consistent instructions. Additionally, when combining CoT with a checklist, we observe more inconsistency, suggesting that CoT elicits the judge's s latent reasoning criteria, which may conflict with the explicitly provided checklist. Furthermore, allowing the judge to declare ties increases non-transitivity, as the judge may opt for ties instead of identifying subtle differences between outputs.
D. Soft Bradley-Terry Model Yields More Accurate Rankings
We explored three methods for computing Wa, in Equation (7). The first method, referred to as hard-BT, directly derives discrete win rates from the judge's 's continuous preferences. In this approach, if J(m, > m; I₂) > 0.5, the outcome
counted as a win (1); if J(m, > m, Ix) < 0.5, is counted as a loss (0); and J(m, > m, Ix) = 0.5, it is considered a tie (0.5).
21

---

Investigating Non-Transitivity in LLM-as-a-Judge
The second method, rounded-BT, incorporates threshold to refine the win/loss definition. Specifically, if J(m, > my TR.) 0.525, it is considered win (I); if J(m; my Tx.) 0.475, it is considered loss (0); and if J[m; my Te) falls within the range [0.475, 0.525], it is treated tie (0.5).
The final method, soft-BTT, follows the formulation presented in the main text. Instead of discretizing preferences into fixed categories, directly uses the judge's s continuous preference scores to compute Wi,, allowing for more nuanced representation of the relative strength between models:
W Jmm: my Tx)-
I&EZ
We evaluate these methods by computing rankings from a round-robin tournament involving 20 models, using GPT-4-Turbo the judge, and measuring their correlation with the Chatbot Arena rankings metrics.
Table 8. Comparison between Round Robin based framework with Bradley- Terry model and AlpacaEval 2.0.
RR + Soft-BT
RR + Hard-BT
RR Rounded-BT
Spearman Correlation 85.4% %
84.4%
84.8%
Kendall Correlation
68.4 %
66.3%
67.4%
Table 8 shows that soft-BT produces the most aligned ranking, demonstrating its ability to better capture the relative strength of models from continuous preferences.
E. Swiss-Wise Iterative Matchmaking tournaments
Algorithm 1 Swiss Swiss-Wise Iterative Matchmaking (SWIM) tournament
1: Input: M unranked models, dataset I and judge model M,-
2: Output: An ordered ranking of all M models.
3: R + empty set Ø to store ranked models
4: U < set of all M models
5: + random model from U
6: + (U(X), + U
while U # 2 do
8:
P random model from U
9:
U- U U. (P)
10:
+ |R], c ((max(logg()) 1)]
11:
X + a random model from H
12:
< {X}
13:
for all 1, I do
Compute J(mp~ mx It)
end for
16:
8 update BT coefficient for R (P)
17:
for = to do
18:
O + arg minoer 130 3rl
19:
T - T (0)
20:
for all 1 E I do
Compute J(mp mo It)
22:
end for
23:
update BT coefficient for RUC (P)
24:
end for
25: R - R U (P}
26:
end while
22

---

Investigating Non- -raniiiiiitivity in LLM-as-a-Judge
F. ELO Scores
We conduct round-robin tournament to obtain pairwise comparisons and apply the Bradley-Terry model to compute ratings, which are then converted Elo scores.
Table Evaluation Results of LLMs in Fully Coner-ontdollled Chatbot Arena, Round Robin Tournament and AlpacaEval.
Round-Robin BT
AlpacaEval 2.0
Model Names
FSC Arena Elo
Elo
LC Elo
Win Rate
LC Win Rate
gPt-40-2024-05-11
1262
1325
1227
51.3%
57.5%
9PC-3-tturoo-2224400099
1241
1306
1217
46.1%
55.0%
gpt-4-1106-preview
1234
1337
1206
50.0%
50.0%
yi-large-prevveew
1204
1377
1205
57.5%
51.9%
claux6-3-0puS-2244029
1238
1180
1156
29.1%
40.5%
1250
1264
1136
39.1%
39.3%
gpt4_
1200
1137
1117
22.1%
35.3%
cLoud=33-oomet-2224022
1197
1152
1110
25.6%
34.9%
Qwen1.5-//BB-hhat
1148
1168
1108
26.5%
36.6%
1193
1210
1093
33.2%
34.4%
mistral-1arge-2402
1158
1110
1090
21.4%
32.7%
claude-2
1144
1043
1060
17.2%
28.2%
mistral-medium
1141
1109
1059
21.9%
28.6%
Yi-34B-Chat
1100
1169
1026
29.7%
27.2%
gemini-pro
1132
1074
1020
18.2%
24.4%
-lama-3-8B-Instrucc
1141
1110
988
22.6%
22.9%
vizardlm-70b
1106
1036
964
14.4%
17.6%
Mistral-1B-Innttuutt-...
1067
1019
947
14.7%
17.1%
Starling-IM-7B-Bn
1083
1021
925
14.2%
14.7%
vicuna-13b
1060
800
800
6.7%
10.5%
23

---

Investigating Non-Transitivin in LL.M-as-a-Jude
Table 10. Ranking L.LMs based evaluation results from the Fully Style-Controlled Chatbot Arena, Round-Rohin Tournament, and AlpacaEval. The numbers in parentheses indicate changes in model rankings after applying the length-controlled debiasing technique, where 1 denotes an increase, + denotes decrease, indicates no change ranking. Round-Ron -ound-Robin BT
AlpacaEval 2.0
Model Names
FSC Arena Rank
Rank
Rank
Rank
LC Rank
gpt-4o-2024-05-22
3
1
2
(11 f)
gp=-4-tuubbo2024400--99
4
2 2 f)
2 f)
gpt-4-1106-previee
2
(1
3
4 4)
yl-large-preview
1
(3
I
3
clauee33-ppu--222029
7
(2
8
5
(lama=3.1-100B--nsttte-------
5
6(1
5
6(1
gpt4_0314
11
7(4
12
8 (4
clouee-3-oonnt-2222422
8
9
Qwen1.5-72B-Gre
9
90-
9
7(11)
-llma-3--70--nstrrue
6
(4
6
14
mistral-1arge-2402
10
12
(1
14
(1
claude-2
12
16
12
16
(3
mistral-medium
13
14
13
13
12(11)
Yi-34H-Chat
8
(4
7
(7
gemini-pro
15
15
15
15
Llara-3-SB-Instrucc
13
16
11
16 ()
wizardlm-70b
17
17
18
17 (1 1)
Wistral-7B-Instructto
19
19
18 1)
17
18 (1
Starling-1M4-B-aaiphee
18
18
(1 4)
19
19 (0-)
vicuna-133
20
20
20 20-)
20
20 (0-)
G. Prompt Template.
G.1. Judge Prompts
Direct Comparison Identical to AlpacaEval 2.0 (Li et al., 2023)
[System Part]
You are a highly efficient assistant, who evaluates and selects the best large language model (LLMB) based on the quality of their responses to a given instruct inn. This process will be used to create Teaderboard reflecting ieiiccing the most accurate and human- preferred answers.
[User Part]
require leaderboard for various large language models. I'll provide you with prompts given to these models and their corresponding outputs. Your task is to assess these responses, select the model that produces best output from human perspective
## Instruction
"instructioa"s """ ""[[insruution]"""
Model Outputs
Here are the unordered outputs from the models. Each output is associated with specific model, identified by unique model identifier.
"m" "cutput"; "*. ("[outpu__l] m*n
24

---

Investigating Non- Ton-Transitivity in LLM-as-a-Judge
"model_identa iiddeiiiieiie"" "M", "output"; """[output[]]
## Task
Evaluate the models based on the quality and relevance of their outputs, and select the model that generated the best output. Answer by providing the model identifier of the best model. will use your ousput as the name of the best model, so make sure your output only contains one of the following model identifiers and nothing else (no quotes, no spaces, no new lines, ...); or M.
## Best Model Identifier
Direct Comparison with Checklist (System Part]
You are highly efficient assistant, who evaluates and selects the best Large Language model. (LLMs) based the quality of their responses to given instruction and the corresponding criteria. This process will used create a leaderboard reflecting the most accurate and hunan-preferred answers.
[User Part]
require leaderboard for various large language models. I will provide you with prompts given to these models and their corresponding outputs. I will also provide one specific ovaluation checklist which contains a list of specific criteria that a good output should fulfill. Your task is to assess these responses to see whether they satisfy the requits of the checklist and select the model that produces the best output from human perspective based the provided checklist.
## Instruction
"instruction"; ""(iinsrrctiinn)"""
## Checklist
Here is the checklist that contains the conditions specified in the question for good output. The more requirements an output meets, the better considered.
checklist: """(checklist)"",
## Model Outputs
Here the unordered outputs from the models. Each output is associated with specific model, identified by unique model identifier.
"model_idde iideeiirdettiiie "m", "output": (output_]]**
"model identifier"; "M" "output"; ""[ootput_2)- "n*
Task
Evaluate the models based on the quality and relevance of their outputs, and select the model that generated the best ousput based on the checklist. Answer by providing the
25

---

Investigating Non- Von-sitasitiiityy in LLM-as-a-Judge
model identifier of the best model. will use your output as the name the best model, ao make sure your output only contains one of the following model identifiers and nothing else (no quotes, no spaces, no new lines, ...); m or M.
Best Model Identifier
CoT Comparison Identical to AlpacaEval 2.0 (Li et al., 2023)
(System Part]
You are highly efficient assistant, who evaluates and selects the best large language model (LLMs) based the quality of their responses given instruction. This process will be used to create leaderboard reflecting the most accurate and human- preferred answers.
[User Part]
require leaderboard for various large language models. I'll provide you with prompts given to these models and their corresponding outputs. Your task to assess these responses, and select the model that produces the best output from a human perspective
Instruction
"instruction": """ ""[iistruction]"""
## Model Outputs
Here are the unordered outputs from the models. Each output is associated with specific model, identified by unique model identifier.
"model identifier": "m", "output"; [""[outut_l]""
"
"model goodel_ddentifier" "M" "output": """ [output_ **"
## Task
avaluate the models based on the quality and relevance of their outputs, and select the model that generated the best output. Answer by first providing a concise explanation and then end your answer by providing the model identifier of the best output. We will the last character of your output 'output[-1] make sure you finish with the token of the model identifiers and nothing else: or 'M" (no quotes, no dots, no backticks, no new lines, ...). For example:
the name of the best model, so
#F Concise explanation some text...
### Which is best, m or M?
M
Now is your turn.
## Your answer: "Concise se expl inolanationn followed by "Which is best, m or M?"
CoT Comparison (Tie Allowed)
[System Part]
You are a highly efficient assistant, who evaluates and selects the best large language model (LLMs) based on the quality of their responses to a given instruction. This
26

---

Investigating Non-Transitivity in LLM-as-a-Judge
process will be used to create leaderboard reflecting the most accurate and human- preferred answers.
[User Part]
require leaderboard for varicus large language models. I'11 provide you with prompts given to these models and their correspording outputs. Your task to assess these responses, and select the model that produces the best output from a human perspective If you determine that both outputs are of equal quality or are unable to decide which one is better, you should indicate tie by providing the identifier 'D'.
## Instruction
"instruction"; "" """listructionn"""
Model Outputs
Here are the unerdered outputs from models. Bach output associated with specific model, identified by a unique model identifier.
"model__ identifier"; "m", "output"; ""[output_1]" 1)*""
"model_ide iideeelieettt "M", *output"; ["[[otput_2] m
## Task
Evaluate the models based on the quality and relevance of their outputs, and select the model that generated the best output. Answer by first providing concise explanation and then end your answer by providing the model identifier of the best output. If you determine that both outputs are of equal quality or cannot decide which one is better, indicate a tie by using the identifier 'D'. We will use the last character of your finish with the
output 'output 1-11' as the name of the best model, make sure token of the model identifies and nothing else: 'm', 'y' or 'D' (no quotes, no dots, no backticks, no new lines, ...). For example:
### Concise explanation
some text...
### Which best, m, or D?
M
Now is your turn.
## Your answer: "Conciss se explanation! followed by "which is best, m, or D)?"
CoT Comparison with Checklist
[System Part]
You are highly efficient assistant, who evaluates and selects the best large language model (LLMs) based on the quality their responses to a given instruction and the corresponding criteria. This process will be used to create a leacerboard reflecting the most accurate and hunan-preferred answers.
[User Part]
require leaderboard for various large language models. I will provide you with prompts given to these models and their corresponding outputs. I will also provide one specific evaluation checklist which contains a list of specific criteria that good
27

---

Investigating Non- Ton-Transitivity in LLM-assa-Judge
output should fulfill. Your task is to assess these responses see whether they satisfy the requirementa of the checklist and select the model that produces the best output from human perspective based on the provided checklist.
## Instruction
"instructica"; """ ""[instruction)""!
# Checklist
Here is the checklist ist that contains the conditions specified in the question for a good output. The more requi rementa an output meets, the better it is considered.
checklist: """[checklist)"";
Model Outputs
Here are the uncrdered outputs from the models. Each output is associated with specific model, identified by unique model identifier.
Mmodel model__denne Leeeelddddeeieeeeeeee "m", *output"; """ """output_l]""
}
"model_iee ied__didentfierrr "M", "output"; """ ""[ootput_2" 2)*""
## Task
Evaluate the models based model that generated the best output based on the checklist. Answer by first providing concise explanation and then end your answer by providing the monel identifier of the best output. We will use the last character of your output 'outputl-1/" as the
quality and relevance of their outputs, and select the
name of the best model, make sure you finish with the taken of the model identifiers and nothing else: 'm" or 'M' (no quotes, no dots, no backticks, no new lines, ...... For example:
### Concise explanation text...
### Which is best, m or M? M
is your turn.
## Your answer: "Conciss se explans iallanation in followed by "which is best, m or M?"
CoT Comparison with Checklist (Tie Allowed)
[System Part]
You are a highly efficient assistant, who evaluates and selects the best Large language model (LLMs) based on the quality of their responses to a given instruction and the corresponding criteria. This process will be used to create a leaderboard reflecting the most accurate and human-preferred answers.
[User Part]
require a leaderboard for various large language models. I will provide you with prompts given to these models and their corresponding outputs. I will also provide one
28

---

Investigating Non- Transiiiittivity in LLM-as-a-Judge
specific fic evaluation checklist which contains a list of specific criteria that good output should fulfill. Your taak is to assess these outputs to see whether they satisfy the requirements of the checklist and select the model that produces the best output from human perspective based on the provided checklist. If determine that both outputs of equal quality should indicate tie by providing the identifies '
unable to decide which is better, you
# Instruction
"instruction": "* (iinstrructoon)"")
## Checklist
Here is the checklist that contains the conditions specified the question for a good output. The more requirements an output meets, the better it is considered.
checklist: """(chcklist))"";
## Model Outputs
Here are the unerdered outputs from the models. Each output associated with specific model, identified by a unique model identifier.
"model identifier"; "m", "output"; ""upoouuu_]1""" 1)*""
"modcl_ iiniilliddntiiier "M", *output"; """ ""[outuut_2)"
## Task
Evaluate the models based the quality and relevance of their outputs, and select the model that generated the best output based the checklist. Answer by first providing then end your answer by providing
concise explanation based on the checklist the model identifier of the best output. If you determine that both outputs are of equal quality or cannot decide which one is better, indicate a tie by using the Adentifier 'D', We will use the last character of your output [utppu[[1]] as the name of the best model, nake sure you finish with the token of the model identifiers and nothing else: 'm', ...). For example:
or (no quotes, no dots, no backticks, no new lines,
### Concise explanation text...
### which is best, m, M or 102 M
Now is your turn.
# Your answer: "Concise explanatica" followed by "Which is best, M oz D?"
G.2. Checklist Generation
We follow Cook et al. (2024)'s prompt tepmplate to generate checklists.
[System Part]
Please help judge an AI assistant's response to an instruction by providing an evaluation
29

---

Investigating Non-Transe Von-Transitivity in LLM-assa-Judge
checklist.
To write a specific evaluation checklist, you get given the following entity each time: INSTRUCTION: An instruction that has been given to an AI assistant.
[User Part]
# Task Details
Your task is to come up with an evaluation checklist 1ist for a given INSIRUCTION.
This evaluation checklist should be a 1ist of questions that ask whether oz not specific met
criteria relevant to the INSTRUCTION Criteria covered by your checklist could explicitly stated in the INSIRUCTION, or be generally sensible criteria for the problen em domain.
AI assistant's respense.
You should, however, try to be concise and not inelude unnecessary entries in your checklist.
Checklist questiens should:
**Be answerable by "yes' or 'no'*x **, with 'yes' meaning that the response successfully met the corresponding requizement.
Be comprchensive, but concise**, meaning that all criteria directly relevant to the INSTRUCTION should represented by a question, but only questions that are very clearly relevant should be included.
**Be precise**, meaning that checklist questions should avoid vague wording and evaluate apecific aspects of response, directly using the phrasing of the INSTRUCTION where appropriate.
You should always analyse the INSTRUCTION before providing an evaluation checklist.
# Response Format Analysis:
Answer: CHECKLIST QUESTIONS (each question should appear on new line)
# Examples
(examples)
## Real Task
### INSTRUCTION (message)
#1F Response
Please analyse the instruction and provide an answer correct fornat. Remember that each question should be phrased such that answering with 'yes' would mean that the zesponse *ssccessfullyy fulfilled the criteria being assessed by the question.
In most cases, your checklist should contain least two questions, but than eight.
30


Reviews:
[{"review_id": "F4vP8iVUTB", "reviewer": "Reviewer_Dskp", "summary": "The paper investigates whether LLM exhibit non-transitive preferences when comparing model outputs. Typically, people use pairwise comparisons against a single baseline model, implicitly assuming transitive preferences. However, the authors find that such judgments can violate transitivity and that rankings can change significantly if a different baseline is used. They propose measuring soft non-transitivity deviation, mitigating position bias by switching model-response order, and moving to a round-robin tournament framework combined with Bradley-Terry scoring. This approach reduces the influence of non-transitivity and aligns more closely with human-preference rankings while being more robust than baseline-fixed methods. They further introduce a computationally lighter Swiss-Wise Iterative Matchmaking method for a cheaper round-robin.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The claim that Non-transitive preferences exist in LLM-based evaluations is largely supported. Position bias partially drives non-transitivity is also supported (this fact is not very surprising). Baseline-fixed methods, such as only compare to GPT-4, yield rankings sensitive to which baseline is chosen is also supported (this fact also not very surprising). Round-robin tournaments plus Bradley-Terry modeling improve reliability and correlate more strongly with human judgments is supported.", "methods_and_evaluation_criteria": "Evaluation primarily relies on comparing final rankings to Chatbot Arena’s human-preference ordering, a recognized baseline for alignment with human judgments.", "theoretical_claims": "I did not find any incorrectness.", "experimental_designs_or_analyses": "The experiments are largely well designed to support most of the claims.\n\nOne limitation is that the data come mostly from AlpacaEval, which may not capture all open-ended domains. Raises question regarding generalizability.\n\nAnother limitation is that System Prompt is a big factor in LLM-as-a-Judge framework that isn't studied in the work. It would be nice if the paper further explore the effects of System Prompt to transitivity (e.g., AlpacaEval system prompt vs MT Bench system prompt vs Arena-Hard system prompt).", "relation_to_broader_scientific_literature": "It relates to recent work on LLM biases (position bias, verbosity bias) and automated evaluators, highlighting that pairwise comparisons can inherit these biases and yield inconsistent rankings.", "essential_references_not_discussed": "I don't think there aren't any essential references not discussed.", "other_strengths_and_weaknesses": "Strength: \n- Thorough empirical analysis of a largely overlooked concern—non-transitive comparisons. I like the author's work, I think more papers like this will improve the LLM-as-a-Judge framework. \n- Practical strategies introduced.\n- Good presentation.\n\nWeakness:\n- As I mentioned above in \"Experimental Designs Or Analyses\".", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Would you guys release the code / implementation for the SWIM method? I think it will benefit the automatic evaluation community.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7KoyQoWt8l", "reviewer": "Reviewer_bBXd", "summary": "This paper investigates the assumption of transitive preferences in LLM-based evaluation frameworks. The authors highlight that non-transitivity exists in LLM judgments, leading to inconsistencies in model rankings depending on the choice of baseline. The authors propose using round-robin tournaments combined with the Bradley-Terry model to generate more robust rankings. Additionally, they introduce Swiss-Wise iterative Matchmaking (SWIM) tournaments to reduce computational costs while preserving ranking reliability. The proposed methods improve correlation with human evaluation benchmarks such as Chatbot Arena.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their positive and insightful comments. These suggestions have significantly enhanced the clarity and depth of our manuscript. Below, we address each concern raised in detail.\n\n> The study does not include explicit human verification of non-transitive cases. While Chatbot Arena provides a reference ranking, a small-scale micro study verification of LLM judgments would add credibility.\n\nWe appreciate the reviewer for highlighting the importance of explicit human verification of non-transitive cases. Although human verification is valuable, it is beyond the immediate scope of this study, as our central objective is to investigate non-transitivity specifically arising from LLM-based judgments. In other words, the phenomenon of interest is the inherent non-transitivity within LLM evaluators themselves which is independent of human judgment. While it would indeed be intriguing to examine whether similar patterns also occur in human evaluators, such an investigation represents an exploration of related but distinct phenomena rather than a direct validation of LLM judgments. Nevertheless, we recognize the significance of this point and have noted this as an important direction for future work in the updated manuscript, indicating our intention to perform a targeted micro-study to explore alignment or divergence between LLM and human judgments in non-transitive scenarios.\n\n> As a former literature, the Bradley-Terry(BT) model has been known to be exposed to 'intransitivity' risk because it relies on scalar variables, where all preferences are transitive by assumption.\n> - The literature below studied representative preference datasets in the real world, where the 'transitive' relationship between preference annotations may not always hold.\n> - https://arxiv.org/abs/2409.19325 (Duan et al, 2017)\n> - Besides evidence and quantitative evaluation of non-transitivity, the paper proposed representation learning algorithms to generalize BT models to a 'non-transitive' setting. To my knowledge, this can be considered related work, and representation learning techniques (profiling of LLMs) are still under-explored in the LLM-as-a-judge topic.\n\nWe appreciate the reviewer’s insightful comment regarding the inherent assumption of transitivity in both the Elo and Bradley-Terry models. We choose the BT model primarily because, despite known cyclic behaviors observed in practice (e.g., in competitive games such as StarCraft II, and Dota 2), their transitivity property is still considered valid for comparative ranking purposes [1] and that is why ELO scores have remained widely used to evaluating agents in non-transitive games [2, 3]. While our study indeed observes non-transitivity at the instance level when using GPT-4-Turbo as the judge, these instances are relatively infrequent, meaning that aggregated model-level evaluations remain predominantly transitive. Consequently, the observed non-transitivity introduces only mild disturbances, effectively manageable within the BT framework. Nevertheless, we fully acknowledge the reviewer’s point that the transitivity assumption in the BT model may not fully capture the nuanced capabilities of models. We have revised the related work section to include this discussion, highlighting representation learning techniques, such as those presented by Duan et al. (2017), as promising and still under-explored methods that could enhance the robustness of LLM-as-a-judge evaluations. \n\nOnce again, we thank the reviewer for their time and detailed feedback. We hope these clarifications address the reviewer’s concerns and welcome any additional feedback to further improve our work.\n\nReferences:\n\n[1] Bertrand, Q., Czarnecki, W. M., & Gidel, G. (2023). On the limitations of the Elo, real-world games are transitive, not additive. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 206: 2905–2921.\n\n[2] Vinyals, O., Babuschkin, I., Czarnecki, W.M. et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 350–354 (2019).\n\n[3] Siqi Liu et al. ,From motor control to team play in simulated humanoid football.Sci. Robot.7,eabo0235(2022).", "claims_and_evidence": "- Extensive empirical evaluation using AlpacaEval supports the existence of non-transitivity in LLM-as-a-Judge frameworks, demonstrating inconsistencies in model rankings when changing baseline models.\n- The study focuses on AlpacaEval as the primary benchmark. Other testing on additional datasets like MT-Bench or WildBench would strengthen generalizability.", "methods_and_evaluation_criteria": "- The authors defined the Percentage of Non-Transitive cases (PNT) and Soft Non-Transitivity Deviation (SNTD) to measure the degree of non-transitivity for a single instruction with a triplet of models. This is a meaningful effort to quantify the existence of non-transitivity.\n- The arguments and remarks give practical guidance for the problem by comparing it to human preference rankings from Chatbot Arena.", "theoretical_claims": "This paper does not have formal theoretical claims but rather focuses on the foundation problem in prevailing RMs.", "experimental_designs_or_analyses": "- The experiments are generally sound, with a high level of attention to detail.\n- The authors used position switching to control and avoid ordering bias.\n- To mitigate verbosity bias and ensure a fair comparison, the authors adopt the generalized linear model with the same weights as Length-Controlled AlpacaEval.\n- The study does not include explicit human verification of non-transitive cases. While Chatbot Arena provides a reference ranking, a small-scale micro study verification of LLM judgments would add credibility.", "relation_to_broader_scientific_literature": "RM is the foundational pillar in RLHF and LLM-as-a-judge emerges as the standard tool for LLM evaluation in broader scientific applications.", "essential_references_not_discussed": "The authors paid special attention to the existence of non-transitivity in the real world. \n\nAs a former literature, the Bradley-Terry(BT) model has been known to be exposed to 'intransitivity' risk because it relies on scalar variables, where all preferences are transitive by assumption. \n- The literature below studied representative preference datasets in the real world, where the 'transitive' relationship between preference annotations may not always hold. \n- https://arxiv.org/abs/2409.19325 (Duan et al, 2017) \n- Besides evidence and quantitative evaluation of non-transitivity, the paper proposed representation learning algorithms to generalize BT models to a 'non-transitive' setting. To my knowledge, this can be considered related work, and representation learning techniques (profiling of LLMs) are still under-explored in the LLM-as-a-judge topic.", "other_strengths_and_weaknesses": "- This paper highlights the existence of 'non-transitivity' in LLM-as-a-judge applications.\n- The proposals in the work, both evaluation metrics and algorithms are simple and relative to existing approaches.\n- This paper effectively connects ranking instability to non-transitive behavior.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YBcm2gbIcj", "reviewer": "Reviewer_BZyP", "summary": "This paper explores an issue in comparison-based evaluation: non-transitivity, meaning that in evaluations based on a baseline, if A > B and B > C, it does not necessarily follow that A > C.  \n\nThe paper first defines how to measure this non-transitivity and establishes a framework for evaluating model performance based on comparisons. The authors then identify the existence of non-transitivity and analyze several influencing factors in detail, including the choice of judge models, the performance gap between compared models, and position bias. These factors make the final ranking results highly sensitive to the choice of the baseline model.  \n\nTo address this, the paper proposes two improvements. First, it refines the ranking method and introduces a more efficient algorithm. Second, instead of using win rate to represent model performance, it adopts the BT model and ELO rating system for estimation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s thoughtful and constructive feedback, which greatly helps us strengthen our manuscript. Below, we provide detailed responses addressing each of the reviewer’s comments:\n\n> …since the authors only select GPT-4 and GPT-3.5 as judges, the robustness of the conclusion \"Weaker Judge is More Non-Transitive\" is limited. This has been pointed out in their limitations.\n\nWe appreciate the reviewer highlighting the importance of robustness in verifying the conclusion \"Weaker Judge is More Non-Transitive.\", and as such we have conducted additional experiments using GPT-4o-mini (gpt-4o-mini-2024-07-18) as a judge on the AlpacaEval dataset across the same four scenarios setting presented in Table 1. The results are shown below:\n\n### GPT-4o-mini as the Judge\n\n|Scenario| Models|PNT|SNTD|\n|-|-|-|-|\n| LL       | gpt-4o > Qwen1.5-72B > Mistral-7B-Instruct                              | 3.35 | 0.1006 |\n| LM       | gpt-4o > Qwen1.5-72B ≈ Claude-3-Sonnet                                  | 3.60 | 0.1070 |\n| ML       | Yi-34B ≈ Qwen1.5-72B > Mistral-7B-Instruct                              | **3.98** | 0.1036 |\n| MM       | Qwen1.5-72B ≈ Claude-3-Sonnet  ≈ GPT-4                                   | 3.60 | **0.1173** |\n\nThese additional results remain consistent with our previous conclusion, indicating that the degree of non-transitivity generally increases as the performance gap between model pairs narrows, which is confirmed by the SNTD metric. Furthermore, according to rankings from Chatbot Arena [1], GPT-4o-mini is ranked higher than GPT-4-Turbo, suggesting that GPT-4o-mini serves as a stronger judge. Compared to GPT-4-Turbo, GPT-4o-mini consistently yields lower SNTD and PNT across almost all scenarios which means it is more transitive, thus further validating the claim that a weaker judge exhibits more non-transitivity.\n\nFor comparison, we also provide the earlier GPT-4-Turbo results below:\n\n### GPT-4-Turbo as the Judge\n\n|Scenario| Models|PNT|SNTD|\n|-|-|-|-|\n| LL       | gpt-4o > Qwen1.5-72B > Mistral-7B-Instruct                              | 3.98 | 0.1121 |\n| LM       | gpt-4o > Qwen1.5-72B ≈ Claude-3-Sonnet                                  | 5.96 | 0.1336 |\n| ML       | Yi-34B ≈ Qwen1.5-72B > Mistral-7B-Instruct                              | 3.98 | 0.1215 |\n| MM       | Qwen1.5-72B ≈ Claude-3-Sonnet ≈ GPT-4                                   | **8.45** | **0.1431** |\n\n\n> The derivation of equation 5 is wrong\n\nWe sincerely thank the reviewer for pointing out this mistake. We confirm that this is indeed a typographical error, where we miss a minus sign before $(s_{AC}^{(i)} - s_{BC}^{(i)})$. However, our actual implementation uses the correct formulation, as verifiable by the `estimate_win_rate` function within the `check_bias.ipynb` file provided in our Supplementary Material. Specifically, defining:\n$$ X = \\phi(o_A^{(i)}, o_B^{(i)} \\mid m_J, I_i), \\quad Y = \\phi(o_B^{(i)}, o_C^{(i)} \\mid m_J, I_i), \\quad Z = \\phi(o_A^{(i)}, o_C^{(i)} \\mid m_J, I_i). $$\nAccording to the Bradley-Terry model, we have:\n$$\ns_{AC}^{(i)} = \\gamma_A^{(i)} - \\gamma_C^{(i)} = \\ln\\left( \\frac{Z}{1 - Z} \\right), \\quad \ns_{BC}^{(i)} = \\gamma_B^{(i)} - \\gamma_C^{(i)} = \\ln\\left( \\frac{Y}{1 - Y} \\right), \\quad \ns_{AC}^{(i)} - s_{BC}^{(i)} = \\ln\\left( \\frac{Z(1 - Y)}{Y(1 - Z)} \\right).\n$$\nWhen substituted back into  $\\hat{\\phi}(o_A^{(i)}, o_B^{(i)} \\mid m_J, I_i) = \\frac{1}{1 + e^{-(s_{AC}^{(i)} - s_{BC}^{(i)})}},$\nthe equation becomes $\\frac{1}{1 + \\exp\\left[-\\ln\\left(\\frac{Z(1 - Y)}{Y(1 - Z)}\\right)\\right]} = \\frac{Z(1 - Y)}{(Y + Z) - 2YZ},$\nwhich aligns precisely with our implementation in `estimate_win_rate_X`. Thus, this typographical error does not affect our experimental conclusions. We have corrected this in the updated manuscript and would like to again thank the reviewer for their diligence.\n\n> line 167: currying -> carrying \n\n\"Currying\" here refers to a concept in functional programming, denoting the process of fixing one argument of a function to create a new function with fewer arguments. \n\n> line 172: (M-1) -> M. when a new model are added, it need to compare with existing M models in round-robin tournament.\n\nWe thank the reviewer for identifying this typographical mistake. Indeed, it should be $M$ comparisons rather than $(M-1)$. This has been corrected in our updated manuscript. \n\nWe once again thank the reviewer for their valuable comments, which significantly improve the manuscript's clarity and rigor.\n\nReferences:\n\n[1] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: an open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning (ICML'24), Vol. 235. JMLR.org, Article 331, 8359–8388.", "claims_and_evidence": "Yes, the authors experimentally prove the existence of non-transitivity and analyze the influence factor of non-transitivity. They also demonstrate the effectiveness of the new methods through experiments.", "methods_and_evaluation_criteria": "Yes, the selection of AlpacaEval and LLMs is appropriate. The Spearman and Kendall correlation are suitable to compare the AlpacaEval and Chatbot Area.", "theoretical_claims": "The derivation of equation 5 is wrong, the correct derivation is as follows: $$\\phi(o_A^{(i)}, o_B^{(i)} \\mid m_J, I_i) = \\frac{1}{1 + e^{-(\\gamma_A^{(i)} - \\gamma_B^{(i)})}} = \\frac{1}{1 + e^{-((\\gamma_A^{(i)} - \\gamma_C^{(i)}  )- (\\gamma_B^{(i)}- \\gamma_C^{(i)} ))}} = \\frac{1}{1 + e^{-(e_{AC}^{(i)} - e_{BC}^{(i)})}} =  \\frac{1}{1 + e^{e_{BC}^{(i)} - e_{AC}^{(i)}}} $$", "experimental_designs_or_analyses": "Mostly good. However, since the authors only select GPT-4 and GPT-3.5 as judges, the robustness of the conclusion \"Weaker Judge is More Non-Transitive\" is limited. This has been pointed out in their limitations.", "relation_to_broader_scientific_literature": "The paper's key contribution is showing that non-transitivity, previously observed in other zero-sum games, also applies to LLMs.", "essential_references_not_discussed": "To the best of my knowledge, the paper adequately covers all the essential related works necessary for understanding its key contributions.", "other_strengths_and_weaknesses": "Strengths: This paper explores non-transitivity in a systematic manner, starting from its formal definition, followed by result analysis, and finally proposing improvements.\n\nWeaknesses: There are a few minor errors, including the derivation errors mentioned above and several spelling mistakes mentioned below.", "other_comments_or_suggestions": "line 167: currying -> carrying\nline 172: (M-1) -> M. when a new model are added, it need to compare with existing M models in round-robin tournament.\n\nIt is recommended to carefully review and correct any remaining unnoticed minor errors.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wkKB9kHWdY", "reviewer": "Reviewer_NNcD", "summary": "The paper shows that LLM judges have non-transitive preference in pairwise comparison, which is not only caused by position bias. Furthermore, non-transitivity can be mitigated by round-robin tournaments combined with the Bradley-Terry model. The efficiency can be further improved by Swiss-Wise Iterative Matchmaking.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback! To our understanding, the main concerns of the reviewer lie in 1) the limited evaluation on only the AlpacaEval datasets, and 2) the claim that round-robin tournaments reduce non-transitivity – we address both in our response. We hope that the reviewer could considers increasing the score if they feel their concerns have been sufficiently addressed.\n\n> Only AlpacaEval dataset is considered. The behavior of LLM judges could be different on other datasets, e.g., Chatbot Arena.\n\nWe appreciate this valuable suggestion from the reviewer and agree that it is crucial to verify the generalizability of our results across different datasets. Consequently, we have conducted additional experiments using GPT-4-Turbo, GPT-3.5-Turbo, and GPT-4o-mini judges on the Arena-Hard-Auto dataset [1], comprising 500 high-quality prompts sourced from Chatbot Arena, with position switching. Models for each scenario are selected based on their rankings in Arena-Hard-Auto's leaderboard. The results are summarized below:\n\n###  PNT and SNTD for Different Judges in Arena-Hard-Auto\n\n| Scenario | GPT-4-Turbo PNT | GPT-4-Turbo SNTD | GPT-3.5-Turbo PNT | GPT-3.5-Turbo SNTD | GPT-4o-mini PNT | GPT-4o-mini SNTD |\n|----------|------------------|-------------------|--------------------|---------------------|------------------|-------------------|\n| LL       | 2             | 0.0820            | 17              | 0.2071              | 1                | 0.0813            |\n| LM       | 3             | 0.1083            | 17.5              | 0.2002              | 1.5              | 0.0880            |\n| ML       | 2.5             | 0.0945            | 24.5              | **0.2370**              | **5.5**              | 0.1085            |\n| MM       | **5**             | **0.1270**          | **28**             | 0.2294              | 5                | **0.1181**            |\n\nThe models evaluated in each scenario are: \n- LL: gpt-4o-2024-05-13 > Qwen1.5-72B-Chat > Mistral-7B-Instruct\n- LM: gpt-4o-2024-05-13 > Mistral-Large-2402 ≈ Qwen1.5-72B-Chat \n- ML: Mistral-Large-2402 ≈ Qwen1.5-72B-Chat > Mistral-7B-Instruct \n- MM: GPT-4-0613 ≈ Mistral-Large-2402 ≈ Qwen1.5-72B-Chat\n\nThese supplementary results align with our original findings, reinforcing that non-transitivity generally increases as the performance gap between model pairs narrows with a strong judge, as quantified by the SNTD metric. This consistency suggests that the observed non-transitivity behavior of LLM judges is robust across different datasets. Due to resource constraints, we conducted this analysis on a sample of 200 questions from Arena-Hard-Auto. We hope the reviewer understands that running extensive model evaluations with positional swaps demands substantial computational resources. \n\n> There is no direct experiment or theoretical analysis towards contribution 3), i.e., round-robin tournaments reduces non-transitivity. Correlation is computed in Section 5, but this is not directly related to transitivity of preference […] Why does round-robin tournaments resolves the non-transitivity issue? Is it from a theoretical guarantee or can be demonstrated through experiments?\n\nWe respectfully request further clarification on this point, as there may have been a misunderstanding. Our claim is not that round-robin tournaments reduce non-transitivity in judge models—non-transitivity is inherent to the judge and cannot be externally mitigated. Instead, round-robin tournaments reduce the negative impact of non-transitivity. By aggregating pairwise comparisons across all model pairs, this approach avoids reliance on a single baseline model, thereby mitigating the cyclic preferences in the model level caused by judges' inherent non-transitivity. We demonstrate empirically that round-robin tournaments do not suffer from the unreliability demonstrated by baseline-fixed approaches, as round-robin tournaments remove the impact of non-transitivity. While we do not prove this claim theoretically, we believe it would be possible to show that a round-robin tournament measures only the transitive component of skill [2], and hence does not suffer from unreliability due to potentially non-transitive judges.\n\nOnce again, we thank the reviewer for their time and detailed feedback. If the reviewer has any further questions or suggestions, we would be more than happy to address them.\n\nReferences:\n\n[1] Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., & Stoica, I. (2024). From crowdsourced data to high-quality benchmarks: Arena-Hard and BenchBuilder pipeline. arXiv preprint arXiv:2406.11939.\n\n[2] Czarnecki, W. M., Gidel, G., Tracey, B., Tuyls, K., Omidshafiei, S., Balduzzi, D., & Jaderberg, M. (2020). Real world games look like spinning tops. In NeurIPS.", "claims_and_evidence": "The claimed contributions 1) and 2) in the paper are supported by experiments on AlpaceEval dataset. The result would be more convincing if more tests are conducted on other datasets, such as Chatbot Arena. This is because the behavior of LLM judges could be different across datasets.\n\nThere is no direct experiment or theoretical analysis towards contribution 3), i.e., round-robin tournaments reduces non-transitivity. Correlation is computed in Section 5, but this is not directly related to transitivity of preference.", "methods_and_evaluation_criteria": "No significant flaws in method and evaluation.", "theoretical_claims": "Not applicable for this paper.", "experimental_designs_or_analyses": "Strength: The experiment considers multiple factors that could impact non-transitivity of preference. \n\nWeakness: Only AlpacaEval dataset is considered. The behavior of LLM judges could be different on other datasets, e.g., Chatbot Arena.", "relation_to_broader_scientific_literature": "This paper suggests using round-robin tournaments in LLM-as-a-judge when transitivity is important, contributing to research in LLM-as-a-judge.", "essential_references_not_discussed": "No missing reference found.", "other_strengths_and_weaknesses": "No additional strength or weakness.", "other_comments_or_suggestions": "No additional comments.", "questions_for_authors": "1. Why does round-robin tournaments resolves the non-transitivity issue? Is it from a theoretical guarantee or can be demonstrated through experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u1eceaYTah", "reviewer": "Reviewer_rsx8", "summary": "The authors argue that the existing automated (using LLM-as-a-Judge) LLM ranking algorithms are unreliable and not aligned with human judgement. The authors propose judgement transitivity as a metric of self-consistency to estimate the quality of judgement. In fact, the authors propose two judgement self-consistency metrics: Percentage of Non-Transitive cases (PNT) and the more reliable Soft Non-Transitivity Deviation (SNTD). The authors apply the Bradley-Terry model to reconcile the pairwise rankings into the ranking through all subject LLMs, and convert the Bradley-Terry Coefficients into Elo Rating for further analysis.\n\nThe original AlpacaEval resource usage is O(N * M). However, in the first approach the authors have to use the round robin tournament with way higher complexity O(N * M^2).  As all-to-all tournament is very resource consuming, the authors propose Swiss-Wise Iterative Matchmaking (SWIM) tournament with O(N * M * logM ) that implements iterative mining of most unresolved LLM pairs.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their insightful and positive comments, which significantly enhance the clarity and impact of our manuscript. Please see below for our detailed response.\n\n>  It is good to see the results for gpt-3.5-turbo, but its low performance is expected and does not carry much insight.\n\nWe appreciate this valuable feedback from the reviewer, and as such we have conducted additional experiments using GPT-4o-mini (gpt-4o-mini-2024-07-18) as a judge on the AlpacaEval dataset across the same four scenarios setting presented in Table 1. The results are shown below:\n\n### GPT-4o-mini as the Judge\n\n|Scenario| Models|PNT|SNTD|\n|-|-|-|-|\n| LL| gpt-4o > Qwen1.5-72B > Mistral-7B-Instruct| 3.35 | 0.1006 |\n| LM| gpt-4o > Qwen1.5-72B ≈ Claude-3-Sonnet| 3.60 | 0.1070 |\n| ML| Yi-34B ≈ Qwen1.5-72B > Mistral-7B-Instruct| **3.98** | 0.1036 |\n| MM| Qwen1.5-72B ≈ Claude-3-Sonnet ≈ GPT-4| 3.60 | **0.1173** |\n\nThese results support our conclusion that non-transitivity increases as model performance differences narrow, as reflected by SNTD. Furthermore, Chatbot Arena ranks GPT-4o-mini higher than GPT-4-Turbo, suggesting it is a stronger judge. Compared to GPT-4-Turbo, GPT-4o-mini consistently yields lower SNTD and PNT across almost all scenarios which means it’s more transitive, thus further validating the claim that a weaker judge exhibits more non-transitivity.\n\nFor comparison, we also provide the earlier GPT-4-Turbo results below:\n\n### GPT-4-Turbo as the Judge\n\n|Scenario| Models|PNT|SNTD|\n|-|-|-|-|\n| LL| ...| 3.98 | 0.1121 |\n| LM| ... | 5.96 | 0.1336\n| ML| ...| 3.98 | 0.1215 |\n| MM| ...| **8.45** | **0.1431** |\n\n> It is not clear when non-transitivity can be detrimental in practice, what are the practical situations when it hurts? What is the definition of the evaluation framework reliability?\n\nNon-transitivity highlights practical risks in commonly used evaluation frameworks, such as AlpacaEval. Our study empirically demonstrates that selecting different baseline models can yield different model rankings, with only 20% of models maintaining consistent rank positions across various baselines, and pairwise rank agreement drops to 61% on average (Section 4.2). This undermines evaluation reliability, which we define as the framework’s ability to consistently produce stable rankings aligned with human preferences, regardless of baseline choice. Additionally, non-transitivity can lead weaker models to appear superior due to cyclic preferences, potentially causing practitioners to deploy suboptimal models. This risk is particularly critical in applications such as chatbots or automated decision-making systems, where model performance directly impacts user trust and safety.\n\n> It is not clear what is “true capability”. Who/what is the oracle in this case? \n\nWe take inspiration from Czarnecki et al. [1] in defining true capability in terms of the vertical (or transitive) component of skill in a distribution of strategies for a given game. They conceptualize the distribution as analogous to a spinning top, where the vertical axis represents skill level (transitive strength), increasing as one moves upward, and the horizontal axis denotes non-transitivity, reflecting strategies’ cyclical relationships (line 127-131). At the widest part of the spinning top, strategies exhibit diverse and strong non-transitive interactions, similar to players with different styles competing against each other. Moving upward, strategies become increasingly homogeneous, and non-transitivity diminishes as skills improve. The \"true capability\" relates to skill progression toward the Nash Equilibrium, represented by the vertical axis in this analogy. In our work, we approximate the oracle using Elo scores from Chatbot Arena’s crowdsourced rankings, as these are generated through randomized anonymous pairings across diverse user queries, effectively mitigating non-transitivity.\n\n> It would be good to see random baselines for PNT and SNTD in Table 1. Not the random choice of the order of the A and B LLM responses in the judge prompt, but the random uniform judge decision.\n\nWe agree with this insightful suggestion and have conducted additional experiments where the judge randomly predicts outcomes with a 50% probability for either preference. Under these conditions, the scenario no longer impacts PNT and SNTD. The results for this random baseline are as follows:\n- PNT: 25\n- SNTD: 0.3465\n\nNote: When calculating SNTD, the default logarithm base is $e$, thus the Jensen–Shannon divergence range is $[0, \\log 2] \\approx [0, 0.693]$.  Using base 2, the range becomes $[0, 1]$, making the SNTD for a random judge precisely 0.5.\n\nWe once again thank the reviewer for their insightful comments, which improve our manuscript significantly.\n\nReferences:\n\n[1] Czarnecki, W. M., Gidel, G., Tracey, B., Tuyls, K., Omidshafiei, S., Balduzzi, D., & Jaderberg, M. (2020). Real world games look like spinning tops. In NeurIPS.", "claims_and_evidence": "The paper closes an important gap in the evaluation of LLMs: understanding and mitigating the deficiencies of LLM-as-a-Judge.\n\nThe paper is clearly written, well structured. All claims are supported. The notation, and definitions are clear.\n\nIt is clever to use the Bradley-Terry model to infer the overall ranking from pairwise comparisons. To avoid the problem of disjoint ranked groups of models, the authors propose the round-robin tournament setup, and further improve over it with the proposed Swiss-Wise Iterative Matchmaking.\n\nThe use of Jensen–Shannon divergence (the symmetrized and smoothed version of KL-divergence) to soften the raw Percentage of Non-Transitive cases (PNT) and arrive to Soft Non-Transitivity is very smart.\n\nThe analysis and visualisation of the experimental results is comprehensive. The extensive investigation of the contribution of position bias and judge’s inherent reasoning limitation is noteworthy.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I checked the expressions conceptually, without detailed verification.", "experimental_designs_or_analyses": "Experimental results are sound. Probably the only weak point is that apart from gpt-4-turbo, the weak gpt-3.5 is used as a judge, and it displays high PNT, and, as the authors point out, is weaker than the majority of the LLMs under investigation. It is good to see the results for gpt-3.5-turbo, but its low performance is expected and does not carry much insight.", "relation_to_broader_scientific_literature": "The topic of the paper is related to ranking and retrieval.", "essential_references_not_discussed": "There are no essential references that are not discussed.", "other_strengths_and_weaknesses": "> Violations of transitivity can result in unstable rankings that undermine the evaluation framework’s reliability\n\nIt is not clear when non-transitivity can be detrimental in practice, what are the practical situations when it hurts? What is the definition of the evaluation framework reliability?\n\nLines 132-134\n> Given the presence of non-transitivity, evaluating a strategy based on  its performance against a single opponent does not reliably reflect its true capability.\n\nIt is not clear what is “true capability”. Who/what is the oracle in this case?", "other_comments_or_suggestions": "It would be good to see random baselines for PNT and SNTD in Table 1. Not the random choice of the order of the A and B LLM responses in the judge prompt, but the random uniform judge decision.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None."}}]

Author rebuttal:

Meta-review: The paper investigates non-transitivity in LLM-as-a-Judge frameworks, proposing round-robin tournaments and SWIM to address ranking inconsistencies. Reviews highlighted several strengths in 
1. Novel Contribution: Identifies and quantifies non-transitivity in LLM evaluations, a critical issue for reliable benchmarking.
2. Empirical Rigor: Comprehensive experiments on AlpacaEval and Arena-Hard-Auto, with position bias controls and multiple judges (GPT-4-Turbo, GPT-3.5, GPT-4o-mini), some of those from additional experiments in the rebuttal messages.

The paper makes a significant contribution to LLM evaluation by exposing and addressing a fundamental issues of LLM-as-a-Judge nontransitivity.  The proposed PNT/SNTD and SWIM offer practical tools for the community.

The authors also mostly adequately address the reviewers' concern, further strengthening the paper's message.
Decision: Accept (spotlight poster)