Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Authors: Lianmin Zheng
Wei-Lin Chiang
Ying Sheng
Siyuan Zhuang
Zhanghao Wu
Yonghao Zhuang
Zi Lin
Zhuohan Li
Dacheng Li
Eric Xing
Hao Zhang
Joseph E. Gonzalez
Ion Stoica
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: large language models
human preference
llm-as-a-judge
benchmark
evaluation
TL;DR: We explore the use of LLM-as-a-judge for evaluating chat assistants and show it matches human preferences with an agreement over 80%.

Abstract: Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences.
To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions.
We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and propose solutions to mitigate some of them.
We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform.
Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80\% agreement, the same level of agreement between humans.
Hence, LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain.
Additionally, we show our benchmark and traditional benchmarks complement each other by evaluating several variants of LLaMA and Vicuna.
The MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Lianmin Zheng's
Wei-Lin Chiang
Ying Sheng*
Siyuan Zhuang
MXXV:220060068554 [cs.CL] 24 24 Dec 2023
Zhanghao Wu
Yonghao Zhuang
Zi Lin
Zhuohan Li
Dacheng Li
Eric P. Xing
Hao Zhang
Joseph E. Gonzalez*
Ion Stoica
1 UC Berkeley
2 UC San Diego
3 Camegie Mellon University
4 Stanford
MBZUAI
Abstract
Evaluating large language model (LLM) based chat assistants challenging due their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges evaluate these models more open-ended questions. We examine the usage and limitations
-a--d----juudee including position, verbosity, and self-enhancement biases, well limited reasoning ability. and propose solutions to mitigate some them. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, multi-turn question Arena, a crowdsourced battle platform. Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain. Additionally, we show benchmark and traditional benchmarks complement each other by evaluating several variants of LLaMA and Vicuna. The MTT-benc questions, 3K expert votes, and 30K conversations with human preferences are publicly available Intpss //github. com 1m- www.loooommmeceeceeeeeeeeeseeeee http:///ws.g.aaaaateee///aaaaaaa
and Chatbot
Introduction
There has been proliferation of LLM-based chat assistants (chatbots) that leverage supervised instruction fine-tuning and reinforcement learning with human feedback (RLHF) to unlock new instruction following and conversational abilities [31 2, 30
g
52
48 [4]. Once aligned with
humans, these chat models are strongly preferred by human users over the original, unaligned models on which they are built. However, the heightened user preference does not always correspond to improved scores on traditional LLM benchmarks benchmarks like MMLU 19 and HELM 24 cannot effectively tell the difference between these aligned models and the base models. This phenomenon suggests that there is a fundamental discrepancy between user perceptions of the usefulness of chatbots and the criteria adopted by conventional benchmarks.
We argue that this discrepancy primarily arises due to existing evaluation that only measures LLMS' core capability on a confined set of tasks (e.g., multi-choice knowledge or retrieval questions), without adequately assessing its alignment with human preference in open-ended tasks, such as the ability to accurately adhere to instructions in multi-turn dialogues. As a demonstration, we show
Joint first authors. This paper an extended version of our earlier blog post g
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Question:
If the FED buys bonds in secondary market
(A) the money supply will increase. (B) the money supply will decrease. (C) the money supply not affected. (D) the discount rate would be affected. Output the answer only.
Assistant A:
Assistant B:
The Federal Reserve buys bonds in the secondary market to increase the noney supply.
(A) the noney supply will increase.
Follow-up Question: How does it affect my daily life? Give 3 examples.
Assistant A:
Assistant B:
The Federal Reserve buys the secondary narket to increase the noney supply. This affects your daily life
When the Federal Reserve buys bonds in the secondary narket, it can have several effects three examples:
econony that can inpact your daily life,
1. Interest rates: The Federal Reserve's actions in the bond narket can influence interest rates, which noney. For exanple, if the Federal Reserve buys honds and drives up demand for then, interest
by:
affect the of borrowing
1. Incroasing the noney supply.
loans, and credit cards may go up, nakinn it nore expensive to horrow noney.
things
nortgages,
2. Incroasing the noney supply and lowering the scount rate.
2. Inflation: When the noney supply increases, it lead to inflation, which is the general rise in prices of goods and services. This can impact your daily life making it
3. Increasing the noney supply and loworing the discount rate and increasing the noney supply.
expensive buy things like groceries,
gas, and other necessities.
3. Employment: The Federal Reserve's actions in the bond market also have an impact on employnent. For example, if the Federal Reserve's actions lead to economic growth and increased demand for goods and services, it can create nore jeb opportunities and boost employnent.
GPT-4 Judgment:
Assistant provided an Incorrect response to the user's question about how the Federal Reserve buying bonds in the secondary narket affects daily life. The answer given is repetitive and lacks Clear examples of how the action inpacts daily life.
On the other hand, Assistant provided a relevant and accurate response to the user's question about the Federal Reserve buying bonds. The answer includes three clear exanples of how the action impacts daily life, such as interest rates, inflation, and employnent.
Assistant A should inprove by providing relevant, clear, and accurate exanples to address the user's question. Assistant B's response is more helpful, relevant, and detailed.
Figure I: Multi-turn dialogues between a user and two Al assistants Liiirattt-LLMM----I1 11B (Assistant A) and Vicuna- 13B (Assistant B) -)-iniitated by a question from the MMLU benchmark and follow-up instruction. GPT-4 is then presented with the context to determine which assistant answers better.
The models LLaMA-13B 33), a pre-trained base model without fine-tuning, and Vicuna-13B, our fine-tuned model from LLaMA-133 13B on high-quality conversations (the training details in Appendix [E) Despite the base LLaMA models showing competitive performance on conventional benchmarks (Table 8) its answers open-ended questions are often not preferred by humans. This misalignment of conventional benchmarks underscores the core problem driving this paper: the need for robust and scalable automated method evaluate L.M alignment with human preferences.
conversation histories with two models on an MMLU question in Figure 1
To study this, we introduce two benchmarks with human ratings as the primary evaluation metric: MT-bench and Chatbot Arena. MT-bench is series of open-ended questions that evaluate chatbot's s multi-turn conversational and instruction-follon following ability two critical elements for human prefer- ence. MT-bench is also carefully constructed to differentiate chatbots based their core capabilities, such as reasoning and math. In addition, we develop Chatbot Arena, a crowdsourced platform fea- turing anonymous battles between chatbots in real-worke scenarios Users engage in conversations with two chatbots at the same time and rate their responses based on personal preferences.
While human evaluation is the gold standard for assessing human preferences, it is exceptionally slow and costly. To automate the evaluation, we explore the use of state-of-the-art LLMs, such as GPT-4, surrogate for humans. Because these models are often trained with RLHF, they already exhibit strong human alignment. We call this approach "LLM-as SLLM-as-a-junn as-a--udge"" This approach has been tried in
our earlier blog post [8] and other concurrent or follow-up work [5] 29
18 40
However, there has not been a systematic study of this approach.
In this paper, we study the LLM-as---judee approach by comparing it to the gold standard of human evaluation. We examine several potential limitations of the LLM-as---judge approach including position bias, verbosity bias, self-enhancement bias, and limited reasoning ability. We show that some of the biases are minor or can be mitigated. Once addressed, our results from 3K controlled expert votes and 3K crowdsourced human votes in the wild verify that GPT-4 judge match

---

human evaluations at an agreement rate exceeding 80%, achieving the same level of human- human agreement ($4.2) Table 4). Consequently, this suggests LLM-as-a-judge is scalable method to swiftly evaluate human preference, serving as a promising alternative to traditional human evaluations.
This paper makes two contributions: (I) a systematic study of LLM-as-a-judge; and (2) human preference datasets with high-quality questions and diverse user interactions from MT-bench and Chatbot Arena. In addition, we argue for the adoption of a hybrid evaluation framework for future LLM benchmarks: by combining the existing capability- based benchmarks and the new preference- based benchmarks with LLM-as-a-jiel judge, one can swiftly and automatically evaluate both the core capabilities and human alignment of models. We publicly release 80 MT-bench questions, 3K expert votes, and 30K conversations with human preferences for future study.
Table 1: Sample multi-turn questions in MT-hench.
Category
Sample Questions
1st Turn
Compose an engaging travel blog post about recent trip to Hawaii, highlighting cultural experiences and must-see attractions.
Writing
2nd Turn
Rewrite your previous response. Start every sentence with the letter A.
1st Turn
Given that f() = 4x 9x 14, find the value of f(2).
Math
2nd Turn
Find x such that [(x)) 0.
1st Turn
Provide insights into the correlation between cconomic indicators such as GDP, inflation, and unemployment rates. Explain how fiscal and monetary policies ...
Knowledge
2nd Turn
Now, explain them again like I'm five.
2 MT-Bench and Chatbot Arena
2.1 Motivation
With the recent advances of LLMs, LLM-based assistants start to exhibit artificial general intelligence across diverse tasks, from writing and chatting to coding [5] 30, T 37.. However, evaluating their broad capabilities also becomes more challenging. Despite the availability of numerous benchmarks for language models, they primarily focus on evaluating models on closed-ended questions with short responses. Given that these chat assistants can now precisely follow user instructions in multi-turn dialogues and answer open-ended questions in a zero-shot manner, current benchmarks are inadequate for assessing such capabilities. Existing benchmarks mostly fall into the following three categories.
Core-knowledge benchmarks, including MMLU 19 HellaSwag [50], ARC 99, Wino- Grande (36), HumanEval (6). GSM-8K [10], and AGIEval
(51), evaluate the core capabilities of pre-trained LLMS using zero-shot and few-shot benchmark sets. They typically require LLMs to generate short, specific answer to benchmark questions that can be automatically validated.
Instructioo-follooingg benchmarks, such as Flan [27 46). Self-instruct [44), NaturalInstruc-
tions 288), Super-Naturallnsturnn NaturallIntruucionn (45), expand slightly more open- questions and more diverse tasks and are used to evaluate LLMs after instruction fine-tuning.
Conversational benchmarks, like CoQA [35). MMDialog [15) and Assntssittant
(23
closest to our intended use cases. However, the diversity and complexity of their questions often fall short in challenging the capabilities of the latest chatbots.
While largely overlooked by existing LLM benchmarks, human preferences serve as a direct measure of a chatbot's utility in open-ended, multi-turn Al interactions. To bridge this gap. we introduce two novel benchmarks expressly tailored to assess human preferences. Simultaneously. these benchmarks are designed to distinguish the core capabilities of state-of-the-an models.
2.2 MT-Bench
We create MT-bench, benchmark consisting 80 high-quality multi-turn questions. MTTbench designed to test multi-turn conversation and instruction-folloiing ability, covering common use cases and focusing on challenging questions to differentiate models. We identify 8 common categories of user prompts to guide its construction: writing, roleplay, extraction, reasoning, math, coding,
3

---

knowledge (STEM), and knowledge II (humanities/social science). For each category, we then manually designed 10 multi-tum questions. Table lists several sample questions.
2.3
Chatbot Arena
Our second approach Chatbot Arena, a crowdsoureing benchmark platform featuring anonymous battles. On this platform, users can interact with two anonymous models simultaneously, posing the same question both. They vote for which model provides the preferred response, with the identities of the models disclosed post-voting. After running Chatbot Arena for one month, we have collected around 30K votes. Since the platform does not use pre-defined questions, it allows gathering wide range of unrestricted use cases and votes in the wild, based the diverse interests of users. A screenshot of the platform can be found at Appendix
3 LLM as Judge
While our initial evaluations using MTTheecch and Chatbot Arena rely on human ratings, collecting human preferences can costly and laborious 44
To overcome this, we aim to develop more scalable and automated approach. Given that most questions in MT-bench and Chatbot Arena are open-ended without reference answers, devising a rule-based program to assess the outputs is extremely challenging. Traditional evaluation metrics based on the similarity between outputs and reference answers (e.g., ROUGE [55,, BLEU 32] are also ineffective for these questions.
38
2
13
As LLMs continue to improve, they show potential in replacing human annotators in many tasks [17] 20 Specifically, we are interested in whether LLMs can effectively evaluate the responses of chat assistants and match human preferences. Next, we discuss the use and limitations of LL-M-as-a-uddg.
3.1 Types of LLM-ass--Judde
We propose 3 LLM-as-a-judde variations. They can be implemented independently or in combination:
Pairwise comparison, An L.LM judge is presented with question and two answers, and tasked to determine which one is better or declare a tic. The prompt used is given in Figure (
Single answer grading. Alternatively, an LLM judge is asked to directly assign a score single answer. The prompt used for this scenario is in Figure 6 (Appendix).
Reference-guided grading. In certain cases, may be beneficial provide a reference solution if applicable. An example prompt we use for grading math problems is in Figure (eguureBAApee (Appendix).
These methods have different pros and cons. For example, the pairwise comparison may lack scalability when the number of players increases, given that the number of possible pairs grows quadratically; single answer grading may be unable discern subtle differences between specific pairs, and its results may become unstable, as absolute scores are likely to fluctuate more than relative pairwise results if the judge model changes.
3.2 Advantages of LLM--asaaJuudge
LLM-as-a aa-uude- offers two key benefits: scalability and explainability. It reduces the need for human involvement, enabling scalable benchmarks and fast iterations. Additionally, LLM judges provide not only scores but also explanations, making their outputs interpretable, as shown Figure
3.3 Limitations of L.M--ss--Judde
We identify certain biases and limitations LLM judges. However, we will also present solutions later and show the agreement between LLM judges and humans is high despite these limitations.
Position bias is when LLM exhibits propensity favor certain positions bias not unique to our context and has been seen human decision-making domains 222 41).
others. This
B 34 and other ML
Figure II (Appendix) shows an example of position bias. GPT-4 is tasked to evaluate two responses from GPT-3.5 and Vicuna-13B to an open-ended question. When GPT-3.5's answer is positioned

---

Table 2: Position bias of different L.MM judges. Consistency is the percentage of cases where a judge gives consistent results when swapping the order of two assistants. "Biased toward first" is the percentage of cases when a judge favors the first answer. "Error" indicates wrong output formats. The two largest numbers in each column are in bold.
Judge
Prompt
Consistency
Biased toward first
Biased toward second
Error
default
23.8%
75.0%
0.0%
1.2%
Claude-vl
rename
56.2%
11.2%
28.7%
3.8%
default 46.2%
50.0% %
1.2%
2.5%
GPT-3.5
rename
51.2%
38.8%
6.2%
3.8%
default 65.0%
30.0%
5.0%
0.0%
GPT-4
rename
66.2% %
28.7%
5.0%
0.0%
Table 4: Judge failure rate on 10 math questions with different prompts. We test LLaMA-13B vs. Vicuna-13B and swap positions. A failure means when GPT-4 says an incorrect answer is correct.
Table 3: Failure rate under "repetitive list" at- tack for different LLM judges on 23 answers.
Judge
Claude-v1
GPT-3.5
GPT-4
Default
CoT
Reference
Failure rate 91.3%
91.3%
8.7%
Failure rate
14/20
6/20
3/20
first, GPT-4 considers GPT-3.5's answer more detailed and superior. However, upon switching the positions of the two responses, GPT-4's judgement flips, favoring Vicuna's 's answer.
To analyze the position bias, we construct two similar answers each first-turn question in MTTbench by calling GPT-3.5 twice with temperature of 0.7. We then try three LLMs with two different 3 (Appendix). "rename" renames the assistants in our default prompt to see whether the bias is on positions or names. As in Table 2 we found all of them exhibit strong position bias. Most LLM judges favor the first position. Claude-vl also shows a name bias which makes it favors "Assistant A", as illustrated by the "rename" prompt. The position bias can be very sigmificant. Only GPT-4 outputs consistent results in more than 60% of cases.
prompts: "default" is our default prompt in Figure
Note that this test is challenging because the answers are very similar and occasionally indistinguish- able even to humans. We will show that position bias less prominent some cases in Appendix [.I] As for the origin of this bias, we suspect that it could be rooted in the training data or inherent to the left-to-right architecture of causal transformers, but leave deeper study future work.
Verbosity bias is when an LLM judge favors longer, verbose responses, even if they are not as clear, high-quality, or accurate as shorter alternatives.
To examine this bias, we design "repetitive list" attack with model answers from MT-hench. We first select 23 model answers from MT-bench that contain numbered list. We then make them unnecessarily verbose by asking GPT-4 to rephrase the list without adding any new information and insert the rephrased new list to the beginming of the original list. For example, if the original response contains items, then the new response will contain 10 items but the first 5 items are rephrased from the original items. An example is shown in Figure 12 (Appendix). We define the attack is successful an LLM judge thinks the new response is better than the old response. Table shows the failure rate of LLM judges under this attack, demonstrating that all LLMs may be prone to verbosity bias though GPTT4 defends significantly better than others. As a calibration, we find LLM judges are able to correctly judge identical answers (i.e., they always return tie for two identical answers) but cannot pass the more advanced "repetitive list" attack.
Self-enhancement bias. We adopt the term "self-enhancement bias" from social cognition litera- ture 4 to describe the effect that LLM judges may favor the answers generated by themselves.
We examine this effect statistically. Figure (b) shows the win rate (w/o tie) of six models under different LLM judges and humans. Compared to humans, we do observe that some judges favor certain models. For example, GPT-4 favors itself with 10% higher win rate; Claude-vi favors itself with 25% higher win rate. However, they also favor other models and GPT-3.5 does not favor itself. Due to limited data and small differences, our study cannot determine whether the models exhibit a self-enhancement bias. Conducting controlled study challenging because we cannot easily rephrase response to fit the style of another model without changing the quality.
5

---

Limited capability in grading math and reasoning questions. LLMs are known to have limited math and reasoning capability [10], which results in its failure of grading such questions because they do not know the correct answers. However, what is more intriguing is that it also shows limitations in grading basic math problems which is capable of solving. For instance, in Figure T3] (Appendix), we present an example of an elementary math question in which GPT-4 makes an incorrect judgment. It's worth noting that although GPT-4 can solve the problem (when asked separately), it was misled by the provided answers, ultimately resulting in incorrect judgment. This pattern can also be seen in a reasoning question example in Figure [4] (Appendix). Both GPT-3.5 and Claude-v show a similar weakness. In Section 3.4 we will introduce reference-guide method to mitigate such issues.
3.4 Addressing limitations
We present a few methods to address position bias and the limited grading ability for math questions.
Swapping positions. The position bias be addressed by simple solutions. A conservative approach is to call a judge twice by swapping the order of two answers and only declare a win when an answer is preferred in both orders. If the results are inconsistent after swapping, we can call it tie. Another more aggressive approach is to assign positions randomly, which can be effective at a large scale with the correct expectations. In the following experiments, we use the conservative one.
Few-shot judge. We assess whether few-shot examples can improve consistency in the position bias benchmark. We select three good judgment examples using MTT-becch-iike questions, GPT-3.5 and Vicuna for generating answers, and GPT-4 for generating judgments. The examples cover three cases: A is better, B is better, and tie. As shown in Table 12 (Appendix), the few-shot judge can significantly increase the consistency of GPT-4 from 65.0% to 77.5%. However, high consistency may not imply high accuracy and we not sure whether the few-shot examples will introduce new biases. Besides, the longer prompts make API calls x more expensive. We use the zero-shot prompt by default in following experiments but leave an additional study in Appendix [D22]
Chain-of-though, and reference-guided judge. In Section 3.3 we have shown LLM's limited capability in grading math and reasoning questions. We propose two simple methods to mitigate this issue: chain-of-thought judge and reference-guided judge. Chain-of-thought a widely used technique to improve LLM's reasoning capability [47]. We propose similar technique to prompt an LLM judge to begin with answering the question independently and then start grading. Detailed prompt in Figure 7 (Appendix). However, even with the CoT prompt, find that in many cases LLM makes exactly the same mistake as the given answers in its problem-solving process (See example in Figure 15 (Appendix), suggesting that LLM judge may still be misled by the context. Hence, we propose a reference- reference-guided guided method, in which we first generate LLM judge's 's answer independently, and then display it as reference answer in the judge prompt. In Table [4] we see significant improvement in failure rate (from 70% to 15%) over the default prompt.
Fine-tuning judge model. We try fine-tuning Vicuna-133 13B on arena data to act as a judge and show some promising preliminary results in Appendix F
3.5 Multi-turn judge
In MT-bench, every question involves two turns to evaluate conversational abilities. Therefore, when comparing two assistants, it becomes necessary to present a total of two questions and four responses, complicating the prompt design. We explore two possible designs, (1) breaking the two turns into two prompts or (2) displaying complete conversations in single prompt. Our finding is the former one can cause the LLM judge struggling to locate the assistant's 's previous response precisely. We illustrate a case in Figure 16 (Appendix) where GPT-4 makes an inaccurate judgment due to faulty reference. This suggests the necessity of displaying a complete conversation to enable the LLM judge to better grasp the context. We then consider the alternative design that presents two full conversations in single prompt in which we ask the I.LM judge to focus on the second question (Figure 9 Appendix)). This approach has been found to significantly alleviate the aforementioned referencing issuc.

---

4 Agreement Evaluation
We study the agreement between different LLM judges and humans MT-hench and Chatbot Arena datasets. On MT--bench, we also study the agreement among humans. MTT-bench represents a small-scale study with controlled human evaluation, while Chatbot Arena represents larger-scale study with crowdsourced human evaluation in the wild.
4.1 Setup
MT-bench. We generate answers for all 80 questions with 6 models: GPT-4, GPT-3.5, Claude-V1
Vicuna-13B, Alpaca-133h [88, and LLLMA- [39]. We then use 2 kinds of judges: LLM judges and 58 expert-level human labelers. The labelers are mostly graduate students so they are considered experts and more skilled than average crowd workers. We let LLM judges evaluate all pairs and let each human evaluate at least 20 random multi-turn questions. This resulted in around 3K votes for all questions. The detailed data collection process Appendix C
Chatbot Arena. We randomly sample 3K single-ture votes from 30K arena data, which covers models including GPT-4, GPT-3.5, Claude, Vicuna-7B/13BB Koala- 13B [16), Alpaca-138 13B, LLaMA- 13B, and DoBly-12B. We use two kinds of judges: LLM judges and collected crowd judges (2114 unique IPs).
Metrics. We define the agreement between two types of judges as the probability of randomly selected individuals (but not identical) of each type agreeing on randomly selected question. See more explanation in Appendix D.3 Average win rate is the average of win rates against all other players. These metries can be computed with or without including tie votes.
4.2 High agreement between GPT-4 and humans
We compute agreement on MT-bench data. In Table GPT-4 with both pairwise comparison and single answer grading show very high agreements with human experts. The agreement under setup $2 (w/o tie) between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (8 (8)%). This means GPT-4's judgments closely align with the majority of humans. We also show that GPT-4's judgments may help humans make better judgments. During our data collection, when human's choice deviated from GPT-4, presented GPT-4's judgments to humans and ask they are reasonable (details in Appendix C.I). Despite different views, humans deemed GPT-4's judgments reasonable in 75% of cases and are willing change their choices in 34% of cases.
The data from Arena shows a similar trend, as illustrated by T Comparing GPT-4 and other LLM judges, we find they reach similar non-tie agreement ratio between humans but the number of non-tied votes from GPT-4 much larger. This means that GPT-4 more affirmative and less suffered from position bias but other models also perform well when they give affirmative answer.
In both tables, GPT-4 with single-answer grading matches both pairwise GPT-4 and human prefer- ences very well. This means GPT-4 has a relatively stable internal rubric. Although it may sometimes perform slightly worse than pairwise comparison and give more tie votes, is more scalable method.
We then perform a breakdown analysis by computing agreement on different model pairs and categories. We only include non-tied votes. In Figure observe the agreement between GPT-4 and human progressively increases in line with the performance disparity of the model pairs (i.e., larger win rate difference). from 70% nearly 100%. This suggests that GPT-4 aligns with humans better when significant performance differences exist between the models.
GPT-4 Judge
GPT-3.5judge
Claude judge
Human
Human (frrtt turn
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.0
Win rate
0.6
0.6
0.6
0.6
0.2
022
0.2
0.2
0.0
0.0
0.0
0.0
4
Claude
GPT.35
Vicuna-12 130
Alpaca-133
LLMMA1138
GPT4
GPT.3.5
Claude
Vicuna-12 138
Alpaca-1 1.30
LLMMA 138
4
Claude
GPT.3.5
Vicuns- Alpaca-1 1.30
138
UaMA 138
GPT4
Claide
G
Vicina 138
Alpaca- 1:30
LLMMA 138
(a) All votes, first turn
[b] Non-tied votes, first turn
[c) All votes, second tum
(e) Non-tied votes, second tur
Figure 3: Average win rate of six models under different judges MT-bench.
7

---

axis value is the win rate difference be- Table 5: Agreement between two types of judges on MT-bench. "G4-Pair" and "G4-Single" denote GPT-4 with pairwise comparison and single-answer grading respectively. The single-answer grading be converted into pairwise comparison results for calculating the agreement. We report two setups: "S1" includes non-tie, tie, and inconsistent (due to position bias) votes and counts inconsistent as tie; "S2" only includes non-tie votes. The agreement between two random judges under each setup is denoted as "R=". The top value in each cell is the agreement, and the bottom gray value is #votes.
Setup
S1 (R 33%)
S2 (R 50%)
Setup
S1 (R 33%)
S2 (R 50%)
Judge
G4-Single
Human
G4-Single
Human
Judge
G4-Single
Human
G4-Single
Human
70%
66%
97%
85%
70%
66% 1325
95%
85%
G4-Pair
1138
1343
662
859
G4-Pair
1161
727
864
60%
85%
59%
84%
G4-Single
1280
739
G4-Single
1285
776
63%
81%
67%
82%
Human
721
479
Human
707
474
(a) First Turn
(b) Second Turn
Table 6: Agreement between two types of judges on Chatbot Arena. "G4-S" denotes GPT-4 with single-answer grading. "G4", "G3.5" and "C" denote GPT-4, GPT-3.5, and Claude with pairwise comparison, respectively. "H" denotes human. The remaining of table follows the same format as Table
1.0
Agromment
0.8
5
0.7
Setup
SI (Random 33%)
S2 (Random 50%)
0.0
0.5
1.0
Win rate difference
Judge
G4-S
G3.5
C
H
G4-S
G3.5
Figure 2: Agreement and win rate dif- ference. Each point corresponds to model pair and counts only the non-tie votes between the two models. The x- 85%
H
72%
66%
66%
64%
95%
94%
95%
87%
G4
2968
3061
3062
3066
1967
1788
1712
1944
60%
62%
60%
89%
91%
G4-S
2964
2964
2968
1593
1538
1761
96%
G3.55
3057
3061
1497
1567
tween the two models. The y-axis value
53%
84%
the GPT-4 and human agreement.
3062
1475
GPT-4 judge
GPT-3.5 Judge
Human
GPT-4-Single judge
0.8
0.8
Win maee
0.6
0.6
0.2
0.2
0.0
0.0
GPF4
Claude
GPF35
Vicuna118 130
Vicuna- 70
Kaala 138
Alpaca- 130
Daily 128
LLaMA 130
GPT44
Claude
GPF335
Vicuna 138
Koals Vicuna
1.8
Alpaca- 138
Do8y-128
LL 138
(al votes
fbt Non-tied vetes
Figure 4: Average win rate of nine models under different judges on Chatbot Arena.
Table 7: Category- win rate of models.
Model
Writing
Roleplay
Reasoning
Math
Coding
Extraction
STEM
Humanities
GPT-4
61.2%
67.9%
49.3%
66.1%
56.3%
66.2%
76.6%
72.2%
GPT-3.5
50.9%
60.6%
32.6%
63.8%
55.0%
48.8%
52.8%
53.8%
Vicuna- 13B
39.7%
39.2%
20.1%
18.0%
36.9%
29.2%
47.0%
47.5%
LLaMA-13B
15.1%
15.1%
7.8%
7.5%
2.1%
9.3%
6.8%
10.1%
4.3 Win rates under different judges
We plot the average win rate of models under different judges on MT-bench and Chatbot Arena Figure 3 and Figure 4 respectively. The win rate curves from LLM judges closely match the curves from humans. On MT-bench second turn, proprietary models like Claude and GPT-3.5 are more preferred by the humans compared to the first turn, meaning that a multi-turn benchmark can better differentiate some advanced abilities of models. We also list the per-category win rate of
8

---

Table 8: Evaluation results of several model variants.
Model
#Training Token
MMLU (5-shot)
TruthfulQA (0-sbot)
MT-Bench Score (GPT-4)
LLaMA-7B
35.2
0.22
2.74
LLaMA-13B
IT
47.0
0.26
2.61
Alpuca-7B
4.4M
40.1
0.26
4.54
Alpaca-13B
4.4M
48.1
0.30
4.53
Vicuna-7B 7B (sclected)
4.8M
37.3
0.32
5.95
Vicuna-7B (single)
184M
44.1
0.30
6.04
Vicuna-7B (all)
370M
47.1
Vicuna- 13B (all)
370M
52.1
0.35
6.39
GPT-3.5 GPT-4
70.0
7.94
86.4
8.99
representative models in to show how MTT-bench differentiates models, in which we see GPT-4 significantly better than others. Vicuna-13B noticeably worse than GPT-3.5/4 in reasoning, math, and coding categories, Note that in math/coding category. GPT-3.5 and GPT-4 have similar overall win-rate because they both failed to answer some hard questions, but GPT-4 still significantly better than GPT-3 in the direct pairwise comparison or single-answer grading. Please see performance breakdown of MTTbench score for each category in Appendix D.4
Human Preference Benchmark and Standardized Benchmark
Human preference benchmarks such MT-bench and Chatbot Arena serve valuable additions to the current standardized I.M benchmarks. They focus on different aspects of a model and the recommended way is to comprehensively evaluate models with both kinds of benchmarks.
We evaluate several model variants derived from LLaMA on MMLU 19 Truthful QA [26) (MCI), and MT-bench (GPT-4 judge). The training details are in Appendix
Since we have shown that GPT-4 single-answer grading also performs well in Section 4.2 we use GPT-4 single-answer grading for MT-bench in favor of its scalability and simplicity. We ask GPT-4 to give score for each turn on a scale of 10 by using our prompt templates (Figure 6o Figure I0 and report an average score of 160 80 x 2 turns. Table Table&Sshoows shows the results. We find that fine-tur the-ttningg high-quality dialog datasets (i.e., ShareGPT) can consistently improve the model performance on MMLU and the improvement scales with fine-tuning data size. On the other hand, small high-quality conversation dataset can quickly teach the model a style preferred by GPT-4 (or approximately human) but cannot improve MMLU significantly, as shown by the Vicuna-7B (selected) which is trained with only 4.8M tokens or 3K conversations. In Table no single benchmark can determine model quality, meaning that approximate
comprehensive evaluation needed. Our results indicate that using LLM-as-a-jues human preferences is highly feasible and could become a new standard in future benchmarks. We also hosting regularly updated leaderboard with more models Notably, DynaBench (1T) research platform dedicated to dynamic data collection and benchmarking, aligns with our spirit. DynaBench addresses the challenges posed by static standardized benchmarks, such saturation and overfitting, by emphasizing dynamic data with human- in-the-loop. Our ALM-as-a-juude approach can automate and scale platforms of this nature.
Discussion
Limitations. This paper emphasizes helpfulness but largely neglects safety. Honesty and harm- lessness are crucial for a chat assistant as well [2]. We anticipate similar methods can be used to evaluate these metries by modifying the default prompt. Additionally, within helpfulness, there are multiple dimensions like accuracy, relevance, and creativity, but they are all combined into a single metric in this study. more comprehensive evaluation separating these dimensions. We propose preliminary solutions to address the limitations and biases of LLM-as-aat -a-a-saaa a-ja-e in Section but we anticipate more advanced methods can be developed.
be developed by analyzing and
Data collection and release. Appendix describes the detailed data collection and release processes, which include the instructions we give users, the sereenshots of the data collection interface, the information about participated users, and the content the released data.
http:://wwggglagtooooo///eee/eep arena- -aaee---eddebbardd

---

Societal impacts. The societal impact of this study is multi-faceted Our evaluation methods can help enhance chatbot quality and user experiences. However, addressing biases in these methods is crucial. Our dataset enables better studies of human preferences and model behavior. Advanced chat assistants may replace certain human tasks, resulting job displacements and new opportunities.
Future directions. 1) Benchmarking chatbots at scale with a broader set of categories 2) Open-source LLM judge aligned with human preference 3) Enhancing open models' math/reasoning capability.
7 Conclusion
In this paper, we propose LLM--as-a-judge for chatbot evaluation and systematically examine its efficacy using human preference data from 58 experts on MT-bench, well thousands of crowd- users on Chatbot Arena. Our results reveal that strong LLMs can achieve an agreement rate of over 80%, on par with the level of agreement among human experts, establishing foundation for an LLM-based evaluation framework.
Acknowledgement
This project is partly supported by gifts from Anyscale, Astronomer, Google, IBM, Intel, Lacework, Microsoft, MBZUAI, Samsung SDS, Uber, and VMware. Lianmin Zheng is supported by a Meta Ph.D. Fellowship. We extend thanks Xinyang Geng, Hao Liu, Eric Wallace, Xuecheng Li, Tianyi Zhang, Qirong Ho, and Kevin Lin for their insightful discussions.
References
[1] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey. Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint 200i::2305.10000. 2023.
[2] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586) 2022.
[3] Niels J Blunch. Position bias multiple-choice questions. Journal of Marketing Research, 21(2):216-200, 1984.
[4] Jonathon D Brown. Evaluations of self and others: Self-enhancement biases in social judgments. Social cognition, 4(4):353-376, 1986.
[5] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 12731 2023.
[6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint wxwv::2077.03333 2021.
[7] Cheng-Han Chiang and Hung yi Lee, Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.0193: 2023.
[8] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1803.0500 2018.
[10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiy preprint arXiv:2110. 14168, 2021.
10

---

[11] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, Advances Neural Information Processing Systems, 35: 15.16444-1635) 2022.
[12] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 0553305./3344 14314, 2023.
[13] Shizhe Diao, Rui Pan, Hanze Dong, Ka Shun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. Lmflow: An extensible toolkit for finetuning and inference of large foundation models, arXiv preprint arXiv:2306. / 12420, 2023.
[14] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305. 14387, 2023.
[15] Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang. Chongyang Tao, Dongyan Zhao, and Qingwei Lin. Mmdialog: A large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation. arXiv preprint arXiv:2211.057/ 2022.
[16] Xinyang Geng, Arnay Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023.
[17]
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303. 15056, 2023.
[18] Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms. arXiv preprini
arXiv:2305.15717, 2023.
[19] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint 1xXiv:2009003300 2020.
[20] Fan Huang. Haewoon Kwak, and Jisun An. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. arXiv preprint arXiv:2302.07736. 2023.
[21] Douwe Kicla, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking benchmarking in nlp. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4110-4124, 2021.
[22] Miyoung Ko, Jinhyuk Lee, Hyunjae Kim, Gangwoo Kim, and Jaewoo Kang. Look at the first sentence: Position bias in question answering. arXiv preprint arXiv:2004. /4602, 2020.
[23] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. Ope- nassistant conversations-dee demarrrtnnsdmmmmrriiiiiingg large language model alignment. arXiv preprint arXiv:2304.07322 2023.
[24] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint 1.09100 2022.
[25] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[26] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint ArXiv:2109.079555 2021.
[27] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le. Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 23X.::2301.13683 2023.
[28] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirii. Cross-task general- ization via natural language crowdsourcing instructions. In ACL, 2022.
[29] OpenAl. Evals is a framework for evaluating 1lms and I1m systems, and an open-source registry of benchmarks. bttpp://gitthb. com/ openai/evasl
[30] OpenAl. Gpt-4 technical report, 2023,

---

[31] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[32] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, -3,, 2002.
[33] Baolin Peng. Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprint arXiv:2304.03277 2023.
[34] Priya Raghubir and Ana Valenzuela. center-of-inattention Position biases in decision-makinn. Organizational Behavior and Human Decision Processes, 99(1 (1 1):66-80, 2006.
[35] Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
[36] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
[37] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint (2X6::2206.04111 2022.
[38] Rohan Taori, Ishaan Gulrajani. Tianyi Zhang. Yann Dubois, Xucchen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followinng llama model. 2023.
https://gi thtp://ggttub.....m watp///ttssuaa/taaaaaaeaaaaa
[39] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[40] Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305. 17926, 2023.
[41] Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. Position bias estimation for unbiased learning to rank in personal search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 610-618, 2018.
[42] Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang. Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. Pandalm: An automatic evaluation benchmark for I1m instruction tuning optimization, 2023.
[43] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.00 14751, 2023.
[44] Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions, 2022.
[45] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. via declarative instructions on 1600+ tasks. In EMNLPP
Super-aauraiistnncioonggnnaaannn 2022.
[46] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint 200.v:2/0900/655:: 2021.
[47] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint 27X1v:2201.11993, 2022.
12

---

[48]
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models follow complex instructions. arXiv preprini arXiv:2304. 12244, 2023.
[49] Zongheng Yang, Zhanghao Wu, Michael Luo, Wei-Lin Chiang, Romil Bhardwaj, Woosuk Kwon, Siyuan Zhuang, Frank Sifei Luan, Gautam Mittal, Scott Shenker, and lon Stoica. SkyPilot: An intercloud broker for sky computing. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 437-455, Boston, MA, April 2023, USENIX Association.
[50] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint OrXiv:/905.07830 2019.
[51] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaoho Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centrin benchmark for evaluating foundation models. arXiv preprint arXiv:2304.066 063640664444...00 2023.
[52] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. Lima: Less is more for alignment. arXiv preprint Xiv::2305 1206, 2023.
13

---

A Prompt templates
We list the prompt templates for LLM judges. Please refer our github repository for full details.
[System]
Please act as an impartial judge and evaluate the quality the responses provided by two AI assistants the user question displayed below. You should choose the assistant that follows the user's s instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail their responses. Begin your evaluation by comparing the two responses and provide short explanation. Avoid any position biases and ensure that the order which the responses were presented does not influence your decision. Do not allow the length the responses influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A better, "[[B]]" assistant is better, "[[C]]" for a tie.
[User Question] (question)
[Th Start Assistant A's Answer]
(answer_ay
[The End Assistant A's Answer]
[The Start Assistant B's Answer]
(answer_De b) [The End of Assistant B's Answer]
Figure 5: The default prompt for pairwise comparison.
[System]
Please act as an ispartial judge and evaluate the quality of the response provided by an AI assistant to the user question displayed below. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response. Begin your evaluation by providing short explanation. Be as objective possible. After providing your explanation, please rate the response on a scale of 1 to 10 by strictly following this format: "[[rating]]", for example: "Rating: [[5]]".
[Question] (question)
[The Start of Assistant's s Answer] (answer) [The End of Assistant's Answer]
Figure 6: The default prompt for single answer grading.
14

---

[System]
Please act as an impartial judge and evaluate the quality the responses provided by two AI assistants the user question displayed below. Your evaluation should consider correctness and helpfulness. You will be given assistant A's answer, and assistant B's answer. Your job to evaluate which assistant's answer is better. You should
independently solve the user question step-by-step first. Then compare both assistants' answers with your answer. Identify and correct any mistakes. Avoid any position biases and ensure that the order which the responses were presented does not influence your decision. Do not allow length the responses to influence your evaluation. Do not favor certain names explanation, output your final verdict by strictly following this format: "[[A]]" assistant better, "[[B]]" assistant is better, and "[[C]]" for tie.
assistants. as objective as possible. After providing your
[User Question] (question)
[The Start Assistant Answer]
(answer_a) __
[The End Assistant A's Answer]
[The Start Assistant Answer]
{answer_d b}
[The End Assistant B's Answer]
Figure 7: The chain-of-thoughe prompt for math and reasoning questions.
[System]
Please act as an impartial judge and evaluate the quality the responses provided by two assistants the user question displayed below. Your evaluation should consider correctness and helpfulness. You will be given a reference answer, assistant A's answer, and assistant B's answer. Your job is to evaluate which assistant's answer better. Begin your evaluation by comparing Identify and correct any mistakes. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A better, "[[B]]" assistant is better, "[[C]]" for a tie.
assistants' answers with the reference answer.
not allow the
[User Question] (question)
[The Start Reference Answer]
(answer_rets
[The End Reference Answer]
[The Start Assistant Answer]
(answer_a) _]
[The End Assistant A's Answer]
[The Start Assistant Answer]
(answer b)
[The End Assistant B's Answer]
Figure 8: The prompt for reference-guided pairwise comparison.
15

---

[System]
Please act ispartial judge and evaluate the quality of the responses provided by two AI assistants the user question displayed below. You should choose the assistant that follows the user's instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide short explanation. Avoid any position blases and ensure that the decision. Do not allow
order in which the responses were presented does not influence the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant is better, and "[[C]]" for a tie.
< The Start of Assistant A's Conversation with User|> ### User: [question 1}
### Assistant A: (answer 1)
### User:
[question 2}
##я Assistant A:
(answer 2)
< The End of Assistant A's Convensation with User|>
< The Start of Assistant B's Conversation with User|> ##я User: {question 1}
ння Assistant B: (answer 1)
#1# User: (question 2}
### Assistant B:
(answer 2)
<he End of Assistant B's Convensation with User|>
Figure The prompt for multi-turn pairwise comparison.
16

---

[System]
Please ispartial judge and evaluate the quality the response provided evaluation should consider correctness and
assistant the user question. helpfulness. will given a reference answer evaluation should focus the assistant's answer with the reference answer. Identify and objective as possible. After providing your explanation, you scale of 10 by strictly following this format:
assistant's answer. second question. Begin your
evaluation by comparing the assistant's correct any mistakes. must rate the response "[[rating]]", for example: "Rating: [[5]]".
< The Start of Reference Answer|> ### User: [question_1)
#h# Reference answer:
(ref_answer._
### User: {question_2) 2}
### Reference
(ref_anm aitwer_wer__ 2}
< The End of Reference Answer]>
<| The Start of Assistant A's Conversation with Userl> нна User:
{question 1}
тun Assistant A: {answer_11 1}
##8 User: [question_2 2}
нua Assistant A:
(answer 2)
< The End of Assistant A's Conversation with User|>
Figure 10: The prompt for reference-guidee multi-tum tur single-answer grading.
17

---

B
Case Study
We list several case studies. The examples are generated by gpt-4-0314. They may not be fully reproducible with future GPT-4 versions.
Question: What are some business etiquette norms when doing business in Japan?
Assistant A:
Assistant
Here are some business etiquette norms to keep nind when doing business in Japan:
when doing business in Japan, it is important to be aware
respect the country's s business stiquette nores. Sone of these noms include:
1. Bowing: Bowing is cormon greeting in
Japan. It sign respect and should reciprocated.
1. Greetings: A handshake and bow the standard greeting in Japan. and to offer a firm handshake.
important
deeply
2. Business cards: Business cards are exchanged the beginning of a neeting. It
Business cards: Japan, business cards are exchanged after the initial greeting. It inportant to handle the card with respect and to read it carefully before placing it the table.
important to receive and give business cards with bath hands and to take a nonent
putting 3. Punctuality: Being on time is inportant in
the card
away.
Japan. Arrive a few minutes early to show respect for your colleagues.
Meetings: Neetings in Japan are typically formal and conducted in hierarchical manner. is important to address people by their titles and avoid direct eye contact.
4. Dress code: Dress conservatively and
should dark suits and ties, while women should conservative business attire. Language: Learn
professionally.
Punctuality: Punctuality is highly valued in Japan. is important to arrive on time for neetings
Japanese
prepared advance.
phrases to show respect for the culture. It also inportant to speak clearly and avoid using slang idions.
5. Gift giving: Gift giving is common practice in Japan, but is inportant to be mindful the occasion and the relationship. Avoid giving gifts that are considered too personal or extravagant.
6. Gift giving: Gift giving is common practice Japan. Bring small gift, such box of chocolates or small souvenir fron your hone country.
Dining: Dining is important part of doing business in Japan. It is important to be aware of the proper table slurping or naking loud noises while eating. Comunication: Connunication in Japan is typically indirect and reserved. It inportant to listen carefully and to avoid confrontational language.
7. Dining etiquette: When dining with colleagues, wait for the host to begin eating before you start. Use chopsticks correctly and avoid leaving food your plate.
avoid
Remember, showing respect for Japanese culture and custons is key to building strong business relationships in Japan.
Overall, it important to be respectful,
punctual, and mindful hierarchy when daing busis in Japan.
GPT-4 Judgment (when A is placed in the first position):
Bath Assistant A and Assistant 8 provide helpful and relevant information about business etiquette norns Japan. Howeven, Assistant A's answer
detailed and organized, covering wider range of topics such as dress cade and language, which are nat nentioned in Assistant B's response.
Assistant also provides nore specific advice on how to handle certain situations, such as is comprehensive
exchanging business cards and dining etiquette. Therefare, Assistant A's and useful for someone looking to understand Japanese business etiquette.
GPT-4 Judgment (when B is placed in the first position):
Both Assistant A and Assistant B provide helpful and relevant information about business etiquette is
norns Japan. However, Assistant B's such as addressing people by their titles, avoiding direct eye contact, and being mindful of comnunication styles. Assistant A's answer is more concise but lacks some the depth found Assistant B's response. Therefore, Assistant B's answer is better in this case.
detailed and
wider range topics,
An example of position bias. When Assistant A is placed in the first position, GPT-4 thinks A is better, but its verdict changes when we swap the position of A and B. We observe similar pattern from other LLM judges such as Claude/GPT.3.5.
Figure 11:
18

---

Question: What is the central dogma of molecular biology? What processes are involved? Who named this?
Assistant A:
Assistant B:
The central dogna of nolecular biology is the fundamental principle that explains how genetic information flows within biological system. states that DNA is transcribed into RNA, which is then translated into proteins. This means that the genetic information stored in DMA is first copied into RNA, which then serves as template for the synthesis of proteins.
The central dogna of molecular biology is the fundamental principle that explains how genetic information flows within biological system. states that DNA is transcribed into RNA, which is then translated into proteins. This means that the genetic information stored in DNA first copied into RNA, which then serves as a template synthesis proteins.
The processes involved in the central dogna are:
The processes involved the central dogna are:
1. Transcription rephrased: The act of copying RNA, 2. Translation rephrased: The act of using genetic information RNA to create proteins. Transcription:
1. Transcription: The process by which the copied 2. Translation: The process by which the genetic infarmation RNA is used to synthesize proteins.
genetic information
genetic information in
process which the genetic information in DNA is copied into ANA. genetic information in RNA is used to synthesize proteins.
4. Translation: process by
term "central dogna" coined by Francis Crick, of the co-discoverers the structure of DNA, 1958.
The term "central dogna" was coined by Francis Crick, one the co-discoverers of the structure of DNA, in 1958.
GPT-3-5 Judgment:
Assistant provides
detailed and accurate response the user's question. While both assistants correctly state the central dogna of molecular biology and who coined the tem,, Assistant provides comprehensive explanation
dogna, including the rephrased definitions of transcription and translation. Therefore, choose as the better assistant.
processes
Claude-v1 Judgment:
assistants answer question by defining central dogna detailing involved, transcription and translation. However, assistant A provides more in-depth and detailed explanation by briefly rephrasing and elaborating on these pracesses. Assistant also nentions who coined the "central dogna", which provides useful understanding of the topic. Overall, assistant gives a fuller and nore insightful response, sa I have chosen as better assistant for this question.
pracesses
and shows comprehensive
GPT-4 Judgment:
central dogna molecular biolagy, the pracesses involved, and who named it. However, Assistant A's answer contains minor redundancy by rephrasing transcription and translation twice. Assistant B"s answer is nore better.
Assistant Assistant provide accurate relevant
cancise avoids this redundancy. Therefore, Assistant B's
An example of "repetitive list" attack to examine verbosity bias. Except for the two rephrased items (highlighted in red), Assistant A's answer is exactly the same Assistant B. Both GPT-3.5 and Claude-vl show verbosity bias towards the longer and repetitive answer. Only GPT-4 successfully detected this attack.
Figure 12:
19

---

Question: Benjamin went a bookstore and purchased variety of books. He bought 5 copies of a sci-fi novel, each priced $20, copies of history book priced at $30 each, and copies of philosophy book for $45 each. What was the total cost of his purchases?
Assistant A:
Assistant total of his purchases
The total
Benjamin's purchases $125.
GPT-4 Judgment:
the user's question. Assistant A's response incorrect. To calculate the total cost of Benjamin's purchases, we need to multiply the number of books by their respective prices
Assistant provided the correct
results:
sci-fi novels $28 $180
history books $38 $98
philosophy
$90
Tatal cast $198 $90 $90
$295
Assistant B's response is accurate and directly answers the user's question. Therefore, my final verdict is: [[B]]
GPT-4's Answer (if asked separately):
The cost of the sci-fi novels is $28 $c<5"28-100>:100
The cost of the history books The of philosophy books is
$38 $c<3338-90>>98.
$<c2"45-90>>98.
The total cost of his purchases is $180 + 599 $98 Sxc188-98-99--2092288
Answer: (boxed(288).
Figure 13: With a default prompt. GPT-4 shows limited capability grading math questions. Despite
being able answer the question itself, its judgment influenced by the given answers, leading to arithmetic mistakes highlighted in yellow.
Question: Each problem consists of three statements. Based on the first two statements, the third statement
may be true, false, or uncertain.
1. Oranges cost more than apples.
2. Oranges cost less than bananas.
3. Bananas cost more than apples and bananas cost more than orange.
If the first two statements are true, then the third statement is
Assistant A:
Assistant B:
false.
False.
If oranges cost nore than apples (statement 1) and oranges cost
If the first two statements are true, then the third statement false. that oranges cost more than apples, and the second statement says that oranges rast less oranges than bananas, which contradicts the third statement that bananas cost more than apples and than oranges. Therefare, the third statement must be false.
bananas (statenent 2), then it is not possible for bananas to cost less than oranges and also cost more than both apples and oranges (statenent 3). Therefore, false.
statement
Judgment:
Both Assistant and Assistant a provided accurate and relevant responses the user's question. concluded the third Assistant B's response is slightly clearer and more concise in explaining the contradiction between the statements, Therefore, choose Assistant as the better response.
false
the first two statenents. However,
[[o]]
GPT-4's Answer (if asked separately):
True.
apples (statement and oranges less logically follows that bananas cost more than apples and bananas cost nore than oranges, which is the third statement. so, the first two statements are true, the third one also true.
oranges
(statement 2),
An example of GPT-4's limited capability in grading reasoning question. Despite GPT-4 knows how to solve the question (if asked separately), made wrong judgement saying both assistants' wrong answers are correct.
Figure 14:
20

---

Question:
How many integers are the solution of inequality |x + 5 < 10??
Assistant A:
Assistant B: To solve the inequality |x + 5| < 10, we need to consider cases:
There are 19 integers in the solution of the inequality 10.
Case 1: x 5 is positive or zero this case, inequality becones which simplifies to <.
10,
Case 2: is negative In this case, the inequality becones -(x + 5) < 1e, which simplifies sides gives -x 15, and multiplying by -1 (and flipping the inequality) gives -15
Adding
So the solution to the inequality is -15 < x 5. The integers in total of 20 integers. Therefore, there are 20 integers in the solution of the inequality.
range =14, -13, -12, .... 4, for
GPT-4 Judgment:
To solve the inequality |x 5| 18, we need to consider two cases:
Case 1: x is positive or zero this case,
inequality becomes
18, which simplifies
Case 2: x is negative this case, the inequality becones 5) which simplifies sides gives -x 15, and nultiplying by -1 (and flipping the inequality) gives x > -15.
Adding
So the solution the inequality is -15 5, The integers in this range are -14, -13, -12, 4, for a total of 20 integers. Therefore, there are 20 integers in the solution of the inequality.
Comparing the answers provided by the two AI assistants:
Assistant A: 19 integers
Assistant
integers
The correct answer is 20 integers, Assistant provided the conrect and more helpful response. verdict: [[B]]
GPT-4's Answer aifasked separatedly):
split inequality into
1) $x+5kge es, then =xx55 so, have $x+5c10s, which means $xc5s. Since $x451ge es, have $xige -55. in this $x5 any integer from $-55 to inclusive, which gives $5-(-5)41-18s integers.
2) If $xe5ces, then (xx=5)= So, have (18x)5 which means $x65>-185, which means $x> 155. Since $x45c0s, we have Sxc-56. So this case, $x$ can be any integer fron $-145 to $-65, inclusive, which gives 3-6-(-14)11195 integers.
total,
(bxee11bo integers in
An example of GPT-4's wrong judgment with thain-of-thouule prompt. We can see GPT-4 exactly copied Assistant B's answer (which contains arithmetic errors) and determined Assistant A's answer is incorrect. This suggest GPT-4's chain-of-thouge process can be significantly influenced by given answers despite we ask to think independently.
Figure 15:
21

---

Share ideas for adapting art masterpieces into interactive experiences for children. List 5 specific artworks and associated ideas.
Question:
Assistant A:
Assistant B: (onitted)
Sure, here five ideas for adapting nasterpieces into interactive experiences for children:
1. "The Starry Night" by Vincent Gogh Create virtual reality experience where children can explore the night sky and see the stars and constellations as van Gogh different colors and brush strakes.
They can
starry night
using
2. "The Persistence Menory" Salvador Dali Create sensory experience where children can touch, feel, and smell different objects to explore the surrealist
Dali's painting. They landscapes using different naterials and textures.
(omitted)
Follow-up Question: Write concrete plan for your second example. Include budget estimates.
Assistant A:
Assistant B: adapting Persistence Memory" by (onitted)
concrete
Dali into an interactive experience for children:
1. Budget: $5,000
2. Objectives:
sensory experience that allows children explare the surrealist thenes of Dali's painting.
To encourage children to create their own surreal landscapes using different materials and textures.
EPT--4Judgmee
Assistant provided a concrete plan adapting "The Persistence Memory" by Salvador Dali into interactive experience for children, while Assistant provided plan for the "Nona Lisa interactive Experience for Children." The user's follow-up question specifically asked for concrete plan the second example, which the Lisa" experience Therefore, Assistant B's answer is more relevant and directly addresses the user's request. My final verdict 1s: [[B]]
by
In this example, despite Assistant A correctly followed user's instruction to generate concrete plan for the second example its previous response, GPT-4 wrongly referred to the second example in Assistant B's response, resulting in wrong judgment. This suggests the prompt design that breaks the questions into two prompts may cause LLM judge struggle locate assistants' previous responses.
Figure 16:
22

---

C Data Collection
We describe our data collection process for both MT-bench and Chatbot Arena.
C.1
MT-bench human evaluation
Figure 17 shows the normal voting interface. Figure 18 shows that we additionally show GPT-4's judgment to users and ask if it is reasonable when a human differs from GPT-4
Instruction
Please act an importia judge and esaluate the qualit of resprrse provnded two arsistacts to the unw quantion displeyed beiew: ihu shoull choese the assistnnt that folloan the uner's digets craatiity
oftter responuss your evaluation by compaiing the nwo resperues provide short enp explasation Avoid any positional buises ensuve that the order in which the resporsies wore presented does nor influence Finulyy indicate your clicking
deciion, De leugth
your
burmnn Flesse use button sponetyy
selemtiicaaly
renewable carehuly hondlingthe ethica dienmoogsumuuudnggh
sources Propose options
"Revokataniiing Benewable Eneage New traukthouugt tha-bnergy Production While Addressing Concemm"
reeaichrough Ramowable tio-tnergy
2. Renowoble o Energy to Power Chiiss for the Nout Years
Erengr for Suata mable future tallncinng Envrrnnental Semerite th ethical
3. Bio Bnerge traithhouuh Gives Hope to Earthis Futura
Consideration"
Blo Energe breakhhough Could Save Eardh's Enviromment
3. "Bea-inn iniendl and tthical the energe How Sclence theing for Greener Nuture"
4, "The othics of bie Energr A Look at Latest Otscoventes in sustrinable Energy Production"
followup question
aduatments
bed advertsamerteo called
Andistant A
Assistant B
Go Gream with Pls Echical and too friendly Bio Enargy Broakthrough""
Benowab a Enargy Power Chiaa the Noet 100 hears
is better
is better
Tie
+ Skip 11/5 left)
The screenshot of MT-bench data collection. We show an instruction similar the prompt we give to GPT-4. We present questions from MT-bench and answers from two random anonymous assistants and ask which one is better. We present the first-turn conversation and ask humans vote, then repeat the same procedure for the second-turn, A user can skip up to 5 questions if they are not confident. For some questions (e.g., math, reasoning), they can also see a reference solution.
Figure 17:
Instruction
choos Please act as Inporraat judge and evalvate the qually ofe responses prowided by two asdssants to the use euestion displayd betow: No showld cloose the assisontt that follows thowsers Instructions
helprutness better enotepptactioo Avckd wny postoonl borser ensoon that onder which repposser newe presested doee inffumnee your your eualustion compusing the nwo repponses and providee
shoulf bectors
cruatiity
the
chccwien. allow leigth the dutoo, use dutton apansely.
irfaenze evsisstonn fever names assitents.
possble, Aisslyy indicate venket clicking
User
Hele construct catchy, sclentificall accuntte heade art dle lenes discavery renovable bio while careluly by handling the sthical dllemma suwwurudngggggggnn blo-enargy oourcas Propone appions.
Asdistant
Assistant
1. "Revolution bing Renowable Energy, New Breakthro gho Bio Energy Production While Addnwasing tih Comoems"
1. rouhh in Renewable Bio Energy
2. Beneweble o Energy to Power chrss for the Med Vears
2. "Blo Energy fora Susta hadle Future Balanding Envrrommntal Benofts Ethical Corsidentiooo"
Blo Ene Bresishiough Gives Hopp Future
Bio-Cneryy Breitthough Sould Save Earth's Envirormnnt
3. TEoo Prtendly and Ethical Ble Energy How Science is Povingme Way for Greener Puturee
4 Thee athics of bic Energe A Look the Leteat Oiscoveriss in Ststainable Energy Production"
NOTICE
Hern lox sddymmnt that diuggness th your cheice. nemonaable?
Judgment
sciertificall address ethikl surouundin other not aódress the etricaal silerrres as any the propose headi hoode nes. Thercfore, Assiston: A's response la botter.
providus
My firal vesticts [[]]
change my
Reesonable; keep my
Wrongnjudgment
Figure 18: The screenshot of MT-bench data collection. When human's 's vote differs from GPT-4, we additionally show GPT-4's judgment (red region the screenshot) and ask the user to click one of the three buttons to decide whether GPT--4's judgment is reasonable.
23

---

To invite participants, we obtained their consent by letting them sign an application form. We pay them $20 for judging 20 questions, which corresponds hourly rate of around $35. The participants are mostly graduate students from more than ten universities.
C.2 Chatbot Arena
shows screenshot of Chatbot Arena. Users are required accept the terms of use, which obtain their consent and give us the right to release the conversation data. The instructions are shown at the top of the interface. This free website. We do not pay users and any user can use this platform without registration. More introductions and analyses can be found at https 7/1msys. wtg/blogg/02333122 05 arena/
Figure T9
X Chatbot Arena X
Char monyen
wer
rowaled feryour Convenatio ldandiy boyworch Cheoftt Woun or wer oounttoed Aoketabonnt
Clict "Caar satary saraness lonn.
Itnd Carael Rtsoet
ay sentica oses regirst net to levvnng temn The vece reserch urevin momeet nancormme voa only anly provito bor mese menures andray greent cneeeeeconen cnsuteetti see dege, heneki undeen, canty, yu yeurns sere sar age the get destoaee a Dealhe Mastsubee auae mewe The dore bndaay desiio anill
Bettle
duuingg Yourennion deteoad part uitu
roven' by tatp Thereado bocc
muke
Rudda
prefess at seek you na persiorrs
Quarterly frnancial Rapport
profess bawking you supe viier feedback Quarterry freancial Repprrt propeepend rak spoc fiuily abous data andyett procntet on stgle the dartt kusiant duewn Keep the and derppinn.
pouperpperal nahsune duclly araut the sulypia, prosonuation styl,, und of con skenn, an inppont..
Sutjert Roquest
Quan fleport
Sugjet Roquest onQuar bear perwisers Narvel.
oeer puperisso''s Marne),
thope this email tindsyn wiill tom wiiting to wwek fwwittork on the currenlyy Finandial The report
hope thic fadt oun ttinng to yneiy thetbaricon the Quommeyy Riaanii
new fiaaniial
confident
your thoughte veluatlerssiie
greats
ggeations
you maay hown shour rass aoady prosentation dry and raiity concuutions dimws
nonuvtation etyn, andt af sdoes
Plaace int me hewwhwwww if taro spectte snose soncerv or shwee in intormatton
The
sancial anning decisios muiking
will inaaustt guiding towards momment thees thave attached the spprrt this emaa for ratorenes yyou cou prow ree feedback
Thanicyca time read amail forwart heari fromm Best rigards Numm))
villa hoould graisttd
regerds.
fouu Numna)
better
better
Buth
press
Regenerate
Clorrinary
Store
Figure 19: The screenshot Chatbot Arena.
C.3
Data Release
We will clean the Personal Identifiable Information (PII) and tag toxic conversations with OpenAl moderation APIs for our dataset release.
24

---

D Additional Experimental Results
We present some additional experimental results.
D.1 Position bias
We test two more prompts and present the full results in "score" changes the default prompt to let the model output two absolute scores instead of which one is better. "short" is simplified version of our default prompt by removing instructions like "Avoid any position bias.." "Begin your evaluation ... and provide short explanation". We can find different prompts have different effects on different models. For example, the "score" prompt can increase the consistency of GPT-3.5 but decreases for Claude-v1 and GPT-4.
Table 9: Position bias on different models and prompts. Consistency is the percentage of cases where a judge gives consistent results when swapping the order of two assistants. "Biased toward first" is the percentage of cases when a judge favors the first answer. "Error" indicates wrong output formats. The largest numbers in each column in bold.
Judge
Prompt
Consistency
Biased toward first
Biased toward second
Error
default 23.8%
75.0%
0.0%
1.2%
rename 56.2%
11.2%
28.7%
3.8%
claude-v1
score
20.0%
80.0%
0.0%
0.0%
short
22.5%
75.0% %
2.5%
0.0%
default 46.2%
50.0%
1.2%
2.5%
gpt-3.5-turbo
rename
51.2%
38. 38.8%
6.2%
3.8%
score
55.0%
33.8%
11.2%
0.0%
short 38.8%
57.5%
3.8%
0.0%
dcfault 65.0%
30.0%
5.0%
0.0%
rename 66.2%
28.7%
5.0%
0.0%
gpt-4
score 51.2%
46.2%
2.5%
0.0%
short 62.5%
35.0%
2.5%
0.0%
As shown in Table 10 position bias more noticeable on open questions like writing and stem/hu- manity knowledge questions. On math and coding questions, L.MM judges are more confident even though their judgments can often be wrong, as we show in Section 3.3 Finally, we study how the model pairs influence position bias by using GPT-4 and the default prompt to judge three different model pairs. As shown in Table IT] the position bias more noticeable for models with close performance and can almost disappear when the performance of the two models differs a lot.
Table 10: Position bias on different categories. The two largest numbers in each column are in bold.
Category
Consistent
Biased toward first
Biased toward second
writing 42.0%
46.0%
12.0%
roleplay
68.0%
30.0%
2.0%
reasoning
76.0%
20.0%
4.0%
math
86.0%
4.0%
10.0%
coding
86.0%
14.0%
0.0%
extraction
78.0%
12.0%
10.0%
stem 44.0%
54.0%
2.0%
humanities
36.0%
60.0% %
4.0%
Table 11: Position bias on different model pairs.
Pair
Consistent
Biased toward first
Biased toward second
GPT-3.5 vs Claude-Vi V1
67.5%
23.8%
8.8%
GPT-3.5 vs Vicuna- 13B
73.8%
23.8%
2.5%
GPT-3.5 LLaMA-13B
98.8%
1.2%
0.0%
25

---

D.2 Few-shot judge
We examine how few-shot examples improve LLM judges. As shown in Table 12 they improve the consistency of all three LLM judges significantly. It almost alleviates the position bias of GPT-4, but moves the position bias of GPT-3.5 from the first position to the second position. We then measure the agreement between few-shot GPT-4 pairwise comparison and humans MTT-hench, but found it performs similarly to zero-shot GPT-4 pairwise comparison.
Table 12: Improvements of the few-shot judge on consistency for position bias.
Model
Prompt
Consistency
Biased toward first
Biased toward second
Error
zero-shot
23.8%
75.0%
0.0%
1.2%
Claude-v1
few-shot
63.7%
21.2%
11.2%
3.8%
zero-shot 46.2%
50.0%
1.2%
2.5%
GPT-3.5
few-shot 55.0%
16.2%
28.7%
0.0%
zero-shot 65.0%
30.0%
5.0%
0.0%
GPT-4
few-shot 77.5%
10.0%
12.5%
0.0%
D.3 Agreement Evaluation
Agreement calculation. We define the agreement between types of judges as the probability of randomly selected individuals (but not identical) of each type agreeing on a randomly selected question. For example, if we comparing GPT-4 and Claude, the agreement is the probability of GPT-4 and Claude agreeing on the vote for a randomly selected question. If we are comparing GPT-4 and humans, the agreement is the probability of GPT-4 and randomly selected human agreeing on the vote for a randomly selected question. The agreement among humans themselves is the probability of randomly selected but not identical humans agreeing on the vote for randomly selected question.
Note that the agreement among humans could be lower estimation compared the agreement of GPT4 and humans. Consider three humans who voted "A", "A", and "B" for a question, respectively. The agreement among them is only } there are three pairs "(A, A)", "(A, B)", and "(A, B)". But the agreement between GPT4 and those three is GPT4 voted "first" and J otherwise.
Therefore, to have a more comprehensive understanding what happened, we introduce a new judge type called human- majaartt which considers the majority of human votes for each question. The agreement between GPT4 and numan-majority then calculated as the probability of GPT4 agreeing with the majority of human votes randomly selected question. The upper bound of the agreement between GPT-4 and humans is the agreement between humaan human-majority and human. When there is no majority vote for question, the agreement is counted by even split. For example, if there an equal number of "A" and "B" human votes for question, and GPT4 votes "A", the agreement is counted 2 on this question.
More results. Table 13 shows more agreement results on MT-bench. In addition to expert labelers (denoted as "Human" we also include author votes (denoted as "Author").
D.4
Category-wise scores with single-answer grading
We use single-answer grading to evaluate 6 models on MT-bench and plot the category-wise scores in
Figure 20
26

---

Table 13: Agreement between two types of judges on MT-bench. "G4-P" and "G4-S" denote GPT-4 with pairwise comparison and single-answer grading, respectively. "C" denotes Claude. "Human" denotes expert labelers (excluding authors). "Human-M" denotes the majority vote of humans. The single-answer grading can be converted into pairwise comparison results for calculating the agreement. We report two setups: "S1" includes non-tie, tie, and inconsistent (due to position bias) votes and counts inconsistent random judges under each setup is denoted as "R=". The top value in each cell is the agreement, and the bottom gray value #votes.
tie; "S2" only includes non-tie votes. The agreement between two
Setup
S1 (R 33%)
S2 (R 50%)
Judge
G4-S
C
Author
Human
Human-M
G4-S
Author
Human
Human-
70%
63%
69%
66%
67%
97%
94%
92%
85%
85%
G4-P
1138
1198
345
1343
821
662
582
201
859
546
66%
67%
60%
60%
90%
94%
85%
85%
G4-S
1136
324
1280
781
563
175
739
473
58%
S4%
55%
89%
85%
86%
C
343
1341
820
141
648
414
69%
65%
55%
87%
83%
76%
Author
428
31
262
46
63%
81%
81%
90%
Human
721
479
631
(a) First Turn
Setup
S1 (R 33%)
(R 50%)
Judge
G4-S
Author
Human
Human-M
G4-S
Author
Human
Human- M
70%
66%
66%
68%
95%
88% 205
85%
85%
G4-P
1161
341
1325
812
727
864
557
65%
59%
61%
89%
84%
G4-$
331
1285
783
193
776
506
67%
68%
63%
87%
86%
84%
Author
49
413
87
31
273
54
67%
83%
82%
91%
Human
707
877
474
629
(b) Second Turn
Writing
model
Humanities
Roleplay
GPT-4
Claude-v1
GPT-3.5-turbo
Vicuna-13B
Alpaca-13B
STEM
Reasoning
LLaMA-13B
Extraction
Math
Coding
Figure 20: Category-we wise scores of 6 models on MT-bench.
27

---

E
Training Details of Vicuna Models
Vicuna is created by fine-tuning a LLaMA base model using user-shared conversations gathered from ShareGPT.com with its public APIs. ShareGPT is website where users can share their ChatGPT conversations. To ensure data quality, we convert the HTML back markdown and filter out some inappropriate or low- low-quality samples, which results in 125K conversations after data cleaning then divide lengthy conversations into smaller segments that fit the model's maximum context length.
f We
We construct three training datasets with different scales from this cleaned ShareGPT dataset. Their statistics are in Table where also compare it with Alpaca (38] dataset. "All" is the full dataset. "Singll" only includes the first turn of each conversation. "Selected" is a small high-quality dataset of 3K sequences. To construct the "Selected" dataset, we pick sequences that include least 3 turns of conversations generated by GPT-4 and run a clustering algorithm to divide them into 3K clusters and pick the centroid of each cluster.
All models (Vicuna-7B/13B are trained with the same hyperparameters: global batch size=128, learning=ee-5, epochs=3, seq length=2048. Except for "Selected", which train for 5 epochs. The training code is built on top of the Alpaca code but additionally handles multi-turn conversations. The training is done with 8x A 100 GPUs. The longest single training takes around 2 days. We utilize SkyPilot (49 managed spot instances for saving training costs and FlashAttention [1] for memory optimizations. The training code is available httttt:t//ggiiieii com 1m- sys/FastChttt
Table 14: Dataset statistics
Dataset Name
Alpaca
Selected
Single
All
#Token
4.4M
4.8M
184M
370M
#Sequence
52K
3K
257K
257K
Avg. turns of conversation
1.0
4.0
1.0
2.9
Avg. response length (token)
65
343
473
373
'In this study, data (125K) than the version earlier blog post (70K).
28

---

F Exploring Vicuna as a judge
In this paper, we mostly evaluate the ability of close-soureed models such as GPT-4 as a proxy for human evaluations. However, model services such as GPT-4 can also become expensive with growing number of evaluations. On the other hand, popular open-sourced LLMs, e.g. Vicuna- 13B shows strong language understanding capability, and are much cheaper than close-sourced LLMs. In this section, we further explore the potential of using Vicuna- as a more cost-friendly proxy.
F.1 Zero-Shot Vicuna
When using as-it-is (zero-shot), Vicuna-13B noticeably suffers from limitations we discuss, c.g. position bias. As shown in Table 13| Vicuna-13B has a consistency rate from 11.2% to 16.2% across different prompt templates, much lower than all the fosed-sourced models. In addition, it has high error rate (from 22.5% to 78.8%) because of its weaker instruction- following capability. In many scenarios, Vicuna- 13B provides responses such as " "Answer A is better than answer B", without following the pre-defined template. These responses are rendered as natural languages and are difficult to be parsed automatically, making the model less useful in scalable and automatic evaluation pipeline.
F.2
Arena Fine-tuned Vicuna
Training Due to the incapability of the zero-shot Vicuna- 13B model, we further finetune the model with human votes from Chatbot Arena. Specifically, we randomly sample 22K single-turn votes from the arena, covering all models supported by the time of this paper submission (GPT-4, GPT-3.5, Claude-v1, Vicuna-138 13b, Vicuna-7b, 7b, Koala- Alpaca- RWKV-4-Raven, MPT-Chat, OpenAssistant, ChatGL.M, and StableI to expose the model with a wider range of chatbot outputs and human preferences. We use 20K votes for training, and 2K for validation. To address the aforementioned weak instruction following problem, we formulate the problem as a 3-way sequence classification problem. Thus, the model simply needs to predict which one of the chat-bot ouiputs is better (or tie), without needing exactly following the provided answer template. In particular, we construct an input by using the default prompt and the two model answers. The labels are A, B, and tie (including noth-bad-voke and toe-v We train for 3 epochs with a cosine learning rate scheduler and a 2e-5 maximum learning rate. We use the 2K validation dataset to choose papre-paraettrrs and test on the same 3K dataset in the main body of the paper.
Dolly-12B. FastChat-TS,
Position bias results The results for position bias are provided in Table 15
The consistency improves significantly from 16.2% to 65.0%. Due the classification formulation, every output is recognizable (error rate 0%). In addition, we measure the classification accuracy over the test dataset.
Agreement results It achieves 56.8% when including all three labels, and 85.5% when excluding tie predictions and labels, significantly outperforming random guesses of 33% and 50% respectively, and show positive signals match GPT4 (66% and 87% respectively). In conclusion, further fine-tuned Vicuna- 13B model shows strong potential to be used as a cheap open-sourced replacement for expensive closed-sourced LLMs. A similar conclusion is also found by concurrent paper 42
Table 15: Position bias of the Vieuna- 13B model without and with further fine-tuning. We denote them as Vicuna-13B Se--zrrrSShoo and Vicuna- Dicuna-13B--o 1-B--ince --nee--TTuee respectively. Consistency is the percentage of cases where a judge gives consistent results when swapping the order of two assistants. "Biased toward first" is the percentage of cases when a judge favors the first answer. "Error" indicates wrong output formats. The largest number in each column bold.
Judge
Prompt Consistency
Biased toward first
Biased toward second
Error
default 15.0%
53.8%
8.8%
22.5%
Mcuua-13B-7eero
rename 16.2%
12.5%
40.0%
31.2%
score 11.2%
10.0%
0.0%
78.8%
Vieuna- 1-3--Bine-Tuse
default 65.0%
27.5%
7.5%
0.0%
29


Reviews:
[{"review_id": "FKc5XCby6g", "reviewer": "Reviewer_BsjF", "summary": "", "questions": "", "limitations": "See above for most of the limitations.", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Systematic analysis: authors present some very impressive systematic study and insights on LLM-as-a-Judge's algorithmic choices and potential limitations. While some of them have been discussed in previous work, it is for the first time they are clearly compared and experimented in a unified setting using GPT-4, with sufficient supporting results and materials.\n2. Original idea on multi-turn evaluation: despite automatic assessment using LLMs have been heated practiced in the past few months, most of them are limited to single-turn evaluation. MT-Bench is definitely an original attempt to evaluate the challenging multi-turn dialogue capabilities of LLMs (though there are some limitations).\n3. Extensive experiments: The third part I do appreciate is that Chat Arena and MT-Bench are evaluations that have been tried in wild and by relatively large number of real users. To lead such effort, which is crucial for AI systems to step out of laboratories, requires conquering much challenges in engineering and operations.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "HgRjKGbxpX", "reviewer": "Reviewer_Zr4f", "summary": "", "questions": "", "limitations": "The authors have accurately identified the potential challenges and societal impact of employing LLMs as evaluators.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The researchers have clearly laid out this well-structured paper, which includes a systematic and data-centric analysis of LLM-as-a-judge, making it comprehensible to readers.\n2. They have executed an exhaustive evaluation of GPT-4 and other high-functioning language models, enriching the pool of research on large language models.\n3. The authors have devised a diverse range of assessment strategies covering both pairwise and single-wise scenarios.\n4. It laudably elucidates LLM-as-a-judge's limitations and offers practical solutions to overcome these constraints.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "BLn67K8iYt", "reviewer": "Reviewer_QiZU", "summary": "", "questions": "", "limitations": "- Other than releasing the dataset of tasks, the authors should include the benchmark, that is, the row evaluations of the judge LLMs. This is crucial. \n- The MT set contains only 80 controlled tasks. This is quite a few. What are the challenges in increasing this to a few hundredth? This will make the dataset much stronger. For example the biases of judge LLMs are calculated on relatively small dataset.\n- Cleaning crowdsourcing data: crowdsourcing responses are known to be noisy, yet it seems like a cleaning process has been done on this part of the dataset. Authors should consider how to include such a process to eliminate any bias in their data. Especially since in contrast to the MT set, the Chatbot arena only includes a single human judment per question. \n", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Dataset of open-ended tasks is needed, the tasks included seem valid. Current LLM evaluation benchmark usually contains only a limited number of such tasks. The dataset has the potential to be used for other case studies not included in the paper.\n- using an off-the-shelf LLM as a judge is now very popular in various of tasks, yet how to evaluate performances is not fully clear. The paper opens the doors on how to evaluate such. \n- Comprehensive discussion about the biases that LLMs as a judge possess, and simple solutions to mitigate this, as well as evaluation of different experimental design choices. \n- Comprehensive evaluation of of-the-shelf LLMs judgments and the agreement with humans\n- Comprehensive evaluation of of-the-shelf LLMs performances in solving the suggested tasks\n- Discussion of how to fine-tune LLM (vicuna) to serve as a judge (although only brief and is not included in other comparisons, see comments below).\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "DdiuwEA4yt", "reviewer": "Reviewer_1xKM", "summary": "", "questions": "", "limitations": "Limitations of the work are well discussed. \n\nAnother potential limitation to be discussed is what would happen if LLMs were to be fine-tuned on LLM judge preference data. Since the paper proposes to use LLMs instead of human evaluators, it is important to think about how the biases discussed in the paper could propagate during fine-tuning.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "* This paper shows that, in the near future, LLMs may be used to evaluate language models on advanced multi-turn language generation tasks. This comes at a much reduced cost compared to employing human evaluators. \n* The authors carefully consider LLM biases in their evaluation method and introduce techniques to mitigate these biases. This list might not be exhaustive, but mitigating for these biases shows that LLMs show a high agreement rate with humans, which suggests that they are well calibrated judges. \n* The paper introduces Chatbot Arena, a popular platform where public users can interact with LLMs. This platform generates good quality data providing plenty of human evaluations.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper investigated a new area of using LLMs to judge the performance of other language models.  It developed benchmarks that can be used to evaluate these LLMs when they are used as a judge.  All reviewers feel positive about this work.  Please check the section of  Opportunities For Improvement carefully for additional improvement ideas.
Decision: Accept (Poster)