PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations
Authors: Ruosen Li
Teerth Patel
Xinya Du
Venue: Accepted by TMLR ()

Abstract: Nowadays, the quality of responses generated by different modern large language models (LLMs) is hard to evaluate and compare automatically. Recent studies suggest and predominantly use LLMs for reference-free evaluation of open-ended question answering. More specifically, they use the recognized “strongest” LLM as the evaluator, which conducts pairwise comparisons of candidate models’ answers and provides a ranking score. However, this intuitive method has multiple problems, such as bringing in self-enhancement (favoring its own answers) and positional bias. We draw insights and lessons from the educational domain (Cho & MacArthur, 2011; Walsh, 2014) to improve LLM-based evaluations. Specifically, we propose (1) the peer rank (PR) algorithm that takes into account each peer LLM’s pairwise preferences of all answer pairs, and outputs a final ranking of models; and (2) peer discussion (PD), where we prompt two LLMs to discuss and try to reach a mutual agreement on the preferences of two answers. We conduct experiments on two benchmark datasets. We find that our approaches achieve higher accuracy and align better with human judgments. Interestingly, PR can induce a relatively accurate self-ranking of models under the anonymous setting, where each model’s name is unrevealed. Our work provides space to explore evaluating models that are hard to compare for humans.

Published in Transactions on Machine Learning Research (07/2024)
PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations
Ruosen Li
Teerth Patel
Xinya Du
Department of Computer Science, The Universily of Texas at Dallas
fruosen.li, teerth.pater patel, zinya. du @utdollos.ed.
Reviewed on OpenRevicw: http:://oognaeesssseeeee///ooo/p
Abstract
Nowadays, the quality of responses generated by different modern large language models (LLMs) is hard to evaluate and compare automatically. Recent studics suggest and pre- dominantly use LLMs for reference-Iree evaluation of open-ended question answering. More specifically, they use the recognized "strongest." LLM the evaluator, which conducts pairwise comparisons of candidate models' answers and provides ranking score. However, this intuitive method has multiple problems, such as bringing in self-enhancement (favoring its own answers) and positional bias. We draw insights and lessons from the ional domain (Cho & MacArthur 2011 Walsh 2014) to improve LLM-based evaluations. Specifi- cally, we propose (I) the peer rank (PR) algorithu that takes into account each peer LLM's pairwise preferences of all answer pairs, and outputs final ranking of models; and (2) peer discussion (PD), where prompt two LLMs to discuss and try to reach mutual agreement the preferences of two answers. We conduct experiments on two benchmark datasels. We find that our approaches achieve higher accuracy and align belter with human judgments. Interestingly, PR can induce relatively accurate self-ranking of models under the anonymous setting, where each modcl's name is unrevealed. Our work provides space to explore evaluating models that are hard to compare for hurnans.
1 Introduction
With a rising number of large language models (LLMs) being developed ever more quickly recently, evaluations become increasingly important as they encode values and priorities that the LLM community should improve upon (Jones & Galliers 1995; Liang et al.] 2022). At the same time, the evaluation becomes harder well. For example, recent models finetuned with reinforcement learning from human feedback (RLHF) demonstrate greater alignment with human preferences, but this capability usually cannot. be reflected by decent performance on standard NLP benchmarks (e.g., MMLU (Hendrycks et al. 2020) and ARC (Clark et al. 2018)). Furthermore, human queries span a diverse range of settings and scenarios, making it nearly impossible to list them all Fan et al. 2019 Ouyang et al. 2022)
To tackle this discrepancy, open-ended questions are being used more often to test LLMs' performance (Chiang et al. 2023). : Then, by default, evaluation is done by collecting human preferences of pairwise comparisons and then calculating scores for each LLM to induce general ranking. Yet the collection process is costly and time-consuming, to automate and scale up the evaluation, most recent works utilize the state-of-the-art LLM as the judge (Dubois et al.) 2023 Lin & Chen 2023). However, various studies show that this method is problemnatic, as the pairwise comparison judgment provided usually contains various biases, such as favoring
LLMs' own answers Liu et al. 22023 Zheng et al. 2023).
Motivated by these limitations, we propose the idea of peer evaluation. The goal is mitigate the biases in automated evaluations while still benefiting from L.M's strong capability in reading and writing reviews. We

---

Published in Transactions on Machine Learning Research (07/2024)
propose Peer Rank and Discussion- based evaluation framework (PRD). The suit consists of two alternatives that share the same format and goal involving peer LLMs' participation as reviewers to reach a more fair evaluation result where all peers mutually agree. We draw insights and lessons from educational psychology research field on methodologies of student peer reviewing (Walsh 2014), as well as their impact and benefits (Cho & MacArthur 2011: Yalch al.) 2019). More specilically, Peer Rank (PR) is utilized for global rankings and induces reviewer weights. It works for the tournament-style benchmarking setting where each LLM in pairwise matches produces an answer for an open-ended question. Instead of using the average vote to decide the final preference scoring, we propose weighted votes based on LLMs reviewers' capabilities. Peer Discussion (PD) facilitates fine-grained pairwise comparison /rankiig. It works for the general pairwise comparison setting. Given two candidate answers, we prompt two other reviewer LLMs to have multi-turn discussions to reach a mutual agreement on the pairwise scoring or preference. The process shares a similar format of LLM interacting with each other through conversations like two communicative agents (Li et al., 2023: Park et al., 2023 Fu ct al.) 2023b). PR and PD are closely interrclated and fall under the same theme of providing a more fair (de-biased) ranking of long- and free-form answers.
We conduct extensive experiments and analysis for measuring PR and PD's capabilities of providing fair pairwisc comparisons. PR is tested on Vicuna80 Chiang et al. (2023), which contains pairwise judgments from human annotators. Our method improves correlations with human rankings substantially. This paradigm also enables a group of LLMs to induce a self-ranking. PD is tested on both Vieuna80 and LFQA (Xu et al.) 2023), which includes annotated pairwise comparisons of Human-Machine and Machine answers. PD enables LLMs to achieve better pairwise comparisons that are more accurate than single model-based reviews. Both PR and PD mitigate the above mention biases especially self-enhancement bias significantly. Further, we provide more analysis for PD, showing: (1) the LLM leading discussions is less likely to alter its opinion; (2) stronger LLMs are more likely to hold their opinions.
2 Related Work
Automatic Evaluations Natural Language Generation (NLG) evaluation methods are mainly of a similarity-based or reference-free free type. For similarity-based metrics, the generated texts are compared to reference texts Papineni et al. (2002); Zhang et al. (2019). In parallel, people have also developed task-specific
metrics such as consistency Kryáciñskii et al. 2020 Wang ct al.] 2020). faithfulness (Fabbri et al. 2022 Gao
et al. 2023) and coherence DDiri el al. 2019). This is similar to our peer discussion idea on designing more
specific prompts for large language mosel-based evaluations. Our prompting- baased method is more flexible
and can act as unified evaluator (Zhong et al. 2022
Specifically, for long-form or open-ended question answering, early work uses ROUGE (in 2004 Lo
measure the similarity between human and machine-generatee answers. However, researchers find that ROUGE is not a fair metric for quality mcasurement due to the open-ended nature of long-form answers (Krishna et al. 2021 Xu et al. 2023). Fu et al. (2023a) propose GPTScore, which evaluates texts with generative pre-training models like GPT-3. Xu et al. (2023) implements similar idea for evaluating long-form answers. Given a prompt consisting of question with two answer candidates, GPT-3 is fine-tuned to output the label answer 1 or answer 2 (pairwise comparisons).
LLMs as evaluators: problems and challenges Most recently, with the trend of developing more LLMs, evaluations for benchmarking the progress have become even more important but also more difficult. They are tested on both standard datascts such as MMLU, and more importantly, on open-ended questions which are much more prevalent in real life (Nakano et al. 2021 Chiang al., 2023) People mostly use GPT-4 (Liu lt al.) 2023; OpenAl 2023) an evaluator for either generating scores or pairwise comparisons (Wang 2023b Zhou et al. 2023). However, such a strategy has fundamental problems because of various
et al. biases, such as (1) positional bias (Dettmers et al. 2023 Wang et al., 2023a where à model favors the first answer in pairwise comparisons: (2) verbosity and length bias (Wang et al. 2023bb; ; (3) and most importantly,
self-enhancement bias, where an LLM favors its own answers (Liu et al.] 2023 Zheng et al. 2023
Efforts have been proposed to tackle them: (1) Using position switching WWang et al.
2023a for mitigating positional bias; (2) Zhenn et al. (2023) proposes Chatbot Arena, where real users ask questions and provide
2

---

Published in Transactions on Machine Learning Research (07/2024)
Reviewers
AI
Better contestant's review weigh more Belter contestants. score higher
W
W
W
Score1
AI
Rank
2
W
W
3
X dot product
Q
Scorez
A
2
A
W
W
ac
Scores
Pairwise Battles
Contestants
Win Rate Matrix
Normalize
Reviewer (Multi-roundi Contestent Weight Vector
Molt-oonddCootestennt Vector
Figure 1: The peer rank process (PR): each LLM model acts both reviewers (A, B, C) and contestants (1, 2, 3). From the battles between contestants (pairwise comparisons), it induces a self-ranking. In this example, models A, B, and C represent GPT-4, Bard, and Claude, respectively.
pairwise judgments of answers generated by two LLMs. But this is time-consuming and requires expert annotation to ensure fairness; (3) Bai et al. (2023) propose using each LLM as an examiner, where each generates questions test other models. Different from PD, their "cxams" are biased with randomly generated questions. Moreover, none of the above works support inducing self-rankings through peer ranking.
Overall, our work, peer evalustion-oriented methods, undergo tests on two benchmarks, each covering variety of tasks, and focuses more on the alignment between LLMs' evaluations and human judgments.
LLM Interactions and Discussion There have been works that explore Multi-agent discussion with LLMs, including Liang et al. (2023); Du et al. 20233 Chan et al. (2023): Chen et al. (2023). All of them are task-oriented frameworks aiming at improving LLMs' performance on general tasks. Du et al. 20233
locused on math (asks and question answering lasks. Liang et al. (2023) tested two lasks, including machine translation and question answering. [hen al. (2023) covered math and reasoning. [Chan et al. (2023) implemented multi-agent debate framework with multiple prompts for evaluation. Prior works utilize LLM interactions to accomplish tasks and improve models" accuracy. For our case, based humans and another from LLM, our approach utilizes LLM interactions to discuss which one is better and in order to achieve better evaluation that align with human preferences.
response from
Peer Evaluation in the Educational Domain Prior works on educational research mainly focus on numan-in-the-loop studies, such as the classroom (Cho & MacArthur 2011 Nicol et al. 2014). They conduct human-oriented data collection and experiments verify the benefits of peer evaluation (Walsh 2014). In contrast, we focus on automatic evaluation, employing peer LLM reviewers conduct pairwise comparisons of LLMs' answers. Moreover, our peer rank process focuses pairwise comparisons, instead of absolute grades.
3 Methodologies
In general, Peer Rank (PR) can be applied to induce self-ranking a ranking of a group of LLMs own capabilities. Peer Discussion (PD) provides more fine-grained and interactive comparison of two models' answers. Both of them aim reducing the bias in automatic evaluations. We claborate on the technical details in this section.
3

---

Published in Transactions on Machine Learning Research (07 (224))
3.1 Peer Rank and Scoring (PR)
Figure | illustratts the peer rank algorithm. The general idea is obtain weighted scores of each battle from the peer reviewer's judgment, and then induce sclf-rankings from the scores. This process is iterated multiple times until the scores converge.
Given a set of questions Q, we generate an answer to each question from each LLM. Let (m) be the answer to question E Q by the model m. Each battle represents two modols (the contestants) answering the samee question q. The comparison of the answers in a battle by the LLM reviewer model r forms a review. Let (z, y) be the score given by the reviewer r to the pair of answers (z,y), We use score of -1 to indicate the first answer better, 0 to indicate a tie, and to indicate the second answer is better. Suppose we have set of reviewer models R and set of contestant models C. We form a set of battle reviews, B = (..,,,).,,)) q E Q, (i,j) E C2, r E R), where 8 = (,(((()), (4)) is the score given by reviewer to the suswers/responses generated by i and for question q. We create a shorthand K2 (q) for this review.
Based on these peer reviews, we can evaluate models based on their performance by calculating metrics such as the win rate of each contestant and the Elo ratings (Section 3.1.2 of each contestant. Since each model is ranked by its peers, we call it Peer Rank.
Specifically, the set of questions should be diverse and cover various tasks, such question answering (12.5%), email writing (12.5%), coding (6%), math solving (4%), etc. Answers srewersrresponess should also vary in format, including concise answers, step-by-step reasonings, detailed explanations, code snippets, long-form answers, etc. Reviewers assess response pairs and indicate preferences in the process ("battle review"). Then, both winrate and Elo metrics can be calculated.
3.1.1 Win rate Calculation
The win rate for a contestant is the number of wins for that contestant divided by the number of battles it participates in. Ties are counted as 0.5 wins for both contestants.
Our win rate calculation assigns differing weight to the scores provided by different reviewers (A, B, C) based on the performance of the corresponding reviewers as a contestant (1, 2, 3). This operates on the assumption that models which are better contestants are also more fit to evaluate and compare answers, so they should be given more weight in evaluation (Equation 2 In another way, since the score is measure of their ability to review/grade correctly, Moreover, the self-rewarding paper by Yuan et al. (2024) also made this assumption. They use model itself for evaluations during the iterations and prove that it makes sense. In their results, the better peaterrperformin models also perform well in providing high-quality evaluations/rewards to themselves.
weigh the win rate LLM gives another LLM by their own score Walsh (0014)
Initially, all reviewers are given the same weight. On each iteration of the calculation, the win rate for each contestant is calculated using the current weights. The win rates are scaled to the range of [o, 1] using a linear scaling. Then, they are scaled again so that their sum is 1. Next, these results are used as the weights for the next round.
Formally, let W be the raw win rate of contestant e C' from the reviews of reviewer r E R. This is equal to the number of times wins a battle plus half of the number of times e ties, divided by the number of battles c participates in.
ΣΣ [s(K] + ((()
(1)
W 2 dec,dyo
2|Q1((C1-1)
where [sscore= = score+1 maps a score of (loss = -1, tie O, win = 1) for the second contestant to a win count of (0, 0.5, 1), so that ties count as half of a win.
Note that we negate (e)(2) when inputting it into f so that the win value of computed instead of d. Also, since there are IQ] questions, C-1] contestants to battle, and 2 orders for two contestants to battle, there are 2|Q]|C 1| battles involving fixed contestant c.
1|
4

---

Published in Transactions on Machine Learning Research (07/2024)
Let af be the weight assigned to reviewer after iteration k. Initially, af 1/|R], so that all reviewers have the same weight, and the weights add to 1. Namely, we assume each reviewer L.I.M has the same capabilities to start. The score of contestant с E C for iteration & is the weighted average of the raw win rates for contestant c. We set the weights for the next itcration to a²:
Σ or 1-1 W,, a Norm((iiMaxxsoore")
score
(2)
EER
where the weights are scaled to range of [0, 1] and finally normalized to have sum equal to 1:
S minrer(S.)
MinMax(S) max,en((,) - min,en(S.)
Given this set of equations, we look for the fixed/ /covvergin point of the framework. This process is reminiscent of the problem faced by the PageRank algorithm (Page et al. 1999). The detailed equivalent implementation of PR is shown in the Algorithm| 2 in Appendix
The whole process is simple but effective. It automatically adjusts weights for all models and mitigates the self-enhancement bias. During the process, every reviewer model's score is considered instead of only one model itself (Equation 2). While reviewer may favor its outputs, other reviewers' scores provide a fair balance. Additionally, weaker models' reviewing weights decrease automatically (to near zero) because of the normalization operation. Empirical tests showed that fixing the welf- at zero resulted in poorer performance.
3.1.2 Elo Calculation
Another method for calculating the performance of contestant relative to other contestants is the Elo rating Elo 1967 Askell et al. 2021), which is utilized for ranking players and widcly used in games (battles) (Dettmers et al. 2023). It measures the relative skill levels of players by predicting the expected win rate against opponents. takes sequence of pairwise reviews and generates ratings for each contestant, with a greater rating indicating better performance. This more fine-grained measurement compared win rate. Based on similar idea, assign different weights to reviewers based on their previous performance such that a review from higher-weight reviewer has greater influence upon Flo ratings.
Similarly to the win rates calculation, we start with equal weights on all reviewers and then normalize the resulting Elo ratings to give weights for the next iteration. We repeat the Elo calculation with the new weights, update the weights based on the new ratings, and continue repeating until converges. [Walsh] (2014) has proved the guarantee of convergence. Compared to Walsh (2014), which involves student grading in the educational domain, our work focuses on peer LLM reviewers conducting pairwise comparisons of LLMs' answers, utilizing winrate and Elo scores. We introduce automatic LLM peer evaluation metric for the machine learning field and extend Walsh's convergence proof, showing our method's reliability through experimental results. More difference details are in appendix
A brief overview the actual Elo ratings calculation follows. All contestants start out with an initial rating of 1000. On each battle, the expected likelihood of cach contestant winning calculated based on the difference between their Elo ratings. The Elo rating of the winner is increased, and the rating of the loser is decreased. The magnitude of the Flo ratings change is inversely related to the outcome's likelihood. In our calculations. weigh reviewers so that reviews by a high-weight reviewer cause larger changes in Elo. For more details, please refer to Algorithm I in Appendix
3.2 Peer Discussions (PD)
In peer discussion, we prompt two LLMs to discuss how to judge two candidate answers, trying to reach a final agreed review. All prompts include detailed instructions and specify the output format for the LLMs. Tn Figure 2 we demonstrate the peer discussion process between two LLMs reviewers (A and B). The input is a given question and two answers, which may be both generated by machines or one by humans and another by
We provide the proof of convergence of our peer rank algorithm in Appendix
5

---

Published in Transactions on Machine Learning Research (07/2024)
Question:
How do credit/debit cards work? What is the process of putting money in and getting out?
credit or debit card basically just an easy way to allow shop to speak to your bank. First you go into the bank, [...]
Debit cards are linked to bank account and whenever you pay [...] amount is deducted[...]
Initial Review
Answer provides a more detailed and narrative explanation, using analogy of [...] Answer 2 the other hand, is more concise and uses more technical language [...]
Answer provides a basic overview of [..]] at high level. However, [...] Answer 2 provides more coherent explanation by separately discussing how [-..]
B
2
Discuss answer of 1 and 2 with reviews from A and B in mind [..]] again output choice in new line
After considering Reviewer B's perspective, [...] While Answer [--], believe that the accessibility and comprehensiveness of Answer
outweigh the concise and technical nature of Answer 2.
can appreciate Reviewer A's perspective on [...] Upon reflection, for the purposes of this question, accessibility and comprehensiveness are most important [...] after considering Reviewer A's perspective, would change my preference to Answer
AV B
The peer discussion process (PD). Bold and italic texts describe the advantages of answer 1 and answer 2. In this example, finally, the two LLM reviewers reach the mutual agreement of selecting answer (hitan-written answer), which correlates with human annotator preference.
Figure 2:
[System] You are helpful and precise assistant for checking the quality of the answer. [Question] {Q}
[Answeri] {A1}
[Answer2] {A2}
[System] We would like request your feedback the performance of answers in response to the user question displayed above.
Firstly, please compare the tion, core information, and coherence. Please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that order in which the responses were presented does not affect your judgment.
based if they contain unsupported informa-
Once you have carefully reviewed both submissions, in new line, choose between the two answers by outputting the number 1 respectively. Do not output anything else other than the number in this last line.
Table 1: It shows the review template for reviewers with three slots (1Q), {A1}, and (A2)). We instruct. reviewer models to focus (2023a), position bias still exists after emphasizing it in the prompt.
aspects. whose definitions are in appendix F As mentioned in Wang et al.
machines (e.g. GPT-3 v.s. human answers). They first conduct pairwise comparisons on answers separately, providing explanations and indicating their preferred answer by outputting the number 1 2 by the end (the prompt for getting initial reviews is listed in Table I.. Then, the two models discuss multiple turns until they reach a fixed number of turns.
The specific prompt for discussion is listed in Table 2 At the very beginning, a system prompt (role prompt) tells models their role- whether it is reviewer or reviewer B (c.g., Claude or GPT-4). Then, all information, including the question, two comparison answers, and the initial reviews, are listed line by line. The order of initial reviews is the same that of reviewers in discussions. In other words, if reviewer A leads the

---

Published in Transactions on Machine Learning Research (07/2024)
[System] You are reviever A, discussing with reviewer B about your reviews of the following answers.
[Question] (Q)
[Answer1] {A1} [Answer2] (A2}
[Init Review A] (Review of reviewer A} [Init Review B] {Review of reviewer B} [System] "Read the reviews and discussions above, and make a decision if to change your preference, and explain. Remember focus unsupported information, and coherence. In a new line, choose between answer and answer 2 by outputting the number or respectively.
information,
not output anything else other than the number this last line." " [Reviewer A] (First-turn output) [Reviewer (Second-turn output) [Reviewer A]
Table 2: The discussion template for reviewer A at the third turn. Similar to the review template, we explicitly indicate aspects that reviewers need to pay attention to. All texts above are chat history which are input to reviewer LLM model.
discussion, reviewer A's initial review is listed first. Right before the start of the discussion, the system prompt specifies the detailed requirements, which provide explicit aspects to focus on.
Specifically, we draw insights from WebGPT (Nakano et al. (2021) Y's annotation guideline OpenAl 2022 For lorg-form question answering, we focus on (1) Unsupported information: detecting information with no support, assume the worst case: that all of it is false. This aspect is most important and often determines the overall rating: (2) Core information: about whether the question has actually been answered; (3) Coherence: generally, it is less important than the two above. Then, the overall preference is finally determined. Moreover, we do not adjust the above prompts based on any datasets we test in experiments.
4 Experiments and Analysis
4.1 Datasets, Metrics, and Setup
4.1.1 Datasets
We select two "mota--valuation" datasets, LFQA (Xu et al. 2023 and Vicuna8O, with human annotations for pairwise comparisons, measure the correlation between our evaluation methods and human judgmnents.
LFQA [Xu et al. 2023) contains 140 long-form questions across seven domains (e.g., economics, history, and biology) and two candidate answers (from either GPT3 or Human) for cach. Similar to ELI5 (Fan et al. 2019). it contains more recent (i.e., after July 2021) questions from Reddit foruns "r/explainlikeimiive and "r/Ask "r/AssHiitrrianss The authors collected expert-level annotations of which answer better (overall preference).
Vicuna80 CCiang et al. 2023) is set of 80 open-ended questions, which spans 9 categories and covers wide range of tasks, including question answering, email writing, math problems, etc. In the QLoRA work (Dettmers et al., 2023), authors annotated pairwise comparison scores (overall preference) across 7 models for each question. The scores include 0, 1, 2, which correspond to tie, model 1 wins, and model 2 wins respectively. We select pairwise comparison annotations of models' answers (i.e., GPT4, ChatGPTT335, PaLM-2, Vicuna-13b). To make our study more comprehensive, we add recent proprictary language models such Claude. Specifically, we also annotate pairwise comparisons between Claude's answers and the other models'. We term this more complete version of the dataset VicunaSO. More details about the annotation process are provided in Appendix [| annotators achieve a fair agreement.
Since answers to open-ended questions are even harder to compare, the
SummEval Fabbrr et al. 2020) a benchmark evaluating summarizations, It contains 1600 summaries over 100 news articles from CNN/Daij Mail dataset (Hermann et al. 2015). Each summary is evaluated in four aspects: coherence, consistency, fluency, and relevance. In our experiments, we take the average of the four metrics the evaluation results.

---

Published in Transactions on Machine Learning Research (07/2024)
GPT-4
All
All (Weighted)
Human Raters
models
Elo
Rank
Elo
Rank
Elo
Rank
Elo
Rank
GPT-4
1282
1165
1
1213 (-23)
1
1236
Claude
1150
1104
2
1125 (-2)
2
1127
Vicuma
883
4
930
3
912 (-8)
3
920
GPT-3.5
890 (+-22)
919
894
4
868
PaT.M-2
804
881
5
856
(+8)
5
847
GPT-4
All
All (Weighted)
Human Raters
models
Win Rate
Rank
Win Rate
Rank
Win Rate
Rank
Win Rate
Rank
GPT-4
0.856
1
0.749
1
0.802
(-0.020)
1
0.822
1
Claude
0.709
2
0.662
0.685 (-0.004)
2
0.689
Vieuna
0.340
4
0.393 (+0.004)
3
0.376
3
0.389
3
GPT-3.5
0.342 (+000028)
3
0.375
0.346
4
0.314
PaLM-2
0.245
5
0.320
5
0.290 (+-0.004)
5
0.286
5
Table 3: The above tables show results performed on the Vicuna80 dataset. The rows represent the contestants in battles, and the columns represent evaluation methods. The upper table shows the correlation of Elo scores between LLM reviewers and human rater. Bottom table shows the correlation between global win rates. Additional results on emerging (more) LLMs Table | further verify the consistent robustness of PR. Boldfaced numbers
the closest. to scores from human raters. Blue numbers show the difference between the scores from LLM reviewers and Human raters. For more detailed pairwise win rates, please refer to the heat maps in Section 4.2
All
All (Weighted)
Human Raters
models
Elo
Rank
Elo
Rank
Elo
Rank
Vicuna
999
2
1011 (-47)
1
1058
1
Zephyr
1010
1007 (+7)
2
1000
2
GPT-3.5
991
993 (+52)
3
941
3
Table 4: Additional Elo Scores for Vicuna, Zephyr and GPT-3.5
In LFQA, questions receive 1-3 expert-level annotations per category, with human agreement ranging from 0.4 to 0.65. Each Vicuna80 question receives 3 human annotations, with human agreement between 0.5 and 0.62. Each SummEval summary is annotated by humans, with an agreement number 0.7. We use the human majority vote as the human preference during battles.
4.1.2 Metrics
For experiments PR, we follow metrics in Wang et al. (2023a). We first conduct erample-level pairwise comparisons. Specifically, each evaluation example (pairwise comparison) consists of question and pair of answers. We compare the model predicted preference score against gold human preference and report Accuracy and Fleiss' statistic that measures the reliability of agreement among multiple raters. Specifically, Fleiss' k is employed to gauge alignment between the model's predictions and human preferences, where a higher score signifies stronger alignment. Following Dettmers et al.] (2023), we also compare model-predicted global ranking scores against human-judged ranking scores. Specifically, we report Elo scores (Elo 1967 and win rate (WR) based rankings (Table 3). We use All to denote our method where each reviewer has equal weights, and use All (Weighted) to denote the setting where the final round weights are applied to each reviewer. Besides experiments on PR and PD respectively, we also compare PR and PD in an experiment of
judging answer qualities of GPT-3.55 v.s. Vicuna-13b (Section 4.2 and 4.3
For experiments on PD, we use peer discussion accuracy (PDA) to describe the correlation of model discussion results compared to human annotators. PDA is calculated by the number of correct answers from peer discussion results over the number of all answers. A high PDA result indicates better correlation with human judgments.
8

---

Published in Transactions on Machine Learning Research (07/2024)
g3f-4
1200
clocde
vizuna
gpt-3
1230
hard
El GPT4
37.75
48.8%
1190
0.18%
GFT4
5.31%
1000
0%
611
000
05
00
000
0011
0371
0000
0012
200
600
Figure 3: Peer rank final round weights of each LLM reviewer. GPT-4 and Claude take 48.8% and 37.7% weights. Bard got close to zero weights.
Match No.
Figure 4: GPT-4 Elo scores every 100 battles on the Vi- cunaS0, Elo scores provided by the GPT-4 reviewer are con- sistently higher than human ratings, while our All (weighted) ralings correlate with humans well.
4.1.3 Setup
For Vicuna-13b, we use the default version from Chiang (002)). For all other APT-based LLM models, use specific versions of each, i.e. GPT-4-0613, 5PT-3.5-turbb-000 for GPT-4, GPT-3.5, Claude, and PaLM-2 respectively. For more details, please refer to appendix B For discussions in the PD method, set the maximum nunber of lurns as 4. Based on our experiments, most. discussions reach mutual agreements at turn 4. Moreover, the default temperature for all models is 0.2.
Claude-1, and Text-Bison0001
4.2 Results for Peer Rank (PR)
On the Vicuna80 dataset, we compare our PR method and representative LLM-based evaluation methods, such as GPT-4 and Claude.
In Table all reviewer combinations listed except Claude, when compared to human reviews at an example level, dis- play Fleiss' к of around 0.40, indicating fair to moderate agreement. There is a significant difference accuracy between LLM reviewers. The worst reviewer is Claude, with an accuracy of only 60.7%. The best individual reviewer is GPT-4, with an accuracy of 64.3%. The combi- nation of reviewers (PR) increases this accuracy by few percentage points, with our PR approach being highest at 67.3%.
Reviewer
Fleiss' x Accuracy
GPT-3.5
0.387
0.621
Claude
0.319
0.607
GPT-4
0.406
0.643
GPT-4 Clande & GPT-3.5
0.403
0.666"
All Reviewers (Weighted)
0.410
0.673*
Table 5: Example-level correlation results, for the fourth and fifth rows, we take the peer reviewers' majority vote weighted by winrate. Two-tailed t-test results statistical significance is indicated with "(p < 0.01). "(p 0.002).
Inspecting Table 3 GPT-4 reviewer ranks GPT-3.55 higher, while our All (Weighted) achieves the same ranking as humans: i.c. GPT-4 Claude > Vicuna GPT-3.5 PALN-2. This shows that a weighted peer ranking provides more accurate evaluation of the global performance of models. Moreover, the ranking corresponds to that in the Chatbot Arena Leaderboard?, Thus, the assumption in section 3.1.1 can be verified. In terms of the Elo ratings provided by the human reviews, we clearly observe that GPT-4 clearly favors its own answers and is prone to self-enhancement bias. Our approach (All (Weighted)) produces the closest Elo ratings. Furthermore, it also produces the closest. win rates (less than a 1% difference for many contestants). In the beginning, when the weight is the same for every reviewer (weights equal to one), the win rate given by " "All reviewers" low at about 0.749 partially because each reviewer treated equally so that each reviewer might have a preference for own answer. After several fair. We display the final round weights in Figure 3
the final win rate becomes more

---

Published in Transactions on Machine Learning Research (07/ (224)
2:48
8.90
614
0.41
0P14
0.41
0.13
9.76
95
400M
000
117
833
043
6-14
6:2
001-2
3:0
0.22
0.38 3.25
5:37
0.2
012
585
001
212
233
04LL
Ead
0:23
Figure 5: Pairwise win rate heatmaps: Fraction of Model A Wins for all vs. Battles (A: rows, B: columns). Left: GPT-4 evaluator; Middle: our method All (weighted); Right: Chatbot Arena pairwise win rate. All results in the three sub-figures are generated separately using the same data.
In Figure we draw the line chart of how the GPT-4 Elo score changes as more battles are fed to the Elo algorithm. GPT-4's score takes off as battle number increases. We can observe that GPT-4 displays self-enhancement across the entire process, while our PR-based evaluation correlates with human pairwise comparisons well.
In Figure we present the detailed pairwise win rates between every two contestants (LLMs). We compare our evaluation with GPT-4 based evaluation, well as the Chatbot Arena leaderboard. The Arena ranking" is based on user queries and their corresponding preferences for two responses. The figure demonstrates that although both approaches favor GPT-4 and Claude answers, the win rates calculated by our approach All (weighted) correlate belter with Arena win rate, especially on weaker models. More pairwise win rate heatmaps are in Appendix
4.3 Results for Peer Discussions (PD)
Prompt for Discussion By preliminary study, find that the template asking for explicit aspects, such as core informa- tion, unsupported information, and coherence, can substan- tially help LLM reviewers gen- erate valuable and informative reviews which correlate better with human annotators.
GPT-4 lead
Claude lead
Random
GPT-4 init
0.729 (+0.014)
Clande init score Generic prompt w/ explicit criteria
0.671 (40.025)
0.714 (#0.018)
0.671 (#0.022)
0.686 (#0.022)
0.729 (#0.014)
0.721 (#0.018)
0.720 (#0.014)
w/ role
0.743 (10.011)
0.729 (10.018)
0.729 (10.011)
w/ role & explicit criteria
0.750 (20.014)
0.721 (20.011) 0.743 (20.011)
Table 6: Different prompting's effect on Peer Discussion Accuracy (on the LFQA datasct). The first two rows are the results before discussions (from GPT-4 and Claude respectively). The last. three rows are results after discussions.
We first conduct preliminary experiments to find a relatively good prompt for facilitating LLM peer discussions. The first two rows in Table lists the Peer Discussion Accuracy (PDA) of GPT-4 and Claude's initial pairwise comparison preference before discussions. They have a moderate agreement with human preference, with GPT-4 leading around 5%. For the discussion-basod evaluators, we report three types. By "GPT-4 lead", we refer to the discussions where GPT-4 first expresses opinions; by "random", refer to discussions where the leader is randomly picked.
In discussions (the last three rows), when we use generic prompt (such as "pick your preferred answer"), the discussion's final preference PDA is around 0.69, higher than Claude's initial judgment's 's PDA but lower than GPT-4's. When we add more explicit aspects into the prompt improvement). When we add the role/identity information (Appendix to each turn's prompt ("w/ role")
3 the PDA boosts significantly (4%
hhttpe://Insye. httpll/yww..rr//////2222222 05-005----5000 leaderboard/
We aspects from WrbGPT annotation guidelines mentioned in the previous section.
10

---

Published in Transactions on Machine Learning Research (07/2024)
Random
R1
R2
R1 lead
R2 lead
Random
(Best Prompt)
GPT-4 & Claude
0.729 (10.014)
0.671 (40.025)
0.743* (10.011)
0.729 (10.018)
0.729 (10.011)
0.740 (=0.011)
GPT-4 GPT-35
0.729 (1000155)
0.579 (20.022)
0.714 (10.011)
0.750 (10.018)
0.731 (10.014)
0.736 (=0.014)
GPT-35 & Claude
0.579 (+0.026)
0.671 (+0.029)
0.700 (+0.118)
0.671 (+0.014)
0.686 (+0.014)
0.693 (-0.014)
GPT-35 & GPT35-0.8
0.579 (+0.026)
0.650 (+0.040)
0.664 (40.018)
0.686 (+0.031)
0.681 (+0.020)
0.686 (-0.014)
Claude & Claude-0.8
0.664 (+0.022)
0.707 (+0.054)
0.693 (+0.018)
0.671 (+0.027)
0.680 (+0.026)
0.693 (=0.014)
GPT-4 & GPT-4-088
0.729 (40.014)
0.757 (+0.022)
0.779 (40.014)
0.757 (40.018)
0.779 (40.018)
0.786 (=0.014)
Table 8: Peer discussion accuracies (PDA) on LFQA. *-0.8" indicates the temperature is 0.8. Statistical significance is indicated with "(p < 0.05). "Best prompt" indicates the discussion uses the best prompt in table (#Numbers) represent standard deviations.
R.1
R2
R1 lead
R2 lead
Random
R1 vs R2
P
T
p
GPT-4 & GPT-35
0.293
0.233
0.262
0.251
0.297
0.266
0.284
0.219
0.292
0.233
GPT-35 & GPT-35-0.8
0.262
0.251
0.211
0.178
0.264
0.207
0.340
0.328
0.334
0.264
GPT-4 & Claude
0.293
0.233
0.234
0.200
0.344
0.268
0.335
0.282
0.341
0.268
Table 9: Summary-level Spearman (p) and Kendall-Tau (r) correlations between model evaluation results and the Summeval Benchmark. Columns RI and R2 contain the results before discussions. The rest of the columns represent discussion results. Variances of all scores are lower than 0.06. P-values for all discussion results (boldfaced) are less than 0.05.
to remind the reviewer, the PDA scores increase for both models, indicating the role information is helpful for LLMs in discussions.
General Accuracy In Table g we report the peer discus- sion accuracy (PDA) of multiple combinations of reviewers' discussion results on LFQA based on the best two discussion prompls in Table 6 We observe: (1) when two reviewer LLMs are of similar capabilities (e.g., GPT-4 and Claude differ by less than 100 in Table 3), they reach stable im- provements upon their initial reviews. In the above example, GPT-4 gets 3% improvement (from 0.729 (=0.014) to 0.743 (=0.011)), and Claude get 8% improvement (from 0.671 (+0.025) to 0.729 (#0.018)) (2) when there a substantial gap between reviewer capabilities (c.g., GPT-4 and GPT-35 differ larger than 100 in Table 3), the PDA of the weaker model always reaches the most improvement and the highest final result. In the above example, GPT-35 receives 30% improvement (from 0.579 (=0.026) to 0.700 (#0.018)). (3) when models "sclf-discuss", for example, create two variants of the same model by setting different temperatures and prompt them to discuss, weaker models (e.g., GPT-3.5) can substantially "self-improve" (from 0.579 (+0.026) to 0.664 (+0.018)). GPT-4's self-discussion brings little improvement (from 0.729 (=0.014) to 0.779 (#0.014)). Future investigations on how to design better self-discussion strategics would be worth working on.
Accuracy
GPT-4
0.3500
(PD) GPT-4 & Claude
0.3675*
(PR) All
0.4375
(PR) (weighted)
0.4625
Table 7: Comparing peer discussions (PD) and peer ranking (PR) on Vicuna80 (random order is applied to the GPT4 & Clande discussion).
Table shows the same trend as Table A higher correlation score indicates discussion results are more aligned with human annotations. Models with similar capabilities (GPT-4 & Claude) get large improvements after discussion. Models having substantial gap (GPT-4 & GPT-35) reach results close to the stronger model. When model self-discuss (GPT-35), it can improve its own performance.
Table 7 reports accuracy on comparisons of GPT-3.5 v.s. Vicuna- 13b answers to Vicuna80 questions, we see the GPT-4 & Claude discussion increases the accuracy by over 1.5%. Also, we compare with PR method and find that the review becomes substantially better after weighted scoring.
11

---

Published in Transactions on Machine Learning Research (07/2024)
Initial Preference
A After Discussion
Reviewers
GPT-3 Answer First
Human First
GPT-3 First
Human First
Human
57.89%
59.46%
57.89%
59.46%
GPT-3.5
73.68%
59.46%
67.11%
58.56%
Claude
63.16%
64.41%
55.70%
55.41%
GPT-4
54.51%
56.37%
58.27%
58.30%
Table 11: GPT-3 answer win rate (in the GPT-3 vs Human battles). Position bias is mitigated.
Peer discussions help mitigate self-entannement bias According what we previously discovered, LLMs endure self-enhancement bias when acting as judges preferring the answers they generate or that of the models under the same series (e.g., GPT-4 and GPT-3).
We conduct experiments on the subset of LFQA ques- tions where we have human-annotated pairwise compar- isons between Human and Machine-generated (GPT-3 cext-davinci-002) answers. Table 10 shows the win rates of GPT-3 judged by humans and three LLMs. We report the LLMs' initial and after-discussion preferences. Both GPT-3.5 and Claude highly prefer GPT-3's answers in their initial reviews. Specifically, GPT-3.5 significantly favors GPT-3 answers with 13.79% higher win rate. After dis- cussing with other LLMs, all models align better with hu- mans. Our peer discussion method largely helps GPT-3.5 mitigate self-enhancement bias. Before discussions, GPT-4's initial preference aligns well with humans and is almost the same humans after peer discussions. Although GPT-4 still has the self-enhancement bias, it does not favor GPT-3's answers.
GPT-3
Reviewers
Tnitial Preference
After Discussion
Human
58.67%
GPT-3355
72.46%
62.22%
Claude GPT-4
63.81%
60.28%
53.50%
58.75%
Table 10: GPT-3 answer win rates judged by different reviewers on LFQA. For all LLM re- viewers, we take the average accuracy of all dis- cussions they participate in. Self-enhancement exists and is mitigated by PD.
Peer discussions help mitigate position bias As indicated by recent work of Wang et al. (023a)) LLMs are prone to position bias, describing that LLMs tend to show a preference for specific positions, even when prompted not to do so (Table ! in Appendix). In Table 11 the win rate of GPT-3 is highly affected by its position when models generate initial reviews. GPT-3.5 highly prefers the answer the first position compared to Claude and GPT-4, The GPT-3 win rate calculated by GPT-3.5 is 15.79% higher than the win rate based on numan-annotate discussion, all LLM reviewers have closer preferences to humans. Second, all LLMs' scores for GPT-3 answers of both positions are closcr as well, indicating that the position bias is mitigated after peer discussions.
pairwise comparisons when GPT-3 appears first (73.68 vs 57.89). After peer
From another perspective, Figure shows the global preference of selecting answers at the first or second positions across different LLM reviewers. Overall, GPT-3.5 prefers answers at the first position. The other two models favor answers in the second position, similar to the position bias shown in Table 11 After pecr discussion, it shows the same trend of miligating position bias as well.
5 Further Analysis
The reviewer who leads the discussion tends to hold its opinion. In discussion between two LLM reviewers, we define the reviewer who leads the discussion as the leader and the other reviewer as the follower. We find that leaders are less likely to be convinced by followers when they insist on their own opinions at the first turn. We name it "Discussion Ordering Effectt". We observe this effect in discussions over the LFQA questions.
We define two phenomenons which may happen during the discussions: (1) Opinion altering (OA): reviewer changing its opinion after the discussion. For example, R2 posts its preference at turn 2, which is different from RI's preference at turn 1, then RI changes its preference at turn 3 that agrees with R2; (2) Opinion holding (OH): a reviewer does not change its opinion even another reviewer disagrees. For example, R1 posts its preference at turn while R2 disagrees with RI at turn 2; then, RI still holds its preference at turn
As shown in Figure 7 all models have OA when they are in the follower position, while their number of OA decreases significantly after they switch to the leader position. This implies that the discussion ordering effect.
12

---

Published in Transactions on Machine Learning Research (07 (07/202) /024)
Prefer2nd
Proter fat
Opinion Holdiing (OH) total
OA (Opinion Altering) total
1035
50
40
30
20
10
0%
GPT36 pn
After Da.
Clauda
Adbr Dis.
GPT4 Ind
Atter
Claude following
Claud leading
GPT35 fallowing
GPT35 leading
GPT4 following
GPT4 leading
Figure 6: The position bias of all three LLMs initial and after peer discussion reviews. Human has an equivalent preference for either position (red dotted line).
Figure 7: The discussion ordering effect of all three models at the leading and following positions.
exists. On the pairwise comparisons of LFQA where two reviewers initially disagree: when in the leader position, GPT-4 has OA, and Claude has two OAs (happens during the discussions with GPT-3.5). When GPT-4 discusses with Claude, both of them hold their initial preferences when they are in the leader position.
Stronger LLMs tend to hold their opinions from Figure 7 we add up the green mass (OH total) for each LLM reviewer obtain their OH cases in both orderings. We see that models that are commonly recognized being stronger (e.g. GPT-4) more firm in their reviews and hold their opinions. For example, GPT-3.5 changes its opinion most often, and GPT-4 usually holds its opinion. More specifically, GPT-4 holds its opinion in 174 discussions, while Claude and GPT-3.5 hold only in 94 and 76 discussions, respectively.
6 Conclusion
In this work, we provide promising prospects for using a peer evaluation method improve LLM-basod evaluations. Our framework mitigates potential bias (e.g. self-enhancement, positional) in previous prevalent methods. Our proposed peer rank process provides more fair ranking of model capabilities. The peer discussion process helps models reach mutual agreements that correlate with human preference. In the future, we plan to investigate how the general peer evaluation process benefits the LLMs in learning to access their own answer and answering new questions Nicol et al. (001))
Limitations
(1) Currently, the complexity of reviews for N models is O(N*). As the number of tested models grows, the number of pairwise model comparisons increases at the square level, and the number of reviews will grow cubically. The PR method's scalability is potential issue. To mitigate the issue, models or utilize the current top K models as reviewers. This significantly simplifies the complexity of our method. (2) Although we can miligate model bias by applying peer discussion, brings position bias which potentially harms the evaluation performance. The simple and straightforward solution is to average the results in two orders for each pair of models or randomly determine the order. However, can only mitigate position bias but not solve it. We encourage future works to focus on reducing the complexity of pairwise comparison and solving the position bias problem.
randomly select K
13

---

Published in Transactions on Machine Learning Research (07 (024))
Acknowledgments
We thank the anonymous reviewers, Jialu Li and Liqiang Jing for their valuable suggestions on various aspects of this work. This research is supported in part by the National Science Foundation CAREER Grant ITS-2340435 and an Amazon Research Award.
References
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova. DasSarma, al. A general language assistant as laboratory for alignment. arXiv preprint ar 1rXic22112.00661 2021.
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking foundation models with aanguag--model-s-----ccaoaa arXiv:2306. 04181, 2023.
orXiv preprint
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shan Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better lim-based evaluators through multi-agent debate. ArXiv, Abs/2308.0722) 2023.
URL httpp://api.
attpp::/aap..semaniccchhooosss www.:ssssmiiiiccoiiss........ www.aaar.rrgg/ooouusi0/22200 260887005
Justin Chih- Chen, Swarnadeep Saha, and Mohit. Bansal. Reconcile: Round-table conference im-
proves reasoning via consensus among diverse llms. ArXiv, abs/2309.13007, 2023. URL https
//api. seapaisemant orgo wwg/CorpusID::622177 262217323
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, and et.al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, https:/ httsss/mmmssssrrg. wtt.oorgg///////ooooooo///oo
blog/ (logg/2023-00 220//2220222222222 03-30-vicuna// 2023.
Kwangsu Cho and Charles MacArthur. Learning by reviewing. Journal of educational psychology, 103(1):73, 2011.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv:1803.0 (tl:18033.54443 05457, 2018.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 1rX1:12305.44314 2023.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. ArXio, 0te//2305.14333 2023. URL
https://api.. httpp:://ap..ssaanniccchhll/ pew.aammae..ceooo....ooorooroppa : 258841118
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.1 14387, 2023.
Nouha Dziri, Ehsan Kamalloo, Kory Mathewson, and Osmar R Zaiane. Evaluating coherence in dialogue systems using entailment. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1812, 2019.
Arpad E Elo. The proposed usef rating system. its development, theory, and applications. Chess Life, 22(8): 242 247, 1967.
A. R. Fabbri, Wojeiech Kryscinski, Bryan McCann, Richard Socher, and Dragomir R. Radev. Summoval: Ro-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9: 391 391-409, 2020. URL. http:///ppp..aaaaaacccccc.a..ooa http:///wh.aaaaaacccccccccoooo/a
14

---

Published in Transactions on Machine Learning Research (07/2024)
Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. Qafacteval: Improved qa-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2587-2601, 2022.
Augela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1568-3567, 2019.
Jinlan Fu, Sco-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2902.4410 2023a.
Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from feedback. ar preprint wrXi:22305.01.42 2023b.
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. arXiv preprint arXiv:2305. 14627, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazcika, Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. ArXio, 0bs/1506.0330 2015. URL
https://api. semanticscholar www.aar.rrroooooe OrggCorrusID:622333 62037777
Karen Sparck Jones and Julia R Galliers. Evaluating natural language processing systems: An analysis and review. 1995.
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. Hurdles to progress in long-form question answering. In Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Longuage Technologies, pp. 9940-4957, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1 /2021 maacl-main.309 URL.
hwwpe/oooyoorgg//// wwwp:///...aanmmaaaaaaaaaaaa mwa.l-aamliann.. 393.
Wojciech Kryšciíski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9332 9346, 2020.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for' mind' exploration of large scale language model society. arXiv preprint arXiv:2303.17766 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint XrXi::2211.000 1.09110, 2022.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking large language models through multi-agent debate. ArXio, abs/2305.19118, 2023. URL https:://api.seeaniicccot www.nticacooollaaaa.....cccocaaa wwtpeiiaaa.oa.oooooooeeeeeeeeeee wttp:/oorpusDD:5556755555
Chin-Yer Lin. Rouge: A package for automatic evaluation of summaries. In Tezt summarization branches out, pp. 74-81, 2004.
Yen-Ting Lin and Yun-Nung Chen. Llm-eval: Unified multi--immensional automatic evaluation for open-domain conversations with large language models. arXiv preprint urXiv:2305. 13711, 2023.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXio preprint ar arx:::2303. 16634, 2023.
15

---

Published in Transactions on Machine Learning Research (07/2024)
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:21 12. 2.0332, 2021.
David Nicol, Avril Thomson, and Caroline Breslin. Rethinking feedback practices in higher education: peer review perspective. Assessment & evaluation in higher education, 39(1):102 122, 2014.
OpenAI. Webgpt annotation guidelines. 2022.
URL
http:://dccs..oooo// com/document/dd
110h5dorAZydWiiDJJaggg.
OpenAl. Gpt-4 technical report. ArXiu, abs/2303.08774, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Newral Information Processing Systems,
2022.
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999.
Kishore Papinoni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeling of the Association for Compulational Linguistics, pp. 311-311 2002.
Joon Sung Park, Joseph C O'Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint ar Xiv: N3w:2304.0344.. 2023.
Toby Walsh. The peerrank method for peer assessment. In Proceedings of the Twenly-firsl Europeon Conference on Artificial Intelligence, pp. 909-914, 2014.
Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5008-5020, 2020.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023a.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hanna Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources. ArXiv, abs/2306.04751, 2023b.
Fangyuan Xu, Yixiao Song, Mohit lyyer, and Eunsol Choi. critical evaluation of evaluations for long-form question answering. In Proceedings of ACL, 2023.
Matthew M Yalch, Erika M Vitale, and J Kevin Ford. Benefits of peer review students' writing, Psychology Learning & Teaching, 18(3):317-3255 2019.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. ArXio, abs/2401.1002) 2024. URL https://app. U.L.attsss//pp...enn..cctooooo stt.aapp...mmancscchllaaar
CorpusID::260335293
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. or Xiu preprint ar 1rXi::1904 2019.
16

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ton Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao; Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards unified multi-dimensional evaluator for text generation. In Proceedings of the 2022 Conference on Empiricul Methods in Natural Language Processing, pp. 2023 2038, 2022.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Xiv preprint arXiv:2305. 1 11206, 2023.
17

---

Published in Transactions on Machine Learning Research (07/2024)
A Proof of convergence of peer rank
We adapt the convergence proof in Walsh 2014) to our peer rank setting, the main difference is that we introduce weighted winrate, instead of student grades. More specifically:
We suppose there are m LLMs, and LLM agent j provides a winrate Wi, for the response LLM agent
Let X;' be the score of agent the nth itcration of the peer rank and 0 < a We define the score each ileration follows:
X²
1
Σ Wis
X
Exy.
Σ,
The PeerRank scores are the fixed point of these of equations. Note that whilst we start with the (unweighted) average grade, this choice not very critical and will typically reach the same fixed point with other initial seeds.
Fixed Point Analysis
A fixed point X* of the iteration satisfies:
Wes X 5
where S" =, Xj. Since X must be a normalized vector (its components sum to 1), let S 1. Then:
x = Xj Wis
3
In vector notation, this is:
X = WTx
where X is the fixed point vector and W the transpose of the matrix W. This implies X is a right eigenvector of WT with eigenvalue 1.
The ProbenuFs theorem for stochastic matrices guarantees that there is a unique eigenvector with eigenvalue 1 (up to scaling factor), and for primitive matrix (which can be made by proper assumption on W), the powers of WT will converge to rank-one matrix projecting onto this cigenvector.
Thus, repeated application of WT followed by normalization ensures that {X"} converges to this unique eigenvector, which is the fixed point X
B LLM details
As mentioned in section 4.1
wC use APIs of GPT-4, GPT-3.5, Claude, and PaLM-2. Currently, the last two
models' APIs are free.
To generate initial reviews for LFQA (140 questions), GPT-4-0613 costs about $20. For the discussion between GPT-4-0613 and Claude-1 on LFQA, the OpenAl API costs about $24. The price of 1TT-3..-tuurb-065 is 1/20-th and 1/30-th of GPT-4-0613 on inputs and outputs correspondingly.
C Major differences between our PR work and Walsh (2014)
Major differences follows:
18

---

Published in Transactions on Machine Learning Research (07/2024)
1. We focus the evaluation setting where peer LLM reviewers conduct pairwise comparisons of two contestant LLMss answers, instead of student giving grades to other students;
2. As a result, our work involves concepts of winrate/elo scores obtained from pairwise battles;
Walsh (2014) targets the oducational domain audience and involves student participation, while our work is the first to propose the automatic LLM peer evaluation metric and targels machine learning area audience;
Regarding convergence, extend Walsh's proof and demonstrate that our winrate/elo will converge, which also correlates with experimental results (See Appendix A
D Major Differences between Self-Discuss and Self-Refine
Our "self-diccuss" and "Self-refine" proposed by Madaan et al. 2023 are different. Major differences are as follows:
Self-refine is task-oriented work aiming at improving LLMs' performance in solving tasks. Our work (including self-discuss) focuses on LLM-based evaluations and proposes the first peer-evaluation framework. Our goal is to improve the alignment between LLMs' evaluations and human judgments.
2. Self-refine improves performance by iteratively evaluating outputs from the same model (with fixed temperature: 0.7). However, we enhance evaluations by discussions between LLMs from multiple perspectives (different decoding strategies) to reach a mutual agreement. Although two reviewer LLMs share the same backbone (e.g. GPTT3355), they adopt varied decoding temperatures (e.g., 0.2 and 0.8), which afford models to explore diverse discussion stralegies.
E Detailed Win rate & Elo Calculation
The algorithm for calculating weighted Elo is described in Algorithm 1 win rate is described in Algorithm 2
The algorithm for calculating weighted
Algorithm 1: Weighted Elo Ratings
Input :B The list of battle reviews
Each review is a 5-tuple
(question, contestant A, contestant B, reviewer, score) where a score of (-1, 13 means (A wins, lie, wins)
W The mapping of reviewers to weights
Output: Elo Elo rating for each contestant
K 3-32;
Define p(z) 1+10=/400
// scale weighta that their is 1.
W + W/mean(W):
Elo + mapping of each contestant B 1000.
foreach (g, (,,i, (),, B do
w +- W[r]
r Blo[i] :
rB < Elofi]:
cA p(rB rA)
10
eB + p(rpppp-1 rB);
// has win value 0, 0.5, or for i loss, tie, or win
11
8A -1+ 3B+1-5A sA
12
13
Increment Eloli] by wK(s eA):
14
Increment Elo[j] by (sB eB):
15 end
16 return Elo
19

---

Published in Transactions on Machine Learning Research (07 (07/202) 2024)
Algorithm 2: Weighted Win Rates
Input :B
The list of battle reviews
Each review is a 5-tuple
(question, contestant A, contestant B, reviewer, score)
where a score {-1, 0, 1}
means {A wins, tie, B wins}
Hers The number iterations to run
Output: :s The win-rate for each contestant
- The resulting weights at the end
C set of contestants
R set of reviewers in B
<- mapping each reviewer R/
for 1 to Ilers do
// No. reviews for each contestant N +- mapping of each e C to 0:
// Weighted wins for each contestant
V mapping of each E C' to 0;
foreach (q, E B do
// Update number of reviews Increment N[1] by 1; Increment N[j] by 1;
+ W[r);
/* maps (loss=-1, tie=0, win=1) to (0, 0.5, 1)
11
Define f(x) = (1 2)
Increase Vi by w- f(-8);
13
Increase VbJ by w f(8);
14
end
$ + mapping of each E C to V'e Nio
15
W Mormalize
17 end
18
return S, W
F Term Definitions
Definitions of the three terms in Table N
are:
Unsupported information: It the information that not related to the question and redundant
the current answer.
Core Information: This type of information is the key to answering the question. Lacking it will lead to a wrong answer.
Coherence: The answer should be tightly structured and coherent, progressing logically from sentence to the next, avoiding disorganized presentation of related information.
Our experiments are already based on prompts, including explanations of terms. We find that this does not affect the performance of PD.
G
Role Prompt
Table 12
describes the prompt added the end of each turn during discussions. The role information in the prompt serves instruction for models in next-turn discussions.
20

---

Published in Transactions on Machine Learning Research (07 (07/2024)
[System]
You are reviewer A/B, discussing with reviewer B/A about reviews of the following answers. Read the reviews and discussions above, and make deciaion change your preference, and explain.
new line, choose between the two answers by outputting number or Do not output anything else other than the number in this last line.
Table 12: The role prompt is added to the end of each turn. is used notify another model to follow the instructions.
H Pairwise win rate heatmap
Figure
shows more pairwise win rate heatmaps.
Model
346-4
claade
101-3.
bard
Viuum
B03-4
claucle
vicure
031-3.5
bard
3.0
0.57
0.71
0.72
0.79
cpc
0.81
0.60
0.62
0.86
0.8
07
Model
0.78
0.45
0.58
0.58
n.S
vicure
0:20
0.11
0.61
0.03
0.5
DOC
03
02
vicumb
321
0.26
0.42
0.48
band
0.14
0.13
0.37
0.50
Fraction Model Wins For All
Battles (Chatbot Arena)
Fraction of Model Wins All
Battles (human reviews)
Figure 8: Pairwise win rate heatmap (Left: arena leaderboard; Right: human).
21

---

Published in Transactions on Machine Learning Research (07 (07/2024)
We sould lix io requee your inectacc parto on of enweru Al anddeni resporon the soe queder cenddnndddennnn eppeeedd bebw
Pireee reen de tergel necs inleereee ooancyy the virt deee rong
Than baak ar paineinn corpesioons nowe led smproes awdi protedieiii renuaing that the cater wowk imquaree ween peroonted dire not aftert jeigreer
Für glleerou elio prowato nokee GPP4 reith oopeenas Sexo 6PT1 ooder tew provide the rowc CPTC tolk (Rsppaned, Pasperou (Resporndd) Rosponeed) (hoictie Aucisio Question:
Use kpprgp 1o0 tomnett structure farmd lettr monantioon for student apoblng prosigious gioducte progran in computt solince.
Peseernc Resprme Rovaue (oouer 3 Dusell Dumpuserr
Response 1:
harw: hean known tirah en for Uirgl of limet in cupecity an jekiror an to qladert al insturtonn numee Shudeen Namms) been on cubllrdding shaden poure couran Foom mo in edtern jrashed hawe fourd [oard[hisher plecr plan eusdent bey aitio tes sccern plat evenss) Hattee has srown dhi lnvdens pi abili coem cattonn oditi an madro by jeserccie and accompg dymenns (fiethee hex elto develppe innoasee un store compies goor paoppen
Frein to horlasn sontact me enquin any further intormuriinn Sincwayy (aly (roir twneg Mlag
heptineo
57.0
por
Figure 9: The form used by human annotators for individual rating of model. Sliders included to rate response several metrics from 0 to 10. Explanations can be entered in the text area below. The tab bar and next button navigate between responses.
Human Annotation for Pairwise Preference
Since completing one HIT can take a considerable amount of time (6-10 min), we added a button that allows saving their work at any stage in the middle of the HIT. This button populates a text area with JSON representation of the current responses, which may be copied into file.
We annotate part of the pairwise comparisons of model answers on Vicuna80. We built an interface form. The screenshot is as illustrated in Figure 10
22

---

Published in Transactions on Machine Learning Research (07 (07/2022 /224)
We'h wali simn you Werteer tn wit saan ownrt viscons woonass cnerg prodic slyydd belie
secatte sannas carpai aar and sapa muildng aoy potrr Rigoni
waps sat passrra dass you
reword oft G/14
unundall CITA flous Riatoeett tiuwe add flauumaaf Housien 2t
Question:
Whst th petna alinaancc cana ane nehwwrr
Compenea Ioppeee compane Compeer Compene Impperaand
Ovia all Compusisers
Resporise
Rassponse
are sorout ireus bicertince
greess decuna
Nake Tate bo stiaccnn
lcki Bi taal mtal arteii dereeees Deair tal 2hu27
Chaelcas latee Teerei den the cieeeed
petogles and cen ie seeutt the ceniee facke Thes while
for moch and
andernatonnn Hepuoees ail chlqguos
repuo LMNCIS mast
Recocio Reppnas backatee ReNCTLY
Seppanes
reparssc EXCCT)
EARCTL
copasiou DSACTE trually
Sequaeee wil
Revarnor 3i wneer Aeraane tie haw (The reporss FoMCTY was
Rupprrss fand oiaapeass Reapore leaa ofe staerss as EXed quiin.
Sak 12Dh bmaa
Figure 10: The form used by human annotators for pairwise comparisons between model answers. Each pair of comparisons has buttons to choose which model is best, along with area to provide explanation. An associated button hides/shows the respective responses and automated comparisons.
23

---

Published in Transactions on Machine Learning Research (07/2024)
J Discussion Examples
In this section, there are four examples showing opinion-altering (OA), opinion-holding (OH), and post- agreement opinion-altering. Tn the following discussions, all texts before colored reviewer's name are the imput for that reviewer. the text before "[System]" is the reviewer's original output and the text after [System]] is added after each round which reminds the nex reviewer about its role.
The following example is a discussion between GPT-3.5 and Clande-1. In this example, GPT-3.5 alters its opinion to agree with Claude-1, while Claude-1 holds its opinion.
GPT-3.5 Claude-1 Discussion (GPT-3.5 Leads)
System: You are reviewer 1, discussing with reviewer 2 about your reviews of the following answers.
Background: [Question]
When the joint-stock company was first invented, was there lot of pushback on the concept? What were some of the concerns? Also any recommended books on the invention of the concept would be much appreciated!
[The Start of Answer 1]
An early form of joint stock ventures appeared in Assyria by the beginning of the 2nd millennium BCE. There does not seem to have been any noticeable pushback against the concept, although trade caravans were often taxed heavily. Of course, the relevant texts were written to and from people engaged in trade, so there's no way of knowing what. other Assyrians thought of long distance trade. >Most of these traders had become more independent by having become managers of a 'joint-stock fund" (called "naruqqum*, "money bag"), usually set up in Assur. This phenomenon appeared for the first time around 1900 BC and seems to have been an Old Assyrian invention that went beyond individual partnerships and cooperation in a joint caravan. The arrangement, rather similar to that of the early medieval *compagnia*, meant enlisting a number (usually about a dozen) investors (fummianum*, 'financiers"), who supplied capital rated in gold, usually in all ca. 30 kilos, idcally consisting of shares of or 2 kilos of gold cach. It entrusted to trader (the traclator), usually for ca. ten years, for the generally formulated purpose of "carrying out trade." The contract contained stipulations on a final settlement of accounts, on paying dividends, on the division of the expected profit, and on fincs for premature withdrawal of capital (meant to secure the duration the business). Investors or shareholders mostly lived Assur, but successful traders in Anatolia too invested in funds managed by others, perhaps also as way of sharing commercial risks. In such cases contract would to be drawn up in Anatolia that obliged the tractator "to book in Assur x gold in his joint stock fund in the investor's name.' Among the investors we find members of the tractator's family, but also business relations and others, probably kind of 'merchannt-ankerss, and other rich citizens, who aimed at fairly safe, long- term investments. ['Ancient Assur: The City, its Traders, and its Commercial by Klaas Veenhof For more on Old Assyrian trade and these early joint stock funds, see Anncintt Kanesh: A Merchant Colony in Bronze Age Ana- by Mogens Larsen
httptttps///wwwwjstoroorrg
225651222
tolia* (tttps://boooss.ppppoooo www.ooosggooll..oomooooo wttp://oooo..oooooooooooooeeeeee /ttp://k/s/ddi00ggggAA00000
[The End of Answer 1]
[The Start of Answer 2]
There was definitely some pushback when the joint-stock company was first invented.
24

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Some people were concerned that would lead to corruption and cronyism, while others were worried that it would concentrate too much power the hands of a few people. Recommended books on the topic include The First Tycoon: The Epic Life of Cornelius Vanderbilt by T.J. Stiles and The Great Transformation: The Political and Economic Origins of Our Time by Karl Polanyi.
[The End of Answer 2]
[The Start of Reviewer I's Evaluation]
Answer 1:
Unsupported information: The answer provides historical information about joint-stock ventures in Assyria, which is not directly related to the invention of the joint-stock company.
Core information: The answer does not directly address the pushback on the concept of joint-stock companies or the concerns associated with it.
Coherence: The answer provides information about joint-stock ventures in Assyria but does not connect to the concept of joint-stock companies or the question at hand.
Answer 2:
Unsupported information: The answer does not provide any unsupported information. Core information: The answer directly addresses the pushback on the concept of joint-stock companies and mentions concerns related to corruption, cronyism, and concentration power.
Coherence: The answer is focused and provides relevant information related to the question.
Based on the evaluation, Answer 2 is the better response it directly addresses the question, provides core information, and maintains coherence.
[The End of Reviewer I's Evaluation]
[The Start of Reviewer 2's Evaluation]
Answer 1 provides comprehensive overview of early joint stock companies in Assyria, citing multiple academic sources. It focuses facts about how these companies operated, who invested in them, how long they lasted, and what pur- poses they served. The answer is coherent and flows logically from one idea to the next.
Answer 2 is much more superficial, making broad claims about pushback against joint stock companics with little evidence or detail. The two book recommendations seem tangential and are not rell-integrated into the main points of the answer. Overall this answer lacks coherence and depth compared to Answer 1.
[The End of Reviower 2's Evaluation]
SSystem]
Read the reviews and discussions above, and make docision if to change your preference, and explain. Remember we focus on unsupported information, core information and coherence.
In a new line, choose between answer and answer 2 by outputting the number
25

---

Published in Transactions on Machine Learning Research (07/2024)
or 2 respectively. Do not output anything else other than the number in this last line.
GPT-3.5:
2
[System]
You are reviewer 2, discussing with reviewer 1 about your reviews of the following answers.
Read the reviews and discussions above, and make a decision to change your preference, and explain. Remember we focus on unsupportod information, core information and coherence In new line, choose between answer and answer by outputting the number or 2 respectively. Do not output anything else other than the number in this last line.
Claude-1:
will not change my preference for Answer 1.
While Answer 2 directly addresses concerns about joint stock companies, it does in superficial way without evidence depth. Answer provides wealth of historical detail about early joint stock companies in Assyria, citing multiple academic sources. Although not directly focused on pushback, Answer 1 demonstrates a strong, coherent understanding of how these early companies worked.
Given our criteria of evaluating unsupported information, core information, and coherence, Answer 1 still superior in my view. The tangential book recommendations in Answer and lack of evidence for its claims reflect poorly for coherence and depth. Answer 1, though broader in scope, shows strong coherence and is well-supported.
[System]
You are reviewer 1, discussing with reviewer 2 about your reviews of the following answers.
Read the reviews and discussions above, and make decision if to change your preference, and explain. Remember we focus on unsupported information, core information and coherence In a new line, choose between answer and answer by outputting the number 2 respectively. Do not output anything clse other than the number in this last line.
GPT-3.5:
I agree with your points. After considering your evaluation and the discussion, I am changing my preference to Answer 1. It provides more depth and coherence with its historical information about early joint stock companies in Assyria. While it may not directly address the pushback on the concept, it still provides valuable context and understanding of how these early companies operated. Answer 2, on the other hand, lacks evidence and depth, and the book rocommendations seem tangential to the main points. Therefore, believe Answer 1 is the betler response overall.
26

---

Published in Transactions on Machine Learning Research (07 (07/2024)
[System]
You are reviewer 2, discussing with reviewer about your reviews of the following answers.
Read the reviews and discussions above, and make a decision to change your preference, and explain. Remember we focus on unsupported information, core information and coherence
In new line, choose between answer and answer 2 by outputting the number or 2 respectively. Do not output anything else other than the number in this last line.
Claude-1:
appreciate your thoughtful consideration of my evaluation. We are agreement that Answer 1 provides the most coherent, well-supported response overall, if not directly focused pushback. Depth of understanding and evidence most. valuable in my view.
The following example is discussion between GPT-4 and Clande-1. In this example, GPT-4 holds its opinion, and Claude-1 alters its opinion to agree with GPT-4's opinion.
GPT-4 Claude-1 Discussion (GPT-4 Loads)
System: You reviewer 1, discussing with reviewer 2 about your reviews of the following answers.
Background:
[Question]
Could Frederick Douglass swim?
[The Start of Answer 1]
think that the best thing to ask about, perhaps, would be *childhood* within the bounds of enslavement. Young enslaved children would often be the playmales of white children, so learning to swim together would not be unusual and Douglass, clsewhere in the extended quotation, makes clear swimming was something from his *childhood*, describing his swim that *"Here all my boyish pranks in the front basin of Baltimore were renewed It was very interesting dichotomy that existed those situations, an intermixing both of friendship *and* mastery, so definitely lot of fodder to write about.
The End of Answer 1]
[The Start of Answer 2]
That's actually very interesting, was under the impression efforts were made make sure slaves couldn't it swim a means of cutting off potential escape routes via rivers ponds etc. it makes sense it wasn't black and white (ha, like it ever was)
'The End of Answer 2]
The Start of Reviewer 1's Evaluation]
Answer provides a more detailed and informative response to the question. It sug- gests that Frederick Donglass could swim, based on his own accounts of his childhood.
27

---

Published in Transactions on Machine Learning Research (07/2024)
The answer also provides some context about the conditions enslaved children and their interactions with white children, which could have included learning to swim to- gether. This answer is coherent and scems be supported by the information provided.
Answer 2, the other hand, doesn't 't directly answer the question. Instead, it offers personal impression about the general conditions of enslaved people and the potential restrictions on their ability to swim. This answer lacks specific information about Frederick Douglass and doesn't provide a clear answer to the question. It's also less coherent than Answer 1, as it doesn't directly address the question and instead offers a personal opinion.
Based these evaluations, Answer 1 is the better response to the question.
[The End of Reviewer I's Evaluation]
[The Start of Reviewer 2's Evaluation] Here is my evaluation of the two answers:
Answer 1 provides some interesting historical context about the intermixing of white and enslaved children during Douglass's childhood, and speculates that Douglass may have learned to swim alongside white children during this time. However, the answer does not directly or conclusively state whether Douglass could swim. The answer coherent but lacks core information fully address the question.
Answer 2 acknowledges the common assumption that slaves were prevented from learning to swim, but speculates that the situation was complex, indicating Douglass may have learned to swim as a child. However, like Answer 1, Answer does not provide a definitive statement on whether Douglass could swim. The answer is coherent but lacks core information.
Overall, neither answer provides conclusive stalement supported by evidence on whether Frederick Douglass could swim. Both answers provide some speculative historical context but lack core information to fully address the question.
(The End of Reviewer 2's Evaluation]
[System]
Read the reviews and discussions above, and make decision if to change your preference, and explain. Remember we focus on unsupported information, core information and coherence.
a new linc, choose between answer and answer by outputting the number or 2 respectively. Do not output anything else other than the number in this last line.
GPT-4:
After reading Reviewer 2's evaluation, I still maintain my preference for Answer 1. While both answers do not provide a direct, definitive answer to the question, Answer 1 provides more context and uses quote from Douglass's own accounts to suggest that he could swim. This gives a stronger basis evidence
28

---

Published in Transactions on Machine Learning Research (07/2024)
than Answer 2, which only offers a personal impression and speculation. Therefore, Answer more coherent and contains more core information relevant to the question.
[System]
You are reviewer 2, discussing with reviewer 1 about your reviews of the following answers.
Read the reviews and discussions above, and make a decision to change your preference, and explain. Remember we focus on unsupportod information, core information and coherence In new line, choose between answer and answer by outputting the number or 2 respectively. Do not output anything else other than the number in this last line.
Claude-1:
Here is my perspective as Reviower 2:
After considering Reviewer 1's preference for Answer 1, still maintain my view that neither answer conclusively addresscs whether Frederick Douglass could swim. While Answer provides more historical context and references Douglass's own writing, it does not contain direct statement on whether he could swim. Answer 2 is more speculative but also lacks definitive conclusion.
Overall, both answers lack core information needed to fully address the question, despite Answer having stronger basis in evidence. Given the lack of conclusive statement in either answer, I do not have clear preference for one over the other. My view is that more information is needed to determine whether Douglass could swim.
2
[System]
You are reviewer 1, discussing with revicwer 2 about your reviews of the following answers.
Read the reviews and discussions above, and make decision if to change your preference, and explain. Remember we focus unsupported information, core information and coherence
a new line, choose between answer and answer by outputting the number or 2 respectively. Do not output anything else other than the number in this last line.
GPT-4:
After considering Reviewer 2's evaluation, I still stand by my ini- tial preference for Answer 1. While it's true that neither answer provides a definitive statement on whether Frederick Douglass could swim, Answer 1 does provide more context and uses a quote from Douglass's own accounts to suggest that he could swim. This gives it a stronger basis in evidence than A Answer 2, which only offers personal impression and speculation. Therefore, Answer more coherent and contains more core information relevant to the question.
[System]
You are reviewer 2, discussing with reviower about your reviews of the following
29

---

Published in Transactions on Machine Learning Research (07 (07/2024)
answers.
Read the reviews and discussions above, and make & decision to change your preference, and explain. Remember we focus on unsupported information, core information and coherence In new line, choose between answer and answer by outputting the number or 2 respectively. Do not output anything else other than the number in this last line.
Claude-1:
After further discussion, I can see Reviewer I's perspective that Answer provides more context and evidence, even without a definitive conclusion. While neither answer conclusively states whether Fredcrick Douglass could swim, Answer 1 does draw on Douglass's own accounts and provides historical details suggesting he may have learned child. This gives Answer stronger basis than Answer which is more speculative.
Therefore, while still believe more information needed for a definitive conclusion, Answer appears to have a stronger grounding in evidence and coherence. On balance, would lean toward preferring Answer over Answer 2, though neither is fully satisfactory.
[System]
Thank you for the discussion. Based on the reviews and discussions, Answer is preferred as it provides more context and evidence, though not definitive, to suggest. Frederick Douglass could swim.
The following example is a discussion between Clande-1 and GPT-4. In this example, Claude-1 alters its opinion after reaching an agreement with GPT-4.
Claude-1 GPT-4 Discussion (Clande-1 Leads)
System: You reviewer 1, discussing with reviewer 2 about your reviews of the
following answers.
Background:
[Question]
Why do people go bald on the top their head and still have some on the sides and not vice-versa?
The Start of Answer 1]
There few reasons why people go bald on the top of their head and still have some hair on the sides. One reason is that the hair follicles on the lop of the head are more sensitive to a hormone called dihydrooestosterrne (DHT). DHT is byproduct testostcrone and is found in both mon and women. When DHT levels are high, it can cause the hair follicles on the top of the head to miniaturize, which leads to hair loss. Another reason that the hair on the sides and back of the head is not as exposed the sun and other environmental factors, which can damage the hair and lead to hair loss.
30

---

Published in Transactions on Machine Learning Research (07 (07/202) 2024)
[The End of Answer 1]
[The Start of Answer 2|
Seems unsettled but here's theory:
"The most recent hypothesis suggests that the hair loss process begins during puberty, when growth of the skull and the muscles in the forchead and neck increases the tension in light band of tissue stretching over the top of the head. The more DHT (a type of testosterone) there is, the more these muscles contract, which causes inflammation and brings in more DHT. Over time, the DHT thickens the band of tissue, restricting the supply of blood and nutrients to the hair follicles above it. The follicles on top of the head get smaller and eventually disappear, while those on the sides of the head remain largely unaffected."
Source https:/ hoss::/wwww http:://www.sciinneefooooooo www.scienceoouus.com www..cooo/neeuuuaaoobbblbooo htwe-/uaaabbboo///wooooeeeeeee www.-ww-----------bblleeee
the-top- of-their-Bee headss
[The End of Answer 2]
[The Start of Reviewer I's Evaluation] Here my evaluation of the two answers:
Answer 1 provides concise and coherent explanation for male pattern baldness, focusing on the sensitivity of hair follicles the top of the head to DHT and less exposure to environmental damage. The information seems well supported by scientific research. However, the answer is lacking in depth and does not fully explore other potential contributing factors.
Answer 2 proposes an alternative hypothesis that relates balding increased tension and inflammation in the scalp tissue over time duc to skull and muscle growth during puberty, well the thickening effects of DHT. This explanation seems more speculative, as the source cited is from general science magazine rather than peer-reviewed research. The coherence is slightly weaker, as the answer jumps quickly from discussing puberty to DHT without fully connecting the idcas. However, the answer provides an interesting alternative perspective and more depth than Answer
Overall, while Answer 1 is more concise and scientifically grounded, Answer 2 explores alternalive, thoughi--roovoiing hypothesis and provides more comprehensive explanation, despite some weaknesses in coherence and evidence. On balance, I would choose Answer as the higher quality response.
[The End of Reviewer I's Evaluation]
The Start of Reviewer 2's Evaluation]
Answer 1 and Answer 2 both provide explanations for why people tend to go bald on the top of their heads while still retaining hair on the sides.
Answer 1 explains that the hair follicles on the top of the head are more scnsitive dihydrotestosterone (DHT), a hormone that can cause hair loss when its levels high. It also mentions that the hair the sides and back of the head is less exposed to environmental factors that can damage hair and lead to hair loss. This answer is
31

---

Published in Transactions on Machine Learning Research (07/2024)
coherent and provides a scientifically supported explanation for the pattern hair loss.
Answer 2 provides different thcory, suggesting that the process begins during puberty when growth of the skull and muscles in the forehead and neck increases tension in band of tissue over the top of the head. This tension, combined with the presence of DHT, lcads to inflammation and restriction of blood and nutrient supply to the hair follicles, causing them to shrink and eventually disappear. This an- swer also provides a source for its information, which adds credibility to its explanation.
Both answers are coherent and provide core information related the question. However, Answer 1 provides more commonly accepted explanation for the pattern of hair loss, while Answer 2 provides more recent hypothesis that may not be as widely accepted or proven. Neither answer contains unsupported information.
Based on the above evaluation, I would choose Answer 1 as it provides a more widely accepted and proven explanation for the pattern of hair loss.
[The End of Reviewer 2's Evaluation]
[System]
Read the reviews and discussions above, and make a decision to change your preference, and explain. Remember we focus on unsupported information, core information and coherence.
In new line, choose between answer and answer by outputting the nunber or 2 respectively. Do not output anything else other than the number in this last line.
Claude-1:
Here is my analysis of the two reviews:
Reviewer prefers Answer 2 due to its depth and exploration of an alternative hypothesis, despite some weaknesses in coherence and evidence. Reviewer prefers Answer due to it providing more scientifically supported and widely accepted explanation.
Upon further reflection, I agree with Reviewer 2 that Answer 1 is superior. While Answer proposes an interesting alternative theory, its explanation seems more spoculative and not as well supported by scientific evidence. Answer provides a concise yet coherent explanation backed by research on DHT sensitivity and environmental factors.
Although Answer 2 explores the topic in more depth, depth alone does not make up for a lack of scientific grounding. Some unsupported information, such as the theory of increased tension and inflammation during puberty leading to balding, weakens its coherence. In contrast, Answer focuses on core information that well established, leading a more compelling explanation overall.
Therefore, based on the evaluations, I would choose:
1
32

---

Published in Transactions on Machine Learning Research (07/2024)
[System] You are reviewer 2, discussing with reviewer about your reviews of the following answers.
Read the reviews and discussions above, and make a decision to change your preference, and explain. Remember we focus on unsupported information, core information and coherence
In new line, choose between answer and answer 2 by outputting the number or 2 respectively. Do not outpul anything else other than the number in this last line.
GPT-4:
[System]
You are reviewer 1, discussing with reviewer 2 about your reviews of the following answers.
Road the reviews and discussions above, and make decision if to change your preference, and explain. Remember we focus on unsupported information, core information and coherence
In a new line, choose between answer and answer by outputting the number 2 respectively. Do not output anything clse other than the number in this last line.
Claude-1:
Upon reviewing the discussion, I do not see a need to change my original preference for Answer 2.
While Answer 1 provides a concise and coherent explanation grounded in scientific research, Answer explores an alternative hypothesis in more depth and provides a source to support its theory. Some speculation does not necessarily equate unsupported information, long the idcas are logically connected and backed by credible references where possible.
Depth and exploration of alternative perspectives are valuable in building understand- ing. Answer 2 acknowledges the rolc of DHT while proposing that increased tension and inflammation may also contribute hair loss over time. This added nuance strengthens rather than weakens its coherence.
Answer is a satisfaciory explanation but lacks the depth and nuance of Answer 2. Based on the additional discussion, continue to believe Answer 2 is the higher quality response due to its more comprehensive analysis of the question, despite its more speculative nature. The source provided also lends some credibility to its alternative hypothesis.
Therefore, maintain my original view that:
2
[System]
You are reviewer 2, discussing with reviewer 1 about your reviews of the following answers.
Read the reviews and discussions above, and make decision if to change your preference, and explain. Remember we focus unsupported information, core information and coherence
In new line, choose between answer and answer 2 by outputting the number
33

---

Published in Transactions on Machine Learning Research (07/2024)
or 2 respectively. Do not output anything else other than the number in this last line.
GPT-4:
After considering the points made by Reviewer 1, still stand by my initial preference for Answer 1.
While Answer 2 does indeed provide more in-depth explanation and proposes an alternative theory, it lacks the scientific grounding that Answer has. The theory proposed Answer not as widely accepted or proven, which could potentially lead to misinformation.
Answer 1, on the other hand, provides concise, coherent, and scientifically supported explanation for the pattern of hair loss. It focuses core information related to the question and does not contain any unsupported information.
Therefore, I maintain my original choice:
The following example is a discussion between GPT-4 and GPT-3.5. In this example, both reviewers hold their opinion all the time.
GPT-4 GPT3.5 Discussion (GPT-4 Leads)
System: You are reviewer 1, discussing with reviewer 2 about your reviews of the
following answers.
Background:
How do credit/debit cards work? What the process of putting money in and getting it out?
[The Start of Answer 1]
A credit or debit card is basically just an easy way to allow a shop to speak to your bank.
First you go into the bank, and you either deposit some cash into your account (debit), you ask the bank nicely to cover you, and promise that you will pay them back at the end of the month (credit).
Then you can go shopping. You get to the till with your products, which you now need to pay for, but instead of handing over cash directly, you ask the shop to deal directly with your bank. The shop phones up the bank and asks them to take amount of money out your account, then nip over to the shops own bank and give it to them on your behalf, which they do.
That pretty much how a credit or debit card works, just automated with computers...
(The End of Answer 1]
[The Start of Answer 2]
Debit cards are linked to a bank account and whenever you pay using a debit card, the amount. deducted from the bank account on completion of the transaction. So, instead of withdrawing and handing over cash for purchase, you can swipe the card. Credit cards come with credit limit and defined billing cycle (generally monthly).
34

---

Published in Transactions on Machine Learning Research (07/2024)
So, you can make purchases using the credit card upto the defined credit limit. The bill is generated as per the defined billing cycle and you are required to settle just like the bills for any other utilitics. Hope this helps.
[The End of Answer
[The Start of Reviewer I's Evaluation]
Both Answer and Answer accurately describe how credit and debit cards work, and neither contains unsupported information.
Answer provides more detailed and narrative explanation, using an analogy of shop speaking to your bank. It explains the process of depositing money into a bank account (for a debit card) or asking the bank to cover your expenses (for a credit card). It then describes how the shop communicates with your bank to transfer money from your account to the shop's account. This answer is coherent and casy to understand, espocially for someone who may not be familiar with banking terms.
Answer 2, on the other hand, more concise and uses more technical language. explains that debit card is linked to bank account and the amount is dedueted from the account when purchase is made. It also explains that credit card has a credit limit and billing cycle, and that you make purchases up to the credit limit. The bill is then generated and needs to be paid, similar to other utility bills. This answer is also coherent and accurate, but may be less accessible to somcone who is not familiar with banking lerms.
In terms of core information, both answers explain the basic functions of credit and debit cards. However, Answer 1 provides more detailed explanation of the process, while Answer 2 is more concise and to the point.
In conclusion, both answers are correct and coherent, but Answer provides a more detailed and accessible explanation, while Answer more concise and uses more technical language.
1
(The End of Reviewer I's Evaluation]
[The Start of Reviewer 2's Evaluation]
Answer
Unsupported information: The answer includes the statement 'you ask the bank nicely to cover you." This statement is subjective and not factual representation of how credit cards work.
Core information: The answer explains that credit cards allow a shop to communicate with the bank, and it describes the process of the shop requesting money from the bank on behalf the cardholder.
Coherence: The answer is coherent and provides a clear explanation of the process.
Answer 2:
Unsupported information: The answer does not contain any unsupported information. Core information: The answer explains that debit cards deduct the amount from the bank account immediately, while credit cards have a credit limit and require monthly bill settlement.
35

---

Published in Transactions on Machine Learning Research (07/2024)
Coherence: The answer is coherent and provides a concise explanation of how debit and credit cards work.
Based the evaluation, both answers contain core information and are coherent. However, answer 1 includes unsupported information, while answer 2 does not. Therefore, answer 2 is the more accurate and reliable response.
2
[The End of Reviewer 2's Evaluation]
[System]
Read the reviews and discussions above, and make decision if to change your preference, and explain. Remember we focus on unsupported information, core information and coherence.
a new linc, choose between answer and answer by outputting the number or 2 respectively. Do not output anything else other than the number in this last line.
GPT-4:
After reading Reviewer 2's evaluation, understand their point about the subjective language used in Answer 1. However, still believe that the analogy used in Answer 1 is helpful for someone who may not be familiar with banking terms, and it does not detract from the core information or coherence of the answer. Therefore, maintain my preference for Answer due to its detailed and accessible explanation.
[System]
You are reviewer 2, discussing with reviewer 1 about your reviews of the following answers.
Read the reviews and discussions above, and make a decision to change your preference, and explain. Remember we focus on unsupportod information, core information and coherence In new line, choose between answer and answer by outputting the number or 2 respectively. Do not output anything else other than the number in this last line.
GPT-3.5:
After considering the points made by Reviewer 1, I agree that the subjective language used in Answer could be seen as unsupported information. Additionally, Answer 2 provides a concise and accurate explanation without any unsupported information. Therefore, I have changed my preference to Answer 2.
2
[Systom]
You are reviewer 1, discussing with reviewer 2 about your reviews of the following answers.
Read the reviews and discussions above, and make decision if to change your preference, and explain. Remember we focus on unsupported information, core information and coherence In a new line, choose between answer and answer 2 by outputting the number
36

---

Published in Transactions on Machine Learning Research (07/2024)
or 2 respectively. Do not output anything else other than the number in this last line.
GPT-4:
After considering Reviewer 2's points, still believe that Answer I provides more detailed and accessible explanation for someone who may not be familiar with banking terms. The subjective language used does not detract from the core information or coherence of the answer. Therefore, maintain my preference for Answer
[System]
You are reviewer 2, discussing with reviewer 1 about your reviews of the following answers.
Read the reviews and discussions above, and make a decision to change your preference, and explain. Remember we focus on unsupported information, core information and coherence
new line, choose between answer and answer by outputting the number or 2 respectively. Do not output anything clse other than the number in this last line.
GPT-3.5:
After considering Reviewer I's points, 1 understand their perspective on the accessibility of Answer 1. While the subjective language used may not necessarily detract from the core information or coherence, is important prioritize accuracy and reliability in providing information. Therefore, I am changing my preference to Answer 2, which provides concise and accurate explanation without any unsupported information.
2
37


Reviews:
[{"review_id": "sJmsjTxQHf", "reviewer": "Reviewer_1dPG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Strengths 1: The idea of facilitating a discussion between two models, in a similar fashion with the discussion period in scientific peer review, is quite interesting. I enjoyed reading the results about Peer Rank and Peer Discussion’s improvements to the LLM-based evaluation quality. \n\n* Strengths 2: The authors described their methods clearly and provided nice intuitions for interpretation of the results. The metrics considered are intuitive and capture the alignment of evaluation outcomes with human preferences. \n\n* Weaknesses 1: While the idea and intuition for using Peer Rank and Peer Discussion to improve LLM evaluation is intriguing, I feel that certain technical descriptions (methods and results) lack support. Please see requested changes of a list of my questions/suggestions. \n\n* Weaknesses 2: I do feel that the writing of the paper could be improved. There are some parts where the writing is unclear, and some minor issues such as typos and notations. For example, the notation of “a set of battle reviews” B (second paragraph in Section 3.1) is overloaded with B as the second reviewer model. There are also some typos, for example, “They as tested on both…” in the second part of Section 2. Below is a non-exhaustive list of places where I found the writing to be unclear: \n    1. Authors mentioned “calculating metrics such as the win rate of each contestant” in Section 3.1, however, at this point, the audiences have no information of the answers and responses (until later in the experiment section) – what types (distributions) of questions were considered? Are they encouraged to be diverse? It might be good to provide some information here, so that the readers can connect the aforementioned “battle reviews” with the win rate matrices. \n    2. In Section 4.1.2, I am confused with the “human-judged ranking scores”, specifically how these scores are acquired. For example, Table 3 shows experiment results on Vicuna80 dataset. If I understood correctly, the human annotations are available in pairwise comparisons (as mentioned in Section 4.1.1). But it is unclear how these pairwise comparisons are converted into rankings of models? \n    3. In the last line on page 10, it is mentioned that before discussions, GPT-4 is “not favoring GPT-3 much and is more fair” – does this contradict with the previously-mentioned self-enhancement bias? Or is it that GPT-4 only favors its own answers and not answers from previous versions such as GPT 3.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes two related methods aiming towards improvement in LLM-based evaluations, with a focus on evaluation of long and freeform answers. Peer Rank takes in pairwise preferences between answers of two “contestants” LLM models, reviewed by a “reviewer” LLM model, and output a final ranking of the “contestants” models. It involves an iterative approach, where the preferences given by different LLMs are weighted according to their (updating) abilities. Peer Discussion focuses on the quality of LLM-based reviews and prompts a discussion between two ‘reviewer’ LLM models. Authors conduct experiments on benchmark datasets to examine the ability of proposed methods to improve the alignment of (i) ranking of models and (ii) pairwise comparison between two answers, with human preferences. In majority of the instances, the proposed methods can improve the correlation with human-annotated preferences.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tWeighting in PR. A question about the method Peer Rank (PR): it is mentioned in the motivation for PR that the current LLMs tend to have self-enhancement bias, meaning that they tend to favor their own answers. It seems that an intuitive way to mitigate this bias would be to put less weight for a model’s evaluation of its own answer. Could the author explain why this was not incorporated in the reviewer weight vector in PR? On a higher-level, I am wondering which procedure(s) in PR counter the self-enhancement bias. \n2.\tConnection with Walsh (2014). I am confused with the convergence analysis of the updating weights in PR (first mentioned in Section 3.1.2). Authors cited Walsh (2014) as a reference which proved the “guarantee of convergence”, but the differences in settings and assumptions are not clearly discussed. There are some descriptions available in Appendix C, but they are not very detailed. I would suggest that author summarize the setting of Walsh (2014) in a more rigorous manner and explain why (or why not) the analysis in Walsh (2014) can extend to the setting studied in this paper. Moreover, it is claimed in Appendix C, bullet point 4 that the authors “extend Walsh’s proof”, but besides this sentence, such extension could not be found. Please do correct me if I missed anything. If such an extension was not available, I strongly suggest the author remove this claim. \n3.\tRelationship with prior works. In general, it is helpful if the authors could discuss how the proposed approach relates to prior works in more details, for example, the prior works in educational psychology research, and prior works which also consider having LLMs “interacting with each other through conversations like two communicative agents” (I do not particularly work on LLMs, so it is hard for me to see the relationship of this paper with the cited works). \n4.\tVariances. All the numerical results presented in this paper are fixed scores – should the variances across trials also be studied and reported?  \n5.\tSelection of models. Table 8 shows experiment results for PD, but the “contestant” model is GPT-3. I am curious to know why GPT-4 was not used as the answering model, given that it seems to be the more superior one from previous experiments. \n6.\tStatistical significance. In Table 6, I am confused about why there are two choices of p-values (0.05 and 0.005). \n7.\tExperiment results. In general, I think that some of the empirical results could be presented in a more rigorous and informative manner. For example, in the experiments of PD, it is unclear to me how many turns there are in the discussion. And whether the number of turns affect the accuracy of the evaluation outcome. Please do correct me if I missed this information. Also, it is reported on top of page 10 that “When we add to each turn’s prompt the role/identity information to remind the reviewer, the performance of GPT-4 leading discussions changes marginally, but Claude-leading PDA drops”. Without an explanation for this phenomenon, it is hard for readers to interpret a clear take-away message.", "broader_impact_concerns": "It is interesting to know that the peer discussion method improves the alignment with human preference in the LFQA dataset, for example, when two LLM reviewers have similar capabilities. However, I wonder whether the biases of LLMs, for example the social bias, would be amplified or reinforced by discussion with a fellow LLM with similar capability.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YBqNFqAvns", "reviewer": "Reviewer_Cu7D", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts**\n\n- The authors address an important and timely problem; LLMs are very often used as evaluators, sometimes without supplementing results with evaluation with human judgements, even though they have position and self-biases.\n- The authors proposed two methods that both improve over the most commonly used method (using GPT-4 as an evaluator)\n- The authors present some interestings results that reproduce existing results (position and self-bias) and extend existing knowledge (discussion leaders tend to stick more to their own opinion)\n\n**Weaknesses**\n\n- The authors propose two method that both require significantly more computational resources than the baseline, and do not discuss this limitation nor talk about to what extent this is mitigated in evaluations (e.g., often when this is the case, the baseline is given more compute by sampling multiple times and taking the average or something like that, to make a fairer comparison, e.g. see https://arxiv.org/abs/2402.06782). This leaves me with the question whether you could get similar gains from single-model evaluation by running it multiple times with different prompts and taking a majority vote.\n- It's unclear from the main text wether you use a separate set of LFQA questions to select the prompt for peer discussion from questions that you use for evaluation. If you don't, PD is unfairly advantaged through hyperparameter selecting on the test set.\n\n**Some questions**\n- Why are there no ranking results based on PD discussed (only accuracy)?\n- Does PR also mitigate position and self-bias?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes two novel methods for using LLMs as evaluators of open-ended questions. The authors show that their proposed methods are both more accurate on an example-level, and produce more accurate rankings of models than commonly used LLM-based evaluation methods (where more accurate means closer to human evaluation). They demonstrate this on two evaluation benchmarks, containing questions from different domains, like history, economics, math, email writing, etc, and using five different LLMs as evaluators (reviewers, in their terminology). The authors present a bunch of additional insights: their method correlates more with a ChatBot arena-based evaluation than GPT-4 evaluation, that their methods can mitigate self- and position bias by evaluators, and that the reviewer that is leading the discussion much more often sticks to its initial position than when its not leading the discussion.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Necessary for acceptance**\n- Add a limitations section that discusses things like computational costs\n- Clear up some confusions in the text about how you do prompt selection (on which part of the dataset, and whether it's held-out or not), and whether or not the ranking in Figure 5 right is based on different evaluation data than Figure 5 left and middle.\n\n**Would make the submission stronger**\n- Consider matching the inference cost of the baselines with the proposed methods for a fairer comparison\n- Some figures are hard to interpret and some results could benefit from using a different style visualisation, e.g. figure 5. \n- It would be useful to know what the inter-annotator agreement is, or to make it more comparable to the metrics you propose, what the correlation of a subset of humans is with another set of humans evaluations.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "I4jSwAnwNA", "reviewer": "Reviewer_RbP6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The introduction of multiple models as a panel is a quite reasonable idea. I believe this is a very useful topic to dig deeper.\n- The study of positional and model's self-favor bias is a good point that could benefit people running similar evaluations. Experiments showed that those biases are mitigated effectively.\n\nWeakness\n- The effectiveness of the added PageRank-style convergence is debatable: it brings the score closer to human, but does not change the ordering (which is more important for the purpose of comparison) and lacks solid significance analysis. Is the added complexity worth it in practice?\n- The effectiveness of Peer-Discussion (PDA) is not clear. The experiment (table 5) showed no raise in the ceiling of accuracy when compared with GPT-4 only.  The data in Table 6 is not very convincing without confidence intervals given the small size of the dataset. This is more concerning given the added cost of running more LLM inference.\n- Lack of statistical significance. As I said in the previous weakness, the size of the eval set is quite small (80 and 140), and given the inherent fuzziness of quality comparison of the open ended text results, it's very important to make sure we are seeing significant changes. A few percentage point change in such a small set is not automatically convincing.\n- Writing can be improved as some parts of the paper are not showing clear conclusions (see requested changes for details).\n\nNote on why I selected \"no\" on Claims And Evidence: it's because of the lack of statistical significance and the unclear conclusion of PD's effectiveness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studied the evaluation of LLMs by LLM themselves. Model-based automatic side-by-side evaluation is a common practice now, but they suffer from positional bias and models' implicit bias to favor their own results. The paper proposed a mixture of model-based reviewers approach, called  Peer Rank (PR) and Peer Discussion Accuracy (PDA). The key idea is to use multiple models as a panel of reviewers, either by combining their separately conducted, position randomized reviews (PR), or by allowing the models to engage in a conversation to convince each other (PDA). The experiments compare the closeness to human review results, which shows that introducing multiple models into the review process reduced positional bias and model's self-favorable bias, and resulted in a closer outcome as human reviewers.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Section 1, last sentence of the second-last paragraph: \"more fire ranking\" do you mean \"fair\"?\n- Section 3.1.1 first paragraph \"the win rate for a contestant is the [ratio] of wins for ..\" should it be \"number of\"?\n- Section 3.1.1 second paragraph. The assumption of better performers are also better judges is indeed an important one but I didn't see clear verification of that in the paper. This is an important missing part.\n- Need to prove that the PR iteration will converge, and at least a short version should appear in the main paper instead of the appendix.\n- Section 3.1.2 it's better to have a short intro of Elo (i.e. what it means) and why we need to see both win rate and elo in our experiments (i.e. what each other can't cover).\n- Figure 2: always have some way more than the color to identify things (color blindness, non-color printer users etc). This applies to later charts too. Thanks!\n- Section 3.1.2 last sentence of paragraph 2: \"Detailed differences between ...\" at least we should have a short summary of the difference, i.e. the reading of the appendix should be optional.\n- Table 2: we used the term of \"unsupported information\", \"core information\" and \"coherence\" without further explanation in the prompt. It's solely to the model's interpretation of those concepts where even human could have different interpretations. Have we tested the model's understanding, or tried to give more concrete definition?\n- About PDA: how do we decide which side win from the discussion history? What if they just hold each other's option without reaching consensus? This detail is missed.\n- Section 4.1.3: As far as I know, Bard (the web-based chat tool) and PaLM-2 (the model) is not the same, the web-based services of different companies update frequently. it's important to use the more specific name (maybe PaLM-2?) and give the specific access point used (through Google Cloud? Which model? Or through any arrangement with Google for some direct access? Or just scraping the Web-UI of Bard? when?).\n- Section 4.1.2: it's better to give a quick explanation of Fleiss' Kappa, what it measures and why it matters to this experiment.\n- Section 4.2 third paragraph: \"we clearly observe that GPT-4 clearly favors...\" duplicative \"clearly\"?\n- Section 4.3. Overall it's not clear to me if PD is effective and what data supports it. Suggest some more revisions and proof-reading. For example, it's not clear why Table 5 has the two \"initial score\" rows and if this table provide any evidence to say if PD is useful. (from the face value it seems to be no?) I will not go into too much detail into this section.\n\nNice to have\n- Theoretical and practical analysis of the convergence of Peer Rank. PageRank's convergence condition has been thoroughly studied. For any similar random-walk algorithm, it's important to show whether and when it will converge.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After three thorough reviews, discussions with authors, and appropriate revisions, we conclude that the manuscript supports all claims with suitable evidence. The topic is undoubtedly of interest to the community.
Decision: Accept (poster)