PiCO: Peer Review in LLMs based on Consistency Optimization
Authors: Kun-Peng Ning
Shuo Yang
Yuyang Liu
Jia-Yu Yao
Zhenhui Liu
Yonghong Tian
Yibing Song
Li Yuan
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
Unsupervised Evaluation
Peer Review
Consistency Optimization
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Existing large language models (LLMs) evaluation methods typically focus on testing the performance on some closed-environment and domain-specific benchmarks with human annotations. In this paper, we explore a novel unsupervised evaluation direction, utilizing peer-review mechanisms to measure LLMs automatically without any human feedback. In this setting, both open-source and closed-source LLMs lie in the same environment, capable of answering unlabeled questions and evaluating each other, where each LLM’s response score is jointly determined by other anonymous ones.  During this process, we found that those answers that are more recognized by other ``reviewers'' (models) usually come from LLMs with stronger abilities, while these models can also evaluate others' answers more accurately.  We formalize it as a consistency assumption, i.e., the ability and score of the model usually have consistency.  We exploit this to optimize each model's confidence, thereby re-ranking the LLMs to be closer to human rankings. We perform experiments on multiple datasets with standard rank-based metrics, validating the effectiveness of the proposed approach.

Published as a conference paper at ICLR 2025
PICO: PEER REVIEW IN LLMs BASED ON CONSIS- TENCY OPTIMIZATION
Kun-Peng Ning', Shuo Yang', Yu-Yang J Yu Yao', Zhen-Hui Liu', Yong-Hong Tian Yibing Song, Li Yuan
Sccoool of Electrical and Computer Engineering, Peking University
PPeng Cheng Laboratory
cn, [liuyuyang13 3, jiayu_yao, yhtian, yuanli- -enei-ecee ēpku edu. en, yibingsong ev@gmail..cm com
(ningkp, Schuu_yandee
ABSTRACT
Existing large language models (LLMs) evaluation methods typically focus on testing the performance on some closed-enviromment and domain-specific bench- marks with human annotations. In this paper, we explore a novel unsupervised evaluation direction, utilizing peer-review mechanisms to measure LLMs au- tomatically without any human feedback. In this setting, both open-source and closed-source LLMs lie in the same environment, capable of answering unlabeled questions and evaluating each other, where each LLMs response score is jointly determined by other anonymous ones. During this process, found that those answers that are more recognized by other "reviewers" (models) usually come from LLMs with stronger abilities, while these models also evaluate others' answers more accurately. We formalize as a consistency assumption, i.e., the ability and score of the model usually have consistency. We exploit this to opti- mize each model's confidence, thereby re-ranking the LLMs to be closer to human rankings. We perform experiments on multiple datasets with standard based metrics, validating the effectiveness of the proposed approach. Our code is re- leased at htttps://qitiii wwtp/o//////oaaae 2ttmm//KK-uuaanrrop///PCCC
INTRODUCTION
Goodhart's Law: "When a measure becomes a target, it ceases to be a good
measure."
Large language models (LLMs) (Brown el all 2020 Achiam et all 2023 Bubeck et all 202K3
Touvron all 2024: Ning et all 2025) have achieved remarkable success across a variety of real- Lu Lw all 2023: Duyang et all, 2T2e; atll, 202: Ning stall 2024: Yang et 2024). With the increasingly widespread application of these models, there an urgent need for an effective evaluation method to ensure that their performance and usability meet the growing demands. To assess the ability level of L.Mss, large number of evaluation benchmarks have been proposed by using some small and domain-specifte datasets with human-curated labels, such as MMLU (Hendrycks et al, 2020), HELM (Liang et all 2022), Big-Bench (Srivastava el all 2022). GLUE (Wang et all 2018). However, these benchmarks can only measure LLMs' capability on a confined set tasks (e.g. multi-choice knowledge or retrieval questions). which fails to assess their alignment with human preference in open-ended tasks adequately (Chiang et all 2023: Lll 2023a; Nakano et all, 2021). On the other hand, these evaluations may suffer from benchmark leakage issue, referring that the evaluation data is unknowingly used for model training, which can also lead to misleading evaluations eall 2023 Zhou ct all. (02Z), Therefore, blindly improving scores on these public benchmarks cannot always yield a large language model that truly satisfies human requirements.
world applications (Zhao et all 2023:
For assessing human preferences, recent studies have focused on building crowdsourced battle plat- forms with human ratings as the primary evaluation metric. Typical platforms include Chatbot Arena
"Coresponding author.

---

Published as a conference paper at ICLR 2025
Peer Review in Large Language Models (LLMs)
Consistency Assumption: Higher-level LLM cun evoluate more accurutely than low-level ones, while achieving higher score.
Consistency Optimization
Avertion Cumpuse pit ing Bky past
LLAMA
Entropy Reduction
RANKING
Howal.
GPT3,5
plosura anborking an tris
Poer Review
Vicuna
the tsxufall....
Liama
Wesae A ata Howail Rezartly
1 eotem el wirinng
VICUNA
We agree!
trand past about reseet tipp tavel Hovail
VOUUMA
The framework of PiCO. this framework, both open-source and closed-source LLMs lie the same environment, capable of answering unlabeled questions and evaluating each other, where each LLM's response score is jointly determined by other anonymous ones. We assign each LLM learnable capability weight to optimize the score ranking based the consistency assumption, while reducing the entropy of the peer-review evaluation system. The goal is to find final score ranking that all LLMs "agree"
Figure I:
(Zheng et all 2023), .T- Bench (Zheng et all (023), and AlpacaEval et 20236) constructs anonymous battles between chatbots in real-world scenarios. where users engage in conversations with two chatbots at the same time and rate their responses based on personal preferences. While human evaluation is the gold standard for measuring human preferences, it is exceptionally slow and costly (Zheng et all, 2023: Ning et all, 2022).. In addition, adding a new LLM to the crowdsourced battle platforms also poses cold-start issue (Chang et all 2023). Thus, fundamental question arises: can we construct an unsupervised LLMs evaluation system without relying on any human feedback?
Actually, in real human evaluation systems, people build the human abiiity hierarchy based on dif- ferent empirical assumptions. For example, majority voting (Feldmani, 2006 Boyer & Moore, 1991; Surowiecki, 2005)) and rating voting (Allahbakhsh & Ignjatovic, 2012) methods are widely used dur- ing the decision-makin process, which are based on the wisdom of the crowds (Surowiecki, 2005: Budescu & Chen, 20SS Wellen, 2007) and have been proven to lead to better results than that of an individual. Moreover, in the established practice of peer-review in academic research, scholars eval- uate their academic level rankings based on the consistency assumption, i.e., scholars with stronger abilities usually have stronger persuasiveness for evaluating others, and these scholars can also ob- tain higher achievements. This paper attempts to explore whether similar phenomenon exists in the LLMs evaluation systems.
In this paper, propose PiCO, a Peer review approach in LLMs based on Consistency Optimization. In this setting, LLMS themselves act as "reviewers". engaging in mutual assessments achieve comprehensive, efficient, and performance evaluations without relying on manually an- notated data. This method aims to address the limitations of existing evaluation approaches and provide insights into LLMs' real-world capabilities. As shown in Figure II, both open-source and closed-source LLMs lie in the same environment and answer the open-ended questions from an un- labeled dataset. Then, we construct anonymous answer pairs, while randomly selecting other LLMs as "reviewers" to evaluate both responses with learnable confidence weight w. Finally, we employ this weight and calculate the response scores G for each L.LM based on the weighted joint evaluation. It is worth noting that the whole peer-review process works in unsupervised way, and our goal is to optimize the confidence weights w that re-rank the LLMs to closer to human rankings.
To achieve this, we formalize it as constrained optimization based the consistency assumption. We maximize the consistency of each LLM's capability w and score G while adjusting the final ranking align with human preference more closely. The key assumption behind this is that high- level LLM can evaluate others' answers more accurately (confidence) than low-level ones, while higher-level LLM can also achieve higher answer-ranking scores. As result, the entropy (contro-
2

---

Published as a conference paper at ICLR 2025
LLMs Paol M
Beer-Bavier
LLke lasderbaard
Medel
Dood and Uoa
VS
Mr
6F135
We agree!
Handowely Sumpling Mo-Mf the porr-resiznee.
Viuun
Ranking worh
Lama
Reviewse
M₁
Canoistercy Optinscenia
(A/LA) (,w")
argmax Com Complstencyy (G.
Stugfting
ΣΣ
Σ cARTR!
1/4 J.
Unlabeled dataset
LLM's response A
Answer Answer-Ranking data TD
Oprioiiaation
Figure The pipeline of the PICO. mainly composed two components: peer-review and consistency optimization stages. Specifically, in the peer-review stage, the unlabeled dataset Q and the LLMs pool A are given. Then, we let all LLMs answer cach unlabeled question to obtain the response set A. We shuffle the set and construct anonymous answer pairs, while randomly selecting other LLMs to evaluate both responses with learnable confidence As result, the partial order between two answers and evaluator's 's confidence weight. In the consistency optimization stage, we update the parameter w by maximizing the consistency of each LLM's capability and score, while re-ranking the LLMs to be closer to human rankings.
obtain the answer-ranking data which is quadruple that records
versy) of the whole peer-review evaluation system can be minimized. In other words, the consistency optimization aims find final score ranking that all LLMs have "disputes" regarding.
We perform experiments on multiple crowdsourcing datasets with standard rank-based metrics, the results demonstrate that the proposed PiCO framework can effectively obtain a large language mod- els' leaderboard closer to human preferences. The contributions of this paper can be summarized as follows:
We explore novel unsupervised LLM evaluation direction without human feedback, iee. utilizing peer-review mechanisms to measure LLMs automatically. All LLMs can answer unlabeled questions and evaluate each other.
A constrained optimization based on the consistency assumption is proposed re-rank the
LLMs to be closer to human rankings.
We conduct extensive experiments three crowdsourcing datasets with three standard rank-based metrics validating the effectiveness proposed PiCO approach.
THE PROPOSED APPROACH
2.1 PROBLEM DEFINITION
This paper aims re-rank the ability of LLMs be closer to human (ground-truth) rankings R* an unsupervised way (without relying on any human annotations). Specifically, we have a large language models (LLMs) pool M = (M, )mm which includes both open-source and closed-source models. Write M1 > M2 to indicate that the LLM M, has stronger capabilities than the LLM My. Thus, can assume that the ground-truth ranking R is as follows,
R [M] Ma M3 ... Mn).
Assuming that the learned ranking R. by different evaluation methods is as follows,
R. = [Mg > M1 x M2 > ...
Mn].
(2)
The goal is to leam LLM ranking R. that aligns with human ranking R* much possible.
2.2 ALGORITIM DETAILS
The pipeline of the proposed PiCO, depicted in Figure A involves peer-review and consistency optimization stages. Next, we will introduce the stages in detail.
Peer Review Stage our peer-review system, we consider an unsupervised LLM evaluation see- nario with an unlabeled dataset Q consisting of open-ended questions, where Q (Q:)2 All
3

---

Published as a conference paper at ICLR 2025
Table 1:
Validation of consistency assumption. Performance comparison of Backward, Uniform, Forward
weight voting, and Consistency Optimization methods with two metrics across three datasets.
MT-Bench Chatbot Chatboo Arena AlpacaEval
Methods
S(t) r(1) S(1)
r(t)
S(()) r(t)
Backward Weight
0.70 0.50 0.72
0.52
0.69 0.50
Uniform Weight
0.74 0.54 0.80
0.58
0.77 0.58
Forward Weight
0.75 0.56 0.82
0.59
0.79 0.60
Random Weight + Consistency Optimization 0.90 0.77 0.89
0.72
0.84
0.68
LLMs will answer each unlabeled question to obtain the set A = {(4()=))) where A is as follows,
A = M2(Q.)
(3)
which infers the model Mj response answer A& with question Q, In addition, LLMs themselves also act as "reviewers" to evaluate other answers. Specifically, for the same question Q, E Q, we randomly construct a battle pair < A,,, A > for review. Each battle pair will randomly assign "reviewers" to determine the winners or declare ties,
A7, w²) = A41:A)))).
Under the same question Qi the quadruples (A AQ, >, w²) indicate that the "reviewer" Mj be- lieves that the answer A4 is better than answer A with confidence w³. Thus, we can collect the answer-ranking data D as follows,
D
A3. w²) -.
(5)
where denotes the question index, and j, k,s indicate the model indices. w E (0, 1] is learnable confidence weight model M,, and > a partial order relationship from {>,<.=}, After that, we can calculate the response score G, of each LLM,
Σ Gj (A
1( > A&) w",
(6)
where {.} is the indicator function that the value is 1 when the condition is met, otherwise, it is 0. We can define the LLM M, is better than M2 as its score is larger, i.e. Mi > M2 = Gi > G2. Thus, we can re-write the learned LLM ranking R as follows,
: [G3 G1 > G2 >
Gm).
Thus, the goal is to learn the confidence weights w to adjust the final ranking R. be closer to ground-truth ranking R
Validation of Consistency Assumption. First of all, we start with toy experiment to study the role of confidence w in Table MI Specifically, we manually construct three methods: Backward Weight, Uniform Weight, and Forward Weight. That is, the ability weights of the model are re- spectively weighted forward (w = [1, 0.9, ..., 0.......... O)), uniformly (w [1, 1,.,, 1),, and backward (w = 0,,00.1.... IJ) according to the ground-truth human ranking. In other words, the Forward Weight means manually assigning higher weights those models with stronger abilitics, and so on for others. Then, we can calculate the response score G, for each model using Eq and obtain the LLM ranking R. We measure the aligmment between R and R* with Spearman's S(1) and Kendall's r(f) rank correlation coefficient in Table II Note that this an ideal experiment, as we only use the ground-truth human ranking to validate the feasibility of our idea.
As shown in Table II can be observed that the Forward Weight achieves better results than the Uniform and Backward ones in all cases, while the Backward one always achieves worse results. validates that assigning larger weights to those models with stronger capabilities can obtain better results. In other words, those answers that are more recognized by other "reviewers" (models) usually come from LLs with stronger abilities. We formalize it as a consistency assumption, i.e. high-level LLM can evaluate others' answers more accurately (confidence) than low-level ones, while higher-level LLM can also achieve higher answer-ranking scores, the ability and score of the model usually have consistency.
4

---

Published as a conference paper at ICLR 2025
Consistency Optimization Stage. Based on this observation, we propose to maximize the consis- tency each LLM's capability w and score G with constrained optimization as follows, argmax Consistency(G,,
(8)
s.t. G==
Σ
1( > AN .
where the Pearson correlation (Sedgwick, 2012) used to measure the consistency between w and
G. Note that only introduce this straightforward implementation validate our idea of PiCO. Other more advanced strategies may be employed to further improve the performance.
Discussion: It is worth noting that the whole process (Eq. 5l and (S) works in unsupervised way. The only thing we do is to adaptively adjust the score of each LLM that match its abilitics. Most importantly we also validate the effectiveness of the proposed consistency optimization in Table Specifically, we randomly initialize the ability weights and employ our consistency optimization to adjust the weight. can observed that the learned by our consistency optimization algorithm (Eq.8) can further improve the performance of the evaluation system, making the LLM ranking R closer to human ranking R'. Another intuitive example is as follows: in real peer-review system, the academic level of three scholars a, b, and satisfies the following relationship, w > w² >> w". So, in the ultimate ideal scenario, the ranking of the scores submitted by these three scholars should also be, G. > G > Ge- In other words, the sorting of G and w satisfies high consistency. On the other hand, scholars with stronger abilities (i.e., scholar a) evaluate A > A have stronger persuasiveness, so scholar b should also receive higher weighted scores 1 * w
Reviewer Elimination Mechanism. Realizing that not all LLMs have sufficient ability evalu- ate the responses of other models. We thus introduce an unsupervised elimination mechanism to remove those LLMs that have low scores. It iteratively removes the lowest-scoring LLM from the "reviewer queue" for the next consistency optimization stage, until 60% of models are eliminated. The discussion of the elimination mechanism can also be found in the Experiment [3,
EXPERIMENTS
Datasets. To validate the effectiveness of the proposed approach, we perform experiments on Chat- bot Arena (Zheng et all 2023), MTT-Bench (Zheng et all, 2023), and AlpacaEval eLl 20236). The MT-Bench dataset assesses six LLMs' responses to 80 multi-category questions. The Chat- bot Arena Conversations Dataset, with 33K conversations from 13K IPs during April-June 2023, evaluates real dialogue performance. AlpacaEval dataset integrates 805 evaluations from diverse tests (e.g., Self-Instrud (Wang et all 2022). OASST, Anthropics helpful (Bar al 2022). Vicuna (Chang et all, 2023) and Koala (Geng al. (IZAR) test sets) to align evaluations real-world inter- actions (Duhois et all, 2023). These datasets are collected by crowdsourcing platforms from human feedback, they have ground-truth ranking L.MSs R to measure the alignment performance of different evaluation methods.
LLMs Pool. In our experiments, we employ 15 LLMs with diverse architectures construct the LLMs pool, including --Trb (OpenAT, 2022), WizardLM-13B (Xu all 20233.. Guanaco- 33B (gua, 2023). Vicuna-7B (Chiang all 2023), Vicuna-13B (Chiang all 2023), Koala-13B
(Geng et all 20235), Mpt-7B (Heam, (023), gpt4all- (Anand et all 2023),, ChatGLM-6B (Zeng
al. 2022). Dasst-sft-4-pythia -ic (Contmbutors 2023). FastChat- -T-BBB (Zheng et all 2023).
StableI.M-75 (AI, 2023),, Dolly-12B (Conover et all, 2023),, ILaMA-13B (Touvron et all, 2(02)).
Alpaca-133 13B (Taori et all 2023). All models use the same prompt template, which can be found in
Appendix 0
Baselines. To validate the effectiveness of the proposed PiCO approach, we compare the following methods in the experiments.
The wisdom of the crowds: The two methods that perform LLMs evaluation based on the wisdom of the crowds (Surowiecki, 20XΣ; Budescu & Chen. 2015 Weller, 2007) are com- pared in this experiment. 1) Majority Voting (Surowiecki, 2005): Multiple review models vote for the better answer for the same response pair, and the model with the most votes gets score; 2) Rating Voting (Allahbakhsh & Ignjatovic, 2012): Multiple review models also vote on the same response pair, and the number of votes obtained is the score.
5

---

Published as a conference paper at ICLR 2025
Table Comparison of all methods on three datasets under data volumes of 1, 0.7, and 0.4, where the top value highlighted by bold font. Higher and 7 scores indicate better performance, while a lower H score signifies improved performance.
Datasets
Chathot Arena
MTT-Bench
AlpacaEvvl 0.7
Methods
0.7
0.7
0.4
0.4
Spearman's 's Rank Coreelation Coefficicnt S(t)
Majority Voting
0.76
0,75
0.73
0.73=0.0
0.77=001
0.75=000
0.80
0.79
0 0,78
Rating Voring
0.74200.0
0.7220.00
0.71
0,80=000
0.78=042
0.74
0.77:50.00
0.7729.01
0.78
GiPTScore (5-xxl)
-00gg
-0.12
0.05
0.01 -001-
0.34
0.34
0.34
GPPTScnreddavinni-0n77
0.15
0.13
-0.220014
0.52=0.00
0.42=000
0.45=000
0.76440.0
0.77
0.75
P
0.43+0.00
0.44
0.44$0.10
0.50=0.00
0.50=0.0
0.52=0.17
0.57#9.00
0.55#0.01
0.48*0.08
PRI
0.84
0.84
0.82#0.00
0.86=0.00
0.84=0.03
0.81=0.03
0.81 0.81#90.00
0.81
0.81#0.02
0.86
0.86
0.86
0.86=0.00
0.84=0.0
0.82=0004
0.83+40.00
0.81=0001
0.8320.022
Claude-3 (API)
0.90
0.80
0.8740.04
0,85=0.0s
0.82=0.0
0.80=0007
0.79=20.03
0.78
0.754
PiCO (Ours)
0.90
0.8g
0.80=
0.89
0.89
0.84
0.8.4
0.83
0.85
Kendall's Rank Correlation Coefficient (t)
Majority Voting
0.38
0.56++0.02
0.52
0.56=0.00
0.61=0.02
0.60
0:62-+0.00
0.60#0002
0.58+00.02
Rating Voting
0.50
0.53
0.52
0.58=0.00
0,57=0.02
0.51=0.01
0.58:+0.00
0.57
0.57
GPTScore flan-t5-xx))
0.00
0000
0.00
0,005
-0.07
-0,02=000
0.25
0,26
0,26
GPTScore(davinciooo 002)
0.20=0.00
0.23#0.02
0.03+0.11
0.36=0.00
0.30=0.s
0.31=0.05
0.60=
0.61#0005
0.59
PandaL.M
0.3024.00
0.31
0.31
0.39=0.00
0.37=006
0.40=0012
0.41
0.3920002
0.32:40.0
PRD
0.68
0.6g
0.67
0.68=0.05
0.66=0.02
0.63=0.03
0.64+
0.63#0000
0.63
PRE
0.71
0.73+0.02
0.72
0.68=0.00
0.68=0.02
0.65=0..s
0.64
0.66#0.01
0.66
Claude-3 (API)
0. 0.7000.04
0. 0.7250
0.67=0.07
0.66=0.11
0.61 =0.10
0.64#*0.06
0.61 #0.04
0.66
PiCO (Ours)
0.77=0.00
0.76
0.77=0.02
0.72
0.7250.05
0.70
0.68::0.00
0.60
0.67
Permutation Entropy H()
Majority Voting
1.27
1.30
1.36#0000
137=000
130=0.00
1,27:0.04
1.26:00.02
1.2g
1
Rating Voting
1.3920.02
1.4320.00
1.4220.07
1332000
1.35-001
1.380004
1.34
1.3720.08
1.34.t0.08
GPTScore (3-xx-
168g0m
1.68
165
L72-002
1.70
LG8 L6g
L55
1552
1.60200r
GiTSecoredavvinci 002)
1.54
1.64
1.6840.00
151=
1.61=001
1.6
1,254002
1.23
1.2644014
PandaLM
165+0001
1.6420.02
1.63+0.00
L.550003
1559000
1.52=00s
1.5640.01
1.58
1.6g#0.00
PRD
1.150004
1.12#0.06
1.13+0.00
1.15=0.0
1.17000s
1.23=004
1.21
1.22=0.06
1.23*0.07
PRE
1.07.0.01
1.03
1.06
1,170004
L11=0005
1.190005
L.1g20.0
1.21
1.15-40.0
PICO (Ours)
0.94=0.02
0.96=0.04
0.95==0.8
1.01
1.02
1.06
1.17#0.02
1.17#0.08
1.13=0.05
State-of-the-arl methody: The four recent SOTA methods of using either single or multi- ple models for self-evaluation are compared in this experiment. PandaLM (Wang et all 2023)): It is a fine-tuned language model based on Llama-7b designed for the preference judgment tasks to evaluate and optimize LLMs. GPTScore (Fu all 2023): employs generative pre-trained models to assess the quality of generated text. It calculates the likeli- hood that the text was generated in response to specific instructions and context, indicative of high quality. In our implementation, GPT-3 (davinci-002) and flan-t5-xxl serve as the base models. PRD (Li all 20233); It transforms the LLMs win rates into weights for competitive ranking, while evaluating each LLM based on its preference for all possible pairs of answers, enabling a tournament-style ranking system. PRE (Chu et all 2024): It employs supervised process to evaluate L.MS using qualification exam, aggregates their scores based on accuracy, and assigns weights accordingly. Claude-3 (API): Another SOTA closed-source LLM developed by Anthropic. PiCO (Ours): the proposed approach in this paper.
Metrics. For all experiments, we employ three popular rank-based metrics to evaluate the afore- mentioned experimental setups and our PiCO method: Spearman's Rank Correlation Coefficient S(t) (Lehmman et all, 2013), Kendall's Rank Correlation Coefficient r(t) (Kendall, (1338) and Permutation Entropy (()) (Bandt & Pompe, 2002). The details of these metrics can be found in the Appendix AL Moreover, we perform the experiments for 4 runs and record the average results over 4 seeds (seed- = 3,4).
3.1 PERFORMANCE COMPARISON
We validate the effectiveness of the proposed PiCO method on three datasets by comparing the following two types of methods, i.e., the wisdom of the crowds and recent SOTA LLMs evaluation methods. The average results with different rank-based metrics and datasets are demonstrated in Table [1 The ratios of response sets D are 1, 0.7, and 0.4, respectively.

---

Published as a conference paper at ICLR 2025
642
300
200
+0.77
6060
18.33
104
+2.00
0.12
4:00
300
4.32
1:00 009
2.60
4003
+4.00
000
591
5.14
5:00
314
201
6:0
5.50
322
6:27
631
6377
Chagee4
321
2.1
6:3
0.33
(33
024
6:1
1.24
2020
02:
2:2
033
3.16
(a) ChatBot Arena (PG)
(b) MT-Bench (PG)
(c) AlpacaEval (PG)
3.1
40:0
+8.10
OCC +000
GEL
4000
OCB
0.10
0:22
2:91 6:23
180
8:2
8:07 6C2 1102
0.0
0:03
0.34 000
1:34
0.3
0.03
4.34
0.0
6:t1
0.07
-.30
033
0.33
302
0.3
ace 0.9
0C5
1:00
0.0
3.06
0.44
304
0.44
3000
Gawer
(d) ChatBot Arena (weighted PG)
(c) MT-Bench (weighted PG)
(f) AlpacaEval (weighted PG)
Figure 3: Hcatmap distribution of preference gap (PG) metric among seven LLMs across three datasets. Higher values (above 0) indicate greater evaluation bias. The first row shows original PG values in three datasets, while the second row displays PG values re-weighted using our learned confidence weights.
The results presented in Table D demonstrate that the proposed PiCO method consistently outper- forms competing approaches across most evaluated metrics, including surpassing all baselines, such as Claude-3 (API). Specifically, PiCO achieves improvements of 0.027, 0.047, and 0. on Spear- man's Rank Correlation Coefficient, Kendall's Rank Correlation Coefficient, and Permutation En- tropy metrics, respectively, compared to the runner-up. These results underscore the superiority of aggregating evaluations from multiple models, such Majority Voting, Rating Voting, PRD, and PRE, as opposed to relying solely on single-model methods like GPTScore and PandaLM. This col- lective model approach, leveraging 'the wisdom of the crowds', aligns with human rankings more accurately in our open-question evaluation framework.
In comparison with existing SOTA evaluation methods(i.e. PRD and PRE), is evident that PiCO exhibits improvements across various evaluation metrics. Despite PRD's adjustment of model weights based on their win rates and PRE's reliance on supervised human feedback data to assign weights through qualification exam, neither method achieves performance superior to the fully unsupervised PiCO approach. These methods rely on predefined criteria and human feedback, po- tentially leading to biases or suboptimal performance. In contrast, PiCO leverages unsupervised learning techniques, allowing it to autonomously adapt and discover patterns in the data without explicit human intervention.
is important to highlight that PandaLM, a language model equipped with 7 billion parameters, was fine-tuned using labels generated by GPT-3.5-turbo as the ground truth, achieving stable per- formance across various datasets. However, in our unsupervised, open-ended experimental setup, which focuses on ranking- metries, GPTScore exhibits less robustness regardless of whether the base model is GPT-3 (davinci-002) or flan-15-xx.
3.2 EXPLORING THE ROLE OF CONFIDENCE WEIGHT
In this subsection, we show that the confidence weight w learned by our consistency optimization can reduce the system evaluation bias. Specifically, we first study whether the "review" model would prefer a particular model's response. Following (Chu all 2024), we employ the preference gap (PG) to evaluate the bias as follows,
PG(i,) P.(i j)- P3 1),
7

---

Published as a conference paper at ICLR 2025
PCCOOOurs
PRE
Chatbot Arena
MT-Bench
AlpacaEval
1.20
1.30
1.19
1.20
1.18
@ 1.15
2 1.17
1.05
1.00
1.05
1.15
0.95
1.00
114
Number Eliminated Reviewers
Number of Eliminnted Reviewers
Number of Eliminated Reviewers
Figure 4: Performance comparison of the PiCO (Ours) and PRE methods on the Chatbot Arena, MT-Bench, and AlpacaEval datasets, with the number of eliminated reviewers on the x-axis. The y-axis is PEN, where lower values indicate better performance.
where (i > j) represents the winning rate of model as the "reviewer" believes that i defeated The heatmap distribution of the PG value PG(i.j) among seven LLMs across three datasets is demonstrated in the first row of Figure B. can observed that the evaluation system exhibits severe bias. Especially on ChatGLM-6B and Mpt-7B models, they often believe that their results are better than other ones, as their PG values are greater than across three datasets.
After the consistency optimization, we assign the learned confidence weight w to the corresponding model and ultimately obtain the re- re-weighting PG value PG((,j as follows,
PG((,j) = w, x P,(i > 5) - w, x (s(i
(10)
The results of the re-weighting PG value (i are displayed on the second row of Figure B. can observed that the learned confidence weight w can significantly mitigate the preference gaps of the whole evaluation system. In our consistency optimization, LLMs such as ChatGL and Mpt-7B have lower weights, and reducing their confidence can effectively alleviate the system evaluation bias.
3.3 STUDY OF ELIMINATION MECHANISM
Performance Comparison of Elimination Mechanisms. The PiCO and PRE methods both employ climination mech- anisms to remove those weakest LLMs from the "reviewer queue" during the evaluation process. As shown in Figure g. the x-axis quantifies the number of reviewers eliminated, and the y-axis measures the PEN, where lower scores denote higher performance. It can be observed that both PiCO and PRE exhibit better performance with an increasing number of eliminated "reviewers". The proposed PiCO approach can achieve better performance than PRE in most cases. is worth noting that the PRE method employs the accuracy of "qualifi- cation exams" to eliminate weak L.LMs, and this process re- quires human annotation (Chu the elimination process of our PiCO method is unsupervised and can still achieve better evaluation results than PRE.
1.2
Apacifaa
1.0
3 0.6
0.4
2
8
10
Number of El minated Reviewers
( On the contrary,
Figure 5: The average loss for different numhers of climinated reviewers(.). It shows how the iterative climination of weaker reviewers affects the overall loss in the peer-review system.
Automatic Learning of Elimination Thresholds. We ob- served that weaker LLMs tend to have poorer evaluation abili- ties, introducing significant noise into the peer-review system. Therefore, eliminating weaker mod- els instead of retaining them enhances the robustness of the system. We employed an unsupervised approach to automatically learn the elimination threshold, as shown in Figure E. by using the average training loss curve as the number eliminated reviewers increases. It can be seen that removing weaker reviewers reduces the average loss of the entire system, indicating that eliminating noisy evaluations benefits the overall process. Notably, when 60% (or 9) of the weaker reviewers are removed, the system's 's loss reaches its minimum. This trend is consistent across all three datasets, suggesting that the climination threshold is learned automatically. However, removing more than stronger reviewers harms the evaluation process.
8

---

Published as a conference paper at ICLR 2025
Table 3: Comparison of more metrics (Precision @ and RBP@K) and token consumption Chatbot Arena.
RBP@K (T)
Precision@ K (*)
Methods
Input Token Output Token Annotation Cost
8
9
10
8
9
10
Chathot Arena Platforms
7500k
10944k
32k
GPTScoree(fan-55xx))
26.2%
29.6%
45.1%
50.0%
55.6%
70.0%
22882k
-12200k
GPTScore(davinci-000)
42.0%
50.6% 53.3%
62.5%
77.8%
80.0%
22882k
-1220K
PandaLM
63.5%
63.5%
2%
62.5%
55.6%
60.0%
22882k
~ 10355k
PRD
67.2%
73.8% 81.3%
87.5%
88.9%
80.0%
25087k
10935k
PRE
78.0%
81.3%
81.3%
87.5%
88 98.9%
80.0%
20k
-1115k
7k
PICO (Ours)
883.2% % 83.2% 2.. 85.9
100.0% % 100.0% 90.0%
23823k
~ 11685k
Loss variacion curve with or bars
0.8
Chatbct Arena
1.4
0.6
0.5
0.8
0.4
0.9
0.2
0.1
0.2
0.0
3.5
Guanaco- 320
Vicuna 1.0
WicaraMM: 130
Vicuma 70
Kodio 1338
Gotdal--330
dass pytmo
Mpt-78
Alpaca- 130
Fustchar 15-30
Chatgim
Stablet M478
Daily
Liame
(b)
Figure 6: Stability validation of consistency optimization. We repeated the experiment with different seeds for 1000 times, and plotted the training loss curve and weight distribution. The results show that the learning process is stable and the learned is convergence.
3.4 OTHER RESULTS
Validation on more metrics (Precision@e @K and R K). We demonstrated the results of preci- sion and RBP (K=8,9,10) with other baselines in Table B (left). The results show that the proposed PiCO approach achieve better precision and RBP performance in all cases. These results once again validate that PiCO can predict the L.MM ranking more accurately than other baselines.
Comparison of tokens consumed. We compute the token consumption of each method in Table (right). It can be observed that the proposed PiCO approach has a similar token consumed with other baselines (e.g., PRD and PRE) while achieving better evaluation performance. Although Chatbot Arena has a smaller token consumption, it requires 33k human annotations, while PiCO does not require any human annotations.
Stability validation of consistency optimization. We repeated the experiment with different seeds for 1000 times, and plotted the training loss curve and weight distribution in Figure R The results show that the proposed consistency optimization process is stable and the learned w is convergence.
Comparing with existing benchmarks. We select the widely-used benchmarks (i.e., MMLU (Hendrycks et all (02T) and GSM8K (Cobhe et all (0211) to evaluate the model performance ranking R. and calculate the Spearman's 's S(t) and Kendall's r(t) rank correlation with the hu- man preference ranking R* The results are demonstrated in Table !l can be observed that these benchmarks can only measure LLMs specific capability assess their alignment with human preference. These phenomena have been widely validated in other literature (Zhou et all, 2123) Zheng et al], 2023: Chang et al] (023) and have almost become a consensus in the community of LLM evaluation.
confined set of tasks, which fails to
4 RELATED WORK
Evaluation Benchmarks for Diversity. LLMs are designed to handle a variety of tasks, necessitat- ing comprehensive benchmarks (Chang et all (023). Notable benchmarks include GLUE and Su-

---

Published as a conference paper at ICLR 2025
Table 4: Comparison with existing benchmarks.
Spearman's 's Rank
Kendall's Rank
Benchmarks
Correlation Coefficient S(t) Correlation Coefficient (t))
MMLU GSM8K
0.53
0.37
0.32
0.15
PICO (Ours)
0.88
0.67
perGLUE (Wang et al, 2018: 2019), which simulate real-world scenarios across tasks such as text classification, translation, reading comprehension, and dialogue generation. HELM (Liang et all 2022) provides holistic evaluation of LLMs, assessing language understanding, generation, coher- ence, and reasoning. BIG-bench (Srivastava ef all 21122) pushes LLM capabilities with 204 diverse tasks. MMLU (Hendrycks et all 2020 measures multitask accuracy across domains like mathe- matics and law. GSM8K (Cobbe et all 202T) including 8K simple math questions with detailed solutions is widely used to assess the mathematical reasoning of models on grade-school-level ques- tions, However, these evaluations can be compromised by benchmark leakage, where evaluation data inadvertently used for training leads to inflated performance metrics (Aryappa et al,, (023)..
Human Evaluation. Human evaluation provides reliable feedback that closely aligns with real- world applications (Chang all MT23). Liang et al. (21122) evaluated summary and misinformation scenarios across multiple models. Ziems et al (2(22) involved experts to assess model outputs in var- ious domain-specing specific tasks. |Bang et all (2023) examined ChatGPT's performance in summarization, translation, and reasoning using forms like Chatbot Arena (Zheng et ar, (02%), relying on human ratings as the primary evaluation metric. Currently, using these anonymous battle platforms has become the primary way to evaluate the large language models, and its success is attributed to the wisdom of the crowds (Surowiecki, 2006: Budescu & Chen. 2015 Weller, (ооτ) and have been proven to lead to better results than that of an individual. Despite its effectiveness, human evaluation is costly and subject to bias and cultural differences(Peng et all (1997).
datasets. The LMSYS initiative introduced plat-
Large Language Models for Evaluation. The trend towards developing open-source LLMs has led to initiatives employing one multiple LLMs evaluators for assessing the outputs of LLMs. GPTScore (En all, 2(2K) uses models like GPT-3 to assign probabilities to high-quality content through multidimensional evaluation. [Bbbeck at (M(23) tested GPT-4, finding it rivaling human capabilities. Lin and Chen introduced LLM-EVAL (Lin & Cheni 2024) for evaluating dialogue quality with single prompts. PandaLM (Wang et all 2023) employs LLMs as "judges" for evaluating instruction tuning. However, reliance on a single model can introduce biases such as positional (Dettmers all 20224), verbosity (Wang et all 2022),, and self-favoring biases (Liu et all 20236
Zheng et all 2023). ChatEval (Chan et all 2023) proposes a multi-agent framework to simulate human evaluation processes. Similarly, PRE CChuetall (OZE)) and PRD (Lietal, (12K3) use LLMs as evaluators, combining multiple evaluation outcomes for automated assessment. However, like the PRE method, which uses human feedback for supervised evaluation throughout the process, the comprehensive assessment of LLMs still incurs a relatively high cost.
CONCLUSION
In this paper, we propose PiCO, a novel unsupervised evaluation method to automatically evaluate Large Language Models (LLMs) without relying on human feedback. PiCO utilizes peer-review mechanisms to autonomously assess LLMs in shared environment, where both open-source and closed-source models can respond to unlabeled questions and evaluate each other. In this setup, each LLM's response score is determined collectively by other anonymous models, aiming maximize consistency across capabilities and scores. The extensive experiment results across multiple datasets and standard rank- metrics demonstrate that PiCO effectively generates LLM ranking that aligns closely with human preferences. In the future, we plan to extend the peer-review mechanism to evaluate the capabilities of multi-modality large models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported in part by the Natural Science Foundation of China (No. 62202014, 62332002, 62425101, 62088 1028, and also supported by the China Postdoctoral Science Founda- tion under Grant Number BX20240013 and 2024M760113.
REFERENCES
Guanaco generative universal assistant for natural-language adaptive context-aware omnilingual outputs.
1o7] 2023. Accessed: 15 April 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Stability AI. Stablelm--tued--alpp----b A fine-tuned language model for diverse applica- tions.
alpha-78
2023. Accessed: 15 April 2024.
Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong-Yeo Ahn. Can we trust the evaluation on chatgpt?, 2023.
Mohammad Allahbakhsh and Alcksandar Ignjatovic. Rating through voting: An iterative method for robust rating. arXiv preprint arXiv: 1210990 2012.
Yuvanesh Anand, Zach Nussbaum, Brandon Duderstadt, Benjamin Schmidt, and Andriy Mulyar. Gpt4all: Training an assistant-style chatbot with large scale data distillation from gpt-3.5-turbo. https://qithi: com/ htt.n/am/aacaaaaaaaaaaa 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
Christoph Bandt and Bernd Pompe. Permutation entropy: natural complexity measure for time series. Physical review letters, 88(17):17410, 2002.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.0400 2023.
Robert S Boyer and J Strother Moore. Mjrtya fast majority vote algorithm. Automated reasoning: essays in honor of Woody Bledsoe. pp. 105-117. Springer, 1991.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. /2712, 2023.
David V Budescu and Eva Chen. Identifying expertise to extract the wisdom of crowds. Management science, 61 (22:266772,,, 2015.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better Ilm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07200 2023.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang. et al. A survey on evaluation of large language models. ACM Transactions Intelligent Systems and Technology, 2023.

---

Published as a conference paper at ICLR 2025
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open sourcs chatbot impressing gpt-4 with 90% chatgpt quality, https://vicung Imsys org, 2023. Accessed: 15 April 2024.
Zhumin Chu, Qingyao Ai, Yiteng Tu, Haitao Li, and Yiqun Liu. Pre: A peer review based large language model evaluator. arXiv preprint arXiv:2401.15641, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 2tff::2/10.1444 2021.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tunne Ilm, 2023. URL https: www wwt.cataabbrooo com wtt.e////o/g//////////o/
12/dolly open commercia. viable instruction -
tuned- 11m
Open-Assistant Contributors. for language understanding.
4 A supervised fine-tuning model
https: httpps/////qg/i//////e aggaae. co co//peee Oopoppssss st.ant/
oasst Beset-sft---oytte
22ochhochh3... 2023. Accessed: 15 April 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv: 14387, 2023.
Allan M. Feldman. Majority voting. SpringerLink, 2006.
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint urXiv:2302.04166. 2023.
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April, 1, 2023a.
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala-13b: Dialogue model for effective human-ai interaction. https://bair: 2023b. Accessed: 15 April 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint
arXiv:2009.00 000002000900300,, 2020.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 1938.
Ann Lehman, Norm O'Rourke, Larry Hatcher, and Edward Stepanski. JMP for basic univariate and multivariate statistics: methods for researchers and social scientists. Sas Institute, 2013.
Charles Eric Leiserson. Ronald Rivest, Thomas H Cormen, and Clifford Stein. Introduction to algorithms, volume 3. MIT press Cambridge, MA, USA, 1994.
Ruosen Li, Teerth Patel, and Xinya Du. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint wxXiv2337.002666 2023a.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-follon followiig models, 2023b.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.099
2022.
Yen- Ken-Ting Lin and Yun-Nung Chen. Llm-eval: Unified open-domain conversations with large language models. arXiv preprint arXiv:2305.1377 2023.
automatic evaluation for
12

---

Published as a conference paper at ICLR 2025
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey prompting methods natural language pro- cessing. ACM Computing Surveys, 55(9):1-35, 2023a.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu, Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint urXiv:2303. 16634, 2023b.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-ansee answering with human feedback. arXiv preprint aIXiv:2112.09332. 2021.
Kun-Peng Ning, Xun Zhao, Yu Li, and Sheng-Jun Jun Huang. Active learning for open-set annotation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 41-49, 2022.
Kun-Peng Ning, Hai-Jian Ke, Yu-Yang Liu, Jia-Yu Yao, Yong-Hong Tian, and Li Yuan. Sparse orthogonal parameters tuning for continual learning. arXiv preprint M.Kiv:2411.0281.. 2024.
Kun-Peng Ning, Jia- Yu Yao, Yu- Yang Liu, Mu-Nan Ning, and Li Yuan. Gpt as monte carlo language tree: A probabilistic perspective. arXiv preprint axXiv:5001.074411 2025.
OpenAL. Introducing chatgpt. sert date here].
2022. Accessed: [in-
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to fol- low instructions with human feedback. Advances in Neural Information Processing Systems, 35: 11730-27744 2022.
Kaiping Peng, Richard Nisbett, and Nancy YC Wong. Validity problems comparing values across cultures and possible solutions. Psychological methods, 2(4):329, 1997.
Philip Sedgwick. Pearsons correlation coefficient. Bmj. 345, 2012.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alons al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.000 1 04615, 2022.
James Surowiecki. The wisdom of crowds. Anehor, 2005.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An moslluition-ollloiing llama model. https: /ttp:::////thbb... wwm.gtttttoooooooo...oooooooooot com/ tattttttsuuu laan-lab/ssa tto/stante anfordd alpaca, 2023.
MosaicML NLP Team. Introducing 7b: A standard for open-source, commercially usable 1lms, 2023. URL mosa cm com/ b wttpptttt//////o/ 7b) Accessed: 2023-05-05.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXhv:2302.13977 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: (804.000000 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang. Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization. arXiv preprint arXiv:2306. 05087, 2023.
13

---

Published as a conference paper at ICLR 2025
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint ar arXiv:2212. 2022.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36, 2024.
Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng. Weiwei LU, Rui Hu, et al. Skywork: more open bilingual foundation model. arXiv preprint arXiv:2310.19344 2023.
Susan C Weller. Cultural consensus theory: Applications and frequently asked questions. Field methods, 19(4)3339-368 2007.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models follow complex instructions, 2023.
Shuo Yang, Kun-Peng Ning. Yu-Y Liu, Jia- Yu Yao, Yong-Hong Tian, Yi-Bing Song, and Li Yuan. Is parameter collision hindering continual learning in Ilms? arXiv preprint 2:011:2400.1019, 2024.
Jia-Yu Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, and Li Yuan. Llm lies: Hallucinations not bugs, but features as adversarial examples. arXiv preprint 10.04423,,.01469, 2023.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm- 13mb: An open bilingual pre-trained model. arXiv preprint
arXiv:2210.024444 2022.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303. 182.23,,,, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your I1m an evaluation benchmark cheater. arXiv preprint arXiv:2311.0199 2023.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large lan- guage models transform computational social science? arXiv preprint arXiv:2305.03514 2023.
14

---

Published as a conference paper at ICLR 2025
A DETAILED EXPLANATION OF METRICS
In this section, we provide a comprehensive explanation the metrics used to evaluate the alignment between learned LLM rankings and human rankings. These metrics assess the strength of correla- tions, complexity, and the level of agreement between rankings. Specifically, we discuss five key metries: Spearman's 's Rank Correlation Coefficient, Kendall's Rank Correlation Coefficient, Permu- tation Entropy, Count Inversions, and Longest Increasing Subsequence, detailing their formulations and intuitive interpretations.
i) Spearman's Rank Correlation Coefficient S(1) (Lehman ef all MIK) measures the strength and direction of the monotonic relationship between two ranked variables. It is computed as:
S(R, R*) = 6
62
tM 1-1
(11)
mmm² 11
where d; = rank (M.)- (M,) 3) the difference between the ranks of L.M M, in the learned ranking π and the human ranking R", and m is the total number of LLMs. A higher Spearman coefficient indicates a stronger correlation between the rankings.
ii) Kendall's Rank Correlation Coeflicient T(t) (Kendall. (938) evaluates the similarity between two rankings by counting the number of concordant and discordant pairs. given by:
C-
(R, R*) = mm(m 1) 1)
(12)
where C rrpreseent the number of concordant pairs, and D represents the number of discordant pairs. pair (M,, Mj) is concordant if M, and M, have the same order in both R and R*, meaning M; > M, in R, then M, > My in R Conversely, a pair discordant if their relative order differs between the two rankings. A higher value indicates closer alignment between the rankings.
iti) Permutation Entropy H()) (Bandt & Pompe, 20122) measures the complexity or randomness of sequences, which is formulated follows:
HRR,
Σ log(())
(13)
where
p(*) # [t|0 S
h. (M M1+4) Ex
++1
denotes different permutations, set l = 3 in this paper. Intuitively, it samples some subsequences and calculates the entropy for all permutation types. And the lower the permutation entropy in the learned LLM rankings, the closer it is to the ground-truth human rankings.
hyper-parameter recommended be
7, and we
iv) Count Inversions C()). Counting inversions (Leiserson et all (994) aims to measure the degree of disorder or "invertedness" in an array or sequence of elements. We thus define it as follows,
C(R, R*)
Σ M..M,,MM
1[M, M, A].
(14)
Where 1(-} the indicator function that the value is when the condition is met, otherwise it Intuitively, the fewer inverse pairs in the learned LLM rankings, the closer it is to the ground-truth human rankings.
v) Longest Increasing Subsequence L(t). The longest increasing subsequence aims to find the length of the longest subsequence in a given sequence of elements, where the subsequence in increasing order. We utilize it to measure the degree of match with human rankings as follows,
L(R, R*) = max (dpli) simm
(15)
where
dp[1] = 1 + max [dpL] |i <iiMi < M
dp[i] represents the length of the longest increasing subsequence that ends with M, LIS allows for a nuanced understanding the degree to which the learned ranking aligns with the ideal human ranking, with higher LIS length indicating greater alignment.
15

---

Published as a conference paper at ICLR 2025
B DATASET FORMAT
Focusing on the MT--Bench dataset, we demonstrate the ensuing data format utilizing dataset Q. As Figure Q illustrates the Question dataset Q contains "Question id," "Category," "Question," and "math." the "Reference" field
"Reference." In categories with definitive answers like "reasoning" is populated with standard answers, otherwise, it remains blank. Each model M in our pool processes the Question dataset Q to generate the LLMS answer data A, consisting of "Question id," " "Answer id," "Model id," and "Answer." Finally, we combine pairs in A and appoint judges to evaluate, creating the Answer-Ranking data D, featuring "Question id," "Model 1," "Model 2, "GI winner," "G2 winner," and "Judge." Here, "G1 winner" and "G2 winner" indicate the outcomes of inputting reversed order responses Model and Model into the judge model, a method employed to mitigate biases stemming from models' preferences for input order.
VS Poop
Question dataset
LLMs answers data A
raswing data
Afestee
Shestara
Acurien alil
Ortggry: Riving
Modi 15-arta
Answer of SMMMA5502220ee220
Aevvls assacing
Modi 2 lmme
Mebilot ge 1irk
2 Reeaact skd
Aefreme (f hose edf néee enseet offers
Arswor ditri rearty the elamber suattii
Jaje 4B-1
Figure 7: Format of the Question dataset Q, L.MSs responses data A, and the Answer-Ranking data DD for Peer Review
C DETAILED PROMPT FOR REVIEWERS
The evaluation prompts, detailed in Section 2.2.1, employed during the Peer Review Stage. These prompts are provided to the Reviewer Language Model Systems (LLMs), enabling them to generate evaluative preferences. In our experimental framework, we devised four distinct prompt settings. For each setting, a tailored prompt template was meticulously crafted as illustrated below:
Template for Single-Turn Interaction: This template designed for single-turn interactions be- tween users and LLMs, where there is no predetermined correct answer. It facilitates open-ended dialogue, allowing for wide range of user inquiries without the expectation of specific responses.
Referenced Template for Single- Turn Interaction: Tailored for single-turn dialogues between users and LLMs, this template incorporates predefined correct answers. is particularly suited for interactions involving factual inquiries, such mathematics or logic problems, where accuracy and reference to correct information are paramount.
This template caters to multi-turn conversations between users and LLMs without predefined answers. It supports extended interactions, enabling users to explore topics in depth through series of interconnected questions and responses.
Template for Multi-Ture Interaction:
Referenced Template for Multi- Interaction: Designed for multi-turn dialogues with prede- fined correct answers, this template is ideal for complex inquiries requiring sequential reasoning problem-solving, such as mathematical computations or logical deductions.
Each template is carefully constructed to match its intended use-case, providing structured frame- work that guides the interaction between users and LLMs towards achieving desired outcomes, whether for open-ended exploration precise problem-solving.
16

---

Published as a conference paper at ICLR 2025
Template for Single-Turr Answer
System prompt: Please act judge and evaluate the quality of the responses provided by two Al assistants to the user question displayed below. You do not need explain, just give your judgment. Output your final verdict by strictly following this format: "[[A]]" if assistant A is better. "[[B]]" if assistant B is better, and "[[C]]" for tie. {question]
User Question:
Assistant A's Answer: (answer a} (answer b}
Assistant B's Answer:
Referenced Template for Single-Turn Answer
System prompt: Please
judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below, with reference to the provided reference answers. You do not need explain, just give your judgment. Output your final verdict by strictly following this format: "[[A]]"if assistant A is better, "[[B]]" if assistant better, and "[[C]]" for tie.
User Question: [question]
Reference Answer:
(reference answer)
Assistant A's Answer: [answer a}
Assistant B's Answer:
(answer b}
Template for Multi-Turn Answer
System prompt: Please act as judge and evaluate the quality of the responses provided by two Al assistants to the question displayed below. You do not need explain, just give your judgment. Output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better, and "[[C]]" for tie Assistant A's Conversation with User:
User: {question 1)
Assistant A: (answer al)
User: (question 2)
Assistant A: (answer a2)
Assistant B's Conversation with User:
User: (question 1}
Assistant (answer bl)
User: (question 2)
Assistant B: (answer b2)
17

---

Published as a conference paper at ICLR 2025
Referenced Template for Multi-Turn Answer
System prompt: Please act judge and evaluate the quality of the responses provided by two Al assistants the user question displayed below, comparison to the reference answers. You do not need to explain, just give your judgment. Output your final verdict by strictly following this format: "[[A]]"if assistant better, "[[B]]" if assistant is better, and "[[C]]" for tie.
Reference Answer
User: (question 1}
Reference answer: (ref answer 1}
User: (question 2}
Reference answer: (ref answer 2
Assistant A's Conversation with User:
User: (question 1)
Assistant A: (answer al)
User: (question 2}
Assistant A: (answer a2)
Assistant B's Conversation with User:
User: [question 1|
Assistant B: fanswer b1]
User: (question 2)
Assistant B: (answer b2)
D SCORING METHODOLOGY
In Section 2.2.2, Equation & delineates the methodology for optimizing scores. Within this frame- work, the function 1 1 > A&) } is more precisely defined S(A), A&). Additionally, the function fAA A/) is not fixed and can be implemented using various computational strategies. We introduce two distinct methodologies in this context: the Elo mechanism and the Rank mechanism.
Within the framework of the Elo mechanism, specified by Equation II. the BASE value is set 10, and the SCALE factor is determined to be 400. This approach facilitates a dynamic adjustment of scores based on the outcomes of pairwise comparisons, allowing for nuanced reflection of performance variations among models.
Conversely, the context of the Rank mechanism, as outlined by Equation IZA rank(j) signifies the current ranking of model j. with the constant K assigned a value of 200. This mechanism employs model's ranking within predefined hierarchy as pivotal factor in score calculation, thereby providing straightforward, yet effective, method for evaluating comparative model performance.
if A A if A A
BISC C SCALET
F(A), At)
0.5
(16)
(IBSS /SCME
0
if A < A
(+BASE
1 + (rank(j) - (knnk
if A3 > A
f(A)
0.5
if A A.
(17)
0
if A A
E OVERALL ALGORITHM OF PEER REVIEW
The overall algorithm, as delineated in Algorithm MI encapsulates the comprehensive process out- lined in Section 2.2. This sequence commences with "Data Collection and LLMs Pool Construc- tion." progresses through "Answer-Be Ranking Data Construction Based on Peer Review." advances to "Consistency Optimization," and culminates with the "Unsupervised Elimination Mechanism."
18

---

Published as a conference paper at ICLR 2025
Algorithm Overall Framework Algorithm of Peer Review
Require: Unlabeled dataset Q, Pool of LLMs M, Active LLM pool M = M
Ensure:
Consistency-optimizzed ranking of LLMs R
1: Initialize response matrix A <- 0
2: for each question q; E Q do
3: Initialize response vector for question qı, A' <-
4: for each model mj E M do
5:
A f- response of model my to question qi 6: A' + A' U
7: end for
8: Shuffle A' to obtain permuted response vector A
9: A < AU {A*}
10: end for
11: Initialize answer-ranking data +-
12: Initialize model weights vector with Gaussian distribution
13: for each permuted response vector A do
14: for each pair of responses (47 A4) do
15: for 1 do
> RRandolly select 5 models for evaluation
Evaluate the pair (A,, Ab) with model m, + D-A A w*)}
17:
18: end for
19: end for
20: end for
21: Initialize scores G, for each model my E M to the Elo initial score
22: repeat
23: while not converged do
for each model my E M do
25:
Compute Gj using updated formula:
26:
= E. Eophers 1(4/ A/) w (A,, w M*) D
27:
end for
28:
Update weight vector w to maximize the consistency of w and G end while
29:
30:
Sort .M" by G, to identify Mmins the lowest-scoring model
31:
if size of M* > threshold then
32:
Remove Mmin from M"
end if
34:
until size of M < threshold
Compute the final ranking R" based on the optimized scores Gj
36:
return R*
F COMPLETE EXPERIMENTAL RESULTS
In Section 3.4, we both employ elimination mechanisms to cull the weakest LLMs from the 'reviewer queue' during the evaluation process. In Figures and E. we present the results for the PEN and LIS metrics, where lower PEN scores indicate better performance, and higher LIS scores denote superior performance. It evident that both the 'PiCO' and PRE approaches demonstrate enhanced performance as the number of eliminated 'reviewers' increases. In most cases, the proposed "PiCO" method outperforms PRE.
In Section 3.5, we validate the effectiveness of the consistency assumption and compare it with the Average Performance of the Reviewer Queue, i.c., employing a single LLM as the 'reviewer' to evaluate all response pairs and then calculating the average results of all LLMs. The comprehensive results compared with the Reviewer Queue are illustrated in Table5i Figure ITC M and M2 reveal- ing that in the full Reviewer Queue, the performance of the vast majority of LLMs is very poor, indicating that the evaluations from most LLMs are noise. However, our 'PiCO' approach nearly matches the evaluative prowess of the pool's most capable LLM,, GPT-3.5. Remarkably, given its
19

---

Published as a conference paper at ICLR 2025
RCOXOurr)
PRE
Groundtruth
AlpazaEval
20.0
20
12.5
15-
10.0
0
10.0
75-
7.5
5.0
2.5
Eliminated Reviewers
Ellmindted Reviewers
Figure 8: Performance comparison of the PiCO (Ours) and PRE (Chu 2IRE) methods on the MT-Bench, Chatbot Arena, and AlpacaEval datasets, with number of eliminated reviewers on the x-axis. The y-axis CIN, where lower values indicate better performance.
PICO(OUrS)
PRE
Groundtruth
Chatbot
MT BTTBench
AlpacaEval
14
14-
14-
12
23
10
10
10
Reviewers
Reviewers
Figure 9: Performance comparison of the PiCO (Ours) and PRE (Chu at (022) methods on the MTT-Bench, Chatbot Arena, and AlpacaEval datasets, with number of climinated reviewers on the x-axis. The y-axis LIS, where upper values indicate better performance.
Table 5:
Comparison of performance across three datasets using Unsupervised methods versus using single models in reviewer queue.
MT-Bench
Chatbot Arena
AlpacaEval
Methods
PEN (L) CIN(L) L
(C) CIN(L LIS(T)
PEN (J) CIN(J) LIS(T)
Gpt-3.5
0.97
12.00
10.00
0.85
11.00
11.00
1.15
16.00
9.00
Guanaco-33B
1.25
21.00
8.00
1.50
28.00
7.00
1.26
20.00
9.00
Vicuna-13B
1.31
20.00
7.00
1.27
23.00
8.00
1.20
17.00
8.00
WizardLLM13BB
1.15
17.00
9.00
1.27
19.00
8.00
1.17
17.00
9.00
Vicuna-7B
1.27
21.00
8.00
1.30
20.00
7.00
1.34
23.00
8.00
Koala-13B
1.67
43.00
6.00
1.34
23.00
8.00
1.54
31.00
7.00
gpt4all-13B
1.74
45.00
6.00
1.60
35,00
6.00
1.73
42.00
6.00
Mpt-7B
1.67
39.00
6.00
1.72
52.00
6.00
1.63
34.00
7.00
Ouss-pythia-125 12B
1.77
50.00
5.00
1.74
42.00
5.00
1.70
47.00
6.00
Alpaca-133 13B
1.77
49.00
7.00
1.60
73.00
4.00
1.63
34.00
7.00
PnsiChat-T5-31
1.45
29.00
7.00
1.53
30.00
7.00
1.30
22.00
7.00
ChaiGLM-6B
1.59
33.00
7.00
1.71
55.00
5.00
1.63
34.00
6.00
StableLM-7B
1.68
63.00
5.00
1.75
44.00
5.00
1.72
56.00
4.00
Dolly-12B 12B
1.76
46.00
6.00
1.57
71.00
6.00
1.75
54.00
6.00
LLaMA-13B
1.60
35.00
7.00
1.76
56.00
6.00
1.70
50.00
5.00
Average Performance of All Review LLMSs
1.51
34.87
6.93
1.50
38.80
6.60
1.50
33.
6.93
PRD (123))
1.15
17.00
8.00
1.13
17.00
8.00
1.21
19.00
9.00
PRE(Chu ctall (122))
1.17
17.00
8.00
1.07
15.00
9.00
1.18
19.00
8.00
PiCO (Ours)
1.01
14.50
8.75
0.94
12.00
10.00
1.17
17.00
9.00
supervised nature, the 'PiCO' method demonstrates the capability to mitigate the influence of noise, reaching the evaluation upper bound (the strongest LLM) within any given unknown LLM pool M, even in the absence of prior ranking information.
20

---

Published as a conference paper at ICLR 2025
Ootimized Mathods
Non- Can-Optiniced Mothods
33
PicO(Curs)
PRD
PRE
G
Guanaco-330
Vicuns-i 138
Wicart 138
Vicuna-7B 7B
Koala 139
Gotdall 130
Oass pythla 122
Mpt-70
Alpaca-11 139
Fastchat-t5 38
Chatgim-66
StabieLM-77
Dally 128
tiama 130
Figure 10: Comparison of performance on the CIN metric across three datasets using Unsupervised methods versus using single models, with Unsupervised methods on the left and Supervised methods on the right. The dotted line represents the average value using single models.
HS Mi-larcch
Cheatees Arana
Optimiced Methads
Nov- Nor-Dptimized Mathods
2.50
3.2
025
0.0
PICOOOurs
PRO
PRE
Gpt-3.
Guanaco-35 338
Vicuna- 138
WizaratMM 138
Vicuna-22 78
Koala 138
Gpodall- 138
Oass-pythia 128
Mpt-78
Alpara- 138
Fastchat-ts 39
Chatgim-66 66
StabieLM 7B
Dally 128
Liama: 138
Figure 11: Comparison of performance on the PEN metric across three datasets using Unsupervised methods versus using single models, with Unsupervised methods on the left and Supervised methods on the right. The dotted line represents the average value using single models.
G SELECTED MODELS AND OPTIMIZED RANKING
For our analysis, we meticulously selected 15 LLMs spanning a variety of architectures, encompass- ing both open-source and closed-source models, detailed in the subsequent table. Our curated selection features prominent LLMs including the closed-source "gpt-335-ttrbo." "chatglm" which is predicated on the encoder-decoder framework, "fastchat-15-35 that leverages Google's 's T5 (Text-to- Text Transfer Transformer) architecture, and "Ilma- 13b" founded on the GPT architectural princi- ples.
21

---

Published as a conference paper at ICLR 2025
UTDbech
Optimizad Mathods
Nan Optimized Mathads
PiCO(Ours)
PRD
PRE
Gpt
Guanaco- 238
Vicuna- 1.38
Wibar M--..--... 139
Vicuna-1
Kosia- 138
Gpt-atl-1318 138
Mpt-
cess-pythi--12 129
Alpaca- 138
Festchat- 35-3
Chatgim-66 68
Stablel 78
Dally 129
Laaaa 138
Figure 12: Comparison of performance the LIS metric across three datasets using Unsupervised methods versus using single models, with Unsupervised methods on the left and Supervised methods on the right. The dotted line represents the average value using single models.
We have comprchensively detailed the ranking outcomes across three distinct datasets for com- parative analysis, incorporating the optimized model rankings, names, and their respective scores. As delineated in Appendix I the PiCO (Ours) is capable of employing various scoring mechanisms, thereby facilitating the presentation of ranking outcomes on three datasets utilizing both the Elo and Rank mechanisms. Furthermore, we have also enumerated the ranking results for PRD and PRE methodologies across the three datasets, offering a holistic view of the competitive landscape.
22

---

Published as a conference paper at ICLR 2025
G.1 PICO
Grade-Elo-Chatbot
#1 Gpt-3.5 | Grade: 1905.162109375
#2 WizardL.--33B Grade: 9143.46875
#3 Guanaco-33B Grade: 9886.9262695315
#4 Vicuma-7B Grade: 336899462890625
#5
Vicuna-13B Grade: 5216.79541015625
#6 Koala-13B Grade: 3545.1171875 Eliminated
#7
Mpt-7B Grade: 062.99462289625 Eliminated
#8 Gpt4all-13B Grade: 652.4602661 132812
Eliminated
#9 Chatglm-6B Grade: 417.1375427246094
Eliminated
# 10 Ousst-pythia-12B Grade: -898.26763916015622
Eliminated
# 11 Fastchat-t5-3BB Grade: -1251.71838377890625
Eliminated
# 12 StableI.M-7B Grade: 63993337953339 Eliminated
#13 Dolly-12B Grade: 3163.5402832031251 Eliminated
14 Llama-13B Grade: 3648.37841796875| Eliminated
# 15 Alpaca-13B Grade: - 14204.3984375 Eliminated
Grade-Elo-AlpacaEEva
#1 WizardLM-13B Grade: 8662.7158203125
#2
Vicuma-13B Grade: 5586.46630859375
#3 Guanaco-33B/ Grade: 5445.34179687
#4
Vicuna-7B | Grade: 5374.2314453125
#5 Gpt-3.5 Grade: 1845.91555733475
#6 Koala-13B Grade: 4338.777832031251
Eliminated
#7 Chatglm-6B Grade: 2293.4208984375| Eliminated
#8 Gpt4all-13B Grade: 2080.511662896625
Eliminated
#9 Mpt-7B Grade: (69449950083359355 Eliminated
10 Fastchat-t553BB Grade: 1371.94287109375 Eliminated
#11 1asst-pythia-12B Grade: -665.6666666 666.86655302333375
EBliminated
StableL.M-BB Grade: -1343.58386230466775
Eliminated
#13 Dolly-12B I Grade: -3777.13427734375 Eliminated
14 Llama-13B Grade: -5847.59130859375 Eliminated
# 15 Alpaca-138 Grade: 13159761122109375 Eliminated
Grade-Elo-MT Benchh
#1 WizardLM-13B Grade: 2178. 1078.1030773337
#2 Vicuna-13B Grade: 1720. 119501914519
#3 Guanaco-33B Grade: 1704. 1832275399225
#4
Vicuna-7B Grade: 2155,2799072265625
#5
Gpt-3.5 Grade: 1535.88 88 Nt..8881188882555
#6
Mpt-7B Grade: (338.5235595703125
Eliminated
#7 Koala-13B Grade: 2267.7747314453125
Eliminated
#8 Chatglm-6B Grade: 1011.7701416015625
Eliminated
#9 Gpt4all-13B Grade: 976.596374511171888 Eliminated
# Oasst-pythia-12B Grade: 779.35736083988438 Eliminated
# 11 StableL.M-7B Grade: 1
Eliminated Elimmittt
4 12 Alpaca-13B Grade: 334.98794555664061 I Eliminated
# Fastchat-t5-3B Grade: 133.5990529788156111
Eliminated
# 14 Dolly-12B I Grade: 72.63818359375| Eliminated
#15 Llama-13B Grade: -35. 399.19991875
Eliminated
23

---

Published as a conference paper at ICLR 2025
Grade-Rank-Chathot
#1 Wizard Grade: 0.308092880272298523
#2 Gpt-3.5 Grade: 0,299962999225761
#3 Guanaco-33B Grade: 0,285875996003366668
#4
Vicuna-7B | Grade: 0,2821291089057999
#5 Vicuma-13B Grade: 0,2790021189636234477
#6 Koala-13B Grade: 0 Eliminated
#7
Mpt-7B Grade: 0.550022995599011Iimmmmee
#8 Gpt4all-13B Grade: 0.2474666225388711555 Eliminated
#9 Chatglm-6B Grade: 0.24669538438320161 aanrssaeeeaa22011 1iiiieeeeeedd
Eliminated
10 Oasst-pythia-128 Grade: 0.2363706996115533322
Eliminated
#11 Fastchat-t5-3BB Grade: 0.2350562959909939) Eliminated
StableLM-7B Grade: 0.228438064455986622
Eliminated
#13 Dolly-12B I Grade: 0.22219440340999578) Eliminna
Eliminated
14 Llama-13B Grade: 0.21656799316440625 Ellmiiittt
Eliminated
15 Alpaca-13B Grade: 0. 0.13975904881954193
Eliminated
Grade-RaankAApacaaEval
#1 WizardLM-13B Grade: 0.4019235074520111
#2 Vicuma-13B Grade: 0.367455290399001465
#3 Guanaco-33BB Grade: 0.36648780100798817
#4
Vicuna-7B | Grade: 0.365411733145713866
#5 Gpt-3.5 Grade: 0,360003650188444664
#6
Koala-13B Grade: 0.3544933795928955
Eliminated
#7 Chatglm-6B Grade: 0.33195710182119944
Eliminated
#8 Gpt4all-13B Grade: 0.3306528677822467
Eliminated
#9 Mpt-7B Grade: 0.326417297124862671 Eliminated
10 Fastchat-t5-3B Grade: 0.32173293828994433 Eliminated
#11 Oasst-pythia-12B Grade: 0.2999681533288265
Eliminated
StableLM-7B Grade: 0,2932431995588683 Eliminated
#13 Dolly-12B I Grade: 0.24777530133722213 Eliminated
Llama-13B Grade: 1584443366444
Eliminated
15 Alpaca-13B Grade: 0.161 146399979114
Grade-Rank-MT Bench
#1 WizardLM-13B Grade: 0,29946511183387146
#2 Vicuna-13B Grade: 0,280926167966493333
#3 Guanaco-33B Grade: 0.2767307460308077
#4
Vicuna-7B Grade: 0.27581477166222177
#5
Gpt-3.5 Grade: 6.272616088390355054
#6
Mpt-7B Grade: 0,263386905193338886
Eliminated
#7 Koala-13B Grade: |201133686344992411I|lmmmm Eliminated
#8 Gpt4aal-13BB Grade: 0,2490888833999663388 Eliminated
#9 Chatglm-6B Grade: 0.248982340097427777
Eliminated
# 10 Oasst-pythia-1228 Grade: 0.24154008922496100
Eliminated
#1 #1 Stablel.M-7B Grade: 0.22990757226944997
Eliminated
12 Alpaca-13B Grade: 0,221714749932289122 Eliminated
# 13 Fastchat-t5-3B Grade: 0.2216777652502266
Eliminated
14 Dolly-12B I Grade: 0.211855100222355596 Eliminated
#15 Llama-13B Grade: 0. 0.192665232288107
Eliminated
24

---

Published as a conference paper at ICLR 2025
G.2 PRD
PRD-Chatbot
#1 WizardLM-13B Grade: 5565.28271484375
#2 Gpt-3.5 Grade: 4613.22900390625
#3 Guanaco-33B Grade:
34765622
#4 Vicuna-7B Grade: 2985.4892578125
#5 Vicuna-13B Grade: 2972.15673828125
#6
Koala-13B Grade: 2237.70751953125
#7 Chatglm-6B Grade: 875.373779296875
#8 Mpt-7B Grade: 602 000244623822125
#9 Gpt4all-13B Grade: 356.06243896484335
# 10 Fastchat-t5-3B Grade: 184.896633962288622
# Dolly-12B Grade: 52.107467651136679
# 12 0asst-pythia-12B Grade: -007.49908847225565
# 13 StableLM-7B Grade: 291.4533333315522
14 Llama-13B Grade: 848.165405273443
# 15 Alpaca-133 13B Grade: 7020.923828125
PRD-AlpacaEval
#1 WizardLM--3BB Grade: 5469.75634765625
#2 Guanaco-33B Grade: 3707.01489257812
#3 Vicuna-13B Grade: 3618.6360 631474337773335
#4 Vicuna-7B Grade: 3569.3898999
35699389892558125
#5
Gpt-3.5 I Grade: 3197.755615233375
#6 Koala-13B Grade: 2893.642578125
#7 Chatgim-6B Grade: 1847. 130044800088
#8 Fastchat-t5-3B Grade: 1.555.699 988.66694335377
#9 Gpt4all-13B Grade: 156..145751953125
# 10 Mpt-7B Grade: 1332.3753662 109375
StableLM-7B Grade: -33.00855255126953
# 12 Oasst-pythia-128 Grade: -92.6833770037597666
#13 13 DoI Grade: -3013.588623446875
# 14 Llama-13B Grade: -3211.0302734375
15 Alpaca-13B Grade: -7432.3701778875
PRD-MT Bench
#1 WizardLM-13B Grade: 181466466766662
#2 Vicuna-13B Grade: 733778084716796855
#3 Guanaco-33B Grade: 1481.1739501953125
#4
Vicuna-7B Grade: 1401.5194091 796875
#5
Gpt-3.5 Grade: 2272.8072509766625
#6
Mpt-7B | Grade: 1188.5518798822812
#7 Chatglm-6B Grade: 016666246637890625
#8 Koala-13B Grade: 1124.2513327734375
#9 Gpt4all-13B Grade: $71.2874755859375
# 10 Oasst-pythia-1225 Grade: 855. 655.3663554455315
# #1 Stablel.M-7B Grade: 782.702880859375
12 Fastchat-15-3B Grade: 636.966064453125
# 13 Alpaca-13B Grade: 414.9374694824219
14 Dolly-12B | Grade: 377.501 80053300997
#15 Llama-13B Grade: 78.90127563476562
25

---

Published as a conference paper at ICLR 2025
G.3 PRE
PRE-Chatbot
#1 WizardLM-13B Grade: 1113:70034115479742
#2 Gpt-3.5 I Grade: 1076. 11666419960S
#3 Guanaco-33B I Grade: 1067,441581415147
#4
Vicuna-13B Grade: 0057.702184441444
#5 Vicuna-7B Grade: 1043.4840340151043
#6
Koala-13B Grade: 1030.4455842015508
Eliminated
#7 Chatglm-6B Grade: 1012.4487557424748 Eliminated
#8 Mpt-7B Grade: (000,487230109001 I Eliminated
#9 Gpt4all-13B Grade: 1000:41113970384922 Eliimnated
# 10 Fastchat-15-3B Grade: 992.37721198323631 Eliminated
# 1 Grade: 977.5217305871272
Eliminated
# 12 MtableL.M-BB Grade: 970.3665926795535| Eliminated
# Llama-13B Grade: 929.622992222 6996688888888149 Eliminated
# 14 Dolly-12B I Grade: 92.. 199994446333397 Eliminated
# 15 Alpaca-13B Grade: 798.6815779514078
Eliminated
PRE-AlpacaEval
#1 WizardLM-13B Grade: 1127.822808841937
#2 Vicuna-7B Grade: 1077.1823399550224
#3 Vicuna-13B Grade: 1075.4338443616666
#4
Guanaco-33B Grade: 1074.80431352294118
#5 Gpt-3.5 I Grade: 0066330573605376
#6 Gpt4all-13B Grade: (039,409163086186)
Eliminated
#7 Koala-13B Grade: (038.205749976473 Eliminated
#8 Mpt-7B | Grade: 1032.28934011621788 Eliminated
#9 Chatglm-6B Grade: 1027.1937466118500 Elliminttdd
# 10 Fastchat-15-3B Grade: 992.34811688911000 Eliminated
#1 StableLM-7B Grade: 979338941414456921 Eliminated
# 12 Oasst-pythia-12B Grade: 940.6434439723215
Eliminated
# 13 Dolly-12B Grade: 886. 14121 Eliminated
14 Llama-13B Grade: 880.0797724297793 Eliminated
# 15 Alpaca- 13B
Grade: 763.7505968602333
Eliminated
PRE-MT_Bende
#1 WizardI-M-13B I Grade: 1065.584377669933
#2 Vicuna-13B Grade: 1062.39341388000302
#3 Guanaco-33B Grade; 1052.2206466556906
#4
Vicuna-7B Grade: 1035.111281724777
#5
Gpt-3.5 I Grade: 0029,8316754711038
#6 Koala-13B Grade: 1024.9307662983267
Eliminated
#7 Chatglm-6B Grade: 1020.5238960907612 Eliminated
#8 Mpt-7B Grade: 0014.06832550810057 Ellmmnnttd
#9 Gpt4all-13B Grade: 991.71426396230117 Eliminated
# 10 StableLM-7B Grade: 779.84432612563271 Eliminated
# Oasst-pythia-12B Grade: 977.99304301 113:00033111122 Eliminated
# 12 Fastchat-15-3B/ Grade: 953.07761151133571 Eliminated
# 13 Alpaca- 13B
Grade: 949. 199777777733 Eliiminttdd
# 14 Dolly-12B I Grade: 928.511065779112 Eliminated
# 15 Llama-13B Grade: 915.0655312591185
Eliminated
26


Reviews:
[{"review_id": "KHnZqjeRKX", "reviewer": "Reviewer_fWmA", "summary": "This paper presents an unsupervised evaluation method for large language models (LLMs) using a peer-review mechanism that operates without human feedback. This method, PiCO, allows LLMs to evaluate each other’s answers to unlabeled questions within a shared environment. The core idea is the consistency assumption, where a model’s evaluation ability correlates with its performance, leading to an optimization of model confidence weights to align LLM rankings more closely with human judgments. The effectiveness of PiCO is validated through extensive experiments.", "questions": "1. How is the ground truth ranking precisely defined and established for each dataset?\n2. How can we mitigate uncertainty and bias in the evaluations conducted by models?\n3. What is the application scenario of PICO?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel peer review mechanism for autonomously evaluating large language models (LLMs), which represents a significant advancement over traditional human-centric review processes. This framework leverages the models themselves as evaluators to assess each other’s answers, potentially increasing the scalability and efficiency of LLM evaluations.\n\n2. The paper is well-written with clear and concise explanations, accompanied by informative figures and tables. These visual aids and structured content greatly enhance the readability and accessibility of the complex concepts discussed, making the methodology and results understandable to a broader audience.\n\n3. The use of multiple well-known datasets like Chatbot Arena, MT-Bench, and AlpacaEval for testing not only demonstrates the practical applicability of the PiCO method but also underscores its effectiveness in aligning model evaluations with human judgment, thus validating the proposed approach through rigorous empirical evidence.", "weaknesses": "1. The paper does not provide detailed information on how the ground truth rankings in the experiments are established, which could impact the transparency and reproducibility of the research. This lack of detail may hinder other researchers' ability to fully replicate or build upon the work.\n\n2. The effectiveness of the peer review process heavily relies on the design of the questions and prompts, which can significantly affect the consistency and scalability of the evaluations. Variability in the outcomes may arise as different prompts may not equally elicit discriminative responses across models. This reliance on prompt design poses a challenge to the general applicability and scalability of using a large number of LLMs for peer review, as consistency in evaluation may not be universally achievable across diverse question sets.\n\n3.Although the paper presents a novel method for evaluating LLMs, it does not discuss specific application scenarios or domains where this method could be particularly beneficial. This omission limits understanding of the potential real-world impact and practical utility of the PiCO method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6VAHewvxmm", "reviewer": "Reviewer_jVng", "summary": "This paper presents an approach to evaluating Large Language Models (LLMs) through a peer-review mechanism without relying on human annotations. The basic idea is to create weights for each model based on the consistency between the weights and the final evaluation results. The proposed PiCO framework shows promising results in aligning LLM rankings with human preferences across various datasets and metrics.", "questions": "- What’s the meaning of “average loss” in the caption of Figure 5?\n- What’s the meaning of Precision@K and RBP@K in Section 3.4?\n- There are other methods that select reviewers unsuperwisely, such PRE with auto exam in the original PRE paper. It would be good to compare the method in this paper with it as well.\n- In Table 2, the variance of multiple baselines are 0, which seems a bit weird. Might be better to explain a bit.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is mostly well-written and easy-to-understand, except for some specific details (discussed below).\n- The topic of this paper is important, and the proposed method is well described. The idea of using consistency to learn reviewer weights is interesting and well motivated.\n- The authors have conducted extensive experiments to compare their method with SOTA baselines.", "weaknesses": "- The assumption that “high-level LLM can evaluate others” answers more accurately” may not be true in all cases, but this is not a big problem. The problem is that, once we assume that  “high-level LLM can evaluate others” and only use high-level LLM as reviewers, it may automatically create new types of biases that cause the cold-start problem: if a model’s response is not preferred by current reviewers, then it won’t get high scores, then it won’t get high reviewer weights, no matter whether it’s a good model or not.\n- The analysis of bias with PG_hat in Section 3.2 is problematic. Because the confidence weights are all smaller than 1.0, it’s not surprising to see that PG_hat is all smaller than PG, so this cannot prove that the bias of the evaluator is reduced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SStw18dBXo", "reviewer": "Reviewer_VBEw", "summary": "The paper proposes PiCO, a model that evaluates LLMs by learning the weights of reviewer LLMs through a peer-review mechanism.\nThe contributions are as follows:\n1. It enhances the peer-review mechanism used in PRD and PRE.\n2. It enables evaluation at a lower cost, as no human annotations are required, unlike in PRE.\n3. The proposed Reviewer Elimination Mechanism is an improvement that can be applied not only to the PiCO model but also to other peer-review methods.", "questions": "I suggest that the authors demonstrate whether the trained PiCO assigns higher ranks to models that are superior to the reviewer models used in its training. Also, it would be more realistic if the authors first separate the LLM models used for training and those used for evaluation, and present the results accordingly. Furthermore, while the paper cites Goodhart’s Law in the introduction, it appears that the current experimental setup may be susceptible to the very concerns highlighted by Goodhart’s Law.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. By combining learnable weights with existing peer review methods, the evaluation process has become more refined and accurate.\n2. The absence of human annotation makes it more practical for real-world applications.\n3. The Reviewer Elimination Mechanism, though simple, is a powerful idea that can be applied not only to the proposed method in the paper but also to other peer review approaches.", "weaknesses": "1. Baselines.\nThe proposed method does not experimentally demonstrate whether it can serve as a replacement for existing benchmarks. While the paper argues that benchmarks fail to adequately capture human preferences and suffer from benchmark leakage issues, it does not compare the rank similarity of the proposed method with that of existing benchmarks in the experiments.\n\n2. Evaluations.\nThe proposed method uses the same models as both reviewers and those used for training and evaluation. This is akin to comparing performance using the training loss, and the experimental results do not reflect the performance on unseen models. However, if the training and evaluation were conducted using different models, then the issue lies in the paper's writing, as it fails to include this crucial information.\n\n3. Writing.\nThe writing lacks academic rigor and professionalism. The notations in Equations (2) and (7) could have been expressed in a more general manner rather than illustrating a specific case. The use of > to denote the set {>,<,=} causes confusion as the set name overlaps with its elements. It is recommended to avoid such overlap for clarity. In Figure 3, while the intended message is conveyed, the exact meaning of the x-axis and y-axis is missing. It is advised to specify these axes more clearly for better precision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)