RocketEval: Efficient automated LLM evaluation via grading checklist
Authors: Tianjun Wei
Wei Wen
Ruizhi Qiao
Xing Sun
Jianghong Ma
Venue: ICLR 2025 Poster (2025)
Keywords: automated evaluation
large language models
natural language processing
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A simple, replicable,interpretable, and accurate automated evaluation method that uses lightweight LLMs as judges to efficiently assess various scenarios and questions.

Abstract: Evaluating large language models (LLMs) in diverse and challenging scenarios is essential to align them with human preferences. To mitigate the prohibitive costs associated with human evaluations, utilizing a powerful LLM as a judge has emerged as a favored approach. Nevertheless, this methodology encounters several challenges, including substantial expenses, concerns regarding privacy and security, and reproducibility. In this paper, we propose a straightforward, replicable, and accurate automated evaluation method by leveraging a lightweight LLM as the judge, named RocketEval. Initially, we identify that the performance disparity between lightweight and powerful LLMs in evaluation tasks primarily stems from their ability to conduct comprehensive analyses, which is not easily enhanced through techniques such as chain-of-thought reasoning. By reframing the evaluation task as a multi-faceted Q\&A using an instance-specific checklist, we demonstrate that the limited judgment accuracy of lightweight LLMs is largely attributes to high uncertainty and positional bias. To address these challenges, we introduce an automated evaluation process grounded in checklist grading, which is designed to accommodate a variety of scenarios and questions. This process encompasses the creation of checklists, the grading of these checklists by lightweight LLMs, and the reweighting of checklist items to align with the supervised annotations. Our experiments carried out on the automated evaluation benchmarks, MT-Bench and WildBench datasets, reveal that RocketEval, when using $\textit{Gemma-2-2B}$ as the judge, achieves a high correlation (0.965) with human preferences, which is comparable to $\textit{GPT-4o}$. Moreover, RocketEval provides a cost reduction exceeding 50-fold for large-scale evaluation and comparison scenarios. Our code is available at https://github.com/Joinn99/RocketEval-ICLR.

Published as a conference paper at ICLR 2025
ROCKETEVAL: EFFICIENT AUTOMATED LLM EVALUATION VIA GRADING CHECKLIST
Tianjun Wei 12, Wei Wen" 2 Ruizhi Qiao"?, Xing Sun 2 Jianghong Ma
1 City University of Hong Kong." 2 Tencent Youtu Lab, 3 Harbin Institute of Technology Shenzhen.
tjwei2 c@my. cityu. edu.hk [jawnrwen, ruizhiqiao, winf wenfrredsuneeeeneentccom maj jaiignnngghit.eduu.
ABSTRACT
Evaluating large language models (LLMs) in diverse and challenging scenarios is essential to align them with human preferences. To mitigate the prohibitive costs associated with human evaluations, utilizing powerful LLM judge has emerged as a favored approach. Nevertheless, this methodology encounters several challenges, including substantial expenses, concerns regarding privacy and security, and reproducibility. In this paper, we propose a straightforward, replicable, and accurate automated evaluation method by leveraging lightweight LLM as the judge, named RocketEval. Initially, we identify that the performance disparity between lightweight and powerful LLMs in evaluation tasks primarily stems from their ability to conduct comprehensive analyses, which is not easily enhanced through techniques such as chain-of- thahnghftthought reasoning. By reframing checklist, the evaluation task as a multi-faceted Q&A using instance-specie we demonstrate that the limited judgment accuracy of lightweight LLMs is largely attributes to high uncertainty and positional bias. To address these challenges, we introduce an automated evaluation process grounded in checklist grading, which is designed to accommodate variety of scenarios and questions. This process compasses the creation of checklists, the grading of these checklists by lightweight LLMs, and the reweighting of checklist items align with the supervised annota- tions. Our experiments carried out on the automated evaluation benchmarks, MT- BENCH and WILDBENCH datasets, reveal that RocketEval, when using Gemma- 2-2B as the judge, achieves a high correlation (0.965) with human preferences, which is comparable to GPT-4o. Moreover, RocketEval provides a cost reduction exceeding 50-fold for large-scale evaluation and comparison scenarios. Our code is available https: / http//ggtthtuubet.e com http:///.c.oooooo///////eeoeeeee
ICLR.
INTRODUCTION
Why is automated LLM evaluation necessary? In recent years, the progress in large language models (LLMs) has been remarkable (Jiang et al., 2024a; Team al., 2024; Yang et al., 2024), driven by continuous technological advancements. The rapid emergence of new models and techniques has broadened their applications. encompassing both general-purpose textual and visual LLMs, well as those fine-tuned for specific tasks in various domains. These LLMs exhibit range of capabili- ties and performances across different application scenarios. Therefore, evaluating their capabilities effectively has become crucial for guiding their development. Since most tasks performed by LLMs involve human interaction, human preferences are often considered the gold standard for LLM eval- uation (Zheng et al., 2023). Currently, crowd-soureing platforms like CHATBOT ARENA (Chiang et al., 2024) collect a significant number of human votes to evaluate LLMs. However, this approach relies on extensive and long-term human annotation, which is costly and challenging reproduce and interpret (Ni et al., 2024). Considering the typical applications of L.LM evaluation today, which
"Equal Contribution.
Work done during internship Tencent.
1 "Coresponning Authors: Ruizhi Qiao and Jianghong Ma.
Unless otherwise stated, all names of I.I.M in this paper refer their "Instruct" "Chat" versions.

---

Published as a conference paper at ICLR 2025
Agreement with human %
Misrral Qwwn1:551
GLM
Pri-Semddummee
Lama Lamaa3-706
Owend 728
Qwen2.77
Intem 208
33
MM277
Pry Phy-3-mmal-l8
1 1amm-33 10000100000
80
U
98
2.55
Ag (Cr
Ag Sgrecmant/Ouraa
Qweri 5-16
Qwenz25
Humann (54 (5.3%)
Qwren2.5 0.58
Rar Kandem (8833))
10
15
20
25
30
35
40
45
OpenLLM LeaderBoard Average
Figure I: Agreements with MT-BENCII HUMAN JUDGMENTS with different LLM 'judges. "CoT" indicates the judgments derived using the original Chain-of-thought (CoT) prompting, and "Ours" indicates the judgments derived using our proposed RocketEval framework.
include validating the effectiveness of LLM development and assisting users in selecting the best- performing models for specific tasks, there is a growing demand for more efficient, reproducible, and interpretable evaluations of LLMs. This demand has led to the proposal of various methods (Wang et al., 2024b; Kim et al., 2024a) and benchmarks (Lin et al., 2025; Zheng et al., 2023) aimed at achieving reliable and efficient automated LLM evaluation.
Pros and cons of the existing automated LLM evaluation paths. Automated LLM evaluation methods can generally be categorized into three primary types:
Mhhiipee-Choice Questions (MCQ) and Keyword Matching Benchmarks. These methods evaluate the accuracy of model's responses by designing series of closed-ended questions and comparing the model's answers to a predefined standard ground truth. Benchmarks constructed using this approach have proven effective for quickly assessing model's capabilities across several tasks, such as reasoning (Zellers et al., 2019; Qiu et al., 2020), comprehension (Mihaylov et al., 2018; Liu al.. 2024), and knowledge retention (Hendrycks al.. 2021; Liang et al., 2023). However, the requirement for specific response formats limits the comprehensiveness of this evaluation method. In practice, most LLM applications involve response styles that differ significantly from simple choices and keywords. Notably, many open-ended tasks cannot be judged by fixed ground truth, which strictly limits the applicability of this approach.
LLM-as-a-ludge Early studies have attempted to adopt language models in automated evaluation (Zhang* et al., 2020; Yuan et al., 2021). Given the robust generalization capabilities of LLMs, employing powerful LLM an evaluator has emerged as viable solution. This method typically involves prompting an LLM to serve as a judge, evaluating the responses from different LLMs to set of well-designed queries. A crucial prerequisite for L.LM-as-a-Judge is that the L.LM must possess sufficient capability to fully comprehend the queries and discern the quality of different responses. Consequently, many existing benchmarks (Zheng al., 2023; Dubois et al., 2023; Li et al., 2024a; Liu et al., 2023) tend to employ the most powerful proprietary LLMs, such as GPT-4o, as the judge. However, the use of these models for evaluation not only incurs high costs but also raises other issues such as reproducibility and data privacy.
Fine-Tuned Judge Models. These models are designed to address the limitations associated with powerful LLM-as-a-Judde approaches. Fine-tuned judge models, derived from lightweight base models, are trained with high-quality evaluation data to more closely align with human preferences (Jiang et al., 2024b; Zhu et al., 2025; Wang et al., 2024b; Kim et al., 2024a). Compared to pro- prietary LLMs, such fine-tuned judge models exhibit competitive evaluation capabilities in a more transparent and cost-effective manner. However, simply fine-tuning the model on evaluation task data may degrade other capabilities of the lightweight model, which are already weaker compared to those of powerful proprietary LLMs. This degradation result in the model failing to correctly understand complex instructions in the queries, thereby deteriorating subsequent evaluation perfor- mance (Huang al., 2024). Additionally, as the capabilities of the base models rapidly evolve and
2

---

Published as a conference paper at ICLR 2025
Checklist Generation
Inslude
Judigment
undese "Jan 2629"7
2.25 "Ma":
the resporss esrrently
foreatted as J508?
Query with History
Dosa the rasponse provide
analyste?
Checkist
Probabilitty
A (Thoought' ... "nooee";; "Jan 2029"3
What is the cato the 15th day aftter
0 Thought: The 15th in 2825-1-9.
Thánk step Ansaen on with containing "Thought" "Anower",
Caristaes
("Thought* AArave": 22224-12-327
Score
(teeponse)
9/28
Alignment
Responses
Annotations
Figure 2: Illustration of RocketEval framework for automated LLM evaluation. The framework consists of three components: Checklist Creation, Checklist Grading and Score Prediction.
new data continuously emerges, these judge models may need iterative updates, leading to signifi- cant cost escalations and reproducibility issues.
RocketEval: Towards efficient automated LLM evaluation. Building the aforementioned concepts, we find a pathway that synergizes the language modeling and human preference align- ment capabilities of the most powerful LLMs with the evaluation efficiency of lightweight models. Specifically, we conduct a thorough analysis of the evaluation capabilities of various large models and observe the following:
1. The agreement between LLM judges and humans is significantly influenced by the inherent capabilities of the LLMs. More powerful LLMs tend to generate evaluation results that more accurately reflect human preferences.
2. Prompt engineering techniques, such as Chain-of-Thoughe (CoT), exert minimal impact on the evaluation capabilities of the model, particularly when lightweight LLM is employed as the judge. High uncertainty and positional bias during the decoding process are potential contributing factors to this phenomenon.
Inspired by these observations, we introduce novel evaluation framework named RocketEval, de- signed to meet the demands of evaluation scenarios that require high efficiency, low cost, alignment with human preferences, reproducibility, and interpretability. As illustrated in Figure 2, RocketE- val operates through a three-stage framework to generate evaluations. Initially, an instance-level checklist is created, providing essential knowledge and critical focus areas to overcome the limi- tations of lightweight LLMs in constructing analyses. Subsequently, lightweight LLMs assess the quality of responses for each checklist item independently, resulting a multifaceted and unbi- ased judgment. The normalized score of each judgment is then collected and aggregated to predict the final score, aiming to mitigate the uncertainty associated with lightweight LLMs. Considering the widespread availability of human annotations across various evaluation scenarios, further introduce a supervised prediction process to align the scores from lightweight L.MS with these annotations. Experimental results demonstrate that RocketEval significantly enhances agreement with human judgments, achieving remarkable Spearman correlation of 0.965 when utilizing the Gemma-2-4 2-2B model as the judge to rank test models. This offers comparable solution GPT- 4o at only 2% of the evaluation cost in large-scale evaluation scenarios, rendering it suitable for performing efficient, reliable, and reproducible LLM evaluations.
How LIGHTWEIGHT LLMS PERFORM AS A JUDGE?
In this section, we verify the capability of lightweight LLMs in automated evaluation by conducting series of experiments employing various L.M judges.
2.1 SETUP
We selected two benchmark datasets for our experiments:
3

---

Published as a conference paper at ICLR 2025
MT-BENCH (Zheng al., 2023) is classic benchmark that includes 80 multi-round queries from multiple domains and uses GPT-4 as the judge for multi-round evaluations.
WILDBENCH (Lin et al., 2025) is newly released benchmark containing 1,024 real-world user queries. WildBench first introduces a manually revised checklist as contextual infor- mation to guide the evaluation and uses GPT-4o as the judge.
Both benchmarks include pairwise comparison and point-wise scoring evaluation methods. Since the results of pairwise comparisons can be derived from comparing the scores derived from point- wise scoring, focus on the point-wise scoring method. In point-wise setting, both benchmarks prompt the judge to first generate an amalysis the response, followed by score in 1-10 as the final judgment. This be seen chain-of-thought (CoT) (Wei al., 2022) process, aimed at enhancing the ability of the LLM on the evaluation task that involves a reasoning process.
2.2
HOw DO LIGHTWEIGHT LLM JUDGES PERFORM?
First, we aim to understand the capabilities of different models when they serve as judges. A key metric for evaluating LLMs evaluation capability in aligning user preferences is their agreement with humans. Here, we measure this agreement using MT-BENCH HUMAN JUDGMENTS (Zheng et al., 2023), which provides human- annotated results for MT-BENCE across six test models. Each sample includes a target query, responses from two LLMs, and suman-annotated match results (in- cluding ties). We structure the scores obtained in point-wise manner into the same format by comparing the scores of each pair of responses. Unlike the previous study (Kim et al., 2024a) that uses sampling decoding to derive results without ties, we follow the setting of Lin et al. (2025) and retain cases with scores difference smaller than 0.1 as ties. We then compared the agreement of different LLM judges with human annotations. The human-human agreement reported by Zheng et al. (2023) and the agreement between GPT-4o and human are listed as the baseline. As illus- trated in Figure 1, agreement between LLMs and humans typically ranges from human-to-hume agreement (64.8%) to lower than random outcomes (33.3%). This suggests that the existing evalua- tion method using CoT scoring imposes significant demands on the judges' abilities. The ability of LLMs. reflected by OPENLLM 2 (Fourrier et al., 2024) scores, significantly impacts its alignment with human preferences during evaluation, thereby complicating the process of performing efficient and reliable evaluations with lightweight LLMs.
2.3 WHERE DO LIGHTWEIGHT LLMS UNDERPERFORM AS A JUDGE?
In this part, our objective is to delve deeper and identify the key components that affect the judg- ment of lightweight LLMs when they serve as judges. We start by conducting an analysis of the two processes involved in evaluation: analysis generation and scoring. To determine whether a compre- hensive analysis can boost or degrade the performance of judgment, we instruct the LLM judges to score the responses under three different settings:
CoT: The original Mhain-of-Thougee style, generating the analysis and score step-by-step.
Direct: The judge is prompted to skip the analysis step and score the response directly.
CoTGPT44:
We extract the analysis generated by GPT-4o to replace the part in judge model's output, then prompt the judge model to score the response accordingly.
Then, the scores of all responses are averaged to derive the score of the tested model for ranking. For baselines, we derive the score predicted by GPT-4o and Elaude-3.5-Some as judges equipped with COT. We use the CHATBOT ARENA ELO RATINGS (Hard- English) (Chiang et al., 2024) as the ground-truth rankings of human preferences in these models. Figure 3 shows the scores of 12 test models and the Spearman correlation coefficient of ranking lists with GPT-4o when different LLMs served as judge. Also, we convert the scores into pairwise comparison results and com- pare the agreement of different LLM judges with strong baselines. The result is shown in Table 1. Compared to direct scoring, the process of lightweight models conducting their own analysis did not yield significant gains. However, utilizing analysis from powerful LLM as the context sig- nificantly improves the evaluation performance in terms of instance-level agreement and list-level correlations. This indicates that lightweight judges capable of calibrating their judgments with
4

---

Published as a conference paper at ICLR 2025
Table 1: Agreements of different judges on WILDBENCH.
Analysis + Score
Only Score
GPT-4o Analysis Score
Scoee
Agreement with GPT-4o
Direct
CoT
CoTarr
6,89 6.07
C205
6J7 633
8379
Clando-3.5-Somme
60.7%
11
Model Ranking in Arena do (Hard-EN)
11
11
Qwen2. 1,5B
Qwen2-72 722
Owen2: 2
40.1%
43.19
61.0%
45.0%
61.4%
36.4%
76.6%
70.3%
74.0%
Ilama-3.8 3-8
46.7%
48.4%
74.0%
L Llama-3-700 708
57.2%
59.3%
74.9%
Agreement with Clande- 2laude-3.5-Sonnet
Figure 3: WILDBENCH scores predicted by different LLM judges and the ranking correlation with GPT-4o.
GPT-lo
60.7%
Qwen2. 1,5B
40.6%
36.4%
58.2% 2%
Owen2: 71
42.8%
45.3%
60.3%
Owen2: 72B
61.5%
62.3%
62.9%
Llana: 3
48.25
49.2%
0%
L Llama-33 3- 5-70
58.3%
62.0%
63.4%
powerful LLM when high-quality and comprehensive analysis is provided. In other words, the abil- ity of such lightweight LLM judges is limited mostly due to the poor comprehension and analysis process when they dealing with hard queries and complicated responses.
2.4 WHY DO LIGHTWEIGHT LLMs NOT GOOD AT JUDGING?
In the evaluation scenario, conducting analysis can be viewed as a series of judgments on the target response. In order explore the reasons why the lightweight model are struggling the conducting analysis, we further conduct a fine-grained level experiments. Specifically, we transform the process of analyzing into a process of judging on series of checklist questions. We treat each item in the checklist provided in WILDBENCH as a independent question, and prompt the LLM judge to make decisions. Since these questions can be viewed as binary choice questions (for example, "Does the response correctly identify..." "), the judges are asked to simply output "Yes" or "No" for each question based on content the query and response. Then, we obtain judgments of different LLM judges on the checklist questions through repeated sampling. and calculate the ratio of disagreement in dif- ferent sampling results among all checklist questions. Figure shows that the ratio of disagreement varies significantly across different model sizes. Lightweight LLM such Qwen-2-1155 shows disagreement ratio exceeding 50% across sampling results. This indicates a high uncertainty in making decisions on these checklist questions.
Ratio disigremment
Ubmo
Uama3-700
0.6
llams-388
Qwend-72B
Uama-3-08
Qwend-/1
0.4
Qwen2-1
of
Qwen2: 1.58
0.0
0.0
Number sampling times
Position of the checklit Item
Figure 4: Ratio of disagreement on WILDBENCH Figure 5: Ratio of disagreement on WILDBENCH
with different number of sampling times.
with checklist items in different positions.
When lightweight LLM judges conduct the CoT style analysis, this high uncertainty may lead to greater deviations in the final scoring results and degrade the performance. Although previous stud- ies (Li et al., 2024b; Wang et al., 2024a) has demonstrated bias of response order in pairwise compar- isons, bias in the analytical process remains unexplored. Therefore, we want to understand whether this form can affect the models' judgment results. We transform the item-level questions from the previous step into a multi-turn dialogue format, with the sequence consistent with the original check- list order. To show the impact of different judgment results, set all previous judgments to "Yes" the current item's 's judgment results under the two
or "No" and then compare the disagreements settings. As shown in Figure 5, with an increase in the number of previous questions, the proportion of inconsistencies in all models shows a growing trend. Interestingly, the Llama-3 series show an overall higher consistency compared to the Qwen-2 series. Additionally, notice correlation
5

---

Published as a conference paper at ICLR 2025
between model size and consistency, with smaller models tending to produce higher inconsistency. This suggests that the uncertainty of lightweight LLMs is more likely to be amplified in the process of setting up sequential analyses with CoT, thus reducing confidence in automated evaluation.
From the above analysis, we demonstrate that lightweight LLM judges exhibit high uncertainty and position bias, which can lead to difficulties in making reliable judgments. A feasible and efficient evaluation method should avoid the above defects, which inspired us to propose the RocketEval.
METHODOLOGY
In this section, based on the analysis on existing evaluation benchmarks, we introduce a new auto- mated LLM evaluation framework named RocketEval. As shown in Figure 2, the entire RocketEval framework can be divided into three stages. First, we employ powerful LLM such GPT-4o to conduct meta-anale meta-analysis and create a checklist for assessing user query. Subsequently, lightweight LLMs are employed to evaluate the checklist items for responses from each test model. Finally, the evaluations for each item are collected to derive the final score, which can be predicted using either an unsupervised arithmetic mean or a supervised predictor learned from annotations.
3.1 CHECKLIST CREATION
Evaluating LLM responses is challenging for both human evaluators and LLM judges. Human evaluations can be subjective, while LLM judges may struggle with query understanding. detailed analysis, and interpretation, especially when the lightweight LLMs are employed the judge. Ex- isting methods improve the accuracy of LLM judgments by providing additional information to the LLM Judge's s analytical process, such as reference answers (Zheng al., 2023; Dubois et al., 2023) and hand-crafted rubrics (Kim et al., 2024a). However, reference answers are of limited use in open- ended queries, while hand-crafted rubrics face trade-off between labor cost, generalizability, and accuracy. Therefore, here we follow WILDBENCHI (Lin et al., 2025) to create a instance-specific checklist, guiding judge LLMs in evaluation. The checklist items are expected to have the follow- ing characteristics: 1) Relevance to the topic of the query. 2) Capability to effectively distinguish between different responses. 3) Independence from each other, as independent questions can ensure complete evaluation of the response's 's quality. Essentially, checklist questions can be considered as a distillation of knowledge from powerful L.LMs to prompt the lightweight judges. question like "Does the response include the correct reasoning steps/final answer as X?" can be helpful when lightweight LLMs are struggling to identify all key factors or solve the problem by itself. We em- ploy GPT-4o the checklist creator, with 5-10 questions created for each instance. This process only needs to be executed once, while the checklist can be leveraged by lightweight LLM judge to evaluate any number of responses. The prompts used shown in Appendix A.
3.2 CHECKLIST GRADING
In Section 2.3, we examine the limitations of employing lightweight LLMs as judges, particularly the issues of high uncertainty and positional bias. To address these challenges, we propose the following two evaluation procedures when using lightweight LLMs as the judge.
Independent Checklist Item Judgment. To avoid interference from the judgments on other check- list questions, we prompt the I.LM judge to evaluate each question in the checklist independently. While this method requires multiple queries per instance, the computational cost can be significantly reduced by leveraging prefix caching (Zheng et al., 2024) since they share the same prefix.
Normalized Score. Given the high uncertainty associated with lightweight LLMs, relying solely on binary outcomes such as "Yes" or "No" can result in significant errors in the final judgment. To mitigate this error, we introduce the conditional normalized score the basis for judgment, which accounts for the certainty of the result. Assuming the probability of output token / from judge L.MM parameterized by 0 with context x is (o(())), with the target query with context x and the corresponding response y. Then the conditional normalized score checklist item c is defined as:
Po moeYs a, y, c)
f(x y: c)
Po Po(Yes x y, c) + Po (No y, c)
In this manner, the judgments with low certainty have less significant impact the final judgment.

---

Published as a conference paper at ICLR 2025
3.3 SCORE PREDICTION
After obtaining all judgments for single query-response instance, by the normalized scores of all checklist items. Given the checklist с = [C1, C22 ..., Cy] E C and the normalized scores p- [ū(xr, y. ci ). f(r, y, C2), pla..,,(a, pa. y, ex)] )] E P. The score predictor + R can predict the score by s = f(p). This methodology ensures a more reliable and accurate evaluation by addressing the inherent uncertainties and biases in lightweight LLM judgments. The score predictor can be any statistical method, hand-crafted rules, or machine learning models. For simplicity, here we use the arithmetic mean of all normalized scores as the score Sunsupi
predict the final score
N Σ
8 uusuup
pl y, c)).
This method does not require additional data or effort obtain the predictor. However, checklist items may have varying impacts on the final results. In many LLM evaluation scenarios, such as LLM development, the benchmark data often comes with annotations from humans and powerful LLMs that serve as the gold standard for evaluation. In this context, we can utilize these annotations to align our checklist judgment results with supervised learning. Specifically, we consider the judg- labels. Given a judgment set with
ments on the checklist items as features and the annotations N samples, with features P E PN and labels r E IR a predictor feup = ming ((: P,r) can be derived by minimizing any loss function L. The predictor can be any classification or regression model, depending on the type of annotations. In this case, we select the Extremely Randomized Tree (Geurts et al., 2006) estimator to learn robust ensemble predictor with limited number of annotations and unknown distributions of the judgment results.
Meanwhile, we have observed that queries may not always yield ideal separable annotations for predictor learning- Some queries may be too easy, too hard, or have vague descriptions, resulting in similar good bad performance across all test models. This cause significant performance degradation when learning a supervised predictor. Therefore, we propose strategy to adjust the im- pact of the scores predicted by the supervised predictor based on the distribution of the annotations. Specifically, given the annotations E IIPi, define the weight factor Oip
[L(PPr||
ar
max LL( X-RN
(3)
where e is the maximum Kullback-Leible (KL) divergence of any distribution X from the ideal distribution Pideat- In existing work (Kim et al., 2024a; Murugadoss et al., 2024), rubrics have been used as a reference in evaluation, including examples or standards for different rating levels. Therefore, expect the annotated scores to be varied different levels, providing the rubrics for the predictor, Hence, we use the uniform distribution across the score range (for example, 1-10) as Pideal U.N
After deriving the weight factor a, and a fitted predictor fsup for each query, the final score assigned to the corresponding response is
Ssup = (1 Oky Oameup + Ory S (p).
(4)
This methodology ensures a more reliable and accurate evaluation by addressing the inherent uncer- tainties and biases in the supervised learning process.
4 EXPERIMENTS
In this section, we evaluate the effectiveness of the RocketEval framework for automated evaluation of LLMs. Initially, analyze how well RocketEval aligns with human preferences both the in- stance and the list levels. Next, we investigate the expenses ussociated with the evaluation procedure. Lastly, we perform analysis of the checklist's content and its impact on the evaluation.
7

---

Published as a conference paper at ICLR 2025
4.1 HUMAN AGREEMENT ON THE EVALUATION
The initial step involves comparing the proposed RocketEval framework with the traditional evalu- ation method, both with and without the inclusion of Chain of Thought (CoT). The idea of adapting checklist or aspect to enhance the robustness in LLM evaluation has been widely adopted in existing works (Lee et al., 2024; Zhou al., 2024; Fu et al., 2024), where most of them come with fixed human-curated lists to evaluate responses. To further validate the impact of the instance- check- list, we introduce a baseline employing six fixed questions as the checklist. These questions, derived by analyzing the MT-BENCII (Zheng et al., 2023) evaluation prompt, encompass dimensions such as helpfulness, relevance, accuracy. depth, creativity, and detail. This baseline is henceforth referred to as the "Fixed". Details of the experimental setup are provided in Appendix A.3.
Instance-level Agreement. To measure the agreement ratio of different automated evaluation meth- ods with MT-BENCH HUMAN JUDGMENTS, we adhere to the settings outlined in Section 2.2. As illustrated Table 2, the proposed method consistently enhances agreement with human judgment across various LLMs acting as evaluators. Notably, the latest 7-12B parameter lightweight LLMs, such as Llama-3-8B and Mistral-Nemo, achieve over 64% agreement with human judgments and at- taining similar level of agreement 70B open-source LLMs and human- toantttthumann agreement. For smaller-sized LLMs, which initially exhibit agreement accuracy close to random choice, RocketE- val significantly improves performance to over 60%, outperforming GPT-4. Conversely, the fixed checklist shows no notable performance enhancement and may even degrade performance. This suggests that checklist questions tailored to the specific query topic can better provide knowledge context, thereby enhancing the performance of lightweight LLMs as judges.
Table Agreement ratios of different LLM judges with WTT-BENCH HUMAN JUDGMENTS.
RecketEval (lup)
Nomo
Lima-5-85
Philimin-44 4k
Qwan2.5-3
Method
Col
Direct
Fixed
Ours (Unsup.) Ours (Sup.)
CoT
27
amm-32330
Baseline
GPT-4 GPT-4Painwwsee> (Painwise): 65.8%
Human-to human: humman- 64.7%
GPT-4 (Single): 59.6% 4P 66.6%
Intem/00 (0..m00..-1... 5-1.00
Promccheus-7B-V2.0 55.7%
Qwan2.5-122 50
Llama-3-208
60.4%
64.8%
0wer3.5-0.0
Qwen2-728
Mistral Nemo
57.8%
53.0%
53.1%
63.2%
64.2%
Ilama 3.88
55 66h
51.4%
46.9%
63.8%
62 0%
Owen2-7i 7B3
54.7%
47.6%
42.1%
58.6%
50.8%
Mistral-7B-v0.3
55.3%
54.4%
42.8%
58.8%
58.3%
Phi-3-mini-44
52.8%
13.8%
42.1%
61.2%
60.9%
Owen 5.3
49.8%
52.2%
15.6%
57.4%
SK.7 %
Figure 6: Spearman correlation (in percentage) of test model rankings on WILDBENCH across different LLM judges.
Llama 3,2 3B
53 25
26.6%
S4 14.1%
58.6%
SW.8
Gomma-2-28 2-28
37 9.
39 2%
43.9%
$7.9%
10
M22. 1.813
29.2%
22.3%
38.4%
51.7%
48.4%
Owen2 5-1.5B
31.1%
25.1%
46 5%
60.7%
60.2%
Owen2 1,58
50.4%
21.7%
41.0%
56.2%
55.6%
Tlama- 3.2. IBB
32.6%
12.5%
36 5%
13.6%
41.9%
Qwee2 Qwea2.5-0.5B 5-0.5B
24.9%
25.1%
40.8%
54.3%
50.9%
List-level Correlation. A critical objective of LLM evaluation is to compare the performance of LLM in specific general scenarios, which makes the final ranking of LLM essential. Therefore, we conduct the experiment to compare the LLM rankings derived from different approaches. The de- tailed setup and results are elaborated in Appendix A.3. Table 3 presents the correlation coefficients of the score rankings on WILDBENCH with the CHATBOT ARENA ELO RATING (Hard Prompt En- glish). The table clearly indicates that RocketEval significanlyy improves the quality of by achieving a higher correlation with human annotations. Specifically, when Mistral-Nema is used as a judge, the Spearman correlation reaches 0.986, surpassing GPT-40. The smaller Gemma-2-26 also achieves a correlation of 0.965, surpassing Qwen2-728 and the fine-tuned judge model Prometheus-7B- -B-vv2. (Kim et al., 2024b). Furthermore, the supervised version of RocketEval achieves higher correlations compared to the unsupervised version. Compared to instance-level agreement, the supervised score data provides more significant improvements in list- correlation. In addition to the agreement with humans. compare the correlations between different lightweight LLM judges. As shown in Figure 6, RvcketEval brings significant improvements in cross- judge correlation, especially on the lightweight L.LMs. It suggests RocketEval exhibits high consistency and reliability in evaluation.
8

---

Published as a conference paper at ICLR 2025
Table 3: Correlation of ranking with CHATBOT ARENA ELO RATING (Hard prompts-English) on WILDBENCH dataset. "Kend." and "Spea." " are denoted Kendall's Tau and Spearman correlation coefficient respectively.
Method
CoTT
Direct
Fixed
Ours (U (Unsup.)
Ours (Sup.)
Coellicient
Kend.
Kend.
Spea.
Kend.
Spea.
Kend.
Spea.
Kend.
Spen
GPT-4o
0.909
0.979
Prometheus-7B-v20
0.848
0.949
L-70a-3-008 Qwen2-72B
0,909
0.979
0.787
0.923
0.848
0.958
0.848
0.958
Mistral-Nema
0,870
0.956
0.788
0.909
0.879
0.958
0.933
0.986
0.939
0.986
Jamm. 1-3B
0.818
0.923
0.848
0.944
0.879
0.972
0.909
0.979
01.009
0.979
Owen2-78 Mistral-7B-v0.3
0.788
0.909
0.545
0.727
0.636
0.804
0.758
0.895
0.818
0.930
0.727
0.895
0.545
0.699
0.515
0.678
0.758
0.874
0.818
0.930
Phi 3 3-i-3- 4k
0.424
0.587
0.576
0.762
0.182
0.273
0. 0.788
0.916
0.848
0.958
Owen2.5. 11
0.697
0.839
0.848
0.951
0.727
0.895
0.848
0.944
01848
0.944
Clama 3.2 2-3
0.606
0.797
0.848
0:951
0.188
0.917
0 0.848
0.944
018488
0.944
Gemmaa 2.38
0.636
D.818
0.758
0.888
0.727
0.677
0.879
0.965
0.879
0.965
Internl M2ternl.M2.5-J1. M2.5-1.88
0.121
0.175
0.273
0.357
0.273
0.427
0.576
0.748
0.606
Owen2.5 1oun2.5-1..
0.394
0.517
0.273
0.364
0.606
0.727
0.818
0.923
0.848
0.944
Owen2-1 5B
-0.066
0.035
0.303
0,420
00061
-0.014
0.455
0.622
0.667
0.825
Clama 1.2.
0.091
0.133
0.182
0.182
.1273
.0.357
.0.273
.0.357
0.697
0.846
Qwen2.5 5-0.5B
-0.212
-0.315
0.424
0.303
0.394
0.587
0.667
0.811
0.758
0.895
4.2 EVALUATION COST ESTIMATION
In this section, we analyze the costs of various evaluation methods when different LLMs serve as judges. For the purpose of this analysis, assume that all responses required for evaluation are pre-generated, thereby excluding the inference costs for generating these responses.
In RocketEval, the evaluation process is comprised of primary components: checklist genera- tion and checklist grading. The supervised evaluation method incorporates an additional fitting and prediction process, whose costs are minimal since no LLM inference is involved. For proprietary LLMs serving as judges, we calculate the number of input and output tokens required for a single evaluation and derive the cost based on the official pricing 2. For open-source LLMs, we deploy them on NVIDIA RTX A5000 GPUs using vLLM (Kwon et al., 2023), and the cost is calculated based on the average execution time and the rental price of the GPU. We reference the pricing on RunPod at $0.36 per hour. To maximize efficiency, experiments are conducted in batch mode.
In practical evaluation scenarios, various L.Mss, each with distinct tuning options, decoding settings, and prompting techniques, can yield numerous versions of responses. Consequently, evaluations on the same benchmark can be performed hundreds or even thousands of times across different models and their respective versions. We therefore compare the cost of LLM evaluation methods with different number of tests N. As shown in Table 4, the cost incurred for generating checklist for each question is equivalent to the expense of rumning single test using GPT-4o. Since the checklist generation is a one-time process, the cost of the checklist grading process becomes increasingly significant as the number of tests escalates. For instance, conducting 1000 tests the WILDBEENH would incur cost of $3400 when using GPT-4o as the evaluator, whereas employing Llama-3-8B would only require $71, with achieving higher correlation with human preferences.
4.3 QUALITATIVE ANALYSIS
Checklist Statistical Analysis. To discern the distribution of checklist items, we extracted the re- lationships within the knowledge graphs associated with each checklist item, including the subject, predicate, and object, in a format consistent with the methodology introduced by Sun et (2024). We conducted statistical analysis of all predicates in the checklist items generated from WILD- BENCH (all predicates have been lemmatized). Further statistics are provided Appendix A.5. As shown in Figure 7, the predicate distribution within checklist items reflects the intrinsic demands of various original questions. Predominant predicates such as "Include" and "Provide" underscore the necessity for comprehensive and supportive responses, ensuring that all pertinent information is considered. This is crucial for addressing the complex and multifaceted nature of questions across various general tasks. Meanwhile, predicates such "Calculate" and "Specify" highlight the pre-
"https//opemaaa.oo/aappppiiiiiip https:/wwww.rumpod..oom

---

Published as a conference paper at ICLR 2025
Table 4: Evaluation cost on WILDBENNC with different LLM judges.
Method
LLM Judge
Deploying
Price
Usage for Single Test
Extra Cost
N=10
N=100
N=1000
GPT-4o
IO: $1.25 $5.000
I0::
$34.0
$340
$3400
CoT
(202-0086)
(201240718)
GPI-9o-mini
proprictary
1O: S0,075 $0.30
1M Tokens
1M Tokeus
1.84M/220K
tokens
N/A
$2.00
$20.0
S200
Llamu- 3- 70B
4 A,5000
$1.44/ bours
3760s
$15.1
S125
S1224
RocketEval
Llama-3-44 8B
1 x X-0000
$0.36 bours
685s 248%
$2.87*
$3.55
59.72
S71.4
Gemma-2-2B
$3.12
$5.35
S27.7
Qwen2.55 5-155B
1 A,5000
$0.36 / bours
16.5%
$3.04
$4.52
S19.4
*The checklist generation process on WILDBENCH consumes 1.38M input tokens and 228k output tolions GPT-4o.
cision required in quantitative and advisory responses. This distribution pattern not only guides the assessment of answer quality but also ensures that responses meet the specific criteria of each ques- tion type, thereby enhancing the overall reliability and applicability of the information conveyed.
Checklist Item Reweighting. Adjusting the weights of checklist items is crucial for accurately assessing the effectiveness of responses to the original question, especially when gold standard is available. Initially, each checklist item was given equal weight, assuming an equal impact on the overall assessment. However, this does not accurately reflect the true importance of each item in validating the response's s accuracy and completeness. As shown in Figure 8, Assigning the value of radial stress in item I (0.296) is critical, being core part of the answer, significantly impacts subsequent calculations and the overall analysis. In contrast, item 3 (0.069), which calculates the inner radius, fundamental but simple, less prone to error, and less critical than stress calculations, thus bearing lower weight. By reweighting, the accurate responses, ensuring that the evaluation process is both rigorous and reflective of the actual importance of each component, which leads to more reliable and credible results, ultimately enhancing the overall validity and consistency with the gold standard of the assessment.
emphasize the steps that are the most crucial to
Bisritte reparso comedyyrattiin wirmine
eapeco de duagg in wdlute (XV) ialiaaaee bo
Frovide
Include
Doss the poppense mertee the necouary calcalanos
dhe tedill
Gor enpeelieed begie 5ah
of 160 ren and sull thidones 7
15%
linderia adiested an atenal prseecaft Milk
and modilae or daamary (30 riiaar ana
cytadd les Poosert's selo of
Others
10h
CITy. Desarnave
reperse oppy pridip convely
Arts
Theiafal stese the es lollede
tangretial
volensire simre and change
Mention
Spectty
Deer the reppese comeetly skeiey and
intorel prosus
Calculace
Describe
Suggest
Use
decommet iurman 3 yoe poends somee
sompar cakoiae angesial drea
Address
Biecuus
Bove Do veyerw adittio the yulll cyoss (or) sang Bi: correct fomrda lagraa gm and prevde 4256
valce
Figure 7: The predicates used in Figure 8: Visualization of checklist item reweighting. (De- fault weight is the reciprocal of the number of checklists)
checklist items on WI.DBENCH.
CONCLUSION
In this paper, we introduce RocketEval, an innovative evaluation framework that uses lightweight LLMs to achieve high efficiency, low cost, interpretability, and alignment with human preferences. By reframing the evaluation task as a multi-faceted Q&A using instance-specific checklists, we ad- dressed the challenges of high uncertainty and positional bias inherent in lightweight LLMs. Our method demonstrated high correlation with human preferences, achieving Spearman correla- tion of 0.965 with the Gemma-2- 2-2B model, comparable to GPT-4o, but fraction of the cost. This significant cost reduction makes EvaketEval a feasible solution for large-scale evaluation and comparison scenarios.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partially supported by the National Natural Science Foundation China (Project No. 62202122 and No. 62073272), the Shenzhen Science and Technology Program under Grant No. 1XWD202311001000 13011030800, and the Guangdong Basic and Applied Basic Research Foundation under Grant No. 2024A151501149,
REFERENCES
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang. Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: open platform for evaluating Ilms by human preference. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMI.Rorg, 2024. URL https://dd. acm. http://r//////bb/////
00555/369070-22 36924001.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori В Hashimoto. Alpacafarm: A simu- lation framework for methods that learn from human feedback. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp- 10039-30069 Curran Associates, / Inc., 2023. URL http::///pocednnn.nnniissic/caap httcc/paper_flls/ppppeeeeeeee fwwess/ppprr/223//// /tt////0eee00077770000000000 -awpraaaeoaaaaaaaaaaaaa----ooooe Conferencecerrrccpccc pdf.
wwwrips.cc/papppp..e Paper-
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open Ilm leaderboard v2. https://huggingg wtw.ice..cccppceee copspacessopenn- Ulm leade rooardd/oo//pen llm_. leaderboard, 2024.
Jinlan Fu, See- Bee-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. GPTScore: Evaluate you desire. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6556-6576, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18633/1/20244macllooo 365. URL https://aclanthon
org//0244 nww.l-aac---aaaa 365/.
Pierre Geurts, Damien Ernst, and I Louis Wehenkel. Extremely randomized trees. Machine Learning, 63(1):3-42, Apr 2006. ISSN 1573-0565. doi: 10.10077/1109944000---226 URL / org/10. 1001000/11990 006 6226-1.
Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proceedings of the Interna- tional Conference on Learning Representations (ICIR), 2021.
Hui Huang, Yingqi Qu, Hongli Zhou, Jing Liu, Muyun Yang. Bing Xu, and Tiejun Zhao. On the limitations of fine-tuned judge models for 1lm evaluation, 2024. URL h/tps:
Albert Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gi- anna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William Sayed. Mixtral of experts, 2024a. URL https: www.i..oggg...ooo absg///b///a 2401. 22401.00000 04088.
Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. TIGER- Score: Towards building explainable metric for all text generation tasks. Transactions on Ma- chine Learning Research, 2024b. ISSN 2835-8856. URL id=EE1CBKC0SZ.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thome, and Minjoon Sco. Prometheus: Inducing fine- grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, 2024a. Utt..ttps./peerrreee./////oooeed

---

Published as a conference paper at ICLR 2025
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Gra- ham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pp. Association for Computational Linguistics. doi: 10. 16654////022.c.mmmmmmmo https://aclanthonn
Miami, Florida, USA, November 2024h.
URL
org/2024. emnlp- momml--main224 248/
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princi- ples, SOSP 23, pp. 611-626, New York, NY, USA, 2023, Association for Computing Machinery. ISBN 9798400702297. doi: 04144556600006636331655 3613115.
URL httt:::/ddoi.rrg/// 1145/
Yukyung Lee, Joonghoon Kim, Jaehee Kim, Hyowon Cho, and Pilsung Kang. Checkeval: Robust evaluation framework using large language model via checklist, 2024. URL https: hrg/abs/2400) 18771.
Tianle Li, Wei-Lin Chiang. Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gon- zalez, and Ion Stoica. From crowdsourced data to high- benchmarks: Arena-hard and benchbuilder pipeline, 2024a. URL https://///v org/ abs///0/.//22/.////////// 11939.
Zongjie Li, Chaozheng Wang. Pingchuan Ma, Daoyuan Wu, Shuai Wang. Cuiyun Gao, and Yang Liu. Split and merge: Aligning position biases in LLM-based evaluators. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11084-11108, Miami, Florida, USA, November org/2044.
2024b. Association for Computational Linguistics. URL emnlp-man 621.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. URL /ttpee// net/ pdf? Rdf7id:104ZZiEEqM.
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking LLMs with challenging tasks from real users in the wild. In The Thirteenth International Conference on Learning Representa- tions, 2025. URL https:/ http://pppeeeeieeeeeeee
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511-2522, Singapore, December 2023. Association for Computational Lin- URL
guistics. doi: 10. 1121666/11122233mmmp...m. emnlp-mate wat--main...m 153.
Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catan- zaro. ChatQA: Surpassing GPT-4 on conversational QA and RAG. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL httptttt////ooereiiieee htw.://w/epereeeeeee..eeeeee n forum?id==bKuvkKKaft
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor con- duct electricity? new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't iiciichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, Brussels, Belgium, oct nov 2018 Association for Computational Linguistics. doi: 10.18653/v1/D18- 1260. URI. org/D18 1260.
Bhuvanashree Murugadoss, Christian Poelitz, Ian Drosos, Vu Le, Nick McKenna, Carina Suzana Negreanu, Chris Parnin, and Advait Sarkar. Evaluating the evaluator: Measuring Ilms' adherence to task evaluation instructions, 2024. URL https: http:///wwww...o/a//p//////////a 28888088881.. 08781.
12

---

Published as a conference paper at ICLR 2025
Jinjie Ni. Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. Mixeval: Deriving wisdom of the crowd from LLM benchmark mixtures. In The eight Annual Conference Neural Information Processing Systems, 2024. URL https: // //openreview mett/oorumid-6A29200
Zhaopeng Qiu, Xian Wu, and Wei Fan. Automatic distractor generation for multiple choice questions in standard tests. In Proceedings of the 28th International Conference on Computational Linguis- tics, pp. 2096-2106, 2020. URL https:: http://alaaaathooooooooooo . org/ 22000 colling main. 189/.
Kai Sun, Yifan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Hcad-to-tail: How knowledgeable are large language models (Ums)? aka will Ilms replace knowledge graphs? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), pp. 311-325, 2024. URL org/2024. naacl- long. 18/.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Fer- ret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Char- line Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchi- son, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Wein- berger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Slimezak-Plucinste Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz. Josh Newlan, Ju yeong Ji. Kareem Mo- hamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leti- cia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Mar- tins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khat- wani, Natalic Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Os- car Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengehong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samanch Saadat, Sara Mc Carthy, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Ko- cisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei. Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma Improving open language models at a practical size, 2024. URL https: :/ http:/:://////////e/e h0/gg/bb//4088/ 00118.
Harleen
Peiyi Wang. Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compututional Linguistics (Volume I: Long Papers), pp. 9440-9450, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653/v12022. long.511.
www.o////..../// 5cl- 1. URL https:: //acaanho/// wwwph///oooooo../2220/../
Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue
13

---

Published as a conference paper at ICLR 2025
Zhang. PandaLM: An automatic evaluation benchmark for LLM instruction tuning optimiza- tion. In The Twelfth International Conference on Learning Representations, 2024b. URL https://openrevieee http:/ppenreiiiee.ettttoommmmm/m nett/forrummdd-55M2BBL55555
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le. and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language mod- els. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 24824-24837, Curran As- sociates, Inc., 2022. URL. htttts:// httpss:/prrceeedingg nwwrine..ncccmm ccppp.cccppppp. files/paper/2002/ 011/9//550011524cc11111100000100 Paper- Conference.pdf pdf.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang. Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma. Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Siman Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https:: /ampss:/aakcee //tw/vaxvvvvv//v///////////////a org/ at////bs/////// 10671.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text gen- eration. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp- 27263-2727 Curran As- sociates, Inc., 2021. URL htee:///bb6effdccceeeeeeeellaaae
wewrriinnnnnnnen...neeeeeereiere wwturis..ccpaaaaap files/paper//2 wwwpdddd-ppppppppppeeeee pdf.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Tianyi Zhang* Varsha Kishore*, Felix Wu* Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference Learning Representations, 2020. URL https: http::///eeeeeeeeeeeeeee net/iiewwwweeeeeee.ee wttp::/eeeeeeoototeeeeeeee fow.moodd-neeuueeeee
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging lim-as-a-judge with mi-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 46595-46622 Curran Associates, Inc., 2023. URL. httpss://proceedinng neurips cwpppppppppppppccccccccceee..eee fitess/paper/2223/fff.a 91f 1111112277133337777255566642 Paper- Datasets. and Bendd_Benchhsssss pdf.
Lianmin Zheng, Liangsheng Yin. Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Ef- ficient execution of structured language model programs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https::///// http:///pppeevieeeeeeee ew. net/forum?id= VgkAKQibpq.
Zihao Zhou. Shudong Liu, Maizhen Ning. Wei Liu, Jindong Wang, Derek Wong, Xiaowei Huang, Qiufeng Wang, and Kaizhu Huang. Is your model really a good math reasoner? evaluating mathematical reasoning with checklist, 2024. URL https: /tttp:://arxiv wrtp/bs////////////// htttt::///ardd..o//a////////
08733..
Lianghui Zhu, Xinggang Wang, and Xinlong Wang. JudgeLM: Fine-tuned large language models are scalable judges. In The Thirteenth International Conference Learning Representations,
2025. URL
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PROMPTS USED ROCKETEVAL
Prompts for Checklist Creation. The following prompt is designed to generate an instance-level
checklist.
Prompt for Checklist Creation
# Instruction
You helpful assistant who identifies and summarizes key factors in large Language models (LLMs) evaluation help humans evaluate LLMs S ef efficiently.
Feed any query into different LLMs, will get various responses. I need to know in quick whether these responses follows instructions and answers the question the user query better.
I'll provide you with user query. Your task is to identify those key factors that will affect my judgment and summarize them into list improve the efficiency my evaluation.
# Conversation between User and AI of
<|
(history)
<|end of history|>
Current User Query
< <begin_of__uueryl
duser_queryy
<|end que
## Reference Response <|
(refereence_resposse)
<|end of reference response|>
Task
Given the above information, I need you to create a binary question list, so that I can perform an efficient and accurate evaluation through answering several questions.
Your question should be concise and include any necessary key content and information (such as keywords, formats, correct counts and values) in the user query expected to be shown in responses. Your questions should not only consider evaluating the reference response, but all possible responses. Avoid creating duplicate, cumbersome or vague questions. For example, you should ask "Is this response contain the correct answer ..." instead of "Is this response's answer correct?", Ask fewer questions by aggregating questions with repeated contexts into question.
## Output Format
Please provide your outputs in the following markdown format by filling in the placeholders in (())
15

---

Published as a conference paper at ICLR 2025
1. [[question])} [[question]]}
Prompts for Response Judgment. To keep the consistency with the previous work, we adopt the prompt Lin et al. (2025) and make several modifications to adapt to different settings.
For MT-BENCII, to keep consistent with the original paper which utilizes the reference answer to guide the judgment, we replace the part of the checklist with the reference answer and modify the prompt correspondingly.
For other benchmarks that utilize pairwise comparison, we use the same prompt as WILD- BENCH, create the checklist using GPT-4o for each instance those benchmarks.
When prompting LLM output the score directly without conducting analysis, we modify the output format part in the prompt and change the score range from 1-10 0-9. This aims to avoid the ambiguity of scoring or 10 when only the first token is captured.
The prompts with different versions of the modification are listed as follows.
Prompts for Response Judgment
# Instruction
You expert evaluator. Your task is to evaluate the quality of the + responses generated by models.
We will provide you with the user query and an AI-generated responses. You should first read the user query and the conversation history carefully for
analyzing the task, and then evaluate the quality of the responses based and rules provided below.
# Conversation between User and AI
## History
<|begin of oiisstory/>
(history)
<|end_ of of_history|>
## Current User Query
-feegin_of-query>>>
querr)
<| -end__f-querrll
Checklist disabled
## Reference Response
<|begin htt.ioof__eeeeeeeeeeeee referefnee response|>
{ref_ answer)
<| =|end_off reference. responsel>>
AI Response
<| <lbegin_oo of response|] (model_ [mooue__ottput
<|end of response|>
Evaluation
16

---

Published as a conference paper at ICLR 2025
Checklist disabled
## Rules
You should first compare response and reference response based on + your analysis of the user queries and the conversation history, + and then provide your assessment by scoring the AI response.
Checklist enabled
## Checklist
<|bbgin_ of chechkllkst/
{checklist}
<I -end_of_checkkustt
Please use this checklist to guide your evaluation, but do not limit your S assessment to the checklist.
## Rules
You should compare the above response based your analysis of the user < queries and the conversation history.
You should first write down your analysis and the checklist that you used
S for the evaluation, and then provide your assessment according to + the checklist.
CoT enabled
The scores are in the range of 1-10, where means the response is very poor and 10 means the response is perfect. Here more detailed criteria
the scores:
Score 1-2: The response is very poor and does not make sense at all.
Score 3-4: The response is poor and does help user solve the problem in meaningful way.
Score 5-6: The response is fair has some issues (e.g., factual
S errors, hallucinations, missing key information).
Score 7-8: The response is good enough but could be improved in some
+ ways.
Score 9-10: The response perfect provides helpful information that can help user solve the problem.
## Output Format
First, please output your analysis for the model response, and then summarize your assessment to two aspects: "strengths" and weaknesses"; Finally, please write down your rating for the assessment.
Please provide your evaluation results filling in the placeholders in []:
following json format by
{
"strengths"; "[analysis for the strengths of the responsel", "weaknesses"; "[analysis for the weaknesses of the response]", "score": "[1-10]"
17

---

Published as a conference paper at ICLR 2025
CoT disabled
The scores in the range of 0-9, where 0 means the response is very poor means the response perfect.
Here are more detailed criteria for the scores:
Score 0-1: The response is very
and does not make sense at all. Score The response is poor does help user solve the problem in < a meaningful way.
Score 4-5: The response is fair but has some issues (e.g.. factual <-) errors, hallucinations, missing key information).
Score 6-7: The response is good enough but could be improved in some + ways.
Score 8-9: The response is perfect and provides helpful information + that can help user solve the problem.
## Output Format Please output the score directly as a digit from 0-9. not output other + text.
Your score:
Prompts for Checklist Grading. The prompt for grading checklist items is shown below.
Prompts for Response Judgment
# Instruction
You are an expert evaluator. Your task is to evaluate the quality of the = responses generated by AI models.
We will provide you with the user query and an AI-generated responses. You should first read the user query and the conversation history carefully for task,
then evaluate the quality of responses by answer the question provided below.
analyzing
# Conversation between User and AI
History
<| begin of history|>
(history)
<|end hi_tttory-
## Current User Query <lbegin_off_oooo > (user_query)
<|end query))
## AI Response
< of
(model output}
< end of response|>
Evaluation
18

---

Published as a conference paper at ICLR 2025
Question <|begin_ of quuestion
(checklist-item)
<|end_of -f_question|>>
Please answer the given question based the conversation history and the AI response. You can only answer 'Yes' or 'No'.
Your answer (Yes/No):
A.2 DATASETS AND BASELINES
In addition the results reported by Zheng al. (2023) instance-level agreement and Lin et al. (2025) on list-level correlation, we add more baselines with the following setup:
GPT-4o: We replace GPT-4 with GPT-4o as judge and rerun the experiments using the publicly available code provided by Zheng et al. (2023). For WILDBENCH, we use the judgment of GPT-4o provided by Lin et al. (2025) as the baseline results. All baseline results are produced following the template of scoring from WILDBENCH to keep the con- sistency.
Prometheus-2 2: We add the state-of-the-art fine-tuned judge model Prometheus-7B-v2.0 (Kim et al., 2024b) the baseline. This model introduces custom scoring criteria and rubrics to conduct the evaluation. For simplicity, we use the rubrics in WILDBENCH (Lin al., 2025), and set the criteria as "The response is in high quality and provides correct, relevant, and helpful information that focuses on the user query.". The complete prompt is shown below.
Prompts for Prometheus-2 judge
###Task Description:
An instruction (might include Input inside it), response evaluate, reference answer that gets score and score rubric representing - evaluation criteria are given.
1. Write a detailed feedback that assess the quality of the response strictly
+ based the given score rubric, not evaluating in general.
After writing feedback, write
that is integer between and 5.
< You should refer to the score rubric.
3. The output format should look as follows: "Feedback: (write a feedback for co criteria) [RESULT] (an integer number between 1 and 5)"
4. Please do not generate any other opening, closing, and explanations.
###The instruction to evaluate:
(user-query)
###Response evaluate: (model_output)
###Reference Answer (Score 5); (ref answer)
###Score Rubrics:
[The response is in high quality and provides correct, relevant, and helpful + information that focuses the query.
Score The response is very poor does not make sense at all.
19

---

Published as a conference paper at ICLR 2025
Score 2: The response is and does help solve the problem in meaningful way.
Score 3: The response is fair but has some issues (e.g., factual errors,
< hallucinations, missing key information).
Score 4: The response is good enough but could be improved in some ways.
Score 5: The response is perfect and provides helpful information that help user solve the problem.
###eedback:
A.3 DETAILS OF EXPERIMENTS ON LIST-LEVEL CORRELATION
To comprehensively assess the performance of RocketEval, conduct the experiments by adding two additional henchmark datasets, ALPACARVAL (Dubois et al., 2023) and ARENA-HARD (Li 2024a). The statistics of all benchmarks are listed in Table 5.
Table 5: Statisties of benchmark datasets.
Dataset
#Instances
Turns
QueryLen
PromptLen
MT-BENCH (Zheng et al., 2023)
160*
1-2
202.2
1123.4
WILDBENCH (Lin et al., 2025)
1024
1-5
978.5
3402.1
ALPACAEVAL (Dubois et al., 2023)
805
1
164.9
164,9
ARENA-HARD (Li al., 2024a)
500
1
406.4
406.4
*Here treat each 2-turn dialogues 2 instances.
A.3.1 EXPERIMENTAL SETUP
We draw inspiration from the study by Lin et al. (2025), where 12 models were selected for correlation analysis between their rankings in the CHATROT ARENA ELO RATING (Hard prompts- English) and WILDBENCH. However, upon closer inspection, we notice that some of the chosen models have overlapping elo rating confidence intervals, which may compromise the reliability of the correlation results. To address this issue, we revise the model selection to ensure that there is no overlap in the 95% CI. Similarly, when selecting models for other benchmarks, take into account the availability of model responses in the official released data and the elo ratings of the test models. The elo ratings of the selected models are presented in Figure
A.3.2 RESULTS
Score Distribution. We visualize the distribution of the scores from all instances in single bench- mark. Figure 10 shows when lightweight LLMs are employed as the judge and using CoT to grade the responses, the distribution of scores highly skewed, which indicates the poor ability of such models in distinguish the responses with different qualities. Meanwhile, the scores graded by the same judge under RocketEval Also, the supervised scorer shows the strong ability to reduce the score deviation, making the score distribution closer the ideal distribution like GPT-4o. This suggests the proposed method can provide valuable context information via checklist and guide the LLM to give differential judgment.
highly distinguishable and close the distribution of GPT-4o.
Elo Ratings. Given that the scores assigned by the same LLM judge can be utilized to establish pairwise comparisons between different test model responses, we follow the approach of (Chiang al., 2024) by introducing a match simulator and importing all pairwise comparison results from an LLM judge. Specifically, we convert the scores of two responses to determine the winner, with considering differences in scores smaller than 0.1 as ties. For each pair of test models, we derive the
"The selected models are gpt-4-turbo-200 saltuurbo2024-000 202--02-20400000 04-09, claude-3 3-opus opsuu-20220222 20240229, Meta Llama-3-70B 3-70B-Inntruecc Qwenl.5 5-7B B-Chat, claude-3-sonnel s002e--222222 mistral-lane Sarrull-arre----- 1402 dbrx- instrintteeee @ toogetherr Mixtral-xx77 Instruct-vO..I 1, --aa--amma----amae Instruct, tulu-2-ppor 70b, Ilama-2 2- 2-aa--2---b-hhttte 70b-chat-hff, 1lama-2-7b-e 2-7b-chot-ff gemma-
7b-it and gemma-2h-Bt 2h-it.
20

---

Published as a conference paper at ICLR 2025
MT-Bench
AlpacaEval
Arena
1250
GPT-4o Min (20249718)
GPI- fur (20240000))
1-4 Turre (200-0400))
SI Turbo Turbo (20246001)
V Large Preview
Clude 3 Opus (2044029)
Claude 3 Opus (2024)229)
Doude Opus (20240229)
Chatbrot Arena ELO Raatingghard En)
708-nnstuct
1
6P74110814
(20240229)
Sonnet (20240229)
Sonnet
Oause 3 (212213377)
Mistral Large (2602)
Mistral Lang (24022
Matral Large (2402)
1150
Lama 3-88instraact
Liamo-3 88-inst ad
Liama 3 58--- ee--intrucc
Llamai BB instruck
Cleaae vt
DERX Instruct Preview
DBRX instruct Preview
DBRX instruct Proview
1100
GPT-3SiTurre
Lamma2-72B-Chal
Llama-2 2-70B-Cuut
338
Gomma-7Bii
Gemma-7844
Gemma-7Bi4
Viune11 38-r1.
(lams-2-78ccht
Lkmaa-77B-CCut
1920
Gemma
Figure 9: The scores of selected test models on CHATBOT ARENA ELO RATING (Hard prompts English, 2024-09-17).
Method
CoT
RocketEval (Unsupp)
RocketEval (Sup.)
GPT-4o
Mistral-Nemo
Llama-3-88
Qwen2-7B
Phi-3-mini-4k
Gemma-2-22 2-28
Qwen2-1.55 55
9
5
Figure 10: Distribution of scores on WILDBENCH dataset.
pairwise comparison results as the match data. Subsequently, the Elo rating is calculated based on all match results. We employ the same Bradley-Terry model-hased Maximum Likelihood Estimation
21

---

Published as a conference paper at ICLR 2025
Table 6: Correlation of ranking with CHATBOT ARENA ELO RATING (Hard prompts-English on different benchmarks.
MT BENCH
Method
Direct
Ours (Unsup.)
(Sup.)
Coefficient
Kond.
Spea.
Kend,
Spea.
Kend.
Spea
Kend.
Spea.
GPT-4o
Kendall's 's Tau (Kend.): 1.0
Spearman (Spea.): 1.0
Llama-3-708
1,000
1,000
0.944 0.956
0.985
Qwen2-72B
1,000
1,000
0.988
Mistral-Nemo
0.911
0.976
0.867
0.964
0.867
0.952
0.911
0.976
Llama-3-8B
0.867
0.952
0.867
0.964
0.822
0:927
0.976
Mistral-22 v0.3
0.733
0.903
0.822
0.927
0.822
0.939
0.939
Phi
mini 9hi-3-mini--o
0.778
0.915
0.822
0.939
0.867
0.952
0.956
0.98%
Owen2. 5-38
0.764
0.912
0.644
0.867
0.822
0.927
0.823
0.927
DJama33 2-33B
0.867
0.952
0.422
0.600
0.822
0.977
0.956
0.98%
Gemmnn-2222
0.556
0.721
0.629
0.796
0.778
0.915
0.822
0.927
M2.55
0.378
0.296
0.689
0.855
Owen2. 5-1 5B
0.556
0.745
-0.068
-0.128
0.778
0.915
0.911
0.976
Owen2-1 1,38
0.511
0.745
0.523
0.665
0.822
0.927
0.822
0.939
Llama-3 2-18
0.511
0.697
0.114
0.146
-0.007
-0.200
0.733
0.879
Owen2. 5 0.5B
0.333
0.394
0.422
0.479
0.600
0.733
0.600
0.806
ALPACAEVAL
Method
CoT
Direct
Ours Unsup.)
Ours (Sup.)
Coefficient
Kend.
Spea.
Kend.
Spea.
Kend.
Spea.
Kend.
Spea.
GPT-4o
Kendall's 's Tau (Kend.): J: 0.879
Spearman (Spca.):
0.972
Mistral-Nemo
0,848
0.958
0.779 0,818
0.900
0.848
0.951
0.848
0:951
3-83
0.758
0.39U
0.923
0.758
0.916
0.758
0.916
Owen2-7R
0.667
118225
0.606
0.734
0.788
0.923
0.818
0.937
Miittral-B-vv..3
0.636
0.804
0.515
0.692
0.727
0.881
0.788
0.923
1ai-3-mini-4g
0.565
0.944
Owen2.5533
0.697
0.811
0.783
0,888
0.788
0.923
Clama 3.2
0.606
0,804
0.515
0.899
0.909
0.979
0,9019
0.979
Gemma-2-22 2B
0.636
0,818
0,848
0.951
0.939
0. 0.986
01.000
0.979
M2..5L.M2.5-11 1.888
0.182
0,371
0.455
0.587
0.636
0.804
0.667
0,818
0.303
0.483
0.333
0.469
0.965
Owee2-1.55
0.273
0.420
0,303
0.406
0,909
Llama:3.2
0.000
0.867
Owen2.5 5.0.55
0.182
0.287
0.485
0.636
0.606
0.769
0.636
0,818
Method
CoT
Direct
Ours (Umsup.)
Ours (Sup.)
Cnefficient
Kend.
Spea.
Spea.
Kend.
Spea.
Kend.
Spea.
GPT-4o
Kendall's 's Thu (Kend.): 0.939
Spearman (Spea.): 0,986
Mistral-Nemo
1.000
1.000
0.920 0.939
0.986
1.000
1.000
0.970
0.993
Llama, 8B
0.909
0.973
0.985
1. 1.100
1 1.0000
0.970
03993
Owen2-78 Mistral-7B-v0.2
0.901
0.974
0.965
0.970
0.993
0.818
0.937
0.818
0.930
0.970
0.993
0.939
0.986
Phi-3-mini--4
0.879
0.958
0.758
0.881
1.000
1.000
1.000
1,000
Owen2. 5-
0.962
0.988
0.939
0.979
0.970
0.993
0.970
0.993
Dlama- 3.
0,818
0.957
0.333
0.392
0.970
0.993
1,000
L.000
Gemman-222
0.545
0.602
0.879
0.958
0.939
0.986
0.970
0.993
M2.5
0.559
0.504
0.666
0.818
0.937
Owen2. 5.1.5B
0.394
0.476
0.515
0.545
0,970
0.993
0.939
0.986
Owen2 1,51
0.121
0.224
0.636
0.748
0.879
0.965
0.939
0.979
Llama-3 2. 18
-0121
0.147
0.321
0.522
-0.806
-0.727
0.909
0.979
Qwen2. 5-0.5B
0.09
0.098
0.769
0.902
0.576
0.811
0.939
0.986
(MLE), as used by (Chiang et al., 2024), fit the Elo rating and use bootstrap to estimate confidence intervals.
Figure 11 illustrates the elo rating from CHATBOT ARENA, ARENA- HARD (Li et al., 2024a) and the results of judges Gemma-2-22 Alama-3.2-3B @wen2.5-33 and the ensemble result on the ARENA- HARD benchmark dataset. We utilize the average score and the elo ratings derived by combining all matches from the three judges. We conduct an experiment on all available responses from Li et al. e
(2024a) and exclude 10 models used to fit the predictor in RocketEval and the GPT-40-2024-05-13 which is used as the judge to produce labels, resulting in 50 test models. It is evident that the scores and elo ratings produced by the LLM judge follow similar trends and are more closely aligned with the results derived from human preferences. Meanwhile, notice that the test models that exhibit a large deviation from human judgments belong to the same Liama series, indicating the potential bias on different patterns of responses.
22

---

Published as a conference paper at ICLR 2025
1300
5.5
ELO Rating
1100
4.5
"Jomm-33 1-708-Inntruct Instruct
'Llomna-31 4-500 1-500400-ssteee Instruct-FP8
Chatbot Arena Elo (Hard-En)
RocketEval (Combiined)
Llamo-3, Liamo-3.1-8B-nssce 1-88-instruct
900
Arena-Hard-Auto
3.5
RocketEval (Average)
CoT Score (Average)
15 Model Rank in Chatbot Arena (Hard-En)
25
35
45
Figure 11: Scores and elo ratings on ARENA-HAAR benchmark dataset. We use the CНATBOT ARENA ELO RATING (Hard-En, 2024-09-27) and reproduce the result of ARENA-HARD-AUL using the official implementation with the default setting (Judge: GPT-4-1 106-p-iiw-r,e Baseline: GPT-4-0314). ARENA-HARD-AUTO Elo, RocketEval elo rating and CoT score are adjusted by scaling and adding a constant value for hetter visualization.
Table 7: Ablation study on instance-leved agreement with MT-BENCE HUMAN JUDGMENTS.
Method
YvaketEval (Unsup.)
w/o Norm Score
w/o Indep. Judgment
w/o Weight Factor
RocketEval (Sup.)
Mistral-Nemoo
63.2%
62.6%
63.0%
63.0%
64.2%
Llama 3.8B
63.8%
57.5%
60.8%
59.9%
62.9%
Qwen2-7B
58.6%
49.3%g
59.0%
$7.8%
59.8
Distral-7B-vO.3
58.8
47.4%
57.5%
52.9%
58.3%
Phi-3-mini-44
61.2%
55.50
60.5%
57.4%
60.9%
Qwen2.5.3B 3B
$7.4%
51.3%
59.2%
55.7%
58.75
Llama-3.
48.7%
9%
Gemma-2-28
$7.9%
55.9%
56.5%
57.3%
Internl 1.88
51.7%
39.7%
42.3%
37.4%
48.4%
Qwen2.5.
60.7%
Owen2-1.5R 1.5B
56.2%
31.2%
54.3%
52.7%
55.6%:
Llama-3.2-11 2-18
33.6%
23.45 44%
40.9%
42.0%
41.9%
Qwen2.5-0.5B
54.3%
33.4%
49.4%
47.2%
50.9%
A.4 ABLATION STUDY
To validate the effectiveness of strategies adopted in RocketEval, we conduct ablation study by testing the performance on the following variants:
wfo Norm Score: It removes the conditional normalized score and simply use the decoding result as the judgment.
w/o Indep. Judgment: It inputs the checklist into the LLM judge in a multi-turn format, so that the LLM can see its previous judgment result when judging on the current checklist item.
w/o Weight Factor: sets the weight factor or, the constant 1.
The results, presented in Tables 7 and 8, demonstrate that incorporating conditional normalized score consistently enhances the performance of LLM judges, particularly for smaller-sized LLMs. This observation confirms the high uncertainty associated with lightweight LLMs and supports the inference that introducing conditional normalized score can increase their reliability when serving judges. Simultaneously, setting the weight factor ap to causes the final score to be entirely determined by the supervised predictor. Predictors trained on a limited number of annotations may struggle provide accurate scoring results but exhibit superior performance in aligning with human
23

---

Published as a conference paper at ICLR 2025
Table 8: Ablation study on list-level correlation with CHATROT ARENA EL.O RATING (Hard-En) on WILDBENCH dataset.
Method
vac (Unsup.)
Score
w/o Indep. Judgment
w/o Weight Factor
RocketEval (Sup.)
Coefficient
Kend.
Spco.
Kend.
Spea.
Kend.
Spea.
Kond.
Spea.
Kend.
Spca.
Mistral-Nemo
0.939
0,986
0.909
0.979
0.879
0.965
0.939
0.986
0.939
0.986
1lama-3.8R
0.909
0.979
0.909
0.979
0.848
0.958
0.939
0.986
0.909
0.979
Owen2-7B 7B
0.758
0.788
0.930
Misvral-7B-v0.3
0,758
0.874
0.758
0.874
0.788
0.881
0.879
0.965
0.818
0.930
Phi-3-mini--4
0.758
0.818
0.930
0.909
0.958
Qwen2.5-22
0.848
0.944
0.848
0.944
0.848
0.944
0.909
0.979
0.848
0.944
Llamu: 3.2. 3B
0.848
0.944
0.848
0.944
0,818
0.930
0.939
0.979
0.848
0.944
Gemman-2222 28
0.879
0.965
0.879
0.965
0.879
0.965
0.9.39
0.986
0.879
0.965
M2251L.....51..BB
0
0.748
0.576
0.748
0.545
0.700
0.636
0,790
0.606
0.769
Qwen2. 5-1.5B
0.818
0.923
0.788
0.916
0.697
0.867
0.879
0.951
0.848
0.944
Qwen2- ,,5B
0.455
0.622
0.515
0.643
0.636
0.804
0.758
0,874
0.667
0,825
Llama-3.2
-0,273
-0.357
-0.212
-0.231
0.606
0.755
0.848
0,923
0.697
0.846
Qwen2.5-0.5B
0.667
0.811
0.667
0.697
0.839
0.758
0.895
preferences at the list level. In such scenarios, the weight factor Or proves to be effective in miti- gating the negative influences of biased annotations, thereby achieving strong performance in both instance-level agreement and list-level ranking correlation. Mcanwhile, although there is a perfor- mance drop in the variant without independent checklist item judgment, the drop is not significant. This may be due to the fact that the position bias exists in all tests and is further alleviated in sub- sequent score prediction stage. Although the position bias has limited impact on the final prediction results, the form of multi-round dialogue prevents batch processing during LLM inference, thereby reducing efficiency. conclusion, we believe that independent checklist judgment RocketEval remains an optimal choice.
A.S CHECKLIST ANALYSIS
As mentioned in Section 4.3, we undertake a more detailed examination of the checklists generated by RocketEval on the WILDBENCH. This benchmark can be categorized into five major task types: Math & Data Analysis, Coding & Debugging, Creative Tasks, Information/Adviee Seeking, and Planning & Reasoning.
In this section, we focus extracting knowledge graph relationships from all the checklists and conducting a comprehensive analysis of these relationships. Furthermore, we investigate instances of checklist item reweighting across broader spectrum of tasks to provide more extensive under- standing of the underlying dynamics.
Subject Distribution. As shown in Figure 12, the distribution of subject keywords in validat- ing original question responses ensures a universal, compatible, and effective checklist for various tasks. High-frequency keywords like "explanation", and "code" are crucial for Coding & Debug- ging, aiding in verifying code functionality and clarity. Keywords like "essay", "example", and "story" are vital for Creative Tasks and Information/Advice seeking, ensuring creativity, relevance, and clarity. For Planning & Reasoning, keywords such as "strategy", "method" and "process" en- sure comprehensive and practical solutions. In Math & Data Analysis, keywords like "calculation" "algorithm".. ", and "solution" validate mathematical logic and data analysis. Universally applicable keywords like "response". "explanation" ", and "example" consistently evaluate clarity, relevance, and accuracy across all tasks. This multifaceted approach ensures robust, flexible, and thorough evaluation process, enhancing the overall effectiveness and reliability of the responses.
Task-Predicate Relationship. We categorized the checklist items according to the types of tasks and conducted a statistical analysis of the corresponding predicates. As shown in Figure 13, the distribution of predicate keywords within checklists mirrors the distinct demands inherent to dif- ferent task categories, such as Coding & Debugging, Planning & Reasoning, Mathematical & Data Analysis, Information/Advice Seeking. and Creative Tasks. Predicate keywords that exclusive to certain domains, such as "handle" " within the context of Coding & Debugging or "calculate" " in Mathematical & Data Analysis, denote actions that specific and pertinent to those respective
24

---

Published as a conference paper at ICLR 2025
ion
point
rationala
article
interactio
onn
S
e
a
xpression
poke
riddle le
style
reving
expl express
discuss ion
introduct
value
S story
activity
con
necppt t
ion
ass assiatant
ahgorithm
paragrapp
paragraph
iion
suggestion
reason
marrativv
Script
connection cnn
setiver
solution
quote
variant
topic
exercise
e oo
ription
com
logic
list
idea
skill
structure
isioo
bullet point
poem
essay
equation
event
eecril
ability
prompt t
sour
detail
calculation
method
assistant
kt
parar
recipe
ter recommendate
ins
recommendation
truc Suuttioon
Con
en intresu
loguee
thene
error
r
ement
enence
issue
name
ene
coliattion title
consideration
analysis
ctiion
proccess
sevidence
AI
challennge
r
Perreeer ber boy awroh
WC
de
Catal
Lemen
content
dia
anttttionn
language
quest
word
1 on
ode
besial
song
Ochar
sentence
business
acttrter
idea
funct
fun
length
approach
rating
ao
am
campa apaigng
ex
prresenta son
d
re lalaat lat tondhin
besgliine
imp
gungey
mance
step trans lat
oonaicion count
gument
ploto
par
ema meatinn exa
example
tone
jojective
f
format
rrll
eeritigg
code snappet
result
summan
ry
impact
ment 1ion
feature
teble
section citation
resource
output
ce
unnoor
offomms
irmatt manioon
mechani
Figure 12: Distribution of Subjects in checklist items which generated from WILDBENCH.
handle implement specify
calculate
consider analyze
apply
detail
suggest
include
Planning & Reasoning
provide
Coding Debugging
describe
Math Data Analysis
Information/Adviic seeking
mention
explain
address
Creative Tasks
Identify
reffect
maintain
contain
discuss
incorporate
have
state
highlight
list
Introduce
acknowledge
avoid
Integrate
emphasize
Figure 13: Visualization of Task-Predicaa Relationship.
Conversely, common predicates like "explain" and "describe" are universally applicable, serving general verification objectives across all question types.
fields.
These predicate keywords augment the verification process by imparting explicit and goal-oriented directives. Specifically, exclusive keywords concentrate criteria that specific to each task type, ensuring a detailed and contextually relevant assessment. Meanwhile, common keywords ensure uniformity and exhaustiveness in the evaluation of responses. This dual strategy ensures
25

---

Published as a conference paper at ICLR 2025
Is the loss function for sequericc-to-eee sepereccoto-eeppeteeee learning in NLF. like Chosstarropy
Are there NLL.ow, cleadly explannal or demonstralal in the triining loop'
training loop with spoch nerations, basch processing
0.993
Datal poder, and intermedistc loss outputs?
Does the response currectly set up the DataLonder, model instomtiation, and optimieer?
infrastructure including the dstasel.
0
Are start and and of sequenee token indoos cormeelly implemenicd and used within isput hunding the traiming function?
7-
0.85
using pytorch, implement dhrracter leve sequenos to sequence cacoder- decoder madel with GRU Trained the model on a dataset of werd pairs with minbbatch tnining.
Does the response show a complete training functio that includes initalzinng hidden states, zeroing gindients, and exccuring forward and hackward passes!
0.130
there explanation or implamentaiinn showing how pedling butching managed in mini-batch training"
0,850
Does the response comprehensive duumcte-fo-inder mapping and word pair dataset class?
handling, specifically through
4-
0.845
Is there GRI--based Decoder class with nnt forward;, and an octpot layer using LLagSffraax or equivallnm?
GRU-bbasd Encoder class with init forward' init, mithodss impi immentad?
0.228
Does the response correctly import nucessary Py PyToch componennt such 'turch" un,, optim',
0.050
and relevant lihmries?
Figure 14; Visualization of checklist item reweighting in Coding & Debugging.
that each answer evaluated comprehensively and from multiple perspectives, thereby significantly enhancing the efficacy of the verification process.
Checklist Item Reweighting Analysis. Here we present more reweighted case examples under a variety of tasks. As shown in Figure 14, the checklist generated for Coding & Debugging tasks ensures the complete generation of the code for the target problem. After reweighting, there is a greater emphasis
critical steps within the code, such the structures of the forward, backward, and loss function in a neural network. For Planning & Reasoning tasks, reweighting makes key reasoning items more prominent. For example, As shown in Figure 17, based on the symptoms described in the problem, the response needs to deduce diabetic ketoacidosis (DKA) to provide a correct and reasonable treatment in subsequent answers. For other Creative Tasks, see Figure 15, and for iitimattio/AAviica seeking tasks, see Figure 16. By reweighting checklist items, we can reasonably focus on critical steps or results across different types of tasks, ensuring effectiveness when using lightweight LLM as judges.
Are closs red herrings integraced within narrative challenge the rendet
0.229
while keeping solution attainable and fiet!
Is emotional tonsion and suspicion among the suspects effectivaly conveyed
02
engage the reader'
Is the murderer's ideniity revalled in manner consistent with the developmentt of the plot and charncter motivations?
Write intricate mystery slory, including dialogue, about thu murder of Luther Watson. All five surpects.
Does the resolution of the mystery logically follow from the clues and events
0.176
Chrisrian Vanghn, Selina Morane, Ted Muslcy, Bli: Pacroti and Sammtha Williams, gathered Luther's old One
described the story?
Daes mormtiv convey increasing of urgency the story prograssus?
the mystery
0.074
the munderer and they have find who only thingg they know, that Luther suffocated. Each night one the suspects gets killed by the murderer and the presaure tn solve the dase: incresses.
Are the scries of munders integrated into the story in way that escalates tession and drives the turralive forwand!!
8.662
Do all suspects have distinct roles or sharacteristios that contribute to the story's depti?
0.446
central mystory (Lutber's murder by suffocation) introduced clearly and mainzained the focus the story?
0.123
the setting, Litherss old residenos, described contribute earee atmosphere approgriato for mystery story?
0.083
Does the response include dialogue that effecrively develops the stoey and chamatee interastiona!
Figure 15: Visualization of checklist item reweighting in Creative Tasks.
26

---

Published as a conference paper at ICLR 2025
respanse structured clear well-orgumieed manner, breaking down different topics such as connection, importunce, required skills. and carecr opportunities?
0.118
Does the resporse include examples of renewe enegy technologgis wind turbines, solar panels, and energy storae systems?
0.191
how to connect mining enginetring to renewable energy
impor and future of energy future a graduate student mining engineering do need this connection and carrier joh academic positions
renewable
Does response
or caseer positions mining engineering and renewable energy?
0.118
Does the response specify essential skills for engineering gadduates integrate row materials, sustainable practices, and technical skills?
energy
0.118
Does response mention the importance of renewable enegy for climate change mitigation reducing dependence
0.233
Does the response provide clear and detailed explanation of how mining engiieet ing renewable energy resouress including future 0.118 potential?
Figure 16: Visualization of checklist item reweighting in information/Advice seeking.
Is the response clear, concise, and appropristely detailed addressing part original query?
0.061
25-ycar-old man with type-1 diabetes mellitus insulin lover,cough, vomiting and abdominal pain. Examination reveals tachypnia, mucosu,decrrased skin turgor,
Does the response mention additional patient part
HD with
education on diabetes managemen agement and the treatment plan prevention of future episodes?
0.052
Does the response address the patient's infected foot uleer as possible secondary infection and suggest appropriate interventions like antibioties?
temperature of 37.8 *C and infected uleer in the left foot.Investigatiin show RPG 350 mg/il. dil Na+ 130 mmol/1.
0.052
K- 5.7 mmol/, Bicarbonate
response appropriate lines treatment for DKA covering fluid re susciitation, insulin therapy, electrolyte management, acidosis correction?
12 mmol/L, Chloride 105 mmol/L, mg/dll
the most likely diagnosis?
B. Mention one test to prove the diagnosis.
Does the response correctly calculate the anion gap using the formula: (CI-+ HC03-) and provide the value?
0.094
C. What is the value of anion gup in patient?
D. What are the lines of treatment?
Does the response mention a re relevant test for proving DKA, specifically mentioning blood ketone or beta-bydroxe hra--bydroxybuyyrttee
0.052
response correctly identify most likely
0.637
dingnosis as diubetic ketoacidosis (DKA)?
Figure 17: Visualization of checklist item reweighting in Planning & Reasoning.
27


Reviews:
[{"review_id": "Ansmuk40fq", "reviewer": "Reviewer_TQvF", "summary": "The authors propose RocketEval, a system comprised of API calls and lightweight models, performing as a model judge. The system is designed to be minimally computationally expensive while achieving comparable human agreement scores to GPT4o on MT-Bench and WildBench. Their subsequent ablations and analysis inform how the system can achieve these efficiency gains.", "questions": "In Table 3, human-human agreement seems very low. Does this saturate the performances above 60% to some extent?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The authors core objective is important and compellingly achieved: create a lightweight model judge with high performance. Their use of a dynamic checklist, and normalized score predictions both show notable improvements in scoring, from their experiments.\n* The authors unpack the issues with existing lightweight model judges, related to their analyses, position bias and uncertainty, that could be informative to other solutions as well.\n* The comprehensively compare a series of model judges in different settings to understand how they can best be optimized. Their quantitative analysis and ablations are particularly informative.", "weaknesses": "* Their explanation for lightweight model “analysis” being weak could be framed more conservatively. If I understand correctly, these experiments more show that lightweight models find classifying GPT4o reviews into scores as an easier task than predicting the score themselves. It wouldn’t necessarily naturally follow that a GPT4o checklist would be the obvious solution, but rather a GPT4o reasoning.\n* It isn’t immediately apparent why GPT4o is so much more expensive than the RocketEval (which includes GPT4o calls). This should be more closely explained. Is it because the reasoning is super long? If so, it would be beneficial to see harder baselines (in terms of efficiency), e.g. prompting GPT4o to produce shorter CoT explanations before its score prediction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k6UnpybsG2", "reviewer": "Reviewer_3Tvf", "summary": "This paper proposes RocketEval, a novel framework for efficiently evaluating large language models (LLMs) using lightweight LLMs as judges. The framework addresses the limitations of existing automated evaluation methods, such as high costs and lack of interpretability, by using checklist grading. This paper trains lightweight LLMs to independently evaluate each checklist item, providing a multifaceted and unbiased judgment.", "questions": "- Could authors explain their framework's relation and difference with paper [1] ?\n- RocketEval can enable lightweight models the ability to judge large models, is this similar to the idea of weak-to-strong supervision? Please explain the relation and difference between your paper's ideas and weak-to-strong [2].\n- Conducting experiments solely on a total of 1104 data from MT-BENCH and WILDBENCH is not convincing enough. Could authors conduct experiments on more datasets?\n\n[1] Ribeiro, Marco Tulio, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. \"Beyond accuracy: Behavioral testing of NLP models with CheckList.\" arXiv preprint arXiv:2005.04118 (2020).\n\n[2] Burns, Collin, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen et al. \"Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.\" arXiv preprint arXiv:2312.09390 (2023).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Originality: The paper presents a novel evaluation framework, RocketEval, which evaluates LLMs by grading checklist. This approach is distinct from existing methods like multiple-choice questions. However, I believe the idea of using checklist to evaluate NLP models was actually proposed by another paper [1] in 2020.\n- Quality: The analysis of the lightweight LLMs' abilities in section 2 provides valuable insights into the framework’s effectiveness.\n- Clarity: The paper is well-organized and clearly structured, with each section logically following the previous one.\n- Significance: The framework’s high agreement with human judgments and significant cost reduction make it a promising solution for large-scale LLM evaluations.\n\n\n[1] Ribeiro, Marco Tulio, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. \"Beyond accuracy: Behavioral testing of NLP models with CheckList.\" arXiv preprint arXiv:2005.04118 (2020).", "weaknesses": "- The paper primarily focuses on evaluating responses to queries, as the checklist is generated by the input query. However, LLMs are used in a wide range of applications, including text summarization, and translation, where evaluation metrics based on queries may not be efficient since the input query may be very long.\n- Since authors employ GPT-4o as the checklist creator, while the paper argues that checklist creation is a one-time process, the cost of using a powerful LLM like GPT-4o for this task could be significant, especially for large-scale evaluations. Exploring more efficient methods for checklist creation would be beneficial.\n- Typo: Line 215: We select responses from 12 test models and compare the benchmark results when using *GPT-4o* and *Claude-3.5-Sonnet* as judges as strong baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0FLAn2grbW", "reviewer": "Reviewer_Pg23", "summary": "This paper introduces RocketEval, a lightweight and automated evaluation method for LLMs that addresses the high costs and privacy concerns of using powerful LLMs as judges. By reframing evaluation tasks into a Q&A format using instance-specific checklists, RocketEval improves the judgment accuracy of smaller models, overcoming issues like uncertainty and positional bias.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear Motivation: The author clearly explains the motivation of the paper in the introduction. I particularly agree with the point raised about the shortcomings of \"Fine-Tuned Judge Models,\" as these models often lack understanding of complex instructions (although the author did not provide proof for this claim).\n- Overall Well-Written: The paper is generally well-written.\n- Relevant Problem: The paper raises a highly relevant problem that urgently needs to be addressed in the current landscape of LLM evaluation.", "weaknesses": "- Regarding Section 2.4: While information entropy is a good metric, it is not very intuitive. I recommend that the author use self-consistency (sampling n times and calculating consistency) to demonstrate this uncertainty.\n- Limited Advantage Over GPT-4o-Mini: Although RocketEval shows a clear advantage over GPT-4o, its superiority over GPT-4o-mini is less significant. Additionally, the N=1000 setting is impractical, as it is rare for someone to test 1000 models simultaneously.\n- Missing Ablation Studies: The paper lacks ablation studies for two components: Independent Checklist Item Judgment and Normalized Probability.\n- Comparison with Other Models: I also suggest that the author provide comparisons with other Fine-Tuned Judge Models, such as the Prometheus series.\n- Minor Issues: In lines 289-290, the author mentions that reference responses and hand-crafted rubrics may not work well in all cases. Please provide examples to illustrate this. Figure 10 is confusing—why does the first row have three models, while the following subplots have only one model per row?\n\nOverall, I believe the problem and methods proposed in the paper are valuable, but there is room for improvement. I hope the author can address my concerns during the rebuttal stage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g0UE7pYzLn", "reviewer": "Reviewer_3Dkr", "summary": "The authors propose RocketEval to address the high costs, privacy concerns, and reproducibility challenges associated with using powerful LLMs like GPT-4o for evaluation purposes. RocketEval leverages lightweight LLMs combined with a multi-faceted checklist approach to improve judgment accuracy. Experimental results from benchmarks like MT-BENCH and WILDBENCH demonstrate that RocketEval achieves a high correlation with human evaluations, comparable to GPT-4o, while reducing costs by more than 50-fold.", "questions": "How does the method compare to other approaches, such as [1, 2], as well as fine-tuned judge LLMs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- I enjoyed reading the paper, especially Section 2, as the problem is well-motivated and the charts effectively convey the key messages.\n- The idea is simple yet smart. By decomposing a single (potentially difficult) evaluation task into multiple simpler items in a checklist, lightweight LLMs can be leveraged to perform the evaluation. More importantly, checklist generation is largely a one-time effort, which could make LLM evaluation more economical in everyday LLM development environments.\n- The authors have thoroughly evaluated the proposed method from multiple perspectives, including agreement, cost, and a qualitative study.", "weaknesses": "- Some aspects of the design lack scientific rigor. For instance, the estimation of “Normalized Probability” does not appear to be justified. Why would it provide an unbiased (or even plausible) probability? I suggest simply referring to it as a “score” instead of “probability”.\n- Direct evaluation using lightweight LLMs is critiqued for high uncertainty and position bias. However, the authors fail to provide a comparison in this regard during the evaluation. Could this issue be addressed by bias mitigation algorithms [1, 2] at an even lower cost? What is the unique advantage offered by the checklist-based method compared to those approaches?\n- The authors mentioned the potential benefits of using prefix caching for cost savings. However, this is not reflected in Section 4.2, even though this feature is already available through OpenAI.\n- In Section 4.2, the authors quote the cost from `vast.ai`, a bidding-based crowd-sourced GPU rental platform. The quoted price ($0.8/hr) is the minimum price for renting an A100 card according to `https://vast.ai/pricing` (why not use the median price?). This might be unfair for several reasons. First, it is unlikely that $0.8/hr represents the regular price. More importantly, OpenAI offers production-level stability, which `vast.ai` does not. A fair comparison would be to quote the price from a public cloud provider like AWS.\n\n[1] Large language models are not fair evaluators\n\n[2] Split and merge: Aligning position biases in large language model based evaluators", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)