Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
Authors: Jaehun Jung
Faeze Brahman
Yejin Choi
Venue: ICLR 2025 Oral (2025)
Keywords: Large Language Model
LLM
LLM Judge
Evaluation
Alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose Cascaded Selective Evaluation, an LLM-as-Judge framework that dynamically selects when to trust different judge models to reduce evaluation overhead, while providing a provable guarantee of human-judge agreement.

Abstract: We present a principled approach to provide LLM-based evaluation with a rigorous guarantee of human agreement. We first propose that a reliable evaluation method should not uncritically rely on model preferences for pairwise evaluation, but rather assess the confidence of judge models and selectively decide when to trust its judgement. We then show that under this *selective evaluation* framework, human agreement can be provably guaranteed---such that the model evaluation aligns with that of humans to a user-specified agreement level. As part of our framework, we also introduce *Simulated Annotators*, a novel confidence estimation method that significantly improves judge calibration and thus enables high coverage of evaluated instances. Finally, we propose *Cascaded Selective Evaluation*, where we use cheaper models as initial judges and escalate to stronger models only when necessary---again, while still providing a provable guarantee of human agreement. Experimental results show that Cascaded Selective Evaluation guarantees strong alignment with humans, far beyond what LLM judges could achieve without selective evaluation. For example, on a subset of Chatbot Arena where GPT-4 almost never achieves 80% human agreement, our method, even while employing substantially cost-effective models such as Mistral-7B, *guarantees* over 80% human agreement with almost 80% test coverage.

Published as a conference paper at ICLR 2025
TRUST OR ESCALATE: LLM JUDGES WITH PROVABLE GUARANTEES FOR HUMAN AGREEMENT
Jachun Jung' Facze Brahman
Yejin Choii
Uniiersitty of Washington
Allen Institute for Artificial Intelligence
ABSTRACT
We present principled approach to provide LLM-based evaluation with rig- orous guarantee of human agreement. We first propose that a reliable evaluation method should not uncritically rely model preferences for pairwise evaluation, but rather assess the confidence of judge models and selectively decide when to trust its judgement. We then show that under this selective evaluation framework. human agreement can be provably guaranteed-such that the model evaluation aligns with that of humans to user-specified agreement level. As part of our framework, we also introduce Simulated Annotators, novel confidence estima- tion method that significantly improves judge calibration and thus enables high coverage of evaluated instances. Finally, we propose Cascaded Selective Eval- uation, where we use cheaper models as initial judges and escalate to stronger while still providing provable guarantec
models only when necessary-ages of human agreement. Experimental results show that Cascaded Selective Evalua- tion guarantees strong alignment with humans, far beyond what LLM judges could achieve without selective evaluation. For example, on a subset of Chatbot Arena where GPT-4 almost never achieves 80% human agreement, our method, even while employing substantially cost-effective models such as Mistral-7B, guaran- over 80% human agreement with almost 80% test coverage. ?
INTRODUCTION
Imagine we need evaluate million pairs of model generations-a task whose scale makes human annotation impractical, if not impossible. Today, a commonly proposed solution is to 'just ask GPT- 4' 4'(Zheng al., 2023; Dubois et al., 2023), realizing a tempting idea that large language models (LLMs) may serve as a scalable substitute for manual annotation (Chiang & Lee, 2023). However, this compelling prospect comes with crucial caveat-LLLMM evaluation would always remain, at best, an approximation of human judgement. Without a provable guarantee of reliability, it is no surprise that the judge model has to be chosen heuristically, often times to be the strongest and the most expensive model available (e.g., GPT-4). Yet, prior works show that even the strongest judge models suffer from systematic biases (Wang et al., 2023; Thakur et al., 2024) and over-confidence (Xiong al., 2024), casting doubt the dependability of these models. This raises fundamental question: How can we guarantee the reliability of LLM-based evaluation?
In this work, aim improve the reliability of LLM-based evaluation by providing rigorous guarantee of human agreement. That is, given a user-defined risk level a, we provide a guarantee that, for an unseen instance x,
P(LLM preference on agrees with human LLM evaluates x) 1-a.
To provide this guarantee, we posit that reliable evaluation framework should not only consider the preference of model, but also the validity of the preference-i.e. how likely humans would agree with the model judgement. When model cannot confidently evaluate given instance, we should not rely on its evaluated result. This motivates selective evaluation: we evaluate instance with an LLM judge, assess the confidence that humans would agree with its evaluation, then decide whether or not trust the evaluated result. We show that under this framework, human agreement can indeed be guaranteed- both theoretically and empirically- by choosing when to trust the model via fixed sequence testing (Bauer, 1991) small calibration

---

Published as a conference paper at ICLR 2025
Cascaded Selective Evaluation
Wha var Judgo M, 788
Stronger Judge Mj (GPT-3.5)
Strongest My (GPT-4)
Input Which is bathr??
Ess,
Boe
Generation I:...
Check i Ar, comkeent
move
Cleck il M2 conficent
move
Check confident
Else. abstain
Generation 2 ...
CM,(x) 2
Kyee
CM,(x) 2 Az?
CM, 2 2N²
Evaluate with M
Evaluatto with M2
Evaluate with My
Simuuaatdd Annotators ($2.2)
Threshold Calibration (52.1) Callbrate thresho each jadge, by formulating A as multiple lesting problem on smail calloration set
Cx(X)
Estimeete ridence by simulaling annotstore through in-contestt learning with each judge
CM((x) 2 2,-
Human Agreement Guarantee
Callbrated As provide Human Agreement Guarantee: P(model-human agreement 2 a) 2 1
a, 5 User delines risk to erance enror lovel 6
Figure 1: Illustration of Cascaded Selective Evaluation. We start with a small, cost-effective model as initial judge, estimate its confidence, and escalate to a stronger model only when the previous judge is not confident. By calibrating when to trust which judge model, our method provides a rigorous guarantee of human agreement while employing substantially cheaper judge models.
The practicality of selective evaluation lies not only in achieving high agreement with humans, but also in maximizing the coverage of evaluated instances without abstention-a factor that depends on the quality of confidence measure. We find that existing methods for confidence estimation (e.g., predictive probability) are brittle even with the strongest judge model. as they tend to overestimate human agreement. We then propose Simulated Annotators, a novel method to simulate diverse annotator preferences through in-context learning and estimate confidence as an agreement ratio between the simulations. Without relying on any external supervision, Simulated Annotators signif- icantly improves both the calibration and failure prediction of LLM judges. As result, selective evaluation can be done with high coverage while satisfying the prescribed human agreement level.
Moreover, since our framework provides model-agnostic guarantee of human agreement, we no longer have to rely solely on GPT-4 for evaluation. We propose Cascaded Selective Evaluation (Figure 1). where we start from a substantially cheaper LM (e.g.. Mistral-7B) as a judge, and esca- late to a stronger model only when the previous judge is not sufficiently confident all while guar- anteeing high agreement with humans. Importantly, users do not have manually choose when to use which judge model; given a user-specified risk tolerance, the abstention policy is automatically decided maintain risk control.
We test our method across preference domains including summarization and real-world user-chatbot interaction, and find that Cascaded Selective Evaluation significantly reduces the evaluation over- head while guaranteeing high agreement. For example, our method can outperform GPT-4 by achieving over 80% human agreement in ChatArena (Li et al., 2024a), while covering 79.1% of all samples, among which 88.1% 1 are evaluated by substantially cheaper Mistral-7B or GPT-3.5 instead of GPT-4. We also show that our abstention policy closely aligns with the subjectivity perceived by humans, rather than relying shallow features such length ratio or token overlap. Overall, our work suggests a principled approach to make LLM-based evaluation more reliable yet cost-effective, without exclusively counting on the capabilities of the most advanced LLMs as judges.
2 CASCADED SELECTIVE EVALUATION
When performing pairwise evaluation with LLMs, we want type of guarantee that the model agrees with the majority of human annotators. To realize this guarantee, we propose selective evaluation, a framework that employs an abstention policy to decide whether an LLM is sufficiently confident. fIM -> y denote the LLM judge, where the input E X consists of a query and pair of generations (a1, az), and the output y y is preference label between dN and 02 (e.g., 01 > az). Introducing a confidence measure CLM + [0, 1]. we define selective evaluator
to evaluate an instance. More formally,
(fLM. (z) [IM(()) if cf)(() 2 X, 0
(1)
otherwise.
2

---

Published as a conference paper at ICLR 2025
An example of CLM is the probability assigned by fLM to its predicted label (predictive probability), a popular choice of confidence measure selective classification (Geifman & El- Yaniv, 2017). is hyperparameter that trades off the precision (i.e., the accuracy of evaluator aligning with human judgements) against the coverage (i.e. the ratio of instances evaluated without abstention). The key advantage of selective evaluation is that by calibrating À in principled manner, we can provide a rigorous guarantee of human agreement while maintaining high coverage. That is, given a user- defined risk tolerance a and an error level 6, one can provably guarantee that
P(fLM (z) Yhaman (CMM(()))))))))) 2 X) 121-a
(2)
satisfied with probability least 5. In the following sections. we illustrate how search for that satisfies this guarantee ($2.1) I). how to define a good confidence measure CLM ($2.2), and how to extend selective evaluation from single model to cascades of judge models ($2.3).
2.1 PROVIDING HUMAN AGREEMENT GUARANTEE
Our human agreement guarantee can be satisfied by formulating selection of À as multiple hypoth- esis testing problem (Bates et al., 2021; Angelopoulos et al., 2022). Specifically, given access small calibration set Deal ~ P(z, Vhuman) of human preferences' we can measure an empirical risk R(A) of disagreeing with humans when using threshold A:
R(()
Σ
1()M(()) + Yhuman A cLM(x) 2 A},
(a))
(ar.Naamm)) Denl
where n(A) = E(e, (x (rann) DD (x) 2 {}. Since the empirical risk is a binomial random
variable with n(A) trials, we can compute the exact (1 - 6) upper confidence bound of the risk as:
(*) = sup {R: P(Bin(n(A), R) = [m(A) 2 5}.
(4)
Note here that the risk is near- monotonic. i.e., it tends to increase À decreases. This allows to use fixed sequence testing (Bauer, 1991), wherein we test from the largest value of X (e.g., 0.999) to progressively smaller value, and stop at the last time R (A) is below the target risk a.
Ā = {A: R((2' for A v
Theorem Consider threshold X chosen as above, and selective evaluator (fIM, C1M) operating bused on X. Then, Equation (2) is satisfied with probubility least 6.
We present the proof $A.1. Our test procedure resembles that from selective classification litera- ture (Geifman & Yl-yaniv, 2017), but we adopt fixed-sequence testing instead of Bonferroni correc- tion, which may too conservative for a large hypothesis space. Compared to recent works on risk control for LLMs, Yadkori et al. 2024), and guarantee with high probability that the risk is below a conditional on the calibration data (as opposed being marginally centered at a;; Gui et al. 2024).
provide exact, tighter bound on the selective risk (instead of approximating it;
2.2 SIMULATED ANNOTATORS
While human agreement guarantee can be met with any choice of (ncar-monotomic) confidence measure, the coverage of selective evaluation essentially depends on how good this measure is i.e,, whether CLи truly reflects if humans would agree with LLM evaluation. In this section, we first test out popular confidence estimation methods for LLMs and show that they fail to accurately represent model uncertainty. We then introduce Simulated Annotators promising alternative.
Existing Methods. We first consider two types of existing confidence measure? predictive probability: as the most straightforward proxy of confidence, we use the likelihood of preference
When we have multiple human annotation y.s per input x. define Bhuman arg max, Σ, 4mx w). We also consider more sophisticated methods (e.g., sampling chain-of of of thoughts and estimating their se- mantic entrpy; Kuhn et al. 2023) in $B, but find their performance to be mostly on-par with the above methods, despite the increased cost.
3

---

Published as a conference paper at ICLR 2025
Table 1: Performance of confidence measures across judge models. Simulated Annotators consis- tently outperforms baselines both in calibration and failure prediction especially improving the reliability of weaker judge models (GPT-3.5-turbo and Mistral-7BB.
Dataset
AlpacaEval
TL;DR
Method
Acc.
ECE 4
AUROC
AUPRC
Acc.
ECB
AUROC
AUPRC
Predictive Probabiiity
0.724
0.217
0.642
0.852
0.760
0.196
0.731
0.890
Verbalized Confidence
0.724
0.215
0.550
0.774
0.760
0.194
0.548
0.792
GPT-4- turbo
Randomized Annotators
0.720
0.113
0.705
0.866
0.779
0.079
0.734
0.905
Simulated Annotators (Maj.)
0.730
0.106
0.718
0.873
0.783
0.062
0.755
0.921
Sinulated Annotators (lnd.)
0.734
0.095
0.723
0.877
0.788
0.039
0.755
0.921
GPT-3.5- turbo
Predictive Probability
0.644
0.293
0.581
0.691
0.667
0.228
0.653
0.786
Verbalized Confidence
0.644
0.306
0.505
0.595
0.667
0.211
0.607
0.716
Simulated Annotators (Ind.)
0.694
0.058
0.632
0,793
0.725
0.043
0.704
0.842
Predictive Probability
0.618
0.374
0.457
0.579
0.661
0.306
0.613
0.735
Mistral 7B-it
Verbalized Confidence
0.618
0.414
0.627
0.661
0.335
0.578
0.6N0
Simulated Annotators (lnd.)
0,684
0.075
0.632
0,772
0.696
0.103
0.654
0.807
Predictive Probability
Verbalized Confidence
Simulated Annotators
EGE10.217
ECE 0,21
BCE: 0.085 AUROC: 0 923
AUROG: 0.842
AUROO 0.000
0.8
28
Humen Aggaaman mant
Humar Ageeemen
u.-
0.4-
0.2
0.2
0
0.2
0.4 0.5
0.2 0.4
0.8
0.2
04 Estimated dente
Extimatedd Confidence
Estimatid Conficience
Figure 2: Reliability plot for confidence estimation methods, using GPT-4 judge on AlpacaEval. Dashed lines denote perfect calibration, and darker bars denote more samples in the corresponding bins. Simulated Annotators reduces expected calibration error by 50% compared to the base- lines, mitigating over-confidence observed in predictive probability and verbalized confidence.
label predicted by the L.M judge, iee, (LM(T) = max, PLM [);; (2) verbalized confudence: follow- ing Tian et al. (2023b), we directly prompt the LLM judge to express its confidence in a scalar value. We evaluate these methods in terms of the expected calibration error (Naeini et al., 2015), AUROC and AUPRC, using the non-tied instances in two standard benchmarks: AlpacaEval (Dubois et al., 2023) for open-domain chat assistant and TL;DR (Stiennon et al., 2020) for summarization.
Canonical methods overestimate human agreement. The results are shown in Table I and Fig- ure 2 (left, middle). Unlike prior reports that the canonical methods work well for tasks with small label space (Kadavath et al., 2022: Tian et al., 2023b), they consistently lead to over-confidence when used for preference evaluation. Notably, the results are pronounced even with the strongest LLM judge GPT-4-turbo, although its agreement with human majority is known to be comparable to average human annotator (Sottana al., 2023; Zheng al., 2023).
The above results suggest that simply achieving human-level performance may not be sufficient for reliable evaluation: while an LLM judge can be accurate single human annotator, it tends to be over-confident in estimating its agreement with the majority of annotators. This contrasts with the standard practice in human evaluation, which involves collecting multiple annotations per instance and assessing the level of agreement between them; the evaluation is deemed reliable only when there is high inter-annotator agreement. Motivated by this discrepancy, we derive Simulated Anno- tators, a confidence measure that simulates diverse annotator preferences with in-context learning. Coneretely, given K (e.g., 3) examples of preference annotations per N (e.g., 5) human annotators, we simulate annotators by Khohot prompting the model for N times and ensemble the results:
(L) max y
Σ
PLM(U) V1.5,"" (IK.j.UK.j)).
where (zi.s Wis) is the i-th in-context example from the j-th annotator. Likewise, the judge pre- diction fIs((x) is defined as arg max,, Σn PLM(y lr; (#1 w)))." ".. (#KJ.UK))) Intuitively,
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Cascaded Selective Evaluation
Input: A list of judges .M- (M..... MMM)) ). calibration set Deat and test set Drent be evalu- ated, risk tolerance a and error level 5 A set of evaluated results S
Output:
A + Malibrate[M, Deal, 0,,6
Calibrate thresholds À E on Deal ($A.2).
S 5+0 0
Initialize a set of evaluation results.
for x E Diss do
for i
M do
Iterate through the cascade of judge models.
if CM. (z) 2 X; then
Evaluate x only when M, is sufficiently confident.
S - SU (1x, faa, (x))
break
return
Return the evaluated results.
the confidence CLM becomes low when multiple simulated annotators disagree with each other. We typically set K, < 5, and ablate the effect of number of simulated annotators in $3.5.
Simulated Annotators improves reliability, even for weaker judge models. The results with == N = 5 are shown in Table (Simulated Annotators (Ind.) and Figure 2 (right). Simulated Annotators significantly outperforms popular confidence measures, reducing ECE by 50% and im- proving AUROC by 13% for GPT-4. Surprisingly, our method improves the reliability of even the weaker judge while they do underperform GPT-4 in accuracy, their estimated confidence is on-par or even better than GPT-4 when using the baseline confidence measures.
Despite the substantial performance gain in Simulated Annotators, remains unclear whether the gain truly comes from simulating diverse human preferences. We analyze this using two ablations on the few-shot examples given to the LLM judge: (1) randomized annotators: using the same set of inputs Ti but random-assignin
labels Yi.j Ber(0.5), and (2) simulated annotators (majority): using ( Jimmmm ) where Jummn is the majority preference of human annotators given input Zi.j. and N for ablations. As shown in Table Simulated Annotators (Maj.) is
We fix consistently better than Randomized Annotators, but slightly underperforms Simulated Anmotators (Ind.) that models individual preference. The performance majority-baal -base Simulated Annotators, however, is encouraging, the method can be applied to cases where we do not have access to multiple human annotations per each instance r. Overall, the result demonstrates that simulating diverse annotator preferences is helpful, and even in the absence of such data, our method improves the reliability of LLM judges over the existing methods.
2.3 CASCADING SELECTIVE EVALUATORS
The strong performance Simulated Annotators demonstrates that even the weaker LLM judges despite not as accurate as larger judge may accurately predict when they are likely to agree with human annotators. Leveraging this finding. we propose Cascaded Selective Evaluation, as illustrated in Figure 1 and formalized in Algorithm 1. Given a list of judge models M, we start with a weaker yet cheaper model as an evaluator, and only when the model not sufficiently confident, we iteratively move on to stronger model. Notably, the confidence threshold X for each judge model can be chosen following the same process as in $2.1, providing the guarantee risk control across the cascades of models (see $A.2 for details). This way, selective evaluation operate at a significantly lower cost than using the strongest model from the start, while still maintaining a rigorous guarantee of human agreement.
3 EXPERIMENTAL RESULTS
3.1 EVALUATING GENERATED SUMMARIES
Experimental Setup. We first test our approach for evaluating summaries on TL;DR dataset (Sti- ennon et al., 2020). We use cascade of ---as--t-tnttt (Jiang et al., 2023), GPT-3.5-turb
Here, as each input instance is associated with a single majority label Yhumon, we induce difference hetween simulations by using different of inputs J5, per simulated annotator.
5

---

Published as a conference paper at ICLR 2025
Table 2: Comparison against baselines TL;DR, with target agreement level 1 - 0.9. The results are averaged across 1000 runs with random data split. Guarantee Success Rate defined as the ratio of successful runs that achieve empirical human agreement larger than equal Cascaded Selective Evaluation is the only method that achieves high guarantee success rate, while maintaining high coverage.
Evaluator Composition (%)
Guarantee Success Rate
Method
Coverage (%)
Mistral-7B
GPT-3,5-turbe
GPT-4-uabo
No Selection
0.0
0.0
100.0
100.0
0.0
Heuristic Selection
100.0
42.0
Cascaded Heuristic Selection
59.6
15.0
25.5
64.6
0.0
100.0
0.0
0.0
5.6
54.7
Point-Estimate Estimate Calibration
0.0
100.0
0.0
9.4
79.0
100.0
Cascaded Selective Evaluation
28.3
43.5
Target Human Agreement
Empiriaal Agmmmment (66)
Empiriaal Human Agreament
8.6
96.0
Mistral-7B
Eustuator Compositon (96)
66.6
77,3
55
07.6
03.5
20.1
F5.7
12.5
43.5
165
GPT-4-uubo Agreement
80
Target Agroement
Target Human Agrooment
Figure 3: TL;DR results. Cascaded Selective Evaluation guarantees human agreement far be- yond level achievable by GPT-4 without abstention (Left),, while employing substantially weaker judge models (Right). Solid blue line denotes average human agreement over 1000 runs on the dataset, and the light blue region denotes the min max agreement within the 1000 runs.
and GPr-4-turbo (Achiam et al., 2023) as judges. Observing that the dataset provides multiple hu- man annotations per input, we use Simulated Annotators (Ind.) with N = We fix size of calibration set Deal 500 and 5 0.1, and run the experiments for 1000 random splits of calibration and test set. For baselines, we consider: (1) Heuristic Selection: using GPT-4 as a judge and setting À
a, assuming perfect calibration; (2) Cascaded Heuristic Selection: variant of Heuristic Selection using the same cascades of judge models as ours; (3) Point-Estimatte Calibration: setting À as the smallest value that satisfies R(A) < in Deat without hypothesis testing.
In Figure 3, we show that human agreement guarantee is satisfied with our approach across all levels of target human agreement, far beyond what GPT-4 can achieve without abstention. Notably, unlike prior works (Gui al., 2024; Mohri & Hashimoto, 2023) that only controls the risk in expectation over calibration sets (solid blue line), our method guarantees with high probability that each indi- vidual run would satisfy the target agreement level (light blue region). Moreover, shown in right plot, the high agreements can be achieved while the majority of evaluation are done with substan- tially smaller LLMs than GPT-4. For example, our method can outperform GPT-4 with 80% human agreement, while 75% of the evaluations are done by Mistral-7B or GPT-3.5.
We also compare against selective baselines in Table in terms of their coverage and guarantee success rate. All baselines fail to provide meaningful guarantee success rate without significantly sacrificing the coverage. This includes Saitt-Estimate Calibration, which makes use of the test statistics in calibration data (Ramirez al., 2024). On the contrary, Cascaded Selective Evaluation achieves over 90% success rate rate-just as expected by setting 0.1-atttt altastinnng to its reliability.
3.2 EVALUATING LLM-BASED CHAT ASSISTANTS
Experimental Setup. Next, we test our approach for evaluating general- purposs LLM assistants on two datasets: Chat(bot) Arena (Zheng al., 2023) with real-world user-assistant interaction'
We evaluation with 5.2k instances collected by Li al. (2024a).

---

Published as a conference paper at ICLR 2025
Table 3: Comparison to baselines on ChatArena, with target agreement level 1 a = 0.85. The results averaged across 1000 runs with random data split. Consistent with results TL;DR, our method successfully guarantees target agreement level while maintaining high coverage.
Evaluator Composition (%)
Guarantee Success Rate (%)
Method
Coverage
Mistral-7h 78
GPT-4-narbo -erro,--trrb
No Selection
0.0
0.0
100.0
100.0
0.0
Heuristic Selection
0.0
0.0
100.0
95.2
0.1
Cascaded Heuristic Selection
27.7
100.0
Point-Estingn Estimate Calibration
0.0
100.0
0.0
40.5
57.2
0.0
0.0
100.0
60.9
54.4
Cascaded Selective Evaluation
23.7
58.8
17.5
63.2
91.0
100
100
100.0
96,9
GPT-4-turbo
Trrget Agueament
GPT-8.5-turbo Mistral-7E
Emrprical Agroooot
22.5
85
Emprttcll Agroemmnnt
60
Evalustor Comprsttion
38.0
63.2
30
37.2
GPT- Agrereen
31.7
70
11.4
85
Target Agroement
Target
Figure 4: ChatArena results. Our approach guarantees target human agreement level (Left) while majority of evaluations done with weaker judge models, Mistral-7B and GPT-3.5 (Right).
and Auto-J (Li et al., 2023), curated benchmark for meta-evaluation of LLM judges. We employ the same cascades of models as in $3.1, but this time we use Simulated Annotators (Maj.). both datasets only provide one human annotation per input æ. We set N 5 and Deal = 500 for ChatArena, and K N 3. Deal 392 for Auto-J, considering the small size of the benchmark.
The results are shown Figure and Table for ChatArena, and in Figure 5 and Table for Auto- J. Again, confirm that human agreement guarantee can be achieved across all levels of On ChatArena, unlike Point-Estimate Calibration with GPT-4 whose guarantee success rate below 60%, method achieves 91% guarantee success rate while only using GPT-4 for 17.5% of the evaluations. The performance particularly pronounced in Auto-J where GPT-4 without abstention could only achieve 63.2% agreement with humans (Figure 5), potentially due to the fact that the dataset introduces additional tie label unlike the other two datasets. In stark contrast, Cascaded Selective Evaluation guarantees up 80% human agreement with high probability.
Next, we conduct further ablations and analyses to better understand the working of our method.
3.3 UNDERSTANDING ABSTENTION POLICY
When does the model abstain from evaluation? One concern with selective evaluation that its abstention policy may not align with the human- perceived subjectivity of each instance and in- stead rely on shallow heuristies, e.g., choosing to abstain when the pair of generations have large token overlap. To address this concern, we ana- lyze whether there exists significant difference subjectivity between model-
Table 4: Comparison between abstained vs. evaluated samples. Our abstention policy aligns with how humans agree with each other (IAA), exhibiting no significant reliance on shallow heuristics (length ratio, token over- lap).
Dimension
Abstained Samples
Evaluated Samples
in numan--erceinn abstained samples and evaluated samples. We first collect 3-5 additional human annotations per each instance in ChatArena, and measure the
Human IAA
0.815 (0.031)
0.902 (0.025)
Length Ratio
0.242 (0.014)
0.245 (0.025)
Token Overlap
0.623 (0.049)
0.592 (0.054)

---

Published as a conference paper at ICLR 2025
Table Evaluation under distribution shift on ChatArena. We induce distribution shift by sampling the calibration and test set respectively from two disjoint sets of instances with overlap of eval- uated models. We iterate experiments for 1000 random splits and aggregate the results. Cascaded Selective Evaluation guarantees high agreement even under the realistic distribution shift.
Target Human Agreement (%)
Empirical Human Agreement (%)
Guarantee Success Rate (%)
Coverage (%)
70.0
73.4
100.0
100,0
75.0
75.3
91.4
92.5
80.0
80.8
72.1
90.8
85.0
85.2
55.4
91.0
90.0
90.1
31.8
90.7
Table 6: Impact of number of simulated annotators N on ChatArena, with 1 0.85. Larger number of simulations generally leads to better coverage, while human agreement is guaranteed even with small N. For all values of N, Cascaded Selective Evaluation guarantees high agreement with humans while reducing the API cost by 40% % compared to GPT-4 without abstention.
Empirical Human Agreement (%)
Guarantee Success
Relative API Cost
Method
Coverage (%)
Rate (%)
GPT- (N 1) Cascaded Selective Evaluation (N =
77.8
100.0
0.0
1.000
85.2
60.9
90.3
0.655
GPT-4 (N =
78.2
100.0
0.0
2,000
Cascaded Selective Evaluation (N
61.5
90.8
1.288
GPT-4 (N 3)
100.0
0.0
3.000
Cascaded Selective Evaluation (N
85.5
62.1
90.3
1.920
GPT-4 (N 5)
78.5
100.0
0.0
5.000
Cascaded Selective Evaluation (N
85,8
63.2
91.0
2.849
inter-annotator agreement (IAA) as proxy of human-perceived subjectivity". Then, we compare the difference IAA between abstained and evaluated samples when the target agreement level is set to 0.9. We also measure the difference in terms of shallow features, specifically the pairwise length ratio and the token-overlap (ROUGE-L) within each instance. For further details, see $C.2.
Table 4 presents the results. The average inter-annotator agreement is 0.815 (a² 0.031) for abstained samples and 0.902 (o² = 0.025) for evaluated samples, a statistically significant difference in two-sample t-test with p 1e -e-- 8. This is in contrast with both the length ratio and token overlap, for which the differences between the two sets are not significant (p > 0.10). In fact, for token overlap, the abstained examples exhibit higher ROUGE-L average than the evaluated samples. Overall, these results show that the instances abstained by LLM judges tend to be more subjective even for humans (with no evidence of reliance on some spurious heuristics), suggesting that the confidence elicited by Simulated Annotators closely aligns with that of human annotators.
3.4 EVALUATION UNDER DISTRIBUTION SHIFT
Our test procedure in $2.1 assumes that the calibration set Dent is sampled i.i.d. from P(x, Yhuman In real-world scenarios this may not be the case, because often only have access generations from a set of known models for calibration, while in the test time, we need to evaluate outputs from unknown models. In Table we empirically analyze whether our method provides risk control even under this distribution shift. First, we randomly divide ChatArena into two disjoint sets such that there is no overlap between the evaluated models in each set. Then, we induce distribution shift by sampling Deat from one set and testing instances in another set. We follow the same setup as $3.2, and run experiments for 1000 random splits. As shown Table 5, despite a small degradation in coverage, method guarantees high human agreement for more than 90% of the time, consistently across all tested levels a. The result demonstrates that Cascaded Selective Evaluation maintains its reliability even under the realistic distribution shift.
As the label space is binary for preference evaluation, define inter-annotator agreement simply as the density of the majority preference lahel assigned hy human annotators.
8

---

Published as a conference paper at ICLR 2025
Table 7: Impact of judge model composition on ChatArena, with 1 a 0.8. Weaker cascades use Mistral-7B. Mixtral-8x7B: and GPT-3.5 as judge models. Stronger cascades use GPT-4 instead of Mixtral-8x777 Our method guarantees human agreement even with the weaker cascades, while only using 12.6% of the evaluation cost for GPT-4.
Method
Empirical Human Agreement (%)
Coverage (%)
Guarantee Success Rate (%)
Relative API Cost
GPT-4
77.8
100.0
13.9
1.000
Cascaded Selective Evaluation (stronger)
80.2
77.6
90.5
0,215
Cascaded Selective Evaluation (weaker)
80.3
68.3
90.8
0.126
Cascaded Selective Evaluation (weaker GPT-4)
80.4
78.2
90.6
0.192
3.5 IMPACT OF NUMBER OF SIMULATED ANNOTATORS
Next, we analyze the impact of number of simulated annotators for Cascaded Selective Evaluation. In Table 6, we report the results on ChatArena using Simulated Annotators as confidence measure with NN- = 3, 2, 1. We compare the result against using GPT-4 with the same prompt, but with- out abstention. Along with the guarantee success rate and coverage, we report relative API cost for calling OpenAl models, where the cost for full evaluation with GPT-4 is set to 1. The results suggest that (1) using larger number of simulated annotators leads to consistently better coverage, but (2) even with a small number of simulated annotators, our method can still achieve high human agreement while reducing the evaluation cost by up to 40% compared to GPT-4 without abstention.
3.6 IMPACT OF JUDGE MODEL COMPOSITION
We study whether Cascaded Selective Evaluation can be done entirely without GPT-4. We use substantially weaker cascades with Fistral-7B-iintruutt-d... Mixtral&&xTb-instruee (Jiang et al., 2024), and GPT-3.5 as judge models (weaker cascades). We compare the result against (1) zero-shot GPT-4 without abstention, and (2) Cascaded Selective Evaluation using the original cascades, with N = for better cost (stronger cascades). We set the target agreement level 1 level than what is achievable by GPT-4 without abstention (Figure 4).
0.8, higher
The results in Table 7 reveal an interesting finding: even the weaker cascades of judge models ensure a satisfactory level of human agreement, by balancing the the trade-off between the evaluation cost and coverage, instead of compromising the precision. While harder evaluation tasks generally require stronger judge models for high precision, unlike conventional LLM-based evaluation where one has to sacrifice accuracy by using weaker judge, our method allows practitioners to consistently achieve their target level of human agreement. Depending on the requirements, one can opt for stronger cascades for better coverage or weaker cascades for lower costs, all while maintaining this guarantee. Additionally, both configurations of Cascaded Selective Evaluation significantly reduce evaluation costs compared to using GPT-4, achieving savings of up to 78.5 with stronger cascades and 87 44% with weaker cascades.
4 RELATED WORKS
LLM-based evaluation has emerged as a scalable alternative to human evaluation (Zheng et al., 2023; Liu al., 2023), with empirical evidence suggesting that despite its cost, GPT-4 can be as accurate as an average human annotator (Dubois et al., 2024; Li et al., 2024b). Subsequent works attempt to reduce the dependency on the large judge model by distilling a small expert judge (Kim et al., 2024; Zhu et al., 2023), or by ensembling multiple agents through peer review and debate (Verga et al., 2024; Chan et al., 2023). However, these methods often lack a provable guarantee of their reliability. Recent research also indicates that LLM judges are not as robust previously assumed, showing susceptibility to cognitive biases (Zeng et al., 2024; Koo et al., 2023) and self- preference (Panickssery et al., 2024). Our goal in this work is enhance the reliability of LLM- based evaluation desiitt these inherent biases as a better-aligned proxy of human judgement.
Another line of works augment LLMs with a rigorous statistical guarantee, controlling their risk in critical applications such as hallucination rate in factual generation (Yadkori et al., 2024; Mohri & Hashimoto, 2023) and FDR in medical decision making (Gui et al., 2024). These approaches

---

Published as a conference paper at ICLR 2025
Table 8: Comparison to baselines on Auto-J, with target agreement level a 0.8. The results are averaged across 1000 runs with random data split.
Evaluator Composition (%)
Guarantee Success Rate
Method
Coverage (%)
Mistral-7B
GPT-3,5-tarbo
GPT-4-tubo
No Selection
0.0
0.0
100.0
100.0
0.0
Heuristic Selection
0.0
0.0
100.0
72.0
Cascaded Heuristic Selection
36.6
23.2
84.4
100.0
0.0
54.3
Point-Estimen Estimate Calibration
0.0
100.0
0.0
27.9
52.1
0.0
0.0
100.0
42.9
56.5
Cascaded Selective Evaluation
49.3
21.8
28.9
42.6
90.2
"arget Humran Agiemment
GPT--.6-turb
Errpii cul Humann Agpommnt
50
87.4
Mistral-78
Emoricaal Human Agtooment
70
60
40.8
49.6
12.3
65
80
8.3
F1.
70
75
80
60
65
70
75
60
Target Human Agreement [1
Target Human Agreement (] o0
Figure 5: Human Agreement Guarantee on Auto-J. GPT-4 without abstention obtains only 63.2% agreement with humans, while Cascaded Selective Evaluation guarantees target human agree- ment level of up to 80 % with high probability.
often powered by conformal methods (Angelopoulos et al., 2024), offering marginal control over the prescribed risks. Other works study fine-tuning objective for LLMs, either to improve their truthful- ness (Kang et al., 2024; Tian et al., 2023a) to abstain when lacking relevant knowledge (Zhang et al., 2024). the context of model evaluation, prediction prediction-powered inference has been employed provide confidence does not guarantee high human agreement over individual runs of evaluation as in ours. Our work builds upon these prior works, but (1) focuses bound on the disagreement risk conditional on the sampling of calibration set, (2) proposes an un- supervised confidence measure that simulates annotation variability (Plank, 2022: Khurana et al., 2024), and (3) derives cascaded framework that significantly reduces the inference cost (Li et al., 2021) while simultaneously guaranteeing the reliability of evaluation.
of possible rankings among evaluated models (Chatzi et al., 2024), but
L.M-hased evaluation provide an exact upper
5 CONCLUSION
We present Cascaded Selective Evaluation, framework provide LLM-based evaluation with robust guarantee of human agreement. As part of our framework, we also propose Simulated An- notators, novel method that significantly improves confidence estimation for LLM judges without resorting to external supervision. By dynamically selecting when to trust which judge model, Cas- caded Selective Evaluation significantly reduces evaluation overhead while still maintaining its reli- ability, often outperforming the precision achievable by fully relying on the strongest judge model.
As LLM judges become prevalent, principled method utilize them will become increasingly more important. As an initial step toward this direction, we focus pairwise evaluation with popu- lar off-the-shelf LLMS as judges, and on domains where the goodness of generations are subjective in nature. Future research should investigate the integration of recent "expert" judges distilled from larger models, and extend our framework to absolute score- based evaluations. Additionally, apply- ing framework for verifying correctness of reasoning steps, such for mathematical problem- solving, would be an interesting next step. We hope our work sets foundation for exploring these advancements, and broadly for more robust and scalable evaluation with LLMs.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0000 087744 2023.
Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms achieve risk control, 2022. URL https: / //aaxivv. org. absaaass//111 01052.
Anastasios Nikolas Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Con- formal risk control. In The Twelfth International Conference on Learning Representations, 2024.
URL httph:t////// opppreeeeeveeeeeeeeeeeee .ee.ieeww..eeeeee...........eeee forumo 3id.uudddddd33332225.
https://qpenreeview
Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-free, risk-controlling prediction sets. J. ACM, 68(6), sep 2021. ISSN 0004-5411. doi: 10.1145/3478535. URL https: /t/ooii..rtt weoii.org/ii. 1145/1155/55 11445/34738855.
Peter Bauer. Multiple testing in clinical trials. Statistics in medicine, 10(6):871-890, 1991.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and
Zhiyuan Liu. Chateval: Towards better Ilm-based evaluators through multi-agent debate, 2023.
URL httpt:tt///// ptttpp://arxiie. /ttp/xxxxvvv...///. abs/ / 2308. 07201.
Ivi Chatzi, Eleni Straitouri, Suhas Thejaswi, and Manuel Gomez Rodriguez. Prediction-powered ranking of large language models, 2024. URL https::///// /estps://arxvv /tgga//s//42..../////g/ 17826.
Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human eval- uations? In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pp. 15607-15631, Toronto, Canada. July 2023. Association for Computational Linguis- tics. doi: acl-long. 870.
lo0. URL https: /aaaa/aaannnttoooooeeo org/2023.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In Thirty-seventh Conference Neural Information net wttp://wwwweeeetem/tteeaaaaeaa wtt.mmiid-huurraeLee.
Processing Systems, 2023. URL https: ht //tp:://///peeeeeeee/ wttpw/rwwwwwweeeeeeeeeeeeeeeeeee
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2024. URL https::/aaxxv..oo abs/2404. 044
Yonatan Geifman and Ran Bl-Yaniv. Selective classification for deep neural networks, 2017. URL hittps://artt
org/ wtg/abs/////ooooooogo...o/...arr abs//705 00800.00000000....
Yu Gui, Ying Jin, and Zhimei Ren. Conformal alignment: Knowing when to trust foundation models with guarantees, 2024. URL https:t//aaxiiii orgg abs/2405. 10301.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b, 2023. URL https: //arxiv. wttp://////............
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gi- anna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https::////o
.0..440000000.0.. 04088.
11

---

Published as a conference paper at ICLR 2025
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson. Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bow- man, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Ka- plan. Language models (mostly) know what they know, 2022. URL https: /aapps:///axiv.../// eerxiiiorg/// abs/2207. 05221.
Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, and Sergey Levine. Unfamiliar finetuning examples control how language models hallucinate, 2024. URL https:/: httpps////xxxvox///////// abs/ 2403. 05612.
Urja Khurana, Eric Nalisnick, Antske Fokkens, and Swabha Swayamdipta. Crowd-calibrator: Can annotator disagreement inform calibration in subjective tasks? In First Conference on Language Modeling, 2024.
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models, 2024. URL https: waw.iaxiivv......./.aaa org/ abs abs/2405. 01535.
Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models evaluators, 2023. URL https:
/ //aaxxiv.rrg//////............./ /30/:/bb///33..///
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Confer- Learning Representations, 2023. URL VD-AY:POdve.
net e www.rrrriww.eeetoorumii...e.. forum?id=
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment, 2023. URL https: //axxvv. org/ abs 2310.1100000 05470.
Junlong Li, Fan Zhou, Shichao Sun, Yikai Zhang, Hai Zhao, and Pengfei Liu. Dissecting human and IIm preferences, 2024a. URL htttps://aaatt
wwwp/rrg////oooooooro//.///..are /202 11296.
Lei Li, Yankai Lin, Deli Chen, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. Cascade- BERT: Accelerating inference of pre-trained language models via calibrated complete models cascade. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings the Association for Computational Linguisties: EMNLP 2021, pp. 475-486, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v 1/8853/v/220211111 010655//1/2221..ndnnn-eeeee findings-emmlp.44.. URL https: /tta::///llaaain //tp//aaaaaaclcacc.c/c/co org/ 2021. Bindings-emn 433
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gon- zalez, and lon Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024b.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quan- tification for black-box large language models, 2024. URL attpps:/cpenreviiee net/ forum? d== iiN N1 VkgA0.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Ka- lika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 251 tics. doi: 251122122552 1-2522. Singapore, December 2023. Association for Computational Linguis- 11t33//11222.mmmmm// 153. URL https: / whwannthhhllaooooo emr.p-ma 153.
org 2023.
Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guaran- tees. In Forty-first International Conference on Machine Learning, 2023.
12

---

Published as a conference paper at ICLR 2025
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Lim evaluators recognize and favor their own generations, 2024. URL https: https/////xxxx...x/xaea / 130766
Barbara Plank. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 10671-10682, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10.1 36653/1122222.emmppmm 731. URL http:: h/t////caatthooooyyy. org/2022. emn.p-main. 731.
Guillem Ramirez, Alexandra Birch, and Ivan Titov. Optimising calls to large language models with uncertainty- based two-tier selection. ArXiv, abs/2405.02134, 2024. URL https://apii semant semantcssholaar org/ Crgg/orpussDi ID: 269588822
Andrea Sottana, Bin Liang, Kai Zou, and Zheng Yuan. Evaluation metrics in the era of GPT- 4: Reliably evaluating large language models sequence to sequence tasks. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https:
/ //operreiiew ew. formm?? d=SyEwsV52Ds
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 0000883021,20 2020.
Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges, 2024. URL https: / arxiv. org/ abs / 2400. 12624.
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, and Chelsea Finn. Fine-tuning language models for factuality, 2023a. URL httpttttp//// /ttpp:///aaivvv////////e/ https////awwwv..a/ab//eesse 2211 08401.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting calibrated con- fidence scores from language models fine-tuned with human feedback. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 5433-5442, Singapore, December 2023b. Associa- tion for Computational Linguistics. doi: 10. 10.18653/w1/2033 2tt.33/111222....mmmmma.... .mnnlpmmiin.330.. URL https: // /aaclanthology.... httpllooy...g////0 crg/2023. emnlp main. 330.
Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating Ilm generations with panel of diverse models, 2024. URL https:: //rxiv.. org/ abs///b////a 2404 4. 18796.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023. URL https: / www.ivi.rg/aaiie www.aa//3335 22005.35500 11.5..1922 17926.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL https: / opoeneeview. met/ffrum/ ftr.mmidummiraa/m/o jejjiedKKejeeeeee. xFp2.
Yasin Abbasi Yadkori, Ilja Kuzborskij. David Stutz, András György, Adam Fisch, Arnaud Doucet, luliya Beloshapka, Wei-Hung Weng, Yao- Yuan Yang, Csaba Szepesvári, Ali Taylan Cemgil, and Nenad Tomasev. Mitigating I1m hallucinations via conformal abstention, 2024. URL https:
// //arxiv. org. abs 24002 01563.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations, 2024.
13

---

Published as a conference paper at ICLR 2025
Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. AlignScore: Evaluating factual consis- tency with a unified aligument function. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 11988-11348, Toronto, Canada, July 2023. Asso- ciation for Computational Linguisties. doi: 10. 1/t333/111222.c1---1/o 634.. URL https:
//aclanthology org/ acl long. 634.
Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Instructing large language models say 'I don't know'. Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies (Volume 1: Long Papers), pp. 7113-7139, Mexico City, Mexico, June 2024. Association for Computational Linguistics. URL httttt//aaccaaattt http///aaaaaaoooooo.ooo/o org/ naacl naacl-long, -log. 394.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: // //openzeview.
Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges, 2023. URL https: baxxiv.org/o wttpor/bbb....//..3/..
14

---

Published as a conference paper at ICLR 2025
A VALIDITY OF HUMAN AGREEMENT GUARANTEE
A.1 PROOF OF THEOREM 1
Theorem Consider threshold X chosen as in $2.1, and a selective evaluator (fLMs CLM) operat- ing based on X. Then, Equation (2) is satisfied with probability at least 6.
The proof extends that of Theorem B.1 in Bates et al. (2021). Let R(A) denote the true risk of disagreeing with humans at threshold A. suffices to show that P(R(A) I a) 2 1 We first note that n (A) R(A) is a binomial random variable, ie., (()))(()))))))
n (())R(A))
Thus, the lower tail bound for ((A) can be expressed as a function g of t E R and R(A)
P(((()) s = P( (Bin(n(A), R(A)) z [m(A](] s(1 R(A)).
Plugging this into the definition of Rt (A) Equation 4,
(A) = sup R(A) P( Bin(n(A), R(A)) & [m(A) 2 6)
sup R(A): s((((); R(A)) 2 6)
Here, let G denote the CDF of R(A) and (-) sup G(x) S 8). From above, we know that R(A) > (A), then (f(()); R(A)) < 6. Therefore.
P(R(A) > Rt (2)) S P(o(B(A); R(A)) < <)
= =P(G(((A) <))
S P(T(()) <G-
36.
Hence, P(R(A)= < (A)) 1- implying that Rt (A) is the (1 5) upper confidence bound of R(A). Finally, since we have R (x) s a from the definition of X, we obtain
P(R()) S R S a) 2-
A.2 EXTENSION TO CASCADES OF JUDGE MODELS
We illustrate the extension of our test procedure from single model to the cascades of judge models. For notational simplicity, we denote the test procedure for a single judge model in $2. as a function calibrate-single. This function takes input a model M, calibration D, risk tolerance a and emror level and gives a calibrated threshold N as an output.
The calibration procedure for cascades of judge models is shown in Algorithm 2. The procedure sequentially applics calibrate-single to each judge model. Specifically for each judge model M,,
Algorithm calibrate[MM, Deats a, 6)
Input: A list of judges M (M1. M MMM) calibration set Deat and test set D to be evalu- ated, risk tolerance a and error level 8
Output: A set of calibrated thresholds A
Initialize the set of thresholds. Initialize D for calibrating each model.
D -- Deal
for i = 1 to |M| do
A; f calibrate-single Mi, D. 6
Calibrate X; for each model M,.
A < (
D < ((,,)) D (M, < Ar)
:
Update D with only the previously abstained instances. Return the set of calibrated thresholds.
return
15

---

Published as a conference paper at ICLR 2025
calibrate X by testing over the set of instances D that have been abstained by the previous models. This allows us to ensure that for each M,
P [M, (z) Yhummnn CM, (x)
Ai
CM₁ (x) ,g 21 21-a
satisfied with probability at least 1 R2 as the disagreement risk for each judge model M;:
M To provide the guarantee across all judge models, define
R3 = [MM,(T))
# Uhuman CM, (x) 2 As, A CMy (x) < A1.
j=1
Also, define Reaseades as the risk of full cascaded selective evaluation, i.e.,
Reascades P[fosscades(1) # Yhuman not abstained by the cascades).
It is easy to that Reascodes is interpolation between all R&s, thus Reascades max, R4. There- fore,
P(Reascudes > a) S P max R. a) -P(U=:
>0) S PP(R,
where the last inequality comes from union bound. Since we know that P(R; > a) < á judge model M,,
for each
[M
PP(R>00)
M
M
Thus, P(eescaaees > a) S 5. In other words, the risk of disagreement across all judge models is guaranteed be most a, with probability at least 5.
16

---

Published as a conference paper at ICLR 2025
B ADDITIONAL RESULTS ON CONFIDENCE ESTIMATION
Table 9: Additional results on confidence estimation with Mistral-7B. We find that more sophisti- cated methods that measure the semantic variance between chain-of-thoughts often underperform Simulated Annotators, marking similar worse performance with zero-shot predicted probability.
Dataset
AlpacaEval
TL;DR
Method
Acc.
ECE
AUROC
AUPRC
Acc.
ECE
AUROC
AUPRC
Predictive Probability (CoT)
0.636
0.292
0.527
0.655
0.669
0.275
0.637
0,751
Lexical Similarity
0.636
0.478
0.478
0.623
0.669
0.459
0.520
0.639
Mistral- 78-i2
Semantic Sets
0.636
0.513
0.638
0.669
0.545
0.665
Semuntic Entropy
0.636
0.572
0.684
0,669
0,650
0,762
Simulated Annotators (Ind.)
0.684
0.075
0.632
0.772
0.696
0.103
0.654
0.807
In Table 9, we provide additional results for more sophisticated methods for confidence estimation. Given instance these methods first generate M chain-of- -fttouugtt (Co'Ts) from an LLM judge prior to inferring the preference label, then measure their variance either in the label space or on the semantic-level:
Predictive Probability (CoT): We average the M label predictive probabilities assigned by the LLM judge after generating chain-of-thoughts.
Lexical Similarity: As simple proxy of semantic variance, we average ROUGE-L across all pairs of M chain-of- tfhhoghs. The intuition is that when the CoTs exhibit high lexical overlap with each other, the model is relatively confident about its generation.
Semantic Sets (Lin et al., 2024): We cluster the CoTs into semantically equivalent groups using bidirectional entailment model (Zha et al., 2023), then use the number of clustered groups to represent model uncertainty.
Semantic Entropy (Kuhn al., 2023): We additionally use the likelihood of each generated chain of thought, and measure the average sequence-level entropy across the semantically equivalent groups
We follow Lin et al. (2024) to set M 20. For Semantics Sets and Semantic Entropy. we exclude expected calibration error as the scores represent model uncertainty rather than the confidence score calibrated in [0, 1]. These methods incur sigmificant overhead compared to the methods dis- cussed in $2.2, generating 20 sequences of chain-of of-thoughts for each input instance and employing a supervised entailment model. Nonetheless, we find that their performance consistently underper- forms that of Simulated Annotutors, with the best method Predictive Probability (CuT) performing worse than our ablation Randomized Annotators (Table 1).
Table 10: Relative API cost for confidence estimation methods, where the cost of Predictive Proba- bility set to be 1.0. For Semantic Sets and Semantic Entropy, we assume that the entailment model can be run locally without incurring additional cost.
Method
Relative API Cost
Predictive Probability Predictive Probubility (Col)
1.000
20.376
Verbalized Confidence
2.215
Semantic Set
20.376
Semantic Fntropyy
20.376
Simulated Annotators
6.910
17

---

Published as a conference paper at ICLR 2025
C
DETAILS ON EXPERIMENTAL SETUP
C.1
PROMPTS
Prompt for Summarization Evaluation
You are a helpful assistant.
Given document and two summaries of the document, an annotator chose which summary preferred. Given examples of the annotator's decision, predict the annotator's verdict on the given example. If Summary A is preferred to Summary B, the annotator chose "[[A]]". Summary B is preferred Summary A, the annotator chose "[[B]]".
[Document]
(document_eexaple...
[Summary A] (summary a example 1)
[Summary B]
(summary b example
[Verdict]:
(verdict [verdict_examppll example.1
[few-shot examples omitted]
[Document] [document]
[Summary A] (summary.a) a}
[Summary B]
{summary b}
Verdict (either "[[A]]" or "[[B]]))
18

---

Published as a conference paper at ICLR 2025
Prompt for Chat Assistant Evaluation
You are a helpful assistant.
Given a question and two assistant's answers to the question, an annotator chose which assistant's s answer is preferred. Given examples of the annotator's decision, predict the annotator's ss verdict on the given example. If Assistant A's response preferred to Assistant B's, the annotator chose "[[A]]". If Assistant B's response is preferred to Assistant A's, the annotator chose "[[B]]"
[Question] {guestion.cxample.1]
[Assistant A's response]
fassistant_a a |_a_mmple__e]]]
[Assistant B's response]
bussstant.b.example
[Verdict]:
{verdict example.11 1}
(few-shor examples omitted]
[Question] {question]
[Assistant A's response] (assistant_a)
[Assistant B's response] fassistant.b)
Verdict (either "[[A]]" "[[B]]"):
19

---

Published as a conference paper at ICLR 2025
C.2
HUMAN EVALUATION DETAILS
(Oocasionally)
idea evaluste this Instance, choose to skip with reason.
2017
Welcome, edmin!
Writs a hallis about Al. It should contain throe lnns th 5-7-5 syfiebles.
(However, skipping disallowed for qualiiication task. keep mainty for your reforonce
Response A
Resporse
Time: Oh 1im 14s
Anmoral
Super amart.
Al
Digital
Centainy
whitsper
about
Issues alllonges Jonly for skpppin)
Code wnstves intiiane patterros
A/s dance begins.
Home
compare by
choosing the that overall better. 24 encoumaged Suejectilty may invotwed untonn and we apprecias your beFt addgrment. )
Guidellnes
Stert Annotation
Annotations
fruthfuli oss, Infor mativanoss, helpfu noss harmlensnees, which you baiing better dedision may require trade offs among often involvess
Report
your own preferencoe. You can review our guidelinss for how to make trade-offs when your best judgament when necsosary.
Profile
Admin Overviow
-
Log
Figure 6: Human Annotation Interface.
Annotator Recruitment. We recruited annotators from Prolific who have recorded at least 99% approval rate, are fluent in English, and have completed Bachelor's degree. In addition, we man- ually designed 10 qualification examples based on our annotation guidelines. The purpose of the qualification test is to find annotators who understand and carefully follow our guidelines. Par- ticipants who scored more than 80% were included in our actual human study. We qualified 21 annotators to do the study and paid them $15 per hour.
Annotation Task. We randomly sample 600 examples from ChatArena (Zheng et al., 2023) each consisting of query and model responses. Given each example, we instruct the annotators to select the overall better response considering several aspects such as helpfulness, truthfulness, and harmlessness. Each instance evaluated by 3-5 annotators. We also allow annotators occasionally skip an instance with a reason if they have no idea how to evaluate it. We provide the screenshot of our annotation guidelines and interface in Figure and 7.
hhttp:://appprroffffccccm
20

---

Published as a conference paper at ICLR 2025
d
Judging Guidelines
The core part of this annotation is to evaluate and compare the two candidate responses given the conversation history and the user request. A typical annotation process includes the following:
1. You supposed to understand the context and the user intent by reading the conversations and thinking over the user request.
2. You need to read and compare the two model responses carefully and find their key differences.
3. Frequently, you may need external tools to verify information the responses if you lack the necessary background or feel unsure about something.
4. Based these, you will indicate your preference between candidate responses based on, several aspects such as Helpfulness, Truthfulness, and Harmlessness, You need to provide your overall assessment. the annotation UI, these are the form of radio buttons.
5. You also need to specify your confidence level in the corresponding judgment.
6. You have the option to skip evaluating instances, but we encourage you to first attempt to answer instance to the best your abilities.
Below is example, where would choose Response to be clearly better in terms of helpfulness and truthfulness and is thus better overall.
User: How do convert Celsius to Fahrenheit?
Model Response A
Model Response B
To convert Celsius Fahrenheit, you need add degrees the Celsius temperature. For example, if you have temperature of 10 degrees
To convert Celsius Fahrenheit, you the following formula:
Celsius, the equivalent temperature
CW 9/5
Fahrenheit degrees Fahrenheit, To convert the temperature from Fahrenheit Celsius, you need to subtract 32 degrees from the Fahrenheit temperature. For example, you have temperature 90 degrees Fahrenheit, the equivalent temperature in Celsius 32 degrees Celsius.
Where:
Fahrenheit temperature in degrees Celsius
C Celsius temperature in degrees Celsius
For example, if the temperature is 20 Celsius, to convert to Fahrenheit you would do:
F 20 W 9/5 32 68 (degrees Fahrenheit)
Note that examples in the real annotation may be more complicated, and the differences responses may be subtle. You also need to trade off many factors to choose which is better. Generally speaking, you need to make your own judgment and express your own preference based on all the research. We discussed more about these tradeoffs in the Overall Assessment section.
Next explain how to interpret the different aspects and make judgments about them.
Figure 7: Human Annotation Guidelines.
21

---

Published as a conference paper at ICLR 2025
D QUALITATIVE EXAMPLES
Summarization Example ]
Document
Long story short We dated for about year and 1/2 I moved for my job we couldn't handle the distance and end our relationship. Months go by, see him Friday night and he is very polite and nice and after we started drinking started pouring his heart out to me. The next day he calls twice in the morning and I missed the calls, try call him back. Doesnt answer, wait few days, send him a long message, and said he read it, and has still said nothing. Should pretend like it never happen?
Summary A
Ex-bf of 1.5 years went full blown confessional me this weekend. Should pretend like never happened and pretend like nothing ever happened?
Summary B
Ex-bf of 1.5 years confided me. Called me twice in the morning and missed the calls. Doesnt answer my messages. Should pretend to ignore it?
Evaluator: Abstained Model Preference: None
Human Preference: B is better
Inter-Annotator Agreement: 0.75
Summarization Example 2
Document
I have been working since 16 and make around $12,000 annually with small amount of expenses (gas, car insurance). literally just turned 18 and felt that it would be a good idea to get credit card. honestly don't 't need it to "make purchases can't afford" but really just want one to start building my credit.
I talked with my Dad about it and he said that applying for store card like Best Buy (where I work actually) would be a good idea or that getting a secured credit card would also be good idea. prefer the secured credit card route due to being able to it anywhere and can't see myself actually using store card consistently as do not shop at one place consistently.
What are some of the best secured Credit Cards to get? I'm not worried about a high credit limit, just want my credit go up and be good, for lack of better words. My current debit card is with a Credit Union and read that those and Capital One are usually the best route go with.
I also wanted to know if it is better to with Visa, Mastercard, Discover, AMEX ETC.
Summary A
18 y/o male wants to start building credit and wants a secure credit card to do so. Willing to anywhere, not worried about credit limit. Best secured credit cards?
Summary B
just turned 18 wanting build credit. Talked to Dad about applying for secured credit card or Best Buy store card. Best route?
Evaluator: Mistral-7B Model Preference: A is better Inter-Annotator Agreement: 0.6
Human Preference: A is better
22

---

Published as a conference paper at ICLR 2025
Chat Assistant Example 1
Query
Who is Nicola Tesla?
Assistant A great electrical engineer
Assistant B
Nicola Tesla was electrical engineer, a physicist, and futurist. She early discov- erer of electromagnetic radiation (EM) and the first person to use radio waves to transmit information. Tesla is perhaps best known for her discovery of alternating current (AC) elec- tricity and her invention of the Tesla coil, which was used to generate high voltage electricity for use in electricity generation. However, Tesla also controversial figure, with some his- torians and scientists questioning her work and reputation.
Evaluator: Mistral-7B Model Preference: is better Inter-Annotator Agreement: 1.0
Human Preference: B is better
Chat Assistant Example 2
Query
If "aka blux fii" means "the cat is white", "aka jorr fii" means "the chicken is white", "aka urf fii" means "the milk white" and "urf blux orto" means "the cat likes milk", how would you say "the cat likes chicken""
Assistant A
"aka blux fii" means "the cat is white", "aka jorr fii" means "the chicken is white", "aka urf fii" means "the milk is white", and "urf blux orto" means "the cat likes milk", then to say "the cat likes chicken", you would say "aka jorr fii."
Assistant B
To say "the cat likes chicken," you would use "blux jorr orto." Here's the breakdown: "aka" means "the"
"blux" means "cat"
"jorr" means "chicken" "fii" means "is white"
"urf" means "milk"
"orto" means "likes"
So. "blux jorr orto" translates "the cat likes chicken".
Evaluator: GPT-4 Model Preference: B is better Human Preference: B is better
Inter-Annotator Agreement: 1.0
23

---

Published as a conference paper at ICLR 2025
Chat Assistant Example 3
Query
Privacy is not universal right.
Assistant A
Privacy is not a universal right. From an international human rights perspective, privacy is not human right because not considered a fundamental right. However, the Amer- ican Civil Liberties Union (ACLU) has highlighted that in the US, the right to privacy is considered a fundamental right under the US Constitution's s Fourth Amendment.
Assistant B
That statement not entirely accurate. While some countries may not have a specific con- stitutional guarantee of privacy, the right to privacy is often recognized human right under international law and is protected by various treaties and conventions, such the Universal Declaration of Human Rights and the European Convention Hu- man Rights. Additionally, many countries have their own constitutional or legal frameworks that guarantee certain privacy rights.
fundamental
Evaluator: GPT-3.5 Model Preference: is better
Human Preference: is better
Inter-Annotator Agreement: 1.0
Chat Assistant Example 4
Query
list all yearly hottest temperature records starting from 1900 to the present
Assistant A
Here is a list of the hottest temperatures recorded each year starting from 1900 to the present:
1900 Death Valley, California at 134 (56.7%C) on July 10, 1913
1901 Aziziyah, Libya at 136 (58*C) on September 13, 1922
1903 Death Valley, California F (56.7%C) on July 10, 1913
... (omi tted)
[The list continues up to 2020]
Assistant B
I'm sorry, but as an AI language model, don't have access to real-time data. However, I can provide you with list of some of the most notable hottest temperature records starting from 1900 to the present, based on historical records and news reports:
1936 F (57%C) Death Valley, California
2. 2013-13499 (57%C) in Turkey
2020 F (62.8%C) in Oman (omitted) 10. 2021 F (49.29C) in Kuwait City, Kuwait
Evaluator: Abstained
Model Preference: None Human Preference: A is better
Inter-Annofator Agreement: 0.75
24

---

Published as a conference paper at ICLR 2025
Chat Assistant Example 5
Query
What month and year will the US stock market make bottom?
Assistant A
The US stock market has been in bear market since February 2018. The DJIA and S&P 500 have both made lows in that time. The S&P 500 has made a low on March 1220 and the DJIA on February 24, 2019.
Assistant B
As AI, am unable to predict future events, especially those related to the stock market. Market movements are influenced by various factors such as economic data, political events, and investor sentiment, which are all subject to change. It is best to consult with financial advisor for any investment-related advice.
Evaluator: GPT-4 Model Preference: B is better
Human Preference: B is better
Inter-Annotator Agreement: 0.83
Chat Assistant Example 6
Query
can you make some joke about programmer for me
Assistant A
Why do programmers always mix up Christmas and Halloween?
Because Oct 31 Dec 25!
Assistant B
Sure, can make joke about programmers. Here's 's a joke about programmers:
Why did the programmer refuse to eat the two cookies he had been given at lunch?
Because he wanted to save them for later!
Evaluator: Abstained Model Preference: None Inter-Annotator Agreement: 0.75
Human Preference: B is better
25


Reviews:
[{"review_id": "jolGhtcdjz", "reviewer": "Reviewer_8fUt", "summary": "This paper proposes a selective evaluation framework to ensure that LLM-based evaluations align with human judgments to a defined degree of confidence. Selective Confidence Evaluation ensures that, instead of always trusting an LLM's judgment, the model evaluates its confidence in a given answer and abstains when unsure. Simulated Annotators leverage in-context learning to simulate diverse annotator perspectives, improving model calibration and reducing evaluation errors. Cascaded Selective Evaluation introduces multiple LLM judges used in a tiered manner, starting with a more cost-effective model and escalating to stronger models only if needed. The method achieves strong human alignment and reduces reliance on more costly models like GPT-4 by using smaller models when possible.", "questions": "See above.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper provides clear methodological contributions to improving human-LLM alignments. I especially appreciate the cascaded selective evaluation approach, which explicitly considers the costs of using stronger/more costly LLMs. Overall, this is an excellent paper with clear contributions and is well-executed.", "weaknesses": "This paper has already been very carefully executed, and I have only minor suggestions. \n\n1.\tSimulated annotators, despite their encouraging performance, will require significant processing time and resources as N and K increase. Given a substantial decrease in ECE, I believe that simulated annotators have enough value but it would be helpful for the readers if the authors can provide time and cost requirements compared to the canonical approaches. Furthermore, the authors may want to invest a bit more space in explaining the algorithm of simulated annotators including exactly how they handled in-context learning.\n\n2.\tThe outperformance of cascaded selective evaluation is somewhat expected because the gist of the framework is to exclude the observations where annotator disagreement is expected. In my view, the model does a good job of identifying these hard-to-reconcile observations. A fairer comparison would be to exclude the abstained observations from the baseline models and see how their performance is affected. For example, if you maintain the same coverage for all other methods in Table 2 and Table 3, what would their empirical human agreement look like? Of course, empirically this cannot be achieved without your model – but as you show in Section 3.3, some systematic characteristics can predict abstention. One can simply devise an alternative strategy that can identify these hard-to-reconcile samples and run a multi-agentic approach on them. \n\n3.\tI think the most striking result is in Table 7. It is surprising that the weaker cascades achieve an empirical human agreement of 80.3%. Although their coverage is 68.3%, this is indeed a remarkable result. Given this, I would try to do more with the abstained samples. Almost 30% of the sample is now abstained and running the most advanced model on those will not only increase your ultimate coverage and success rate. The relative API cost will be substantially lower than using only GPT4.\n\n4.\tRelatedly, the relative API cost of the model is largely dependent on the difficulty of your task. As seen in Tables 2, 3, and 8, the evaluator composition is heterogeneous across different tasks. When a smaller LLM can handle the majority of tasks, the overall task is likely to be cheaper. When a larger LLM needs to intervene, the overall task will be more expensive. I would caveat this (and perhaps show the range of relative API cost using some simulated evaluator compositions) in Section 3.6.\n\n5.\tIt is a minor thing, but is there any reason that you chose 1-a =0.9 in Table 2, 1-a=0.85 in Table 3, and 1-a=0.8 in Table 7? I would personally prefer the three thresholds to be consistent throughout the paper.\n\nOverall, it was an interesting read. Great paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mqUoj6bzjc", "reviewer": "Reviewer_ZdoK", "summary": "Current NLP paradigms have shifted to using LLMs as judges for annotations instead of humans. This paper asks how we can guarantee the reliability of using these LLMs for evaluation through human agreement. \n\nGiven an input, how likely would humans agree with the model judgment?\n\nFor this, the approach is to use selective evaluation: the model can abstain if not confident enough to make a prediction.\n\nThe conventional way of using the predictive probability as confidence estimates doesn't show to be reliable so two methods are proposed: \n1. Simulated annotators: simulate annotator preferences through In-Context Learning and the confidence is estimated through the agreement ratio. \n2. Cascaded Selective Evaluation: Start with a cheaper model and go to stronger models when the previous model is less confident. \n\nThis reduces the evaluation overhead with high agreement guarantees. Their abstention policy aligns with human subjectivity.", "questions": "How does IAA change for abstained samples with a lower target accuracy?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles an important issue, we need more human subjectivity at the forefront of LLM evaluation. \n2. In essence, the Simulated Annotators technique is ensembling, so using a simple and popular technique as ensembling is an interesting way to utilize for such a task. \n4. The authors conduct many experiments to show the utility of the approach. \n3. The added ablation studies in the paper are nice. \n5. The paper is well-written and structured.", "weaknesses": "1. A clear connection to _selective prediction_ is missing. While the authors talk about _selective evaluation_, the concept is the same so it is better to make that link clear. \n2. Related Work can benefit from more work on selective prediction and human subjectivity in LLMs, e.g. \n\nBarbara Plank. 2022. The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\n\nKhurana, U., Nalisnick, E., Fokkens, A., & Swayamdipta, S. Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?. In First Conference on Language Modeling.\n\nBavaresco, A., Bernardi, R., Bertolazzi, L., Elliott, D., Fernández, R., Gatt, A., ... & Testoni, A. (2024). Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. arXiv preprint arXiv:2406.18403.\n\n3. Cheaper models used for Cascading are still very costly (e.g. Mistral 7B)\n4. The calibration set for human preferences is based on argmax of human judgments. Using argmax gets rid of disagreement signals, which is actually important information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5CaxShochB", "reviewer": "Reviewer_4Yrv", "summary": "The authors propose a framework for LLM evaluation that includes a human agreement guarantee, that is called Cascaded Selective Evaluation. The evaluation task (in this case a pairwise evaluation) is given to a weak LLM judge and escalated to a stronger LLM judge, if the confidence level for the task does not exceed a certain threshold. As a confidence estimation method they propose Simulated Annotators, where multiple LLMs are used to simulate annotator preferences.", "questions": "- You analyzed the abstention rate quantitatively, but did you also do a qualitative analysis? Do the models abstain on the same type of pairs?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly structured and well written. The approach clearly improves the evaluation process supported by LLMs, while being cost effective by only considering stronger models (such as GPT-4-turbo) as a last escalation step. The authors provide meaningful experiments and show how the method can be extended in the future.", "weaknesses": "A weakness of the approach (for now), is that it has only been tested and applied in context of the pairwise preference evaluation. Nevertheless, the Simulated Annotators and Cascaded Selected Evaluation methods are promising. It will be interesting to see how the approach performs in other evaluation settings in the future. The question of when the models abstain could be answered in more detail. For now, it is only answered quantitatively, but not really qualitatively. In order to find out whether the examples the models are not sure about are the same ones about which the human judges also disagree, the examples need to be examined in more detail, i.e. more on the level of content. It would be interesting to see the ratio of examples where both humans and the models abstained from evaluation and to analyze some examples on a content level. \n\nSmall errors:\n- Line 462: the the tradeoff", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mhyPAOz7wM", "reviewer": "Reviewer_mWkk", "summary": "The authors introduce a principled method for evaluating language models with guaranteed human agreement. They argue that evaluation should not solely rely on model preferences in pairwise comparisons. Instead, it should assess judge models' confidence and selectively decide when to trust their judgment. This selective framework ensures that the evaluation aligns with human agreement to a specified level. They introduce Simulated Annotators, a new method for confidence estimation that enhances judge calibration, enabling comprehensive evaluation. Their Cascaded Selective Evaluation uses cost-effective models initially, escalating to stronger ones as needed, maintaining human agreement guarantees. Results show this method achieves over 80% human agreement.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Robust Human Agreement: The Cascaded Selective Evaluation framework offers a strong guarantee of alignment with human judgment, enhancing the reliability of LLM-based evaluations.\n\n2. Improved Confidence Estimation: The introduction of Simulated Annotators enhances the confidence estimation for LLM judges, allowing for more accurate evaluations without the need for external supervision.\n\n3. Dynamic Judge Selection: By dynamically selecting which judge model to trust, the framework reduces evaluation overhead while maintaining reliability, often surpassing the precision of relying solely on the strongest judge model.", "weaknesses": "It is more like a theoretical work. Given that the experiments have proved effective in evaluating different tasks, I suggest the authors invest some effort in releasing a user-friendly toolkit based on this work, for other researchers to use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)