Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement

## Abstract

Abstract We present a principled approach to provide LLM-based evaluation with a rigorous guarantee of human agreement. We first propose that a reliable evaluation method should not uncritically rely on model preferences for pairwise evaluation, but rather assess the confidence of judge models and selectively decide when to trust its judgement. We then show that under this selective evaluation framework, human agreement can be provably guaranteed—such that the model evaluation aligns with that of humans to a user-specified agreement level. As part of our framework, we also introduce Simulated Annotators , a novel confidence estimation method that significantly improves judge calibration and thus enables high coverage of evaluated instances. Finally, we propose Cascaded Selective Evaluation , where we use cheaper models as initial judges and escalate to stronger models only when necessary—again, while still providing a provable guarantee of human agreement. Experimental results show that Cascaded Selective Evaluation guarantees strong alignment with humans, far beyond what LLM judges could achieve without selective evaluation. For example, on a subset of Chatbot Arena where GPT-4 almost never achieves 80% human agreement, our method, even while employing substantially cost-effective models such as Mistral-7B, guarantees over 80% human agreement with almost 80% test coverage. \faGithub

## 1 Introduction

Imagine we need to evaluate 1 million pairs of model generations—a task whose scale makes human annotation impractical, if not impossible. Today, a commonly proposed solution is to ‘just ask GPT-4’(Zheng et al., ; Dubois et al., ), realizing a tempting idea that large language models (LLMs) may serve as a scalable substitute for manual annotation (Chiang & Lee, ). However, this compelling prospect comes with a crucial caveat—LLM-based evaluation would always remain, at best, an approximation of human judgement. Without a provable guarantee of reliability, it is no surprise that the judge model has to be chosen heuristically, often times to be the strongest and the most expensive model available (e.g., GPT-4). Yet, prior works show that even the strongest judge models suffer from systematic biases (Wang et al., ; Thakur et al., ) and over-confidence (Xiong et al., ), casting doubt on the dependability of these models. This raises a fundamental question: How can we guarantee the reliability of LLM-based evaluation?

In this work, we aim to improve the reliability of LLM-based evaluation by providing a rigorous guarantee of human agreement. That is, given a user-defined risk level $\alpha$, we provide a guarantee that, for an unseen instance $x$,

$$ $P(\textit{LLM preference on $x$ agrees with human}\,|\,\textit{LLM evaluates $ x$})\geq 1-\alpha.$ $$

To provide this guarantee, we posit that a reliable evaluation framework should not only consider the preference of a model, but also the validity of the preference—i.e., how likely humans would agree with the model judgement. When a model cannot confidently evaluate a given instance, we should not rely on its evaluated result. This motivates *selective evaluation*: we evaluate an instance with an LLM judge, assess the confidence that humans would agree with its evaluation, then decide whether or not to trust the evaluated result. We show that under this framework, human agreement can indeed be guaranteed—both theoretically and empirically—by choosing when to trust the model through fixed sequence testing (Bauer, ) on a small calibration set.

The practicality of selective evaluation lies not only in achieving high agreement with humans, but also in maximizing the coverage of evaluated instances without abstention—a factor that depends on the quality of confidence measure. We find that existing methods for confidence estimation (e.g., predictive probability) are brittle even with the strongest judge model, as they tend to overestimate human agreement. We then propose Simulated Annotators, a novel method to simulate diverse annotator preferences through in-context learning and estimate confidence as an agreement ratio between the simulations. Without relying on any external supervision, *Simulated Annotators* significantly improves both the calibration and failure prediction of LLM judges. As a result, selective evaluation can be done with high coverage while satisfying the prescribed human agreement level.

Moreover, since our framework provides a model-agnostic guarantee of human agreement, we no longer have to rely solely on GPT-4 for evaluation. We propose Cascaded Selective Evaluation (Figure 1), where we start from a substantially cheaper LM (e.g., Mistral-7B) as a judge, and escalate to a stronger model only when the previous judge is not sufficiently confident—all while guaranteeing high agreement with humans. Importantly, users do not have to manually choose when to use which judge model; given a user-specified risk tolerance, the abstention policy is automatically decided to maintain risk control.

We test our method across preference domains including summarization and real-world user-chatbot interaction, and find that Cascaded Selective Evaluation significantly reduces the evaluation overhead while guaranteeing high agreement. For example, our method can outperform GPT-4 by achieving over $80\%$ human agreement in ChatArena (Li et al., ), while covering $79.1\%$ of all samples, among which $88.1\%$ are evaluated by substantially cheaper Mistral-7B or GPT-3.5 instead of GPT-4. We also show that our abstention policy closely aligns with the subjectivity perceived by humans, rather than relying on shallow features such as length ratio or token overlap. Overall, our work suggests a principled approach to make LLM-based evaluation more reliable yet cost-effective, without exclusively counting on the capabilities of the most advanced LLMs as judges.

Figure: Figure 1: Illustration of Cascaded Selective Evaluation. We start with a small, cost-effective model as initial judge, estimate its confidence, and escalate to a stronger model only when the previous judge is not confident. By calibrating when to trust which judge model, our method provides a rigorous guarantee of human agreement while employing substantially cheaper judge models.
Refer to caption: x1.png

## 2 Cascaded Selective Evaluation

When performing pairwise evaluation with LLMs, we want a type of guarantee that the model agrees with the majority of human annotators. To realize this guarantee, we propose selective evaluation, a framework that employs an abstention policy to decide whether an LLM is sufficiently confident to evaluate an instance. More formally, let $f_{\textit{LM}}:\mathcal{X}\rightarrow\mathcal{Y}$ denote the LLM judge, where the input $x\in\mathcal{X}$ consists of a query $q$ and a pair of generations $(a_{1},a_{2})$, and the output $y\in\mathcal{Y}$ is a preference label between $a_{1}$ and $a_{2}$ (e.g., $a_{1}\succ a_{2}$).
Introducing a confidence measure $c_{\textit{LM}}:\mathcal{X}\rightarrow[0,1]$, we define selective evaluator as:

$$ $(f_{\textit{LM}},c_{\textit{LM}})(x)=\begin{cases}f_{\textit{LM}}(x)&\text{if }c_{\textit{LM}}(x)\geq\lambda,\\ \quad\emptyset&\text{otherwise}.\end{cases}$ (1) $$

An example of $c_{\textit{LM}}$ is the probability assigned by $f_{\textit{LM}}$ to its predicted label (predictive probability), a popular choice of confidence measure in selective classification (Geifman & El-Yaniv, ). $\lambda$ is a hyperparameter that trades off the precision (i.e., the accuracy of evaluator aligning with human judgements) against the coverage (i.e., the ratio of instances evaluated without abstention). The key advantage of selective evaluation is that by calibrating $\lambda$ in a principled manner, we can provide a rigorous guarantee of human agreement while maintaining high coverage. That is, given a user-defined risk tolerance $\alpha$ and an error level $\delta$, one can provably guarantee that

$$ $P(f_{\textit{LM}}(x)=y_{\textit{human}}|c_{\textit{LM}}(x)\geq\lambda)\geq 1-\alpha$ (2) $$

is satisfied with probability at least $1-\delta$. In the following sections, we illustrate how to search for $\widehat{\lambda}$ that satisfies this guarantee (§2.1), how to define a good confidence measure $c_{\textit{LM}}$ (§2.2), and how to extend selective evaluation from a single model to cascades of judge models (§2.3).

### 2.1 Providing Human Agreement Guarantee

Our human agreement guarantee can be satisfied by formulating selection of $\lambda$ as a multiple hypothesis testing problem (Bates et al., ; Angelopoulos et al., ). Specifically, given access to a small calibration set $D_{\textit{cal}}\sim P(x,y_{\textit{human}})$ of human preferences(^1^11When we have multiple human annotation $y_{i}$s per input $x$, we define $y_{\textit{human}}\coloneqq\operatorname*{arg\,max}_{y}\sum_{i}\mathbbm{1}\{y_ {i}=y\}$.), we can measure an empirical risk $\widehat{R}(\lambda)$ of disagreeing with humans when using a threshold $\lambda$:

$$ $\widehat{R}(\lambda)=\frac{1}{n(\lambda)}\sum_{(x,y_{\textit{human}})\in D_{ \textit{cal}}}\mathbbm{1}\{f_{\textit{LM}}(x)\neq y_{\textit{human}}\land c_{ \textit{LM}}(x)\geq\lambda\},$ (3) $$

where $n(\lambda)\coloneqq\sum_{(x,y_{\textit{human}})\in D_{\textit{cal}}}\mathbbm{1
}\{c_{\textit{LM}}(x)\geq\lambda\}$. Since the empirical risk is a binomial random variable with $n(\lambda)$ trials, we can compute the exact $(1-\delta)$ upper confidence bound of the risk as:

$$ $\widehat{R}^{+}(\lambda)=\sup\big{\{}R:P(\operatorname*{Bin}(n(\lambda),R)\leq \lceil n(\lambda)\widehat{R}(\lambda)\rceil)\geq\delta\big{\}}.$ (4) $$

Note here that the risk is near-monotonic, i.e., it tends to increase as $\lambda$ decreases. This allows us to use fixed sequence testing (Bauer, ), wherein we test from the largest value of $\lambda$ (e.g., 0.999) to a progressively smaller value, and stop at the last time $\widehat{R}^{+}(\lambda)$ is below the target risk $\alpha$.

$$ $\widehat{\lambda}=\inf\big{\{}\lambda:\widehat{R}^{+}(\lambda^{\prime})\leq \alpha\text{ for }\forall\,\lambda^{\prime}\geq\lambda\big{\}}.$ (5) $$

###### Theorem 1

Consider a threshold $\widehat{\lambda}$ chosen as above, and a selective evaluator $(f_{\textit{LM}},c_{\textit{LM}})$ operating based on $\widehat{\lambda}$. Then, Equation (2) is satisfied with probability at least $1-\delta$.

We leave the proof in §A.1. Our test procedure resembles that of selection with guaranteed risk (Geifman & El-Yaniv, ), but we adopt fixed-sequence testing instead of Bonferroni correction, which may be too conservative for a large hypothesis space. Compared to recent works on risk control for LLMs, we provide exact, tighter bound on the selective risk (instead of approximating it; Yadkori et al. ), and guarantee with high probability that the risk is below $\alpha$ conditional on the calibration data (as opposed to being marginally centered at $\alpha$; Gui et al. ).

###### Theorem 1

### 2.2 Simulated Annotators

While human agreement guarantee can be met with any choice of (near-monotonic) confidence measure, the coverage of selective evaluation essentially depends on how good this measure is—i.e., whether $c_{\textit{LM}}$ truly reflects if humans would agree with LLM evaluation. In this section, we first test out popular confidence estimation methods for LLMs and show that they fail to accurately represent model uncertainty. We then introduce Simulated Annotators as a promising alternative.

**Table 1: Performance of confidence measures across judge models. Simulated Annotators consistently outperforms baselines both in calibration and failure prediction, especially improving the reliability of weaker judge models (GPT-3.5-turbo and Mistral-7B).**
| Dataset | AlpacaEval | TL;DR |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Acc. | ECE $\downarrow$ | AUROC | AUPRC | Acc. | ECE $\downarrow$ | AUROC | AUPRC |  |
| GPT-4-<br>turbo | Predictive Probability | 0.724 | 0.217 | 0.642 | 0.852 | 0.760 | 0.196 | 0.731 | 0.890 |
| Verbalized Confidence | 0.724 | 0.215 | 0.550 | 0.774 | 0.760 | 0.194 | 0.548 | 0.792 |  |
| Randomized Annotators | 0.720 | 0.113 | 0.705 | 0.866 | 0.779 | 0.079 | 0.734 | 0.905 |  |
| Simulated Annotators (Maj.) | 0.730 | 0.106 | 0.718 | 0.873 | 0.783 | 0.062 | 0.755 | 0.921 |  |
| Simulated Annotators (Ind.) | 0.734 | 0.095 | 0.723 | 0.877 | 0.788 | 0.039 | 0.755 | 0.921 |  |
| GPT-3.5-<br>turbo | Predictive Probability | 0.644 | 0.293 | 0.581 | 0.691 | 0.667 | 0.228 | 0.653 | 0.786 |
| Verbalized Confidence | 0.644 | 0.306 | 0.505 | 0.595 | 0.667 | 0.211 | 0.607 | 0.716 |  |
| Simulated Annotators (Ind.) | 0.694 | 0.058 | 0.632 | 0.793 | 0.725 | 0.043 | 0.704 | 0.842 |  |
| Mistral-<br>7B-it | Predictive Probability | 0.618 | 0.374 | 0.457 | 0.579 | 0.661 | 0.306 | 0.613 | 0.735 |
| Verbalized Confidence | 0.618 | 0.414 | 0.490 | 0.627 | 0.661 | 0.335 | 0.578 | 0.680 |  |
| Simulated Annotators (Ind.) | 0.684 | 0.075 | 0.632 | 0.772 | 0.696 | 0.103 | 0.654 | 0.807 |  |

Figure: Figure 2: Reliability plot for confidence estimation methods, using GPT-4 as judge on AlpacaEval. Dashed lines denote perfect calibration, and darker bars denote more samples in the corresponding bins. Simulated Annotators reduces expected calibration error by 50% compared to the baselines, mitigating over-confidence observed in predictive probability and verbalized confidence.
Refer to caption: x2.png

#### Existing Methods.

We first consider two types of existing confidence measure(^2^22We also consider more sophisticated methods (e.g., sampling chain-of-thoughts and estimating their semantic entropy; Kuhn et al. ) in §B, but find their performance to be mostly on-par with the above methods, despite the increased cost.)—(1) predictive probability: as the most straightforward proxy of confidence, we use the likelihood of preference label predicted by the LLM judge, i.e., $c_{\textit{LM}}(x)=\max_{y}p_{\textit{LM}}(y|x)$; (2) verbalized confidence: following Tian et al. (), we directly prompt the LLM judge to express its confidence in a scalar value. We evaluate these methods in terms of the expected calibration error (Naeini et al., ), AUROC and AUPRC, using the non-tied instances in two standard benchmarks: AlpacaEval (Dubois et al., ) for open-domain chat assistant and TL;DR (Stiennon et al., ) for summarization.

#### Canonical methods overestimate human agreement.

The results are shown in Table 1 and Figure 2 (left, middle). Unlike prior reports that the canonical methods work well for tasks with small label space (Kadavath et al., ; Tian et al., ), they consistently lead to over-confidence when used for preference evaluation. Notably, the results are pronounced even with the strongest LLM judge GPT-4-turbo, although its agreement with human majority is known to be comparable to an average human annotator (Sottana et al., ; Zheng et al., ).

The above results suggest that simply achieving human-level performance may not be sufficient for reliable evaluation: while an LLM judge can be as accurate as a single human annotator, it tends to be over-confident in estimating its agreement with the majority of annotators. This contrasts with the standard practice in human evaluation, which involves collecting multiple annotations per instance and assessing the level of agreement between them; the evaluation is deemed reliable only when there is high inter-annotator agreement. Motivated by this discrepancy, we derive Simulated Annotators, a confidence measure that simulates diverse annotator preferences with in-context learning. Concretely, given $K$ (e.g., 3) examples of preference annotations per $N$ (e.g., 5) human annotators, we simulate annotators by $K$-shot prompting the model for $N$ times and ensemble the results:

$$ $c_{\textit{LM}}(x)=\max_{y}\frac{1}{N}\sum_{j=1}^{N}p_{\textit{LM}}(y|x;(x_{1, j},y_{1,j}),\cdots,(x_{K,j},y_{K,j})),$ $$

where $(x_{i,j},y_{i,j})$ is the $i$-th in-context example from the $j$-th annotator. Likewise, the judge prediction $f_{\textit{LM}}(x)$ is defined as $\operatorname*{arg\,max}_{y}\sum_{j=1}^{N}p_{\textit{LM}}(y|x;(x_{1,j},y_{1,j}
),\cdots,(x_{K,j},y_{K,j}))$. Intuitively, the confidence $c_{\textit{LM}}$ becomes low when multiple simulated annotators disagree with each other. We typically set $K,N\leq 5$, and ablate the effect of number of simulated annotators in §3.5.

#### Simulated Annotators improves reliability, even for weaker judge models.

The results with $K=N=5$ are shown in Table 1 (Simulated Annotators (Ind.)) and Figure 2 (right). Simulated Annotators significantly outperforms popular confidence measures, reducing ECE by 50% and improving AUROC by 13% for GPT-4. Surprisingly, our method improves the reliability of even the weaker judge models—while they do underperform GPT-4 in accuracy, their estimated confidence is on-par or even better than GPT-4 when using the baseline confidence measures.

Despite the substantial performance gain in Simulated Annotators, it remains unclear whether the gain truly comes from simulating diverse human preferences. We analyze this using two ablations on the few-shot examples given to the LLM judge: (1) randomized annotators: using the same set of inputs $x_{i,j}$ but random-assigning labels $y_{i,j}\sim\operatorname*{Ber}(0.5)$, and (2) simulated annotators (majority): using $(x_{i,j},y_{i,\textit{human}})$ where $y_{i,\textit{human}}$ is the majority preference of human annotators given input $x_{i,j}$.(^3^33Here, as each input instance is associated with a single majority label $y_{\textit{human}}$, we induce difference between simulations by using different set of inputs $x_{i,j}$ per simulated annotator.) We fix $K=5$ and $N=5$ for all ablations. As shown in Table 1, Simulated Annotators (Maj.) is consistently better than Randomized Annotators, but slightly underperforms Simulated Annotators (Ind.) that models individual preference. The performance of majority-based Simulated Annotators, however, is encouraging, as the method can be applied to cases where we do not have access to diverse human annotations per each instance $x$. Overall, the result demonstrates that simulating diverse annotator preferences is helpful, and even in the absence of such data, our method improves the reliability of LLM judges over the existing methods.

### 2.3 Cascading Selective Evaluators

Figure: Algorithm 1 Cascaded Selective Evaluation

The strong performance of Simulated Annotators demonstrates that even the weaker LLM judges—despite not as accurate as a larger judge—may accurately predict when they are likely to agree with human annotators. Leveraging this finding, we propose Cascaded Selective Evaluation, as illustrated in Figure 1 and formalized in Algorithm 1. Given a list of judge models $\mathcal{M}$, we start with a weaker yet cheaper model as an evaluator, and only when the model is not sufficiently confident, we iteratively move on to a stronger model. Notably, the confidence threshold $\lambda$ for each judge model can be chosen following the same process as in §2.1, providing the guarantee of risk control across the cascades of models (see §A.2 for details). This way, selective evaluation can operate at a significantly lower cost than using the strongest model from the start, while still maintaining a rigorous guarantee of human agreement.

## 3 Experimental Results

**Table 2: Comparison against baselines on TL;DR, with target agreement level $1-\alpha=0.9$. The results are averaged across 1000 runs with random data split. Guarantee Success Rate is defined as the ratio of successful runs that achieve empirical human agreement larger than or equal to $1-\alpha$. Cascaded Selective Evaluation is the only method that achieves high guarantee success rate, while maintaining high coverage.**
| Method | Evaluator Composition (%) | Coverage (%) | Guarantee Success<br>Rate (%) |  |  |
| --- | --- | --- | --- | --- | --- |
| Mistral-7B | GPT-3.5-turbo | GPT-4-turbo |  |  |  |
| No Selection | 0.0 | 0.0 | 100.0 | 100.0 | 0.0 |
| Heuristic Selection | 0.0 | 0.0 | 100.0 | 89.6 | 42.0 |
| Cascaded Heuristic Selection | 59.6 | 15.0 | 25.5 | 64.6 | 0.0 |
| Point-Estimate Calibration | 100.0 | 0.0 | 0.0 | 5.6 | 54.7 |
| 0.0 | 100.0 | 0.0 | 9.4 | 79.0 |  |
| 0.0 | 0.0 | 100.0 | 57.7 | 47.5 |  |
| Cascaded Selective Evaluation | 28.3 | 28.2 | 43.5 | 55.7 | 90.8 |

Figure: Figure 3: TL;DR results. Cascaded Selective Evaluation guarantees human agreement far beyond a level achievable by GPT-4 without abstention (Left), while employing substantially weaker judge models (Right). Solid blue line denotes average human agreement over 1000 runs on the dataset, and the light blue region denotes the min / max agreement within the 1000 runs.
Refer to caption: x3.png

### 3.1 Evaluating Generated Summaries

#### Experimental Setup.

We first test our approach for evaluating summaries on TL;DR dataset (Stiennon et al., ). We use a cascade of Mistral-7B-instruct-v0.2 (Jiang et al., ), GPT-3.5-turbo and GPT-4-turbo (Achiam et al., ) as judges. Observing that the dataset provides multiple human annotations per input, we use Simulated Annotators (Ind.) with $K=N=5$. We fix the size of calibration set $|D_{\textit{cal}}|=500$ and $\delta=0.1$, and run the experiments for 1000 random splits of calibration and test set. For baselines, we consider: (1) Heuristic Selection: using GPT-4 as a judge and setting $\lambda=1-\alpha$, assuming perfect calibration; (2) Cascaded Heuristic Selection: a variant of Heuristic Selection using the same cascades of judge models as ours; (3) Point-Estimate Calibration: setting $\lambda$ as the smallest value that satisfies $\widehat{R}(\lambda)\leq\alpha$ in $D_{\textit{cal}}$, without hypothesis testing.

In Figure 3, we show that human agreement guarantee is satisfied with our approach across all levels of target human agreement, far beyond what GPT-4 can achieve without abstention. Notably, unlike prior works (Gui et al., ; Mohri & Hashimoto, ) that only controls the risk in expectation over calibration sets (solid blue line), our method guarantees with high probability that each individual run would satisfy the target agreement level (light blue region). Moreover, as shown in right plot, the high agreements can be achieved while the majority of evaluation are done with substantially smaller LLMs than GPT-4. For example, our method can outperform GPT-4 with 80% human agreement, while 75% of the evaluations are done by Mistral-7B or GPT-3.5.

We also compare against selective baselines in Table 2, in terms of their coverage and guarantee success rate. All baselines fail to provide meaningful guarantee success rate without significantly sacrificing the coverage. This includes Point-Estimate Calibration, which makes use of the test statistics in calibration data. On the contrary, Cascaded Selective Evaluation achieves over 90% success rate—just as expected by setting $\delta=0.1$—attesting to its reliability.

**Table 3: Comparison to baselines on ChatArena, with target agreement level $1-\alpha=0.85$. The results are averaged across 1000 runs with random data split. Consistent with results on TL;DR, our method successfully guarantees target agreement level while maintaining high coverage.**
| Method | Evaluator Composition (%) | Coverage (%) | Guarantee Success<br>Rate (%) |  |  |
| --- | --- | --- | --- | --- | --- |
| Mistral-7B | GPT-3.5-turbo | GPT-4-turbo |  |  |  |
| No Selection | 0.0 | 0.0 | 100.0 | 100.0 | 0.0 |
| Heuristic Selection | 0.0 | 0.0 | 100.0 | 95.2 | 0.1 |
| Cascaded Heuristic Selection | 57.1 | 15.2 | 27.7 | 79.7 | 0.3 |
| Point-Estimate Calibration | 100.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 0.0 | 100.0 | 0.0 | 40.5 | 57.2 |  |
| 0.0 | 0.0 | 100.0 | 60.9 | 54.4 |  |
| Cascaded Selective Evaluation | 23.7 | 58.8 | 17.5 | 63.2 | 91.0 |

Figure: Figure 4: ChatArena results. Our approach guarantees target human agreement level (Left) while majority of evaluations are done with weaker judge models, Mistral-7B and GPT-3.5 (Right).
Refer to caption: x4.png

### 3.2 Evaluating LLM-based Chat Assistants

#### Experimental Setup.

Next, we test our approach for evaluating general-purpose LLM assistants on two datasets: Chat(bot) Arena (Zheng et al., ) with real-world user-assistant interaction(^4^44We use an evaluation set with 5.2k instances sampled by Li et al. ().) and Auto-J (Li et al., ), a curated benchmark for meta-evaluation of LLM judges. We employ the same cascades of models as in §3.1, but this time we use Simulated Annotators (Maj.), as both datasets only provide one human annotation per input $x$. We set $K=N=5$ and $|D_{\textit{cal}}|=500$ for ChatArena, and $K=N=3$, $|D_{\textit{cal}}|=392$ for Auto-J, considering the small size of the benchmark.

The results are shown in Figure 4 and Table 3 for ChatArena, and in §C for Auto-J. Again, we confirm that human agreement guarantee can be achieved across all levels of $\alpha$. On ChatArena, unlike Point-Estimate Calibration with GPT-4 whose guarantee success rate is below 60%, our method achieves 91% guarantee success rate while only using GPT-4 for 17.5% of the evaluations. The performance is particularly pronounced in Auto-J where GPT-4 without abstention could only achieve 63.2% agreement with humans (Figure 5), potentially due to the fact that the dataset introduces additional tie label unlike the other two datasets. In stark contrast, Cascaded Selective Evaluation guarantees up to 80% human agreement with high probability.

Next, we conduct further ablations and analyses to better understand the working of our method.

### 3.3 Understanding Abstention Policy

Figure: Table 4: Comparison between abstained vs. evaluated samples. Our abstention policy aligns with how humans agree with each other (IAA), exhibiting no significant reliance on shallow heuristics (length ratio, token overlap).

One potential concern with selective evaluation is that its abstention policy might not align with the human-perceived subjectivity of each instance and instead rely on shallow heuristics, e.g., choosing to abstain when the pair of generations have large token overlap. To address this concern, we analyze whether there exists a significant difference in human-perceived subjectivity between model-abstained samples and evaluated samples. We first collect 3-5 human annotations per each instance in ChatArena, and measure the inter-annotator agreement(^5^55As the label space is binary for preference evaluation, we define inter-annotator agreement simply as the density of the majority preference label assigned by human annotators.) (IAA) as a proxy of human-perceived subjectivity. Then, we compare the difference in IAA between abstained and evaluated samples when the target agreement level is set to 0.9. We also measure the difference in terms of shallow features, specifically the pairwise length ratio and the token-overlap (ROUGE-L) within each instance. For further details, see §E.2.

**Table 5: Impact of number of simulated annotators $N$ on ChatArena, with $1-\alpha=0.85$. Larger number of simulations generally leads to better coverage, while human agreement is guaranteed even with a small $N$. For all values of $N$, Cascaded Selective Evaluation guarantees high agreement with humans while reducing the API cost by 40% compared to GPT-4 without abstention.**
| Method | Empirical Human<br>Agreement (%) | Coverage (%) | Guarantee Success<br>Rate (%) | Relative<br>API Cost |
| --- | --- | --- | --- | --- |
| GPT-4 ($N=1$) | 77.8 | 100.0 | 0.0 | 1.000 |
| Cascaded Selective Evaluation ($N=1$) | 85.2 | 60.9 | 90.3 | 0.655 |
| GPT-4 ($N=2$) | 78.2 | 100.0 | 0.0 | 2.000 |
| Cascaded Selective Evaluation ($N=2$) | 85.7 | 61.5 | 90.8 | 1.288 |
| GPT-4 ($N=3$) | 78.1 | 100.0 | 0.0 | 3.000 |
| Cascaded Selective Evaluation ($N=3$) | 85.5 | 62.1 | 90.3 | 1.920 |
| GPT-4 ($N=5$) | 78.5 | 100.0 | 0.0 | 5.000 |
| Cascaded Selective Evaluation ($N=5$) | 85.8 | 63.2 | 91.0 | 2.849 |

Table 3.3 presents the results. The average inter-annotator agreement is 0.815 ($\sigma^{2}=0.031$) for abstained samples and 0.902 ($\sigma^{2}=0.025$) for evaluated samples, a statistically significant difference in two-sample t-test with $p<1e-8$. This is in contrast with both the length ratio and token overlap, for which the differences between the two sets are not significant ($p>0.10$). In fact, for token overlap, the abstained examples exhibit higher ROUGE-L on average than the evaluated samples. Overall, these results show that the instances abstained by LLM judges tend to be more subjective even for humans (with no evidence of reliance on some spurious heuristics), indicating that the confidence elicited by Simulated Annotators closely aligns with that of human annotators.

### 3.4 Evaluation under Distribution Shift

Our test procedure in §2.1 assumes that the calibration set $D_{\textit{cal}}$ is sampled i.i.d. from $P(x,y_{\textit{human}})$. In real-world scenarios this may not be the case, because we often only have access to generations from a set of known models for calibration, while in the test time, we need to evaluate outputs from unknown models. In §D, we empirically analyze whether our method provides risk control even under this distribution shift. First, we randomly divide ChatArena into two disjoint sets such that there is no overlap between the evaluated models in each set. Then, we induce distribution shift by sampling $D_{\textit{cal}}$ from one set and testing instances in another set. We follow the same setup as §3.2, and run experiments for 1000 random splits. As shown in Table 9, despite a small degradation in coverage, our method guarantees high human agreement for more than 90% of the time, consistently across all tested levels of $\alpha$. The result demonstrates that Cascaded Selective Evaluation maintains its reliability even under the realistic distribution shift.

### 3.5 Impact of Number of Simulated Annotators

Next, we analyze the impact of number of simulated annotators for Cascaded Selective Evaluation. In Table 5, we report the results on ChatArena using Simulated Annotators as confidence measure with $N=5,3,2,1$. We compare the result against using GPT-4 with the same prompt, but without abstention. Along with the guarantee success rate and coverage, we report relative API cost for calling OpenAI models, where the cost for full evaluation with GPT-4 is set to 1. The results suggest that (1) using larger number of simulated annotators leads to consistently better coverage, but (2) even with a small number of simulated annotators, our method can still achieve high human agreement while reducing the evaluation cost by up to 40% compared to GPT-4 without abstention.

### 3.6 Impact of Judge Model Composition

We study whether Cascaded Selective Evaluation can be done entirely without GPT-4. We use a substantially weaker cascades with Mistral-7B-instruct-v0.2, Mixtral-8$\times$7b-instruct (Jiang et al., ), and GPT-3.5 as judge models (weaker cascades). We compare the result against (1) zero-shot GPT-4 without abstention, and (2) Cascaded Selective Evaluation using the original cascades, with $N=1$ for better cost (stronger cascades). We set the target agreement level $1-\alpha=0.8$, a higher level than what is achievable by GPT-4 without abstention (Figure 4).

The results in Table 6 reveal an interesting finding: even the weaker cascades of judge models ensure a satisfactory level of human agreement, by balancing the the trade-off between the evaluation cost and coverage, instead of compromising the precision. Unlike conventional LLM-based evaluation, where one has to sacrifice accuracy by using a weaker judge, our method allows practitioners to consistently achieve their target level of human agreement. Depending on the requirements, one can opt for stronger cascades for better coverage or weaker cascades for lower costs, all while maintaining this guarantee. Additionally, both configurations of Cascaded Selective Evaluation significantly reduce evaluation costs compared to using GPT-4, achieving savings of up to 78.5% with stronger cascades and 87.4% with weaker cascades.

**Table 6: Impact of judge model composition on ChatArena, with $1-\alpha=0.8$. Weaker cascades use Mistral-7B, Mixtral-8$\times$7B, and GPT-3.5 as judge models. Stronger cascades use GPT-4 instead of Mixtral-8$\times$7B. Our method guarantees human agreement even with the weaker cascades, while only using 12.6% of the evaluation cost for GPT-4.**
| Method | Empirical Human<br>Agreement (%) | Coverage (%) | Guarantee Success<br>Rate (%) | Relative<br>API Cost |
| --- | --- | --- | --- | --- |
| GPT-4 | 77.8 | 100.0 | 13.9 | 1.000 |
| Cascaded Selective Evaluation (stronger) | 80.2 | 77.6 | 90.5 | 0.215 |
| Cascaded Selective Evaluation (weaker) | 80.3 | 68.3 | 90.8 | 0.126 |

## 4 Related Works

LLM-based evaluation has emerged as a scalable alternative to human evaluation (Zheng et al., ; Liu et al., ), with empirical evidence suggesting that despite its cost, GPT-4 can be as accurate as an average human annotator (Dubois et al., ; Li et al., ). Subsequent works attempt to reduce the dependency on the large judge model by distilling a small expert judge (Kim et al., ; Zhu et al., ), or by ensembling multiple agents through peer review and debate (Verga et al., ; Chan et al., ). However, these methods often lack a provable guarantee of their reliability. Recent research also indicates that LLM judges are not as robust as previously assumed, showing susceptibility to cognitive biases (Zeng et al., ; Koo et al., ) and self-preference (Panickssery et al., ). Our goal in this work is to enhance the reliability of LLM-based evaluation—despite these inherent biases—as a better-aligned proxy of human judgement.

Another line of works augment LLMs with a rigorous statistical guarantee, controlling their risk in critical applications such as hallucination rate in factual generation (Yadkori et al., ; Mohri & Hashimoto, ) and FDR in medical decision making (Gui et al., ). These approaches are often powered by conformal methods (Angelopoulos et al., ), offering marginal control over the prescribed risks. Other works study fine-tuning objective for LLMs, either to improve their truthfulness (Kang et al., ; Tian et al., ) or to abstain when lacking relevant knowledge (Zhang et al., ). Our work builds upon these prior works, but (1) focuses on LLM-based evaluation to provide an exact upper bound on the disagreement risk conditional on the sampling of calibration set, (2) proposes an unsupervised confidence measure instead of a supervised estimator (Kapoor et al., ; Gupta et al., ), and (3) derives a cascaded framework that significantly reduces the inference cost while simultaneously guaranteeing the reliability of evaluation.

## 5 Conclusion

We present Cascaded Selective Evaluation, a framework to provide LLM-based evaluation with a robust guarantee of human agreement. As part of our framework, we also propose Simulated Annotators, a novel method that significantly improves confidence estimation for LLM judges without resorting to external supervision. By dynamically selecting when to trust which judge model, our method significantly reduces evaluation overhead while still maintaining its reliability, often outperforming the precision achievable by fully relying on the strongest judge model.

## 6 Acknowledgements

This work was funded in part by NSF through DMS-2134012 and ONR via N00014-24-1-2207.

## Appendix A Validity of Human Agreement Guarantee

### A.1 Proof of Theorem 1

###### Theorem 1

Consider a threshold $\widehat{\lambda}$ chosen as in §2.1, and a selective evaluator $(f_{\textit{LM}},c_{\textit{LM}})$ operating based on $\widehat{\lambda}$. Then, Equation (2) is satisfied with probability at least $1-\delta$.

The proof extends that of Theorem B.1 in Bates et al. (). Let $R(\lambda)$ denote the true risk of disagreeing with humans at threshold $\lambda$. It suffices to show that $P(R(\widehat{\lambda})\leq\alpha)\geq 1-\delta$. We first note that $n(\lambda)\widehat{R}(\lambda)$ is a binomial random variable, i.e.,

$$ $n(\lambda)\widehat{R}(\lambda)\sim\operatorname*{Bin}(n(\lambda),R(\lambda)).$ $$

Thus, the lower tail bound for $\widehat{R}(\lambda)$ can be expressed as a function $g$ of $t\in\mathbb{R}$ and $R(\lambda)$ as

$$ $P(\widehat{R}(\lambda)\leq t)=P\big{(}\operatorname*{Bin}(n(\lambda),R(\lambda ))\leq\lceil n(\lambda)t\rceil\big{)}\eqqcolon g(t;R(\lambda)).$ $$

Plugging this into the definition of $\widehat{R}^{+}(\lambda)$ in Equation 4,

$$ $\displaystyle\widehat{R}^{+}(\lambda)$ $\displaystyle=\sup\left\{R(\lambda):P\big{(}\operatorname*{Bin}(n(\lambda),R( \lambda))\leq\lceil n(\lambda)\widehat{R}(\lambda)\rceil\big{)}\geq\delta\right\}$ $\displaystyle=\sup\left\{R(\lambda):g(\widehat{R}(\lambda);R(\lambda))\geq \delta\right\}.$ $$

Here, let G denote the CDF of $\widehat{R}(\lambda)$ and $G^{-1}(\delta)=\sup\{x:G(x)\leq\delta\}$. From above, we know that if $R(\lambda)>\widehat{R}^{+}(\lambda)$, then $g(\widehat{R}(\lambda);R(\lambda))<\delta$. Therefore,

$$ $\displaystyle P(R(\lambda)>\widehat{R}^{+}(\lambda))$ $\displaystyle\leq P(g(\widehat{R}(\lambda);R(\lambda))<\delta)$ $\displaystyle=P(G(\widehat{R}(\lambda))<\delta)$ $\displaystyle\leq P(\widehat{R}(\lambda)<G^{-1}(\delta))$ $\displaystyle\leq\delta.$ $$

Hence, $P(R(\lambda)\leq\widehat{R}^{+}(\lambda))\geq 1-\delta$, implying that $\widehat{R}^{+}(\lambda)$ is the $(1-\delta)$ upper confidence bound of $R(\lambda)$. Finally, since we have $\widehat{R}^{+}(\widehat{\lambda})\leq\alpha$ from the definition of $\widehat{\lambda}$, we obtain

$$ $P(R(\widehat{\lambda})\leq\widehat{R}^{+}(\widehat{\lambda})\leq\alpha)\geq 1-\delta.$ $\blacksquare$ $$

###### Theorem 1

### A.2 Extension to Cascades of Judge Models

Figure: Algorithm 2 $\,\operatorname*{calibrate}(\mathcal{M},D_{\textit{cal}},\alpha,\delta)$

We illustrate the extension of our test procedure from a single model to the cascades of judge models. For notational simplicity, we denote the test procedure for a single judge model in §2.1 as a function $\operatorname*{calibrate-single}$. This function takes as input a model $M$, a calibration set $D$, risk tolerance $\alpha$ and error level $\delta$, and gives a calibrated threshold $\widehat{\lambda}$ as an output.

The calibration procedure for cascades of judge models is shown in Algorithm 2. The procedure sequentially applies $\operatorname*{calibrate-single}$ to each judge model. Specifically for each judge model $M_{i}$, we calibrate $\lambda_{i}$ by testing over the set of instances $D$ that have been abstained by the previous models. This allows us to ensure that for each $M_{i}$,

$$ $P\left(f_{M_{i}}(x)=y_{\textit{human}}\Bigg{|}c_{M_{i}}(x)\geq\lambda_{i}, \bigwedge_{j=1}^{i-1}c_{M_{j}}(x)<\lambda_{j}\right)\geq 1-\alpha$ $$

is satisfied with probability at least $1-\frac{\delta}{|\mathcal{M}|}$. To provide the guarantee across all judge models, define $R_{i}$ as the disagreement risk for each judge model $M_{i}$:

$$ $R_{i}\coloneqq P\left(f_{M_{i}}(x)\neq y_{\textit{human}}\Bigg{|}c_{M_{i}}(x) \geq\lambda_{i},\bigwedge_{j=1}^{i-1}c_{M_{j}}(x)<\lambda_{j}\right).$ $$

Also, define $R_{\textit{cascades}}$ as the risk of full cascaded selective evaluation, i.e.,

$$ $R_{\textit{cascades}}\coloneqq P(f_{\textit{cascades}}(x)\neq y_{\textit{human }}|\,x\textit{ not abstained by the cascades}).$ $$

It is easy to see that $R_{\textit{cascades}}$ is an interpolation between all $R_{i}$s, thus $R_{\textit{cascades}}\leq\max_{i}R_{i}$. Therefore,

$$ $\displaystyle P\big{(}R_{\textit{cascades}}>\alpha\big{)}$ $\displaystyle\leq P\big{(}\max_{i}R_{i}>\alpha\big{)}=P\bigg{(}\bigcup_{i}R_{i }>\alpha\bigg{)}\leq\sum_{i}P\big{(}R_{i}>\alpha\big{)},$ $$

where the last inequality comes from union bound. Since we know that $P(R_{i}>\alpha)\leq\frac{\delta}{|\mathcal{M}|}$ for each judge model $M_{i}$,

$$ $\sum_{i}P\big{(}R_{i}>\alpha\big{)}\leq\frac{\delta}{|\mathcal{M}|}\cdot| \mathcal{M}|=\delta.$ $$

Thus, $P(R_{\textit{cascades}}>\alpha)\leq\delta$. In other words, the risk of disagreement across all judge models is guaranteed to be at most $\alpha$, with probability at least $1-\delta$.

## Appendix B Additional Results on Confidence Estimation

**Table 7: Additional results on confidence estimation with Mistral-7B. We find that more sophisticated methods that measure the semantic variance between chain-of-thoughts often underperform Simulated Annotators, marking similar or worse performance with zero-shot predicted probability.**
| Dataset | AlpacaEval | TL;DR |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Acc. | ECE $\downarrow$ | AUROC | AUPRC | Acc. | ECE $\downarrow$ | AUROC | AUPRC |  |
| Mistral-<br>7B-it | Predictive Probability (CoT) | 0.636 | 0.292 | 0.527 | 0.655 | 0.669 | 0.275 | 0.637 | 0.751 |
| Lexical Similarity | 0.636 | 0.478 | 0.478 | 0.623 | 0.669 | 0.459 | 0.520 | 0.639 |  |
| Semantic Sets | 0.636 | - | 0.513 | 0.638 | 0.669 | - | 0.545 | 0.665 |  |
| Semantic Entropy | 0.636 | - | 0.572 | 0.684 | 0.669 | - | 0.650 | 0.762 |  |
| Simulated Annotators (Ind.) | 0.684 | 0.075 | 0.632 | 0.772 | 0.696 | 0.103 | 0.654 | 0.807 |  |

In Table 7, we provide additional results for more sophisticated methods for confidence estimation. Given an instance $x$, these methods first generate $M$ chain-of-thoughts (CoTs) from an LLM judge prior to inferring the preference label, then measure their variance either in the label space or on the semantic-level:

- •
Predictive Probability (CoT): We average the $M$ label predictive probabilities assigned by the LLM judge after generating chain-of-thoughts.
- •
Lexical Similarity: As a simple proxy of semantic variance, we average ROUGE-L across all pairs of $M$ chain-of-thoughts. The intuition is that when the CoTs exhibit high lexical overlap with each other, the model is relatively confident about its generation.
- •
Semantic Sets (Lin et al., ): We cluster the CoTs into semantically equivalent groups using a bidirectional entailment model (Zha et al., ), then use the number of clustered groups to represent model uncertainty.
- •
Semantic Entropy (Kuhn et al., ): We additionally use the likelihood of each generated chain of thought, and measure the average sequence-level entropy across the semantically equivalent groups.

We follow Lin et al. () to set $M=20$. For Semantics Sets and Semantic Entropy, we exclude expected calibration error as the two scores represent model uncertainty rather than the confidence score calibrated in $[0,1]$. These methods incur significant overhead compared to the methods discussed in §2.2, generating 20 sequences of chain-of-thoughts for each input instance and employing a supervised entailment model. Nonetheless, we find that their performance consistently underperforms that of Simulated Annotators, with the best method Predictive Probability (CoT) performing worse than our ablation Randomized Annotators (Table 1).

## Appendix C Additional Results on Chat Assistant Evaluation

**Table 8: Comparison to baselines on Auto-J, with target agreement level $1-\alpha=0.8$. The results are averaged across 1000 runs with random data split.**
| Method | Evaluator Composition (%) | Coverage (%) | Guarantee Success<br>Rate (%) |  |  |
| --- | --- | --- | --- | --- | --- |
| Mistral-7B | GPT-3.5-turbo | GPT-4-turbo |  |  |  |
| No Selection | 0.0 | 0.0 | 100.0 | 100.0 | 0.0 |
| Heuristic Selection | 0.0 | 0.0 | 100.0 | 72.0 | 2.4 |
| Cascaded Heuristic Selection | 36.6 | 40.2 | 23.2 | 84.4 | 0.8 |
| Point-Estimate Calibration | 100.0 | 0.0 | 0.0 | 12.1 | 54.3 |
| 0.0 | 100.0 | 0.0 | 27.9 | 52.1 |  |
| 0.0 | 0.0 | 100.0 | 42.9 | 56.5 |  |
| Cascaded Selective Evaluation | 49.3 | 21.8 | 28.9 | 42.6 | 90.2 |

Figure: Figure 5: Human Agreement Guarantee on Auto-J. GPT-4 without abstention obtains only 63.2% agreement with humans, while Cascaded Selective Evaluation guarantees target human agreement level of up to 80% with high probability.
Refer to caption: x5.png

## Appendix D Evaluation under Distribution Shift

**Table 9: Evaluation under distribution shift on ChatArena. We induce distribution shift by sampling the calibration and test set respectively from two disjoint sets of instances with no overlap of evaluated models. We iterate experiments for 1000 random splits and aggregate the results. Cascaded Selective Evaluation guarantees high agreement even under the realistic distribution shift.**
| Target Human<br>Agreement (%) | Empirical Human<br>Agreement (%) | Coverage (%) | Guarantee Success<br>Rate (%) |
| --- | --- | --- | --- |
| 70.0 | 73.4 | 100.0 | 100.0 |
| 75.0 | 75.3 | 91.4 | 92.5 |
| 80.0 | 80.8 | 72.1 | 90.8 |
| 85.0 | 85.2 | 55.4 | 91.0 |
| 90.0 | 90.1 | 31.8 | 90.7 |

## Appendix E Details on Experimental Setup

### E.1 Prompts

### E.2 Human Evaluation Details

Figure: Figure 6: Human Annotation Interface.
Refer to caption: extracted/5755392/figures/annotation_ui.png

#### Annotator Recruitment.

We recruited annotators from Prolific(^6^66[https://app.prolific.com](https://app.prolific.com)) who have recorded at least 99% approval rate, are fluent in English, and have completed a Bachelor’s degree. In addition, we manually designed 10 qualification examples based on our annotation guidelines. The purpose of the qualification test is to find annotators who understand and carefully follow our guidelines. Participants who scored more than 80% were included in our actual human study. We qualified 21 annotators to do the study and paid them $15 per hour.

#### Annotation Task.

We randomly sample 600 examples from ChatArena (Zheng et al., ) each consisting of a query and two model responses. Given each example, we instruct the annotators to select the overall better response considering several aspects such as helpfulness, truthfulness, and harmlessness. Each instance is evaluated by 3-5 annotators. We also allow annotators to occasionally skip an instance with a reason if they have no idea how to evaluate it. We provide the screenshot of our annotation guidelines and interface in Figure 6 and 7.

Figure: Figure 7: Human Annotation Guidelines.
Refer to caption: x6.png

## Appendix F Qualitative Examples

## Peer Reviews

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: See above.
soundness: 3
strengths: This paper provides clear methodological contributions to improving human-LLM alignments. I especially appreciate the cascaded selective evaluation approach, which explicitly considers the costs of using stronger/more costly LLMs. Overall, this is an excellent paper with clear contributions and is well-executed.
summary: This paper proposes a selective evaluation framework to ensure that LLM-based evaluations align with human judgments to a defined degree of confidence. Selective Confidence Evaluation ensures that, instead of always trusting an LLM's judgment, the model evaluates its confidence in a given answer and abstains when unsure. Simulated Annotators leverage in-context learning to simulate diverse annotator perspectives, improving model calibration and reducing evaluation errors. Cascaded Selective Evaluation introduces multiple LLM judges used in a tiered manner, starting with a more cost-effective model and escalating to stronger models only if needed. The method achieves strong human alignment and reduces reliance on more costly models like GPT-4 by using smaller models when possible.
weaknesses: This paper has already been very carefully executed, and I have only minor suggestions. 

1.	Simulated annotators, despite their encouraging performance, will require significant processing time and resources as N and K increase. Given a substantial decrease in ECE, I believe that simulated annotators have enough value but it would be helpful for the readers if the authors can provide time and cost requirements compared to the canonical approaches. Furthermore, the authors may want to invest a bit more space in explaining the algorithm of simulated annotators including exactly how they handled in-context learning.

2.	The outperformance of cascaded selective evaluation is somewhat expected because the gist of the framework is to exclude the observations where annotator disagreement is expected. In my view, the model does a good job of identifying these hard-to-reconcile observations. A fairer comparison would be to exclude the abstained observations from the baseline models and see how their performance is affected. For example, if you maintain the same coverage for all other methods in Table 2 and Table 3, what would their empirical human agreement look like? Of course, empirically this cannot be achieved without your model – but as you show in Section 3.3, some systematic characteristics can predict abstention. One can simply devise an alternative strategy that can identify these hard-to-reconcile samples and run a multi-agentic approach on them. 

3.	I think the most striking result is in Table 7. It is surprising that the weaker cascades achieve an empirical human agreement of 80.3%. Although their coverage is 68.3%, this is indeed a remarkable result. Given this, I would try to do more with the abstained samples. Almost 30% of the sample is now abstained and running the most advanced model on those will not only increase your ultimate coverage and success rate. The relative API cost will be substantially lower than using only GPT4.

4.	Relatedly, the relative API cost of the model is largely dependent on the difficulty of your task. As seen in Tables 2, 3, and 8, the evaluator composition is heterogeneous across different tasks. When a smaller LLM can handle the majority of tasks, the overall task is likely to be cheaper. When a larger LLM needs to intervene, the overall task will be more expensive. I would caveat this (and perhaps show the range of relative API cost using some simulated evaluator compositions) in Section 3.6.

5.	It is a minor thing, but is there any reason that you chose 1-a =0.9 in Table 2, 1-a=0.85 in Table 3, and 1-a=0.8 in Table 7? I would personally prefer the three thresholds to be consistent throughout the paper.

Overall, it was an interesting read. Great paper.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: How does IAA change for abstained samples with a lower target accuracy?
soundness: 3
strengths: 1. The paper tackles an important issue, we need more human subjectivity at the forefront of LLM evaluation. 
2. In essence, the Simulated Annotators technique is ensembling, so using a simple and popular technique as ensembling is an interesting way to utilize for such a task. 
4. The authors conduct many experiments to show the utility of the approach. 
3. The added ablation studies in the paper are nice. 
5. The paper is well-written and structured.
summary: Current NLP paradigms have shifted to using LLMs as judges for annotations instead of humans. This paper asks how we can guarantee the reliability of using these LLMs for evaluation through human agreement. 

Given an input, how likely would humans agree with the model judgment?

For this, the approach is to use selective evaluation: the model can abstain if not confident enough to make a prediction.

The conventional way of using the predictive probability as confidence estimates doesn't show to be reliable so two methods are proposed: 
1. Simulated annotators: simulate annotator preferences through In-Context Learning and the confidence is estimated through the agreement ratio. 
2. Cascaded Selective Evaluation: Start with a cheaper model and go to stronger models when the previous model is less confident. 

This reduces the evaluation overhead with high agreement guarantees. Their abstention policy aligns with human subjectivity.
weaknesses: 1. A clear connection to _selective prediction_ is missing. While the authors talk about _selective evaluation_, the concept is the same so it is better to make that link clear. 
2. Related Work can benefit from more work on selective prediction and human subjectivity in LLMs, e.g. 

Barbara Plank. 2022. The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Khurana, U., Nalisnick, E., Fokkens, A., & Swayamdipta, S. Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?. In First Conference on Language Modeling.

Bavaresco, A., Bernardi, R., Bertolazzi, L., Elliott, D., Fernández, R., Gatt, A., ... & Testoni, A. (2024). Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. arXiv preprint arXiv:2406.18403.

3. Cheaper models used for Cascading are still very costly (e.g. Mistral 7B)
4. The calibration set for human preferences is based on argmax of human judgments. Using argmax gets rid of disagreement signals, which is actually important information.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: - You analyzed the abstention rate quantitatively, but did you also do a qualitative analysis? Do the models abstain on the same type of pairs?
soundness: 3
strengths: The paper is clearly structured and well written. The approach clearly improves the evaluation process supported by LLMs, while being cost effective by only considering stronger models (such as GPT-4-turbo) as a last escalation step. The authors provide meaningful experiments and show how the method can be extended in the future.
summary: The authors propose a framework for LLM evaluation that includes a human agreement guarantee, that is called Cascaded Selective Evaluation. The evaluation task (in this case a pairwise evaluation) is given to a weak LLM judge and escalated to a stronger LLM judge, if the confidence level for the task does not exceed a certain threshold. As a confidence estimation method they propose Simulated Annotators, where multiple LLMs are used to simulate annotator preferences.
weaknesses: A weakness of the approach (for now), is that it has only been tested and applied in context of the pairwise preference evaluation. Nevertheless, the Simulated Annotators and Cascaded Selected Evaluation methods are promising. It will be interesting to see how the approach performs in other evaluation settings in the future. The question of when the models abstain could be answered in more detail. For now, it is only answered quantitatively, but not really qualitatively. In order to find out whether the examples the models are not sure about are the same ones about which the human judges also disagree, the examples need to be examined in more detail, i.e. more on the level of content. It would be interesting to see the ratio of examples where both humans and the models abstained from evaluation and to analyze some examples on a content level. 

Small errors:
- Line 462: the the tradeoff

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: NA
soundness: 3
strengths: 1. Robust Human Agreement: The Cascaded Selective Evaluation framework offers a strong guarantee of alignment with human judgment, enhancing the reliability of LLM-based evaluations.

2. Improved Confidence Estimation: The introduction of Simulated Annotators enhances the confidence estimation for LLM judges, allowing for more accurate evaluations without the need for external supervision.

3. Dynamic Judge Selection: By dynamically selecting which judge model to trust, the framework reduces evaluation overhead while maintaining reliability, often surpassing the precision of relying solely on the strongest judge model.
summary: The authors introduce a principled method for evaluating language models with guaranteed human agreement. They argue that evaluation should not solely rely on model preferences in pairwise comparisons. Instead, it should assess judge models' confidence and selectively decide when to trust their judgment. This selective framework ensures that the evaluation aligns with human agreement to a specified level. They introduce Simulated Annotators, a new method for confidence estimation that enhances judge calibration, enabling comprehensive evaluation. Their Cascaded Selective Evaluation uses cost-effective models initially, escalating to stronger ones as needed, maintaining human agreement guarantees. Results show this method achieves over 80% human agreement.
weaknesses: It is more like a theoretical work. Given that the experiments have proved effective in evaluating different tasks, I suggest the authors invest some effort in releasing a user-friendly toolkit based on this work, for other researchers to use.

Meta-review: This paper proposes a framework called Cascaded Selective Evaluation for improving LLM-based evaluation with guaranteed human agreement. The authors demonstrate that their approach can achieve over 80% human agreement while being more cost-effective than relying solely on frontier models like GPT-4.

Strengths: It addresses a critical need for reliable automated evaluation. Theoretical guarantees and empirical results both strongly support the proposed method.

Weaknesses: only validated on pairwise preference evaluation tasks, more qualitative analysis of abstention patterns might also be helpful

Overall this paper is a clear acceptance with all reviewers agreeing on the contribution of the paper. It makes significant methodological and practical contributions to improving LLM-based evaluation. The theoretical framework is sound, and the empirical results demonstrate clear improvements over existing approaches.

Decision: Accept (Oral)