Limits to scalable evaluation at the frontier: LLM as judge won’t beat twice the data
Authors: Florian E. Dorner
Vivian Yvonne Nastl
Moritz Hardt
Venue: ICLR 2025 Oral (2025)
Keywords: Evaluation
Benchmarking
Model-as-a-judge
Theory
Primary area: learning theory

Abstract: High quality annotations are increasingly a bottleneck in the explosively growing machine learning ecosystem. Scalable evaluation methods that avoid costly annotation have therefore become an important research ambition. Many hope to use strong existing models in lieu of costly labels to provide cheap model evaluations. Unfortunately, this method of using models as judges introduces biases, such as self-preferencing, that can distort model comparisons. An emerging family of debiasing tools promises to fix these issues by using a few high quality labels to debias a large number of model judgments. In this paper, we study how far such debiasing methods, in principle, can go. Our main result shows that when the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth labels by more than half. Our result speaks to the severe limitations of the LLM-as-a-judge paradigm at the evaluation frontier where the goal is to assess newly released models that are possibly better than the judge. Through an empirical evaluation, we demonstrate that the sample size savings achievable in practice are even more modest than what our theoretical limit suggests. Along the way, our work provides new observations about debiasing methods for model evaluation, and points out promising avenues for future work.

Published as a conference paper at ICLR 2025
LIMITS TO SCALABLE EVALUATION AT THE FRONTIER: LLM AS JUDGE TON'T BEAT TWICE THE DATA
Florian E. Dorner*122 Vivian Y. Nastl. 1.3. and Moritz Hardt
Max Planck Institute for Intelligent Systems, Tübingen ?Tübingen Al Center *ETH Zürich
ABSTRACT
High quality annotations are increasingly bottleneck in the explosively growing machine learning ecosystem. Scalable evaluation methods that avoid costly anno- tation have therefore become an important research ambition. Many hope use strong existing models in lieu of costly labels to provide cheap model evaluations. Unfortunately, this method of using models as judges introduces biases, such as self- seff-refferencing that can distort model comparisons. An emerging family of de- biasing tools promises to fix these issues by using a few high quality labels to de- bias large number of model judgments. In this paper, we study how far such de- biasing methods, in principle, can go. Our main result shows that when the judge is more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth labels by more than half. Our result speaks to the severe limitations of the L.M-as-a-judge paradigm at the evaluation frontier where the goal is to assess newly released models that are possibly better than the judge. Through an empirical evaluation, we demonstrate that the sample size sav- ings achievable in practice are even more modest than what our theoretical limit suggests. Along the way, our work provides new observations about debiasing methods for model evaluation, and points out promising avenues for future work.
INTRODUCTION
As large models continue to advance in their capabilities, it is increasingly challenging for human experts to evaluate newly released models. Expert data annotation is not only slow and costly. Traditional benchmarking also struggles to keep up with rapidly changing model capabilities across an expanding range of tasks. Yet, as models become more powerful, there is hope that models themselves could become powerful tools for scaling up evaluation. An intriguing idea is strong existing model to provide judgments about other models. Such a "model-as-judge" could provide labels to classification instances, compare model outputs, and replace human annotators across a variety of tasks.
Already implemented in practice in numerous instances, the model-as-judge paradigm, however, runs into some roadblocks. When used as judges, models exhibit a range of biases that can skew model comparisons and result misleading model rankings. Recently proposed debiasing methods, however, promise a compelling way forward. Using a small number of ground truth labels, these methods can potentially debias large number of model predictions, thus restoring their utility for benchmarking purposes.
Scalable evaluation most needed and most challenging the evaluation frontier: Newly re- leased models for which we have little intuition as of yet. What makes this case so challenging that the new model is likely better than the judge some ways. In this work, we address the question whether debiasing methods together with the model-as-judge paradigm can, in principle, provide an adequate solution to scalable evaluation at the frontier. However, our theory, supported by empirical evaluation, makes a sobering prediction: Whenever the judge model performs worse at its task than the evaluated model, the optimal debiasing method is no better than using twice the ground truth data. This shows that, although there is merit to debiasing, at the evaluation frontier its economic gains are not greater than a factor-two savings in annotation cost.
"oorreponningg author; forian.domerr wwwee@@@nnccnnncncccccc

---

Published as a conference paper at ICLR 2025
1.1 OUR CONTRIBUTIONS
In our theoretical work we focus on standard evaluation setup that encompasses classification, question answering, arena-style comparisons, and safety evaluations. Given input æ, model m produces an output max)) that receives a binary ground-truth score s(2, m(x)). ) For example, in classification the accuracy score is s(x, m(x)) is the model's output m(x) matches the ground- truth label ((z) and 0 otherwise. A judge model provides binary proxy score š(r, m(æ)). We imagine that obtaining ground-truti scores is costly and proxy scores are significantly cheaper.
For fixed model our goal is to estimate its true score E ssx, m(x)), where the expectation is taken over an input drawn from a distribution. However due to judge bias, simply using cheap proxy scores 8 can lead to substantially distorted estimates. To motivate the issue, Figure shows how proxy scores can cause highly misleading rankings of model performance.
1
1
The Labalk
GPT4
2
Obuut Oper
Claste Sanner
OPT.41
GPT4 lirto
2
Chuut Sonner
Coude Senvie
Rank
Rank
3
Ghute-Sereet
Coude
Gamin Pro
Clauda Ozsa
Qker2 instuct 1723
GPT-4
4-
Coudt-Seentt
8
GPT4 Tarto (prouied
Owend Inttrct (730)
5
oPT4
The Labels
Tirts (prsiend
10
LoMo53773
Gamin 25 Pro
Thue
GPT-4 Labels
LlaMa-3 708 labels
(a) GPT-4 as judge (84% accuracy).
(b) LLama3-70b as judge (79% accuracy)
Model ranks on MMLU based on true labels compared to LLM labels in a semi-synthetic setting (a) and for the top-10 10 models on HELM by July 2024 (b). Using LLM labels heavily perturbs the ranking despite high judge accuracy.
Figure I:
Rather than solely relying on proxy scores, we assume that we have a small number of samples of ground-truth scores s, in addition to larger number N of proxy scores debiasing method is any estimator that provides an unbiased estimate of the true score from the sample of available proxy scores and ground truth scores. particular, debiasing methods guarantee that ranking models by their estimated performance produces correct model rankings with high probability.
We focus on sample size savings terms of the ground-truth scores that debiasing methods can achieve by making use of the proxy scores 3.
We first show that the best possible savings determined by the Pearson correlation between and & (Theorem 5).
Our main result (Theorem 6) then shows that this correlation is small whenever the judge model performs worse than the evaluated model. In that case the best possible outcome is an effective doubling of the number of samples s, no matter how many samples of the proxy score are available (Corollary 7). This result remains true in slightly weaker sense, when we allow for the proxy to continuous rather than binary (Theorem 10).
In experiments on MMLU and MT-Bench, we empirically confirm that sample size savings of more than a factor two are rare: We only observe them when current state-of-the-art models are used to judge substantially weaker models.
In addition, we show that the commonly reported agreement rate between the judge and the ground truth neither meaningfully restricts the judge bias, nor the sample size savings for debiasing methods.
FORMAL SETUP AND MOTIVATION
In this section, introduce the formal model to analyze the model-as-judge paradigm. We then provide empirical and theoretical evidence that judge bias can indeed cause model-as-judgee to produce misleading results, even when judge models are quite accurate.
2

---

Published as a conference paper at ICLR 2025
We focus our analysis aggregated binary evaluations: Given prompt x, model receives a binary score s(x, mxx)) depending on its output m(x). As both model outputs and evaluation protocols can be non-etterministic, we treat s(z, m(x)) random variable, even when the prompt I and model output mx)) are fixed. The model mn is then evaluated based on its expected score (X,, m( (X)). Here, the expectation is taken over the prompt distribution X DD well as additional randomness in the scores s(-, m(-)) introduced by the evaluation protocol.
This setup covers several important benchmarking and evaluation settings:
Here, the prompt is an instance to be classified or a question to be answered. Letting m(x) denote the model's output on input æ, the accuracy score (xx, m) equals one whenever m(xx equals the label y(æ).
Accuracy in classification and Q&A benchmarks:
Arena-style benchmarks: In this setting I is a prompt. We sample the model's re- sponse m(z) and the response mx(x) of another randomly selected model m². The score szz, m) indicates whether m(x) was judged to be a better response than m' m'(x).
Safety benchmarks: For this, the prompt x is designed to potentially elicit unsafe be- haviour. The score s(ar m) indicates whether the model's response m(x) is safe.
In each of these settings, we can use a language model as judge model to score models more cheaply than via ground truth evaluation. To do so, we compute a prexy score with the help of the judge model. We assume that the proxy score is binary as well. In the classification setting, for example, the judge models could provide the labels for evaluation. The proxy score would then be the classification accuracy with respect to the judge model's labels. In arena comparisons, the LLM could replace human raters. And in safety benchmarks, an LLM could evaluate the safety of model outputs. Although our motivation focuses on language models as the source of proxy scores, formal work applies to any proxy score, e.g., scores provided by crowdworkers rather than experts.
Precisely modeling the specific relationship between x, s(z, m) and s(z, m) is generally infeasible: Even in the simplest case of classification, this would require a precise formal model of not just the judge's 's prediction ff()), but also the relationship between and the correct labels y(r)). We therefore treat the scoring functions s and 3 as black boxes, as is standard, and focus on the joint distribution of the two induced random variables s(m) == s(X, m) and 5(m) := m))
As both scores smm) and 5(m) are binary, we only require three parameters to specify their joint distribution:
b(m) P(s(m) = =1}
p(m) ((5(m) ) = s(m) (mm) = 1} ((m) (mm(m) = =s(m) smm) = 0}
Here, the parameter b(m) represents the model's expected real score [m(m), while p(m) represents the probability that the evaluations s(m) and 5(m) match, conditional on the score being equal to one (i.c., the true positive rate). Similarly, ((m) represents the probability of s(m) = 5(m) conditional on s(m) = 0 (i.e., the true negative rate).
2.1 JUDGE BIAS CAN STRONGLY PERTURB MODEL RANKINGS
We argue that estimating the expected proxy score E (m)) instead of the real expected score EE (m) problematic for benchmarking. In the spirit of Salaudeen & Hardt (2024), benchmark has two purposes: quantifying model performance and ranking models. In turn, a useful LLM judge should accurately estimate each model's 's performance, and induce the same ranking of evaluated models as the ground truth. For the first goal, the main quantity of interest is the judge bias, defined as
JB(m) = (F)((m) s(m) = (1 - ((m)) (1 b(m)) - (1 - p(m)) b(m).
Whenever JB(m) has large magnitude, the proxy score ş misrepresents the performance b(m) of model m, even with large sample sizes.
When it comes ranking models, the relationship between judge bias JB and model rankings is more subtle: If the judge bias JB(m) was constant across all models, rankings would be unaffected. However, (m) can strongly depend the evaluated model m. If JB(m) is positive for models
3

---

Published as a conference paper at ICLR 2025
with low performance, and negative for better performing models, using the score of the judge š can easily perturb model rankings. Our next proposition provides such an example inspired by empirical work. Specifically, real-world classifiers not only highly similar to each other (Mania et al., 2019), Mania & Sra (2020) also observe that when one classifier is better than another on real data, it tends to be strictly better in point-wise sense, for the most part. We therefore consider using a classifier m to evaluate a set M of strictly better classifiers. In that case, using the proxy evaluations based on m fully reverses the correct model ranking:
Proposition 1. Consider a binary classifier in and a set of strictly better binary classifiers M such that m(x+) v(z) implies ma(a)) y(x)) for all my E M.. Let Bs(m)) represent the accuracy of model m, evaluated on the correct labels, and E 3(m²) its accuracy evaluated on predictions of model m.. Then for m,, m E M, E [smm)) > E ((m)) implies Es(m,) < <E <m,))
Appendix C.5. An illustration is provided in Figure 1a. We use GPT-4 labels to evaluate Claude Sonnet, as well as a set of (ficticious) models on MMLU (Hendrycks et al., 2021). The ficticious models are denoted as "Claude-Sonne" + x" and designed to be strictly better than Claude. This is done by changing wrong predictions of Claude into correct ones until accuracy is improved by x%. Figure l shows a more realistic setting, with LLama-3 70B judging the top-10 10 MMLU models according to HELM (Liang et al., 2023). While the ranking is not fully reversed, it is strongly perturbed. This effect can also be observed in prior work: For MTT-Benhh, the ranking of the best models is not consistently preserved when using LLM judges (Zheng et al., 2024).
We prove Proposition
2.2 HIGH AGREEMENT RATE IS INSUFRICIENT FOR EVALUATION
The results from the previous subsection might appear surprising, considering the strong perfor- mance of the judge models: Lama-3 70B has 79% accuracy on MMLU, while GPT-4 has 84%, and the models used in MT-Bench have 85% agreement with expert annotators. Our next proposition ex- plains how it is possible for model rankings to be strongly perturbed despite accurate judges. Before stating it, we need to formally define the agreement (mmm) between score and proxy score 5 as AG(m) == (m) {mm) b(m)p(m) + (1 - b(m)) q(m).
In the classification setting, the judge model's accuracy lower bounds the agreement AGmm): Clearly s(2, m) 3(x, m) whenever the judge model m is correct. But when there are more than two classes, we also get s(x, = $(x, m) = 0 whenever the true label y(r) and the predictions m(x)) and mxx) are all different from each other. In particular, this highlights that the agreement AG(m) can indeed depend on the evaluated model m, rather than just the judge m.. For binary classification, however, the agreement rate equals the judge model's accuracy and is thus constant across evaluated models. Our next proposition now relates judge bias to agreement:
Proposition 2. Fix any model score b(m). Then for any with b(m) I r, there are values of q(m) and p(m), such that AG(m) = and we obtain positive judge bias JB(m) = 1 11-r. Similarly, for any r with b(r) S r. there are values of q(m) and p(m), such that (G)m) = r and we obtain negative judge bias JB(m) = -
Proposition 2 shows that for agreement AG(m) r, there can be judge bias of 1 in either direction. This means that even if AG = r was constant across models m, we could only reliably rank models for which the true score E (m) differs by more than 2 2(1-r). is is common for -fthe-of-the-art models to differ in performance by only low single digit percentages, without further assumptions an agreement rate above 99% (for each evaluated model!) would be required to ensure (asymptotically) correct rankings. For a proof, see Appendix C.6.
3 DEBIASING LLM JUDGMENTS
In the previous section, we demonstrated how biased LLM judges can lead to strongly perturbed model rankings. This motivates the need for debiasing methods. We begin this section by discussing a simple sufficient condition to guarantee asymptotically correct rankings: Ensure that our estimates of E a(m) are unbiased for all models m. Statistical bias correction methods can make use of large number of proxy scores and a smaller number of gold standard labels to construct an unbiased estimator. We discuss one such method in detail, and show that it is essentially optimal in terms of estimator variance. Afterwards we prove our main result, i.c., that LLM judges offer limited benefits for evaluating state-of- -f-the-r- models.
4

---

Published as a conference paper at ICLR 2025
3.1 UNIFORMLY CONTROLLING JUDGE BIAS ENSURES CORRECT RANKINGS
We begin with a simple and natural sufficient condition for correct rankings. This is that all model evaluations are approximately unbiased. The next proposition shows that this, in fact, entails correct rankings in expectation and hence also with a sufficient amount of data.
Proposition 3. Let M be a finite set of models such that for any m, m' E
[R(((m)) - [E(((m')) 2
Let å be an estimator for E s(m) such that Var (m)) converges to zero in the dataset size. Then if E {(m) E(s(m)) < for all models m, using 9 to rank models yields the correct ranking with high probabiliry.
Proof. By construction, E (m) induces the same model rankings Es(m). Chebyshev's inequal- ity and a union bound imply correct rankings Var ((m) goes to zero.
Proposition 3 implies that if we could guarantee sufficiently small estimator bias, large amount of proxy samples 3 would yield correct rankings with high probability. However, it is important for bias to be small for all evaluated models. In light of the worst-case results from Proposition 2, cannot rely on this to be true when solely basing our estimates on the proxy scores
3.2 BACKGROUND ON PREDICTION POWERED INFERENCE
We next discuss how to leverage a small amount of ground truth samples to debias the proxy scores This yields an unbiased estimator for E and thus asymptotically correct model rankings.
Specifically, we follow the Prediction Powered Inference (PPI) framework (Angelopoulos et al., 2023a) and an for our case equivalent method by Chaganty al. (2018): Alongside large iid sample of model judgments (;(m) 8(1,, m) for integer i E [1, N + rt). we assume access to the corresponding ground truth label sm s(z,, m) for a small subset of integer j E [1, n]. With this, we can construct an unbiased estimator @PP for E (m)) by using the small parallel sample to estimate (Bm)., and subtracting from our estimate of Es(m). The PPI estimator thus equals
N+n
1
OPP
Σ
(.
(s)(m) (m))
N
1=1+1
Clearly, the PPI estimator is unbiased
R PP
B8(m) + Rs(m) - B8(m) = E Es(m).
and has variance
Var PPP
1
Var 8(m) + Var(s(m) - s(m)).
N Whenever N >> n, the first term on the right hand side is small. Moreover, when s(m) and 5(m) are strongly correlated, the second term is small. Assuming both, we have Var @PP < Var (GT where gOT is the ground-tme -ruth sample average estimator gGT (m)).
(y))
However if the correlation between s(m) and 8(m) small, the inequality is reversed. This issue be fixed by interpolating between the PPI estimator and the classical estimator (Chaganty al., 2018; Angelopoulos et al., 2023b), setting
9.r - + (1 - (1-X)0CT.
As linear combination of unbiased estimators this remains unbiased. but can now optimize over to minimize estimator variance. We obtain the optimum value x = ((((s(m),3(mm) ((1+ (1++)
+) Var (m)). At this X, the estimator 0.r 0fP never increases vari- compared to the classical ground truth estimator:
Var PPP Var s(m) 1 ((((((m),((m)) < Var(š(m))
1 Var s(m) = Var GGT
In the following subsection, we discuss to what extent the PPI estimator PPP can reduce variance or equivalently improve sample efficiency, compared to the ground truth estimator (GT
5

---

equivalent to Published as a conference paper at ICLR 2025
3.3 THE SAMPLE EFFICIENCY FACTOR
As the variance of joT scales as (1//n), using an estimator with r increasing the ground-trun truth sample size by factor of
Var Var gar
-(0) Var gGT
Var
We call r0) the sample efficiency factor of the estimator Ó. It is main quantity of interest. In order to ensure it is well-defined, we assume b(m), p(m), g(m) E (0, 1) for the remainder of the text. Our next proposition provides upper bound the sample efficiency factor (0pPP) of PPI, based on the squared Pearson correlation between s and
Proposition 4. The sample efficiency factor for the PPI estimator is upper bounded by
T(ORP) ((s))),((m))2²
1
where
b(m) (p(m) E {(m))²
p(s(m), (1 - (1-b(m)) E (8(m)
IE 5(m))
This upper bound is large whenever p² is large. However, the sample efficiency factor 7 (( is finite unless we have perfect correlation p² For any correlation bounded away from the proxy samples can only provide a constant factor improvement. Scaling them up without also scaling the number of ground truth samples has limited benefits. The proof can be found in Appendix C.7.
3.4 PPI NEAR-OPTIMAL FOR BINARY EVALUATIONS
The limited sample efficiency gains of PPI compared to the classical estimator raise the question whether we can find an unbiased estimator with lower variance. For black-box estimators that do not model the prompt-conditional distribution P[(s(m), {(m))]xr), we show that the answer is no: Theorem 5. Let ② be the set of all unbiased estimators for Es(m) that observe n joint samples (s(m), 3(m)) and N independent proxy samples 5(m). Then, any Ô O fulfills the variance bound Var Ô 2 Var 07.
This means that the sample efficiency factor of 0 bounded by
T())
< max max T(OPP) - 1 p p(s(m), 3(m))2
BE0
Theorem follows from an application of the Cramér-Rao bound, combined with extensive algebraic manipulation that was assisted by computer algebra system. Appendix C. has the details.
3.5 LIMITED GAINS FOR EVALUATING FFATE-OF-TEES MODELS
With Theorem 5 at hand, we can analyze the best sample efficiency factor r an unbiased estimator can achieve by making use of model judgments frontier models that outperform older models, including the models used to judge them. We use the evaluated model's score 6(m) and the judge's 's agreement rate AG(m) as indicators for the respective model's 's capabilities. We focus on tasks for which performing and judging the task are of similar difficulty, such that these indicators are comparable. As prime example, for binary elassification, AG(m) equals the judge's accuracy, while b(m) equals the evaluated model's accuracy.
this subsection, we focus on the evaluation of
We thus capture the evaluation of frontier models with the assumption that the evaluated model's score b(m) is larger than the judge's agreement rate AG(m). The next theorem shows that the squared correlation p(s(m), 3²m))² at most one half in this setting.
Theorem 6. Assume 0.5 AG(m) < b(m). Then.
p(s(m),8(m) < 0.5.

---

equal the Published as a conference paper at ICLR 2025
We prove Theorem 6 in Appendix C.2. Combining Theorem 6 with Theorem 5 allows to bound the sample efficiency gains that any unbiased estimator can achieve for evaluating frontier models: Corollary 7. Assume 0.5 AG(m) S b(m). Then,
max r(0) 2 dee
Tmax
Corollary implies that when evaluating state- the-art models, the best we can expect from using LLM judges is factor-two improvement in sample efficiency. This is unless the judge's task is substantially easier than the evaluated model's task.
3.6 HIGH AGREEMENT RATE IS INSUFFICIENT FOR SAMPLE EFFICIENCY GAINS
Theorem 6 holds for arbitrarily high levels of agreement. Therefore, the theorem shows that high agreement rates are not just insufficient for avoiding judge bias bias-they re also insufficient for obtain- ing a meaningful sample efficiency factor т. Our next proposition goes step further and shows that the agreement (mmm) does not provide any lower bound on the sample efficiency factor т, even when the judge bias (Bm is zero.
Proposition 8. For any agreement rate 0.5 s AG(m)
<
1, there exist values of
b(m), p(m), ((m) E (0, such that
JB(m) = pls(m), 3(m))) 0 and thus Tmax 1.
We prove the proposition in Appendix C.8. implies that in some cases despite access to š, no unbiased estimator has less variance than jGT It might be surprising that there are cases in which cannot make productive use of the proxy evaluations despite zero judge bias. However, without further assumptions, JB)m) can vary wildly between models, so that we cannot skip estimating it anew for each model. Now, even if has zero judge bias, the estimator does not "know" that, but rather has to estimate JB(m) from joint observations (s, 3). In the configuration from the proposi- tion, estimating the judge bias JB(m) turns out as hard as directly estimating the real score IE s(m).
3.7 EXPERIMENTS
So far, have shown that even optimal debiasing yields limited sample efficiency gains at the frontier (Section 3). In this section, we have a more detailed look into the sample efficiency gains on popular benchmarks that can be achieved by using current flagship models judges.
Our experiments in this section focus on two settings: Multiple-choice question answering on MMLU (Hendryeks et al., 2021) and chatbot evaluation MT-Bench (Zheng et al., 2024). Ad- ditional experiments on TruthfulQA can be found in Appendix B.2. In all cases, we calculate the empirical correlation p(s, 5) between the score and the proxy to obtain and plot an upper bound for the sample efficiency factor Tmax based on Theorem 5.
We first focus MMLU, multiple-choice question answering benchmark for the world knowl- edge of language models. We obtain model predictions from the HELM (Liang et al., 2023) leaderboard. We then aggregate subtasks of MMLU into single test set and set accuracy score s(x, m) = [z((z) = y(z)). Here is the indicator function, m(x) is the model's answer and y(x) is the correct answer. For model-based evaluations using in as a judge, we replace the correct answer y(x) by the judge's prediction m(xr) setting $(æ, m) = I[(m(x) = m m²(x)).
In the second setting, we evaluate model performance on MT-Bench, an arena-style benchmark designed to evaluate the ability of models to follow instructions. The benchmark comes with results for six models. For each pair of these models m and m', both models are queried with the same prompt The test set for each model then consists of all triples of the form (m, m', xx), for other models in the benchmark m and prompts æ. For each triple in the test set, an expert evaluator decides, whether model produced better response than m'. In that case, the score s(x: m) for model m is one, otherwise it is zero. For the proxy score $(x, m) we replace the expert's 's answer by the GPT4 response provided by Zheng et al. (2024). For more details, consider Appendix B.
Figures and 3 show the optimal sample efficiency factor Tmax on MMLU and MT-Bench respec- tively. The sample efficiency factor Trax consistently stays below the value of two suggested by

---

Published as a conference paper at ICLR 2025
Sample efficiency Tmaa
Judge model: Claude-Sonnet
GPT-4
LLaMa3-70b
3-
2-
Claude 3.5 Sonnet
GPT-4
LLama3.1-4055
Gemini 1.5 Pro
LLaMa3-70b
LLama2-7b
Figure 2: Best possible sample efficiency factor Tmax according to Theorem 5, using different judges (colors) evaluating different models (x-ticks) on MMLU. Error bars show 90% confidence intervals.
Sample efficiency gains stay below two, unless SOTA models are used to evaluate weak models.
Sample efriciency Tmax
3.0
Judge model:
GPT-4
2.0
1.5
1.0
GPT-4
Claude-V1
GPT-3.5-Turbo
Vicuna-13b-V1.2
Alpaca-13b
LLama-13b
Figure 3: Best possible sample efficiency factor Tmax according to Theorem 5, using GPT-4-as-a- Judge, evaluating different models MT-Bench. Error bars show 90% confidence intervals. Sample efficiency get close to two in some cases, but consistently stay below that value.
Corollary 7, except when current flagship models are used to judge the significantly worse LLama2- 7b on MMLU. This exception is not surprising, in that case AG > such that the assumptions for Corollary are heavily violated. Interestingly, Tmax stays below two in all other cases, even when stronger models like GPT-4 used evaluate weaker models like LLama3-70B. This sug- gests that the upper bound on maax from Corollary 7 is fairly robust. We also note that Tmax is often substantially smaller than two, especially on MMLU. This highlights that Corollary 7 provides best-case upper bound rather than guarantee that Tmax will be close to two in practice.
4 GOING BEYOND BINARY EVALUATIONS.
So far we have assumed the proxy score 3 to be binary. In this section, we relax this assumption. By not forcing the judge to fully commit to single answer, hope obtain more useful proxy For example, in a Q&& task, we can make use of the uncertainty of a judge model m' by setting $(z, m) P (m)) (m(x)) equal to the probability the judge m' assigned to the model answer m(x).
In terms of analysis, the non-parametric joint likelihood of (s, 8) resulting from non-binary proxies makes exceedingly hard prove an analogon to Theorem 5 for all unbiased estimators. Instead, we focus on the PPI estimator, DPP. It is a natural choice estimator, given its proven optimality in the special case of binary proxies 5. In order state condition similar to AG < b from Theorem 6, we assume the proxy to be bounded in [0, 1] and define the soft agreement as
SO(3) = E + (1 (1=s)(1- (01--3) 3).
For binary scores 3, the soft agreement S3(3) simply reduces to the agreement AG(3). For non- binary proxy scores 5, interpret them probability estimates, such that SO equals the estimated probability of the real score s. To make this interpretation valid, we focus on the recalibrated proxy R($) P(s 1.3). The following proposition suggests way generalize the condition AG < b: Proposition 9. For any binary proxy & such that 0.5 < AG < b, we have 0.5 < SO(R(3)) <
Proposition 9 states that if the agreement AG(3) of binary proxy is below the evaluated model's accuracy b, the same is true for the soft accuracy SO SO(R(5) of the recalibrated proxy R(3). Based on this, SO(R (R))) & b generalize the condition AG(5) < b from Theorem 6, and prove:
8

---

Published as a conference paper at ICLR 2025
Theorem 10. For any proxy score with SO(R(3)) b, have (s, S 0.5. Correspondingly, the sample effficiency of PPI is bounded: T(0 - 2.
We prove more general versions of Proposition 9 and Theorem 10 in Appendix C.3. In these, we allow for the judge to make times many mistakes as the evaluated model for rather than being strictly worse. In that case, we obtain a maximal sample efficiency factor r (ÔPP) of
Experiments on MMLU Using the prompt format from HELM, we extract I.Lama3. 1-00B'ss next-token predictions Ps: for all questions in MMLU, From these, we take the probabilities Pa (t) corresponding to the four tokens E {A, B, C, D} =: 'I' representing the answers to question We then renormalize them to P (m) (t) = (f- (t)) Ever Pz (t)) and define the proxy score $(z; m) as the probability P. (=) assigned to the evaluated model's answer m (x). as described above.
Figure 4 shows the sample efficiency factor T(fPP) for PPI in the limit of infinite unlabeled data N + op. As expected, using non-binary proxy consistently improves sample efficiency. How- ever, as suggested by Theorem 10, the sample efficiency factor 7 (((PP) remains below two when we LLama3.1-40S 1-405B to evaluate the stronger Claude 3.5. As in the previous experiments, the same true for evaluating slightly weaker models such as GPT-4 or Gemini 1.5. This shows that our main the frontier" is robust. That said, claim of "no sample efficiency gains of more than factor exceeded for LLaMa3-70BB (which is unlike in the binary case, the factor two is already (slightly) 5% less accurate than LLama3. 1-405B). rather than just for the much weaker LLaMa2-7B.
Sample efficiency factor
Judge model:
4-
LLama3.1-4055 1-405b (Binary)
LLama3.1 3.1bb05b (Non-Binary)
Claude 3.5 Sonnet
GPT-4
LLama3.1-4052
Gemini 1.5 Pro
LLaMa3-70b
LLama2-7b
Figure 4: Sample efficiency factor T(PPP) for PPI in the N - co limit, using LLama3. 1-4051---BB- Judge on MMLU, with binary and non-binary scores. Error bars: 90% confidence intervals. Non- binary scores improve the sample efficiency factor T()PP), but stays below two at the frontier.
RELATED WORK
JLd--e----uugee With the success of large transformer-based language models, the idea using model predictions to provide feedback for another model has become very popular. Initially, spe- cialized fine-tuned models were used to provide training signals for a second model (Ouyang et al., 2022; Bai et al., 2022: Dorner et al., 2023). The approach quickly expanded to using general purpose LLMs like GPT-4 (Achiam et al., 2023), not just for providing training signal but also for evaluating other models. This new paradigm, dubbed ALM-as-a-Judge has been applied to evaluate a variety of model capabilities (Yu et al., 2023: Chiang & Lee, 2023; Fu et al., 2023; Li et al., 2024; Weyssow et al., 2024; Raju al., 2024: Vu et al., 2024; Kumar al., 2024). In some cases, just the ratings, but also the prompts given to evaluated models are designed by LLMS (Bai et al., 2024). -uduue----ddee is often paired with another emerging paradigm for model evaluation: As generative tasks often do not have single correct answer. models are evaluated in so-called arena benchmarks (Chiang et al., 2024), where different models' responses to the same prompts are ranked to determine the best model. Beyond evaluating models, LLM judges have also been employed to evaluate red-teaming (Mazeika et al., 2024) and jailbreaks (Souly et al., 2024). The use of LLM judges rather than experts crowd workers is often justified by high agreement rates (Gilardi al., 2023; Zheng et al., 2024) between both types of judges. However, Thakur et al. (2024) find that seemingly high agreement does necessarily imply accurate judge scores. In this work, pro- vide a theoretical justification for that finding and show that agreement not necessarily a good indicator of judge quality, even when judge scores debiased via PPI.

---

Published as a conference paper at ICLR 2025
Bias in LLM judges. LLM judges can be biased in numerous ways, making their evaluations unreliable: Their outputs often correlate poorly with expert annotations (Bavaresco et al., 2024; Koo et al., 2023). Models are known to rate their own outputs more favorable than the outputs of other models (Liu et al., 2023; Panickssery et al., 2024), prefer longer outputs and outputs containing lists, regardless of quality (Dubois et al., 2024b; Wei et al., 2024). They also exhibit choice-order bias et al., 2023; Wang et al., 2023; Shi et al., 2024) as well as a variety of other biases (Koo et al., 2023). While correcting for these known biases (Dubois et al., 2024a; Zheng et al., 2024) is an important first step, manually enumerating and correcting for all judge biases appears infeasible. As one potential alternative, Jung et al. (2024) propose to combat judge bias by having judges abstain based on their confidence. However, if abstention correlates with the evaluated model's performance, this approach can introduce its own biases to model evaluations.
(Dominguez-Olmee)
Debiasing methods. Usually, the biases listed above are found by identifying patterns in how LLM judgments deviate from (a smaller set of) ground truth labels. Instead of trying to identify and fix specific biases, another line of work uses ground truth labels to directly estimate the bias an LLM judge introduces and correct for it. This approach was already suggested by Chaganty et al. (2018) for debiasing classic automated NLP metrics like BLEU (Papineni et al., 2002) and ROUGE (Lin & Och, 2004). The authors find that their method, which is essentially equivalent to PPI, only improved data efficiency by around 10% using 2018's s automated metries. In addition, they show that their method has optimal worst-case variance. In comparison to their worst-case result, our Theorem 5 shows that 05. is always optimal in our binary evaluation setting.
For modern LLM-based metrics, using PPI has been suggested by Boyeau et al. (2024) and Chatzi et al. (2024), and applied as part of more complicated evaluation pipelines (Saad- Mald-Falcon et al., 2023; Tyser al., 2024). Furthermore, Fisch et al. (2024) combine PPI with stratified sampling for model evaluation. These works consistently show that PPI improves efficiency in terms of ground truth labels, but gains in effective sample size rarely exceed 50% and are always below 100%. Our upper bounds on the sample efficiency factor Tmax show that gains larger than that are indeed unlikely, especially when evaluating state-of-the-art models.
DISCUSSION
Our results show that for evaluating frontier models, LLM judges might fall short of the promise of largely replacing expert labelers: While doubling the effective sample size can be useful for practitioners, the order of magnitude of required ground truth labels remains the same with and without access to LLM judges. That said, there are ways to circumvent our negative results:
First, our results pertain to uniform sampling rather than more sophisticated sampling strategies. Approaches like stratified PPI (Fisch et al., 2024) might be able to obtain a somewhat better sample efficiency factor, for example by breaking the assumptions of Theorem 6 per-stratum. That said, Theorem 5 still applies per-stratum. Empirical results presented in Appendix B.3 suggest that at the frontier, stratified PPI rarely improves sample efficiency by more than factor of two, compared to standard stratified sampling. Second, Theorem 6 only holds if the proxy 8 is less capable at predict- ing than the evaluated model m is at its task. Thus, for tasks in which evaluation is substantially easier than the task itself, or tasks for which data to train specialized evaluator is abundant, gains in sample efficiency of more than a factor two are possible even when evaluating models at the frontier. In addition, weaker models might find their way into many applications due to cost saving reasons, such that efficient evaluations of these models are still valuable.
Finally, we would like to reiterate that our results do not only apply to LLM judges, but any form of biased evaluators including (poorly instructed) crowdworkers. Whenever the crowdworker majority does not consistently agree with the desired ground truth labels, bias correction is required for valid evaluation results. Similarly, if the crowdworker majority vote is less accurate than the evaluated model, Theorem 6 implics that crowdworker labels are of low value compared to ground truth labels.
7 ACKNOWLEDGEMENTS
We would like to thank Anastasios Angelopoulos, Amin Charusaie, Yatong Chen, André Cruz and Guanhua Zhang for helpful discussions and/or feedback draft versions of this work. Florian
10

---

Published as a conference paper at ICLR 2025
Domer and Vivian Nastl are grateful for financial support from the Max Planck ETH Center for Learning Systems (CLS).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint urXiv:2303.08774, 2023.
Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I Jordan, and Tijana Zrnc. irediction-poweree inference. Science, 382(6671):669 674, 2023a.
Anastasios N Angelopoulos, John Duchi, and Tijana Zmic. Ppi++: Efficient ceddiction-powerts inference. arXiv preprini arXiv:2311.01453, 2023b.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862, 2022.
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang. Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking foundation models with anguage-model-as-ee as-an-exaainee.. Advances in Neural Information Processing Systems, 36, 2024.
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi. Desmond Elliott, Raquel Fernández, Al- bert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanma, Alexander Koller, et al. Lims instead of human judges? large scale empirical study across 20 nlp evaluation tasks. arXiv preprint DrXiv:2406.18403, 2024.
Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael I Jordan. Au- toeval done right: Using synthetic data for model evaluation. arXiv preprint a7X00.2403.07008 2024.
Arun Tejasvi Chaganty, Stephen Mussman, and Percy Liang. The price of debiasing automatic metries in natural language evaluation. arXiv preprint arXiv:1807.02202, 2018.
Ivi Chatzi, Eleni Straitouri, Suhas Thejaswi, and Manuel Gomez Rodriguez. deddiction-powere ranking of large language models. arXiv preprint arXiv:2402./7826, 2024.
Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evalu- ations? arXiv preprint urXiv:2305.019377 2023.
Wei-Lin Chiang. Lianmin Zheng. Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating 1lms by human preference. arXiv preprint arXiv:2403.04132. 2024.
Ricardo Domingucz-Olmen Olmeeo, Moritz Hardt, and Celestine Mendler-Dünner. Questioning the survey responses of large language models. arXiv preprint arXiv:2306.07951, 2023.
Florian E Dorner and Moritz Hardt. Don't label twice: Quantity beats quality when comparing binary classifiers on a budget. In Forty-first International Conference on Machine Learning, 2024.
Florian E Dorner, Momchil Peychev, Nikola Konstantinov, Naman Goel, Elliott Ash, and Martin Vechev. Human guined fair classification for natural language processing. In The Eleventh Inter- national Conference on Learning Representations, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsumori B Hashimoto. Length-controlled al- pacaeval: A simple way debias automatic evaluators, arXiv preprint 04X4::2404.04445, 2024a.
11

---

Published as a conference paper at ICLR 2025
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024b.
Adam Fisch, Joshua Maynez, R Alex Hofer, Bhuwan Dhingra, Amir Globerson, and William W Cohen. Stratified mrddcciion-ooweeed inference for hybrid language model evaluation. arXiv preprint arXiv:2406.04291, 2024.
Jinlan Fu, See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint wwXiv::2302.041666 2023.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text- annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e230001612) 2023.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359 378, 2007.
Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representutions, 2021.
Jachun Jung, Faeze Brahman, and Yejin Choi. Trust or escalate: Llm judges with provable guaran- tees for human agreement. arXiv preprint 1rXiv:2407.18370 2024.
Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zac Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators. arXiv preprint arXiv:2309. 02.3009177122. 2023.
Shachi Kumar, Saurav Sahay, Sahisnu Mazumder, Eda Okur, Ramesh Manuvinakurike, Nicole Beckage, Hsuan Su, Hung-yi Lee, and Lama Nachman. Decoding biases: Automated methods and Ilm judges for gender bias detection in language models. arXiv preprint arXiv:2408.03907, 2024.
Richard Larsen and Morris L Marx. An introduction to mathematical statistics. Hoboken, NJ, 2005.
Prentice Hall
Erich L Lehmann and George Casella. Theory of point estimation. Springer Science & Business
Media, 2006.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gon- zalez, and lon Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 10066///939 1939, 2024.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023.
Chin-Yew Lin and FJ Och. Looking for few good metrics: Rouge and its evaluation. In Ntcir workshop, 2004.
Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. Llms as narcissistic evaluators: When ego inflates evaluation scores. arXiv preprint arXiv:231 2023.
Horia Mania and Suvrit Sra. Why do classifier accuracies show linear trends under distribution shift? arXiv preprint arXiv: 15483, 2020.
Horia Mania, John Miller, Ludwig Schmidt, Moritz Hardt, and Benjamin Recht. Model similarity mitigates test set overuse. Advances in Neural Information Processing Systems, 32, 2019.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint 240.::2002.042499, 2024.
12

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Arjun Panickssery, Samuel R Bowman, and Shi Feng. Llm evaluators recognize and favor their generations. arXiv preprint arXiv:2404.13070 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th anmual meeting of the Association for Computational Linguistics, pp. 311 311-318, 2002.
Ravi Raju, Swayambhoo Jain, Bo Li, Jonathan Li, and Urmish Thakkar. Constructing domain- specific evaluation sets for llm-as-a-judge. arXiv preprint arXiv:2408. 08808, 2024.
Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. Ares: An automated evalu- ation framework for retrieval-augmented generation systems. arXiv preprint arXiv:2311.09470 2023.
Olawale Salaudeen and Moritz Hardt. Imagenot: A contrast with imagenet preserves model rank- ings. arXiv preprint arXiv:2404.02112 2024.
Lin Shi, Weicheng Ma, and Soroush Vosoughi. Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by Ilms. arXiv preprint arXiv:2406.07791, 2024.
Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, et al. A strongreject for empty jailbreaks. arXiv preprint arXiv:2402. 10260, 2024.
Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the judges: Evaluating alignment and vulnerabilities in auuuges. arXiv preprint 2rXiv:2406.1200 2024.
Keith Tyser, Ben Segev, Gaston Longhitano, Xin- Yu Zhang, Zachary Meeks, Jason Lee, Uday Garg, Nicholas Belsten, Avi Shporer, Madeleine Udell, et al. Ai-driven review systems: Evaluating Ilms in scalable and bias- academic reviews. arXiv preprint arXiv:2408. 10365, 2024.
Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung- Foundational autoraters: Taming large language models for better automatic evaluation. arXiv preprint arXiv:2407.1083 10817 2024.
Peiyi Wang. Lei Li. Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint 2305. 2023.
Hui Wei, Shenghua He, Tian Xia, Andy Wong, Jingyang Lin, and Mei Han. Systematic evaluation of llm-as-a-judge in Ilm alignment tasks: Explainable metrics and diverse prompt templates. arXiv preprint arXiv:2408.1300 2024.
Martin Weyssow, Aton Kamanda, and Houari Sahraoui. Codeultrafeedback: An llm-as-a- judge dataset for aligning large language models to coding preferences. arXiv preprint 200.002403.09032. 2024.
Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-mix: A flexible and expandable family of evaluations for ai models. arXiv preprint arXiv:2310.17567 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging alddae-a-jude with mt- bench and chatbot arena. Advonces Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
A ADDITIONAL THEORETICAL RESULTS
A.1 SAMPLE EFFICIENCY FACTOR FOR STRICTLY BETTER CLASSIFIERS
In the of strietly better classifiers discussed in Proposition 1, can provide upper bound: When using classifier in with accuracy æ > 0.5 to evaluate strictly better classifier m with accuracy x - 6, we get ((m) 0, b(m) x p(m)
stronger
In that case we can only
reduce variance by a factor of 1 5. or equivalently improve sample efficiency by factor of 1.125:
Proposition 11. Fix ((m) 0 and assume b(m) ô 0 and + S 1. Then
as well as p(m) z+5 for > 0.5,
p(s(m), (mm)
Correspondingly, the sample efficiency factor is bounded by Tmax I 1.125.
Proof. Fixing ((m) = O, we get
(5 x 1)
x+6
p(s(m).5(m))
1-
Taking the 8 derivative yields
x(26 1) (6 - (6 8 + x)2"
ds p( ((m)))
This is zero precisely when d 1- and negative for larger than that. Correspondingly, ő 1-z is the global maximum. Inserting this back, we obtain
2
2
(o(m),,(m))) (x 1)2
(x + 1)2
with negative derivative (x-1)²
Correspondingly, the upper bound is maximized the (z+1)3-
4(z-1)
(x+1)=
smallest possible x 0.5, where equals
0.5² 1.52 9
A.2 BALANCED AGREEMENT RATE AS AN ALTERNATIVE
Intuitively, high agreement fails to constrain the squared correlation p² because predictors without any signal, such as constant predictors, can achieve high agreement. This analogous to how accuracy can be misleading for imbalanced classification problems. In classification, this is solved by focusing on balanced accuracy instead. By analogy, we introduce the balanced agreement with
BA(m) g(m) + p(m)
The next theorem confirms that controlling BA(m) yields meaningful lower bounds the squared correlation.
Theorem 12. For any value of BA(m), have:
4b(m)( (2 1) & p(s(m),5(m)) < (2BA(m) - 1/.
Theorem 12 is proven in Appendix C.4. It also provides another upper bound sample efficiency gains from access to ă: Whenever BA(m) is close to half, access to can not improve sample efficiency by much and the sample efficiency factor Tmax I-p² is close to one. As corollary, show that whenever the proxy has reasonable balanced accuracy. the minimum of the true positive rate p and true negative rate upper bounds the squared correlation p²:
Corollary 13. Whenever BA(m) 2 0.5, we have
p(s(m), 3(m))² < (pn()) g(m)).
Proof. Given BA(m) 2 0.5, (2BA(m) - 1| is maximized at the maximal value of BA(m). But with min (p(m), g(m) < r, that is at 111 Inserting into Theorem 12 yields the result.
n
14

---

Published as a conference paper at ICLR 2025
Figure 5 shows actual value of the maximal sample efficiency and its lower and upper bounds Tmax derived from Theorem 12 for a variety of models. In line with our theoretical results, Tmax lies consistently between the lower and upper bounds. Interestingly, the lower bound appear to be considerably tighter than the upper bound, that is, Tmax is much closer to the lower bound than the upper bound throughout the experiments.
Judge Model
Metric Upper Bound
6
GPT-4
Claude-Sonnet
Tmax
Sample efficiency
LLaMa3-70b
Lower Bound
3
X
1
Claude 3.5 Sonnet
GPT-4
Gemini 1.5 Pro
LLaMa3-70b
LLama2-7b
Figure 5: Best possible sample efficiency factor Tmax and lower/upper bounds based on the balanced accuracy BA for different judges (color) and evaluated models (x (x-ticks).
A.3 ESTIMATING THE GAP
ranking rather than accurately estimating model performance is the main goal, we need to estimate the difference B(s(m)) EE( EEs(m')) for model pairs (m, m.). Because of potential cancelations, the necessarily the
difference of the optimal estimators for both terms (x-(m) (m)) @ (mn' optimal estimator for the difference. Correspondingly, further improvements can be made by jointly optimizing and X' to minimize the variance of (m) - ( Solving the corresponding
optimization problem yields:
(mmm)= Cov(s(m'), 5(m')) (S)m),
((mm) (ov()(m),8(m))
+ / (Var(š(mm)) Var($(m)) Cov(5( 5(m))) Cov(s(m), 5(m)) Var($(m')) - Cov(s(m/),, 5(m)) Var(š(m'))
(1 + (Var(((mm)) Var(5(m)) (o) 5(m²))
X² (mmm') Cov(s(m)) 3(m)) (ov" (6m3
-
(1 + N (Var((((m)) Var(5(m)) - (ov() 5(m()) Cov(s(m'), 3(m')) Var(5(m)) (ov(s(m), (mm))(m))))))) Var(S(m))
(1 / ) (Var()(m))) Var($(m)) -
However, using different values of for different comparisons means that there are multiple different score estimates for model such that the resulting comparisons might not yield valid transitive ranking.
B ADDITIONAL DETAILS ON EXPERIMENTS
For MMLU, we obtain most model predictions from the HELM (Liang et al., 2023) leaderboard and focus the top 10 models'. As HELM does not document model uncertainty, and results for LLama3. 1-05B were initially not available, we evaluated L 1-405B in bfl6 using 8 A.00
Cutoff date: 23.07.2024
15

---

Published as a conference paper at ICLR 2025
GPUs and the accelerate library for offloading, ourselves. We use the prompting format from HELM and extract the predicted probabilities p(y) corresponding to the four tokens E that represent the answer options. For our uncertainty experiment, we renormalize them, setting p(y) Ey'er Pla")'
p(y)
For MT-bench, we use the results for six models released by the benchmark's authors Zheng et al. (2024). For each triple consisting of two different models m, m' and prompt x there is varying amount of human expert judgments, as well as judgment by GPT-4. We aggregate the expert judg- ments using majority vote, and discard all triples for which the expert's tied, GPT-4's judgment amounts to a tie. We then calculate the win-rate for model m by averaging over all remaining triples that include m. Note, that this is slightly different from the evaluation in the MT-Bench paper. where win-rates are calculated per model pair, and then averaged over models. MT-Bench also includes follow-up prompt, with separate judgments for the first and second response. For simplicity, we focus on the first answer.
B.1 HIGH AGREEMENT NOT NECESSARY FOR RANKING.
Proposition 2 indicates that high agreement (m is not sufficient for the proxy score to yield similar model rankings as the real score s. However, it turns out that high agreement is also necessary for stable rankings: For binary classification with judge errors that independent of the performance of different models, a sufficiently large sample size guarantees correct rankings as long as the error rate below 50% (Dorner & Hardt, 2024). We demonstrate this phenomenon in Figure 6. In that experiment, the judge reproduces the correct label 60% of the time, but picks random wrong label otherwise. Despite this judge's low agreement with the correct labels, model rankings preserved near-perfectly
Clsude
Obude Sannet
Fus Labds
Yus Latels
JO S 2
Oput
2
Caudle Sonnei 10
Oasce Sans. 10
off-4e
Offi4e
Rank
Rank
g714
0m14
3
Cludo Sonnet 5
Osice Somi
Gemili 1.5
Ower2 inmunt (7201
(7240
Gemin 1S kns
4
Claudo sonns
Onde Samis
CPTi4
cm-44
cm14
Gamen 1.5 (prester)
Gamini ho sprenker
10
LEatra 328
LUe453 no
acourate labels
60% accurate labels
Figure 6: Model rankings on MMLU using 60% accurate judge with random emror.. Ranks are preserved despite agreement substantially below LLM judges.
B.2 ADDITIONAL RESULTS ON TRUTHFULQA
Figure 7 shows additional results on TruthfulQA. As on MMLU, we extract model answers to the Q&& prompts from HELM² (Liang et al., 2023) and define the proxy score š using the judge's answers mı(x) as described in Section 2. We use the top three models in the leaderboard as judges, and consider evaluations of these as well as some vorse-performing models. As predicted by our theory, the sample efficiency factor Tmax is consistently below when weaker models used to evaluate stronger ones. Beyond that, the sample efficiency factor Tmax even stays below two when use the strongest model we considered (Palmyra-x) to judge the weakest (LLaMa2 7B).
B.3 ADDITIONAL RESULTS ON STRATIFICATION
We consider debaising method for more sophisticated sampling strategies, namely stratified PPI (Fisch al., 2024). Note that efficiency improvements from stratification occur independently from
We accessed the data 14.1 1.2024, but it appears if the leaderboard has not been updated to include recent models such GPT-4.
16

---

valid Published as a conference paper at ICLR 2025
Sample efficiency Tmme
3.0
Judge model:
2.5
Palmyra-x
GPT-3 (Davinci)
GPT-3.5
2.0
1.5
Palmyra-x
GPT-3 (Davinci)
GPT-3.5
LLaMa2 70B
Vicuna 13B
LLaMa2 7B
Figure 7: Best possible sample efficiency factor Tuax using different judges (colors) evaluating different models (x-ticks) on TruthfulQA. Error bars show 90% confidence intervals.
the model-as-judge regime. In order to disentangle gains from stratification from the gains of (strat- ified) PPI (Fisch et al., 2024), we therefore focus on the per-stratun sample efficiency factor Tmax- Theorem 6 still holds per-stratum when applying stratified PPI, as long the assumptions for every stratum. It might however happen that the assumptions do not hold for every stratum, even they are true globally: judge model that weaker than the evaluated model on average could still be stronger on certain strata, thus achieving sample efficiency factor larger than two (on these strata).
Figure 8 suggests that this might be rare in practice: It shows the per-stratum sample efficiency factors Tmax on MMLU, using subtasks as strata. Using GPT-4 to judge the stronger Claude (Figure 8a) and ILLMM3.1 405B (Figure 8b), the sample efficiency factors Tmax varies significantly across strata. They are larger for most strata than the total non-stratifieed) sample efficiency factor. However they consistently stay below two. Even when using the stronger LLaMa3.1 405B (Figure 8d) and Claude (Figure 8c) to judge GPT-4 sample efficiency gains above two remain rare: When only observe them in out of 57 (Claude) and I out of 57 (LLaMa) cases.
Total
Total
2.5
Subtasks
Sample effillen
2.5
Subtasks
2.0
2.0
Sample
1.5
1.5
1.0
1.0
MMLU
(a) GPT-4 judging Claude
(b) GPT-4 judging LLaMa3.1 405B
Sumple efficiency Trax
Sample efficiency eecceenyc Trax
Total
Total
4
Subtasks
4
Subtasks
2
MMLU Subtasks
MMLU Subtasks
(c) Claude judging GPT-4
(d) LLaMa3.1 405B judging GPT-4
Figure 8: Per-stratum sample efficiency factor Tmax on MMLU, The strata are the 57 MMLU sub-
tasks, in alphabetical order.
C PROOFS
C.I THEOREM 5
We use a multivariate version of the Cramér- Rao bound, as presented in Lehmann & Casella (2006):
Theorem 14 (Lehmann Casella (2006) Theorem 6.6). Consider an experiment with parameter 0 E Ω for a product of open sets Ω E I2",, Assume the likelihood ((Z, 0) has the same support for all and finite derivatives d1(200)
8² log 2200)
Assume the Fisher information (()) = Ez~1(2.0)
d8
66"
17

---

Published as a conference paper at ICLR 2025
positive definite. Then for any statistic we have Varø(ō) dEa
I(0)-1 ((0)-1dmg) 6 In particular, if
du
Eg equals the i-th component of 8, we have
Var(o) 2 (I(0) (1(0))))(.)) ().)))
We again drop the dependence in order to declutter notation. We then use that the likelihoods of independent observations factor and the linearity of derivatives to obtain
1(b, = nl, (.p) + NIN(b,q.P).
where [bb, q. q) is the Fisher information for our experiment and I, (b, (b. 9. p) and In (b q.p) are the Fisher informations for single (s(m), 8(m) or single 8(m) sample, respectively. Denoting the corresponding likelihoods L, and Iy respectively, we have:
(1 b)q
(,(s = 0, 1) (1-b)( 4)
L,(s = 1, 0) = =b(1 p)
(,(s = 1,
bp
1x(š 0) [1 +)+ bp
1x(3 1) b(1 - p) + - (1-b)q.
Computing the Fisher information' yields
(I(0)) (b -10---/---11100-----0--000-----0
b 600-1110000111---100-----0-01110
n 100+0000((0-1000---++000--0-0---
6 (bp ())-)())))))((()))------)
1))
(b-1)²
(b (b-1)
b
b
(I(0)) =N
+
+ n
-b(p 1) (6
bp + (b 1) (g 1)
9
1/2
1/
b
b
(T(0)) = N
(b(p 1) (b 1) bp + 1) (q 1)
p 1
p
N ( ( hp by + + p + 1)
(I(O)) b2pp 2b2p 2b2p + b²q2 2b 122 2bpq bp 2bg² + 3bq
+ q² q
Nb (-p-q+
(I(0)) + 12 222g + 12 2bpy + hp 2bq + 3by - + q²
Nb (b (p 1) (b 1) (9 1))
(I(0))(g.))
(b- (b 1))
We used Sympy for automatic symbolic differentiation and simplification for the remainder of the proof.
The code can be found in Proofs.ipynb in the supplementary material.
18

---

Published as a conference paper at ICLR 2025
With this, calculate
(T.b))
n + -000--10000000000000000000000000
b [1
n (N
nNN B E3)
NN (ES -p))
b(1 b)
n
n (N + n) E E 8)
Vars
N (bE bp))² )
71
n (N + n) E R)
Vars
1
Cov(s, 8)2
Var PPP.
n
n² N
Var
C.2 THEOREM
Proof. To proof the theorem, reparameterize to simplify the constraints, namely setting AG(m) 2 0.5 and 8 = b(m) - AG(m) > 0. For notational convenience, we also set g q(m). We then optimize the squared correlation p² over the parameters in succession: At first, we take the derivative with respect to finding two roots. Thus, the maximum of p² with respect to is either at one of these roots or the boundary values for g. We consider each candidate separately, and optimize over x and 5 in a similar manner. This way. we enumerate all points q, rr, 5 that are candidates for the global maximum, i.e. all such points that are feasible, and either hit the boundary constraint or have zero derivative respectively, in all of the three variables.
We now delve into the details of the proof: Note that the agreement rate given by AG(m) (1 b)g + by. We therefore replace p(m) (b- (G(m)) 0² based on q, and 8:
(6+-- D2t+= and obtain a formula for
8 6+ 2qx 9)² p(s(m), 5(m))) (0 + x) (2ōg - 2gxx 2q) (2ōg - 5 + 2q 2g + 1)
(6 + x 1) (25g-
Maximizing in E [0, 1] We then take the derivative with respect to to obtain
plo(m)) 5(m))² 2 (6 + x 1) (2óg 5 + 2qzz 9) (2ògz ōg 6x + 2qx² 3ggz + 9) dy
d
(6 + x)) (25g- 8 + 2qx -29)² (25g 8 2qr 2g 1)²
The numerator is quadratic in q. so it has at most two zeros. These turn out to be at 21 = 28+22 2x- 8x
and 22 [2=-110=2-I]- We note, that there are singularities q 2 and g 210+0 However, as + x 1, the first singularity occurs at a value of q S 0. Similarly, the second singularity can be seen to occur at 1, as 5 - 1 < 26 + 2r - 2 is equivalent to 1 < 5 2z, which true as 0.5 and 8 2 O. Correspondingly, the singularities do not introduce any discontinuities in y for q E (0, 1). This means that for fixed æ and 6, p² is either maximized at #1 22, or at the
8
extreme values q = or q = 1.
Case 1: Inserting g = žy turns the numerator of p² to zero, such that p² = O.
6x Case 2: We next analyze 22 (2x (2==1)(0+-11) While ör and 2x are positive by assumption, + x 1 is negative. Correspondingly. q = 22 is not a valid value for q and the maximizer must be one of the other three values.
19

---

Published as a conference paper at ICLR 2025
Case 3:
Next, inserting yields
+ + x-
pls(m), 8(m))² (0 - (8 xx)
Maximizing a [0.5, 1) We take the derivative with respect to to obtain
dx (0mmm),,(m)))) 66 1) (6 xx
This is negative because 8 is positive while 1 is negative. Correspondingly, it is maximized
the smallest possible value of x = 0.5. Inserting x = 0.5 yields
5 (5 0.5)
p(s(m), 3(m))²
0.58 - 0.5
Maximizing in E (0, - x) We take the derivative with respect to 6, obtaining
d
(6 0.25)
db ((((m),)))))))))mm))) 8(m))) (6 1)-1)" (6 0.5)²
This zero = 0.25 and negative for larger 6, indicating a local maximum. thus insert 5 = 0.25, obtaining
0.25 (0.25 0.5)
((s(m), ((m))) 0.0625 - 0.125 0.5
9
Case 4:
This leaves us with q = 1. We again insert, obtaining
(ssmm).(mm)) (6 + 1) () + 2x
(6 6 x) 2x 2)
Maximizing in E (0, z) Taking the derivative with respect delta yields
d do (((((m),((m))
(x 1) (26 + 3z - 2)
(6 + x) (6 2x 2)
Note that this only has singularities at 6 = 1 and 8 + 2x 2, both of which are ruled out by the constraint + x < The derivative zero at
3z and negative for larger 8, indicating
2
local maximum. When x >
2 this value violates the constraint of 5 > 0, such that the maximum instead.
3-
has to be at or
Local maximizing (0, 3) We insert the local maximizer 8
3x
x²2
((m)))
(x 2)²
with positive derivative
d (s(mmmmmm))))m))))))())) 5(m))²
4r
dx
(o - 2)3
This means that the maximum value is at where
((s(m),5mm))
Boundary maximizing in x: E (0, 1) We check the boundaries for d, i.e., 0 < 5 < 1 - r. First, we notice that for 8 x, p² O. We thus focus on 8 0. Inserting yields
2x
p(s(m), 3(m))²
2x
with positive derivative
dr p(s(m), 3(m))²
2x2²
Correspondingly, it is maximized for where it clearly goes to 0.5. With this, the maximum attainable value for o equals 0.5.
20

---

Published as a conference paper at ICLR 2025
C.3 THEOREM 10
Instead of Theorem 10, we prove a stronger version that allows for the judge to be better than the evaluated model. Concretely, the soft error rate of the judge (-(((S(())) is allowed to be smaller than the soft error rate of the evaluated model - b by a factor < 1. In that case, our upper bound on becomes 2.
Theorem 15. For any proxy score 8 with c(1 b) s (1 (O(R()))) we have ps(s, S Correspondingly the sample effficiency of PPI is bounded: r(07.)) <
Proof. First, we note that the because the Mean squared error/Brier score is proper scoring rule (Gneiting & Raftery, 2007), the Bayes-optimal predictor R($) minimizes the mean squared error MSE(g(5), 8) for all post-processing functions y. At the same time, we have that the squared cor- relation (s)
(s,
MSB(a.f(E) falls monotonously in MSE(s, ((5), where Var(s)
the affine transformation with the smallest value of MSF(&, f()) (Larsen & Marx, 2005). This means that the bayes-optimal predictor R(3) fulfills (s, R()) 2 (s, As the Bayes-optimal predictor is calibrated, it is sufficient to prove our upper bound for calibrated scores á (i.e. scores $ such that 5 7) = for all 7 [0.
In the following, we assume to be calibrated. This means that
MSE(s,5) [[s - 3)² (11-5) 5² - 3))p(3)dd J. (1 5)5p()d,
where we use that because of calibration, conditional on a prediction 8, 8 is one and the squared error is (1 2)2 with probability 8. Similarly, the squared error is 82 with probability (1 8).
At the same time, the soft accuracy of & equals
SO / (82 (1- 111-3)7 ()))ps = I (23(3 1) + 1)p($)ds
for calibrated scores 3, as s = yields a score of á and happens with probability 5, and = happens with probability (1 8) and gets score of 1 - 3.
By the linearity of the integral, this implies that
SO 2 MSE
or
SO
MSE = 2
As is bayes-optimal with respect to itself, the optimal affine transformation in terms of MSE is just the identity. Thus, we have that
MSE
1-80
1- 2h(1 - b)
(s, (B.
Var(s)
b(1 b)
This is decreasing in 1-SO and thus maximized the minial value of SOO, which equals c(1 b) by assumption. Inserting yields
(3, <1 <(1 - b) = 2b
2b(1 b)
This is increasing in b and thus maximized b 1, where we have
(3,,8) 51 e(1 b) 2b(1 b) 2
L
Together with a generalized version of Proposition 9, Theorem 15 allows us to generalize theorem from the paper:
21

---

Published as a conference paper at ICLR 2025
Theorem 16. Assume that 2 0.5. Then for any 0 < 1, and any binary proxy 3 such that
AG 2 0.5 and c(1 - b) S (1 - AG), we have p² 1 2' Correspondingly, the sample efficiency factor Taax is at most
Proof. We first state the generalized version of Proposition
Proposition 17. Assume that b 2 0.5. Then for any 0 c S 1, take any binary proxy 3 such that AG 2 0.5 and c b) S (1 - AG). Then, the recalibrated proxy R(3) fulfill e ([1-6) b) S (1 (O).
We now simply apply Theorem 15. By Proposition 9, the recalibrated version R(š) fulfillss c(1 b) = (1 SO), yielding the desired bound. In the following, we prove Proposition 17:
Proof. We first characterize the recalibrated proxy R(s) and its relationship with the real score R(3) equals IP(s = 1|3 = 1) whenever ă = and (( 1|3 = 0) whenever & = O. This means that
bp bp+(1 (1(1_))) b(1 (1lpp)) p)
21 w.p. bp + (1 b) 9)
R(x)
6[1 (1- z0 w.p. 6(1 - p) + (1 -b4g.
Plugging this into the definition of soft accuracy yields
SO(R(3)) (bp+ + - () ((r)) (1---1000)))=+(1-1))))))(----))- x1)) (b(1 p) + (1 (200 zz)²
We want to show that (1 SO) <(1 b) 2 0. We begin by inserting the values, obtaining
(1 (O-S0) - e(1 b) = (c (b 1) 11000-0---1000110000000--0-00000
(bp (0)()))))))0))000---0---1---1--1
(bp + (0000-011011((00--)33((0111111
()) (172+
(1)))(00000) 1)²
1))
is easy to see that the denominator negative, so is sufficient show that the numerator NU is negative. We obtain
NU = (b - 1) (eb²p² + 2eb 2eb- - 2eb2 cb -2b2 eb² 2cbpy + cbp - 2eby
+ (b 1) (3eby eb + =gg² -eq - p2 + 262p 26²g² 22b4 22-22 + 2b4)
NU is positive. We take the derivative with respect As 1 negative, it sufficient show that b
to e:
NU de b 6- -
(bp + bq - b- q) (bp + bg - b- q 1).
But the first term equals b(p - 1) + (b 1)q = P(3 0) and is thus between minus one and zero. Correspondingly, the second term is positive, such that the derivative is negative overall. This 1. We insert that value and take two derivatives with respect
means that ( (6-1) is minimized to q:
d d NU dg dg b 5- 662 86 +
This has two zeros at = 1/2 and and is negative between them. Correspondingly, under our NU is concave in q and thus minimized at either of the two extrema 0
assumption of b 2 0.5, b-1
or = 1. At q = 0, we have
NU
b(p 1) (bp + b 1).
22

---

Published as a conference paper at ICLR 2025
This is positive whenever
bp + b 0.
But at q = 0, we have AG bp such that the condition becomes
AG +b 0, which is because we assumed both AG and b be least 0.5.
Meanwhile, At q = 1, we have
NU 1 bb 2b + 1).
Here is sufficient to show that
bp 26 + 1 0.
This time, AG hp b 1, such that the condition becomes
AG - bb 5,
which follows from c(1 b) S AGG as was fixed to one.
C.4 THROREM
Proof. We again drop the dependence for notational convenience.
p(s(m),š(m))
(p ((1 -1--00 b)
b ((1 -1 - pb)( (1- pb))(1 b)
((1 b)
b (1 9 - b qb + pb)(1 (1- q b qb + pb)) b)
(B - 11)2
6(1 b) (1 q - b+ BA) (1 + 2h BA)
Clearly, this equals zero when BA 0.5. Fixing BA 0.5 and taking the derivative with respect to yields
d p(s p(s(m).
b)BBA 11188-1122 (4b -2h 1 24 +
dg
(1-q-b- b 26 + 26BA
Assuming < b and BA * 0.5, this is zero if and only if BA -2h 2q + 1) is zero long the singularities in p² lie outside of the
and clearly positive for larger q. Correspondingly. q godomain, 2b BA -b + is a unique minimum, while p(s(m), 3mm))² 2 is maximized either at the minimal or the maximal possible value of g.
We first insert the minimum, noting that for lower bound we not need to worry about whether this value is indeed attainable. This yields
p(s(m), 8(m))) 4b(1 6)(2BA -1)
as both denominator terms reduce to 7.
Next, we assume BA > 2. As both p and are in (0, 1), we get that q= 2BA -p 2BA -1 Now, the singularities occur at q = b(2 BA- -1) and = 2BA b+1, 1, the first of which is smaller than 2 BA -1, while the second is larger than 1. Correspondingly, we can ignore them and insert minimal = 2 BA -1. We obtain
BA -1)
((s(m),5(m)) = 2b -2BA-BBA +
23

---

Published as a conference paper at ICLR 2025
and inserting q 1, we obtain
p(s(m), ((m(m) (b (2
2b BA -b -
These two terms are symmetric in the sense that replacing b in the first term with b yields the second term and vice versa. Thus, it is sufficient to maximize the second term with respect to b. We obtain
d db p(s(m), BA (2 -1)
(26 BA - - 1)2
which is negative such that (ms(m), {m)) 2 is maximized at = 0, where it equals 2 BA -1.
Similarly, for BA 1.2 q can range from 2 BA. Again, the singularities are at q = 6(2 BA -1) and 2 BA b - b ++1 This time, the first one is negative, while the second one can easily be seen be larger than 2 BA. This again allows us to ignore the singularities. Inserting the extreme values yields (2BA (2BA-1)(4e 226BA-641 -1) and M2BA - which again are the same after replacing with - b.
Taking the derivative of the first term with respect to yields
(2 (2 -1)
Te ((s)m) 5(m))²
(2b BA -b+ 1)2
which is negative as BA < 1. Correspondingly, point it equals BA. Joining both cases, we thus obtain
0 now maximizes p(s(m),8(m)) at which
p(s(m), 3(m))² < [2BA -
C.5 PROPOSITION
Proof.
E E3(x, m²) P(ñ((x m,(x))
(P(((x) =
m=())) y(x)) + P(m(x)= = m.(x), m((x) # v(x))
smm) 1 - ((),(() # y()), m(x) v(z))
(m)) + (x), # y()))
(m)m) +1 +1-Es(m)) {m))).
which is monotonously falling E (m,;)
C.6 PROPOSITION 2
Proof. We again drop the dependence on m. In the first case, we need that
bp + (1 b)g AG r JB (1 () + (1 p)b.
This is equivalent to
bp + q - by q -bb+bb+0 + bp
bp = b + b bp.
i.e. 1. With this, the constraint of AG - turns into b+ q(1 - b) = r. The left side is clearly continuous and monotonous in q and inserting the extreme values of zero and one, obtain and 1, such that we can find solution for all intermediate values of r.
In the second case, we want
bp + (1 - b)g = AG =-= r (1 (1 - b) (1 - p)b 1.
We simplify to get
bp + q bg = 11 1111--10 b qb - + pb +
or equivalently
q - bg - - b b
29 2bg + 2b = 2,
which is achieved at 9 = In that case, the constraint of AG = r turns into bp - (1 - b) = r with extreme values again lying at b = and
24

---

Published as a conference paper at ICLR 2025
C.7 PROPOSITION 4
Proof. We first note that Var)(m) b(m) b(m)) and Var 8(m) = - 8(m)) as both 8 and & binary. Furthermore, ([[s())((m)] b(m)p(m), such that Cov(s(m)) ((m)) ((m))((m)BB)))))
8(m)),
With this, it is easy to see that the formula for n² in the proposition statement indeed correct.
We then note, that for fixed values of n and N, the relative variance reduction of 0 compared to gCT is determined by the squared Pearson correlation coefficient p between the ground truth labels 8 and model judgements 8:
Var P.P
=1 1+
p(s(m), 8(m))) {(m))
Var GCT
Using the definition of this yields
-(0kt)
1+ I ((s(m)) a(m))
Finally, we take the limit of N > oo for an upper bound.
C.8 PROPOSITION 8
Proof. We drop the parameters' dependence on m and set g
and p Then JB
b(1 - b) (1 b)b = 0. At the same time, there is no judge bias, E ((m) b = p. such that
((s(m),3(m)) Lastly AG 62 + (1 (1, which can attain any value between 0.5 and O
25


Reviews:
[{"review_id": "NbQQfPO4UD", "reviewer": "Reviewer_hSz6", "summary": "This paper addresses an important challenge in LLM evaluation: the reliance on costly, high-quality annotations, which has become a key bottleneck as model complexity grows. The authors investigate the potential of using large, pre-trained models as \"judges\" to evaluate newer models, thereby reducing the need for extensive human annotation. They present a theoretical analysis showing a significant limitation: when the judge model’s accuracy is not superior to that of the evaluated model, debiasing methods cannot reduce the necessary amount of ground-truth labels by more than half. Empirical results further underscore this limitation, as practical savings on annotation requirements fall even shorter than the theoretical prediction.", "questions": "See weaknesses section", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors identify and tackle the misalignment between the ranks produced by the ground-truth score and those produced by LLMs. \n2. The motivation for the submission is formalised from the judge's bias and agreement rate perspective. \n3. The effect of real samples and the limited benefit of proxy samples for sample efficiency are well modelled and explained.", "weaknesses": "1. The writing sometimes is not very easy to follow. \n2. The submission only focuses on binary evaluation, can it generalised to more complicated cases?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eBopqIS2i7", "reviewer": "Reviewer_US4Q", "summary": "LLM-as-judge has become a popular practice for assisting humans in labeling and evaluating model performance. While this method shows promise, it still faces critical challenges related to inherent biases. This paper examines the biases inherent in LLM-as-judge setups and reviews existing debiasing methods. Especially, with the rapid advancement of models, there is a growing likelihood that evaluated models may surpass the judge model in performance. The authors demonstrate that when the judge model performs worse than the evaluated model on a given task, even the best debiasing methods offer no advantage over simply doubling the amount of ground truth data.", "questions": "Line 133: Why is it stated that precisely modeling the specific relationship is infeasible? Could you provide an example to clarify this?\n\nProposition 1: The explanation is unclear. What does \"point-wise\" mean in the context of line 166?\n\nIn the second setting in the experiment, arena-sylye benchmark, we see sample efficiency lower than 2 when using GPT 4v. but when evaluating on llaam2 -7b it's over 2, what's the examplantion here? is it due to the hugh performance gap between llama2-7b and the judge model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "Originality: To the best of my knowledge, this paper is the first to theoretically analyze biases in different judging styles and examine the impact of using state-of-the-art (SOTA) models as judges.\n\nClarity: The paper covers many important topics, but the structure could benefit from refinement to improve readability. Currently, some sections feel challenging to follow due to a somewhat disjointed flow of ideas.\n\nSignificance: With the growing use of LLM-as-judge setups in the research community, this work addresses a highly relevant issue. The paper provides theoretical analysis of inherent biases and explores how sample size impacts the effectiveness of debiasing methods.", "weaknesses": "**Structure**: The structure of the paper feels somewhat disjointed, making it difficult to follow the flow of ideas.\n\n**Clarity of Results**: The key takeaways from the final set of experiments (line 452) are not well articulated. The experimental setup and the expected outcomes are unclear. \n\n**Balanced Agreement**: In Section 3.6, where the concept of balance agreement is discussed, but the usage and its validation is missing. I am confused by its usage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HsiPWA0R9V", "reviewer": "Reviewer_eL89", "summary": "The paper investigates the model-as-judge evaluation framework when a calibration set with ground-truth scores is available to debias the judging LLM. It first shows the gap between prediction accuracy and ranking and then uses the prediction-powered inference as a calibration method to debias the judging LLM. With the debiased prediction, the paper shows that the sample efficiency is upper bounded by 2. The theoretical result is validated in the experiment with several popular LLMs.", "questions": "Will the experiment result still be consistent with Theorem 6 if the tasks are challenging such as TruthfulQA?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The theoretical results are solid and the experiment result matches well with the conclusion from the theoretical analysis, which makes the paper very strong.\n\nNovelty and Significance: Giving an upper bound for the sample complexity of LLM-as-a-judge is novel and significant.\n\nSoundness: The theoretical results and the derivation are sound.", "weaknesses": "The limitation is discussed in Section 6, which I pretty much agree on. It will be better if weighted PPI such as Stratified PPI can be evaluated in the experiment to show the difference.\n\nClarity: The clarity can be improved by adding examples of Q&A.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HUajXmPh9V", "reviewer": "Reviewer_PERR", "summary": "This paper studies the paradigm of model evaluation in which one uses a capable existing model", "questions": "- I am not very familiar with PPI (S3.3), what is the relation between PPI and doubly robust estimation from causal inference? At least on the surface level \\theta^{PP} looks similar to many estimators from causal inference.\n\n- What is the intuition behind the constant factor equalling two exactly?\n\n- Is it straightforward to extend the theory from the binary case to the continuous case (as in the experiments)? Or are there techniques specific to binary outcomes.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper studies an interesting problem, and presents both theoretical and empirical results supporting its central claim. The theoretical results are non-trivial and actually shed light on the practical question of LLM evaluation. Overall I enjoyed reading the paper.", "weaknesses": "The clarity of writing could be somewhat improved, especially the exposition around the experiments, the section feels a bit rushed and it was challenging to relate the content of the figures to the central message of the paper and the theoretical results. I would also suggest increasing the descriptiveness of the figure captions.\n\nThe experiments could also be more thorough - currently all experiments are on MMLU, but more datasets would strengthen the empirical results.\n\nSee also questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)