Accelerating Unbiased LLM Evaluation via Synthetic Feedback
Authors: Zhaoyi Zhou
Yuda Song
Andrea Zanette
Venue: ICML 2025 poster (2025)
Keywords: LLM evaluation
synthetic evaluation
variance reduction
Primary area: deep_learning->large_language_models
TL;DR: We construct an unbiased LLM evaluation method with synthetic feedback to reduce human annotation cost.

Abstract: When developing new large language models (LLMs), a key step is evaluating their final performance, often by computing the win-rate against a reference model based on external feedback. Human feedback is the gold standard, particularly for capturing nuanced qualities like coherence, readability, and alignment with human expectations. However, human evaluations are costly—even for large tech companies—and when conducted with active users, they may negatively impact user experience.
A promising alternative is synthetic feedback, where evaluations are conducted by other large language models, including reward models. While this eliminates the need for costly human annotations, it introduces biases that may distort the evaluation process.
In this work, we propose a statistically principled framework that integrates human and synthetic feedback to reduce reliance on human annotations while maintaining unbiased win-rate calculations. 
Our experiments demonstrate a reduction in human annotations by up to 12.2\% with an off-the-shelf synthetic evaluator and up to 24.8\% with a finetuned variant. Apart from being generalizable, scalable, and free of hyper-parameter tuning, our method offers predictable annotation savings, which can be estimated based on data-dependent characteristics.

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
Zhaoyi Zhou
Yuda Song'
Andrea Zanette
Abstract
2024). However, this approach demands substantial time and financial resources due to human involvement. When conducted with active system users, it may also diminish user experience, see Figure 2.
When developing new large language models (LLMs), a key step is evaluating their final perfor- mance, often by computing the win-rate against a reference model based on external feedback. Hu- man feedback is the gold standard, particularly for capturing nuanced qualities like coherence, read- ability, and alignment with human expectations. However, human evaluations are costly- - even for large tech companies - and when conducted with active users, they may negatively impact user experience. A promising alternative is synthetic feedback, where evaluations conducted by other large language models, including reward models. While this eliminates the need for costly human annotations, it introduces biases that may distort the evaluation process. In this work, we propose a statistically principled framework that integrates human and synthetic feedback reduce reliance human annotations while maintaining unbiased win-rate calculations. Our experiments demonstrate a reduction in human annotations by up to 12.2% with an off-the-shelf synthetic evaluator and up to 24.8% with a finetuned vari- ant. Apart from being generalizable, scalable, and free of syper-parameter tuning, our method offers predictable annotation savings, which can be esti- mated based on data-dependent characteristics.
In order to mitigate these challenges, recent works have explored cost-efficient alternatives, most notably the use of synthetic feedback generated by other LLMs, a concept often referred as "LLLM-as-a-judgee" (Zheng al., 2023; Dubois et al., 2024), to compute the head-to-head win rate. This approach leverages the computational efficiency of LLMs to evaluate other models, reducing the need for exten- sive human involvement. Despite its promise, synthetic feed- back often introduces biases since LLM can not perfectly re- flect human preference, undermining the evaluation reliabil- ity (Zheng et al., 2024). As result, critical need remains for evaluation methods that reduce the cost of human anno- tation while maintaining the reliability and generalizability.
Besides replacing the evaluator, recently there has been a growing interest in accelerating LLM evaluation (Ye et al., 2023; Polo et al., 2024a; Zhou et al., 2024) with smaller datasets. However, previous methods only focused on reducing the number of prompts in specific benchmark with predefined answers (e.g., math problems). Thus it is unclear if these methods generalize or apply to other tasks. For example, in math benchmark it is easy to find some problems that are "representative" of the whole benchmark, but in general the prompts are more diverse and less struc- tured, and sometimes they are generated on the fly, such as when a user interacts with language model via APIs.
1. Introduction
Towards reliable and cost-efficient LLM evaluation, in this work we propose to leverage LLM generated synthetic feed- back to reduce the number of human annotations, in the standard head-to-head win rate setting (Chiang et al., 2024). Specifically, we propose Control Variates Evaluation (Fig- ure left), an unbiased LLM evaluation method based on the classical control variates technique (Lavenberg & Welch, 1981) that combines human annotations and synthetic feed- back. Note that there are previous works (Chaganty et al., 2018; Boyeau et al., 2024) that apply control variates to machine learning evaluation, but they study settings like single-response natural language evaluation or BT mod- elling (Bradley & Terry, 1952). Therefore, the performance of control variates in head-to-head win rate estimation still
Accurately evaluating the performance of large language models (LLLMs) is crucial before large-scale deployment. Human judgment remains the gold standard for this eval- uation, as it captures nuanced qualities such as coherence, harmlessness, and readability (Bai et al., 2022), while also ensuring alignment with human values (Ouyang et al., 2022). widely accepted performance metric is the win rate, as- sessed by humans against a reference model (Chiang al.,
'Camegie Mellon University. Correspondence to: Zhaoyi Zhou <zhaoyizı andrew.em..ddu.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
Synthetic Annotation
uMi
LIM
10 -2
cheap
y
Human Control Variates
Synthetic
Human
Y
Averaged MSE
+ urbiesed expersive
10-3
Control Variates Eveluation
10
200
400
600
800
1000
cheap unbiased!
Number Human Annotations
Figure I: (Left) Illustration of Control Variates Evaluation, which makes use of possibly inaccurate synthetic evaluator to reduce the variance of evaluation, reducing the need of human annotations while preserving unbiasedness. (Right) Averaged mean square error v.s. number of human annotations for Human Evaluation, Synthetic Evaluation and Control Variates Evaluation using the finetuned Skywork-8B evaluator on Chatbot Arena. The Synthetic Evaluation has high bias, while the bias of Human and Control Variates Evaluations are negligible. Control Variates Evaluation reduces the variance of Human Evaluation.
requires thorough investigation.
We introduce Control Variates Evaluation to reduce the number of human annotations in head-to-head win rate estimation with zero bias, resulting in a reliable, cost- efficient and task-agnostic LLM evaluation method.
In work, we theoretically show that Control Variates Evaluation enjoys lower variance, and thus it requires fewer human annotations to achieve the same level of accuracy on the win rate estimation. Empirically, Control Variates Evaluation enjoys significant human annotation saving for various types of synthetic evaluators, from a small reward model with 2B parameters to LLMs such as GPT-4. In addition, we can further reduce human annotations by finetuning the synthetic evaluators on existing human annotations for other LLMs. Note that the cost of control variates is minimal as it only requires some additional synthetic feedbacks, which can be generated at a low cost. Somehow surprisingly, the synthetic evaluators that contribute to such achievement are inaccurate themselves and have high prediction bias (c.f. Figure right).
2. We demonstrate the viability of improving human an- notation saving through fine-tuning.
3. We propose the human annotation saving ratio as the data-dependent metric to predict the saving human data when using the Control Variates Evaluation.
We believe work is first step towards principled efficient LLM evaluation and can be combined with various existing and future works. Our code is avail- able at attps::/githii com/ Dae///zeeeeeeeeeee LLass controo__vriates peaittes__vlluuatin
2. Related Work
Besides the advantage of reducing the number of human annotations, Control Variates Evaluation also has predictable saving, one that can be estimated from the data and one which depends on how strongly the synthetic feedback correlates with human judgments. This is in contrast to the all existing methods that do not provide predictions on the potential saving. Based on the theoretical guarantee, we propose human annotation saving ratio as metric to evaluate our method, which can be computed through a few human annotations without actually rumning the evaluation. We demonstrate through experiments that this metric perfectly reflects the practical variance reduction effect in Control Variates Evaluation.
2.1. LLM Evaluation: Metric, Benchmark and Systems
The earliest attempt for LLM evaluation includes rule based metrics such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), which only measures the similarity between the model generation and the reference text. Going beyond rule-based metrics, LLM evaluation has been proposed, with earlier works using L.MM to compute similarity (Zhang et al., 2020; Yuan al., 2021). Recently, LLM-as-a-judge has been proposed to evaluate LLMs (Zheng et al., 2023; Dubois et al., 2024; Gu et al., 2024; Li et al., 2024a; Lan et al., 2024), by querying powerful LLMs to generaaeli2024ggeeratio preference of generations between different models, with the hope that the powerful LLMs can serve as a proxy for human evaluation. Towards real human evaluation, very few public
In summary, contribution is three folds:
2

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
Do a one line TLDR:
sosliingg ofter rely on lang chain-af-hooggts. making inference costly. We address by post-training models reinforcement learning reason effciently- g tokan usa while maintaining accuracy.
reasoning models
You're re giving feecback on new version of ChatGPT,
Which response do you orefer? Responses may take- moment to load.
Response
Response
Optimized reasoning models use reinforcement learning to reduce token usage while prescrving accuracy.
Optimized reasoning models use reinforcement learning to reduce taken usage while preserving accuracy.
reaporou
resportus
Figure 2: OpenAl's prompting users for feedback; excessive requests may negatively impact user experience.
systems exist due to their high cost and time-consuming nature, with the large-scale community collective effort Chatbot Arena (Chiang et al., 2024) being the most notable one.
& Hanawal, 2021), optimization (Yuan et al., 2024), com- puter graphics (Rousselle et al., 2016; Müller et al., 2020). In particular (Chaganty et al., 2018) uses control variates to evaluate natural language metrics, but it is restricted to single response evaluation. In work, extend control variates evaluation to pairwise LLM comparison.
2.2. Speeding Up LLM Evaluation
Recently there has been a surge of research on speeding up LLM evaluation, with the goal of reducing the cost and time of evaluating LLMs. One approach is use heuristics to minimize the number of prompts tasks to evaluate, with the hope that the selected subset can represent the whole distribution of the prompts or tasks (Ye et al., 2023; Perlitz et al., 2023; Polo et al., 2024a).
Prediction- Powered Inference (PPI, and PPI++) (An- gelopoulos et al., 2023a;b; Boyeau et al., 2024) is a related technique which uses variance reduction to improve the MLE objective. (Boyeau et al., 2024) applies PPI++ to estimate practical metrics in machine learning, such as accu- racy, correlation and BT model (Bradley & Terry, 1952) in pairwise model comparisons. It differs from our work which conducts in-depth study of control-variates to accelerate head-to-head win rate estimation.
The other approach to leverage active learning or ban- dit algorithms to select a subset of the prompts: (Polo et al., 2024b; Zhou al., 2024; Li et al., 2024b). How- ever, these methods are still limited by the requirement to operate within specific benchmark with prefined answers, and thus can not be applied to human evaluation, the focus of our work. In addition to the essential benefit that human evaluation can provide, note that is more challenging be- cause it is agoniiiiii and typically has less structure than any specific benchmark.
3. Preliminaries
3.1. LLM Evaluation
We consider the problem of evaluating LLMs performance through head-to-head comparisons, via human preference judgments. Given set of prompt X, we compare two LLMs El and E2 by estimating the win rate of EL over E² on X..
Formally, we independently sample a prompt x E X, and sample two responses yy² A- x)) and y² ~ E²(. |x) from el and e respectively. We then ask human annotators to choose the better response with label 2 z(y² y²). where
2.3. Control Variates, Application, and related techniques
Control variates is well-known variance reduction tech- nique in Monte Carlo sampling (Owen, 2013), with appli- cations to finance (Broadie & Glasserman, 1998; Hester- berg & Nelson, 1998; Kemma & Vorst, 1990: Glasserman, 2004). In recent years, it has also been applied to vari- ous areas of machine learning, such as variational inference (Geffner & Domke, 2018 8; Phan et al., 2023), bandits (Verma
I
if y is preferred over y if y" is preferred over y
2 =(y > v2) =
0.5
if tie.
We will use the shorthand 2 sometimes in the rest of the text when the context clear. The win rate of El over l on the
3

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
prompt is defined as
Algorithm 1 Control Variates Evaluation
1: Input: Evaluation dataset Deval = (2 vi v3 3f) human annotation budget k;
p(e2 > () = -B N² [=(v²= > }
2: Optional Input: Finetune dataset Dfinetune
{(rg u with human annotations {2
3: (Optional) Finetune the synthetic evaluator on D Pinetune-
i.., the averaged human preference over the prompt set, and Em,y² gg [.] := Ex-Uniffr((X) (10))) [=]] To estimate plll > (²) empirically, we collect an evaluation dataset Deval = {(0i, w/ )²) estimate human prefer- ence 2, zy) > y with 2t and output the empirical average frm ( > (2) = -
4: Get synthetic evaluations 21, 22, ... en on D
Sample & data from Deval and get human annotations
Zig Fig." :Zig-
6: Estimate μ Σ 24
n -== zith as the estimate of the win rate. Our goal is to minimize the number of human anno- tations involved in the process while keeping prm close to p.
7: Estimate using {z,,) j=1 and { 1 by Equa-
tion (2)
Output the estimated win rate
3.2. Human and Synthetic Evaluation
k
k
Human Evaluation annotates every sample (I); yi vz) in Deval with human, i.e. let zmm := 24. This makes the eval- uation unbiased. However, leveraging human annotator is extremely expensive, but without enough amount of samples n, the empirical mean ( > 2 can be very noisy due to high variance from a small sample size.
442
ki
j=
4.1. Control Variates
On the other hand, Synthetic Evaluation generates prefer- ence estimates 2yy > v²) using a reward model or LLM (e.g., GPT-4) (Zheng et al., 2023) on every sample. Al- though it completely obviates the need for human annota- tions, the evaluation is biased and can lead to inaccurate win rate prediction.
Given sample (x, ,',""' with human preference = and synthetic preference E, we treat 2 as the random variable for which we want to estimate its mean. Using as the control variate, the classical control variates approach (Lavenberg & Welch, 1981) constructs a new estimated preference:
em coia = =- a(= -12),
(1)
3.3. Other Notations
where μı Emy"" [ē(y' > y²]] is the synthetic win rate, and R is the control variates coefficient used to control the variance of go00, Intuitively, μı cancels out the bias incurred by the control variate 2, keeping the estimate unbiased. In addition, assuming that μı is known, we can guarantee variance reduction compared to human evaluation, as stated by
For two one-dimensional random variables x and y, we use Cov(a, w). Corr(z, y) to denote the covariance and corre- lation coefficient between v and y. respectively. We use Var [z:] to denote the variance of æ. Let {=:) [v: 1 be samples of æ and y, respectively, we abuse the notation and
use Var H= for the empirical variance of (r:))_. and Cov [(x,)==1 {ou Haal. Corr H=: 1 for the empirical covariance and correlation coefficient between {zx))= and (y 1L respectively.
Proposition 4.1 (Control Variates Properties (Lavenberg & Welch, 1981) ). Suppose the expectations, variances, covariances and correlation coefficients, unless otherwise stated, are taken under the distribution x ~ Uniform(X)) y' (- x), y² {²( z). Then the control variates estimate zena enjoys the following properties
4. Efficient LLM Evaluation via Control Variates
In this section, we introduce Control Variates Evaluation, which combines human and synthetic annotations realize a variance-reduced unbiased evaluation method, based on control variates (Lavenberg & Welch, 1981). We first recap the classical control variates method in the context of LLM evaluation, and then formally describe how to adapt the control variates method to make it applicable in practice. Finally, we briefly discuss its application in the LLM-as-a- judge setting (Zheng et al., 2023).
(1) (Unbiasedness) For any a E R. we have [| = pl6 x (2)
(2) (Variance Reduction) Let p= Corr(z, 2) be the corre- lation coefficient between human and synthetic prefer- ence. Then we have
min Varl20v00 = -(1-=) oeR
4

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
The minimum is achieved if and only if equals
control variates evaluation requires an evaluation dataset .Deval { (2g y v ] The Control Variates Evaluation consists of the following steps:
Cov(2,
a
Var
(3) (Human Annotation Saving) Given an evaluation dataset Deval = {(Ts vi Bi)). in which {a:)== are sampled i.idd. from X, st ~ (. xz), vi ~ C æi) (i E [nl). Let (ig be independently sampled from [n]. Then when m = (1 - (1-p²)m, we have
Synthetic annotation gathering (Line 4). We generate syntheric preferences 2; E [0, from an automatic anno- tator for all samples in the evaluation dataset. Synthetic preferences can be generated in various ways depending the type of automatic annotator. For an LLM annotator like GPT-4, query the model to directly generate the preference in natural language. If the automatic annotator is reward model, we can query the rewards ri and r from the two responses y and y respectively, and then compute the synthetic preference as the Bradley-Terry score of the two rewards (Bradley Terry, 1952), i.e.,
I m
Var
Var
2k
Here the variance on the right hand side is taken by the randomness of sampling {(x; ub vτ )-r The vari- ance on the left hand side is taken by the randomness of sampling {1 vi. vt y= well as that of sampling {ūg 3mm
2 = 1 1+ - r!)
Human annotation sampling (Line 5). We query the hu- annotator and obtain human preference E {0, 0.5, 1}. Instead of annotating all the samples like in Human Evalua- tion, we only annotate & samples randomly drawn from the evaluation dataset, in which & is the number of human an- notations we want to use. Increasing & lowers the variance of the estimation but raises the cost of evaluation.
We provide the proof in Appendix A for completeness.
Human annotation saving ratio. Proposition 4.1 imme- diately suggests that the control variates method can reduce the percentage of human annotations by p² while maintain- ing the same variance as that of Human Evaluation, with negligible cost of querying the synthetic evaluator. There- fore, p² is important metrics to measure the variance re- duction effect and cost efficiency of control variates method.
Synthetic win rate estimation (Line 6). Since μıg is un- known in practice, estimate it by averaging the synthetic evaluator's 's preferences on the whole evaluation dataset. In other words, μı := Ei= 24.
We formally define it below.
Definition 4.2 (Human annotation saving ratio). The human annotation saving ratio of a synthetic evaluator w.n.t LLMs El 2 and prompt set X is defined as
Control variates coefficient computation (Line 7). Al- though Proposition 4. (2) already shows the optimal a, the covariance between human and synthetic annotations as well as the variance of synthetic annotations needs to be estimated via sampling. Since human annotations in- volved in the computation, we reuse the human annotations
0² (Corr, wew.ew/ww/..ee (z((y') > 2yy, > ?))1)²
Here z(y' > y²) is the human preference, and Ely² > )²) is the synthetic prefence. The correlation coefficient is computed under the distribution I ~ Uniform(X),. el( 1x),"² y ~ (. x).
{ { &
) j=1
Nonetheless, to apply control variates approach in the con- text of LLM evaluation, we still face the following chal- lenges: How estimate the synthetic win rate ua? 2) How to compute the correlation coefficient in practice to achieve the lowest variance? 3) How to improve the cor- relation coefficient if the off-the-shelf automatic evaluator does not give a satisfactory human annotation saving ratio? In the following, we discuss how to construct the control variates for LLM evaluation.
Cov
{ } {2i
a:
Var {2 }
It is standard practice in control variates estimate with Equation (2) (Owen, 2013, Chapter 8.9). Although it introduces some correlation between and the final estimator, and thus the estimated win rate in Algorithm I is technically biased, the incurred bias is usually negligible, and it is standard practice to ignore such bias (Owen, 2013, Chapter 8.9). We also validate this practice through experiments Section 5.2.
4.2. Control Variates Evaluation
Algorithm 1 describes the full procedure of control variates evaluation. Same as other evaluation methods,
5

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
Win rate estimation (Line 8). After we obtain estima- tions of the synthetic win rate M4, and the control variates coefficient a, we can apply Equation (I) to get the variance-
Summary.
We offer several remarks:
Our construction of control variates is task-agnostic, i.e, we do not leverage any specific structure or knowl- edge of the prompt set X'
reduced preference estimates {
{ cyro 1=ز for the samples
we collected with human annotations. Then we output the win rate estimate by taking the average over the preference estimates:
The method is hyperparameter-free parameters for control variates like the synthetic win rate μı; and con- trol variates coefficient a are estimated directly from data. (If fine-tuning is used, one still needs to choose fine-tuning
k
1
pem
(f. > (2)
cuia
k
j=l
over a validation dataset)
1 k
The performance of Control Variates Evaluation is pre- dictable. By sampling a small subset of evaluation data, collecting human and synthetic annotations, and com- puting the human annotation saving ratio, the reduc- tion in human annotations can be accurately estimated without fully performing the evaluation. In the experi- ment (cf. Section 5.2), we show that the saving ratio of human annotations correctly predicts the observed saving.
Z
2ig M$
(3)
(Optional) Synthetic evaluator finetuning (Line 3). On many popular LLM evaluation benchmarks such as Chat- bot Arena and MT Bench (Zheng et al., 2023), there are abundant off-the-shelff human annotations for pre-generated language model responses. Now suppose we have a new LLM and we want to compare it with the existing ones in the benchmark. Can we make use of these existing human annotations to help reduce the human annotations needed in Control Variates Evaluation?
5. Experiments
To evaluate the performance of control variates in practice, we conduct experiments real-world datasets to mainly answer the following questions:
Recall that the human annotation saving ratio is p², the square of correlation coefficient between human and syn- thetic annotations. One natural idea is to raise the correlation coefficient by finetuning the synthetic evaluator with exist- ing human annotations, to save future human annotations.
1. How does Control Variates Evaluation compare to Hu- man Evaluation and Synthetic Evaluation (c.f. Sec- tion 3.2)?
Formally, suppose that we have a finetune dataset Diinetune- {(rr, 1t. v3) , j=1 with precollected human an- notations {=, m }- We discard the ties and assume 2j E {0, 13 for all 1 S j S m. In case that the synthetic evaluator is a reward model, we finetune the evaluator on Diinetune to maximize the Bradley-Terry score (Bradley & Terry, 1952) on the chosen response:
2. How does the finetuning process of the synthetic eval- uator affect the human annotation saving?
m
5.1. Setup
Synthetic evaluators. Towards comprehensive analy- sis, we experiment with synthetic evaluators across vari- ous model types and sizes, including BRMM-Gemm--2B- sftreg (GRM-2B) (Yang et al., 2024), ArmoRM-Llaman 8B (ArmoRM-8B) (Wang et al., 2024), Skywork- Reward- Llama-3. 1-B--3...--B-.... (Skywork-8B) (Liu et al., 2024) as well as GPT-4 (Achiam al., 2023).
2j BT (3)))))))))) + exp(1) rJ) 1 1+ exp(r) r
1- 2j
After finetuning, we expect an increase in the correlation coefficient p and thus also the human annotation saving ratio when we want to evaluate the win rate between a new LLM pair on the same benchmark. Note that the dataset used for finetuning the synthetic annotator contains responses generated by LLMs that are different from the LLMs that we wish to evaluate, i.c., the responses in evaluation dataset are out of distribution w.r.t. the finetune dataset. We explain in Section 5.1 how to guarantee this in our experiments. Despite the out-of-distribution property, we show in the ex- periment section fc.f. Section 5.4) that the finetuned model still generalizes well in terms of the correlation coefficient to the human annotations.
Finetuning procedure. The testing of Control Variates with finetuning (Line 3 of Algorithm 1) is done in a cross- validation manner. Suppose there are LLMs generating responses in the evaluation dataset. Our finetuning proce- dure trains K reward models, each by leaving out the data for specific LLM. That is, for each LLM k. we finetune the reward model on the head-to-head comparisons over the remaining K - 1 LLMs. This finetuned reward model is then evaluated on the head-to-head comparisons involving LLM against the other K 1 models. When comparing Control Variates Evaluation with finetuning and Synthetic
6

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
1e-4
1e-4
Human
Human
6-
Control Variates
Control Variates
Human (shifted)
Human (shiited)
Averoged MSE
5-
Averaqed MSE
5-
4
2-
14
200
400
600
800
1000
200
400
600
800
1000
Number of Human Annotations
Number of Human Annotations
(a) Skywork-8B
(h) Skywork-8B (ft)
Figure 3: Averaged mean-square eπror versus number of human annotations for Skywork-8D (pretrained and finetuned) on Chatbot Arena. The r-coordinate curves "Human" and "Control Variates" correspond to the number of human annotations (Zheng et al., 2023). The curve "Human (shifted)" is derived by horizontally scaling the Human Evaluation curve by (1 s), in which is the averaged human annotation saving ratio Table 1. The averaged reaane error of Control Variates Evaluation converges near O, indicating that it has negligible bias. The human annotation saving ratio aligns perfectly with the actual variance relationship between Human Evaluation and Control Variates Evaluation.
human- aniaanaenntttedd preferences. The responses gen- erated by 20 models, i.e., 190 LLM pairs in total. There are 121 pairs that have more than 100 annotations.
Pretrained
Pretrained
Finetuned
Averaued Saving Relof%)
20
Averaued Savinn Rallo/
MT Bench (Zheng et al., 2023) contains about 3.36k human- annotated preferences. The responses are gen- erated by 6 models, i.e., 15 LLM pairs in total. There are 14 pairs that have more than 100 annotations.
15
8
6
20:
5.2. Control Variates Evaluation v.s. Human Evaluation
As suggested in Section 4.1, the human annotation saving ratio is a practical metric to measure the performance of Control Variates Evaluation. Therefore, we will first present the human annotation saving ratio on different evaluators and benchmarks. After that, we demonstrate that this theo- retical measure matches perfectly with the actual variance reduction effect.
GRM-28
Skywork-68
GRM-28
Skywork-88
(a) Chatbot Arena
(b) MT Bench
Figure 4: Averaged human annotation saving ratio before and after fine-tuning for GRM-2B and Skywork-8B on Chat- bot Arena and MT-Bench. Under all setups, we observe at least 5% increase in the saving ratio.
Human annotation saving ratio on different benchmarks and synthetic evaluators. For each synthetic evaluator and benchmark, we test the human annotation saving ratio on every L.LM pair that have at least 100 human annotations. In order to clearly present the result, we take the mean of the ratios across different LLM pairs to get the average human annotation saving ratio of that evaluator on the benchmark. The result is presented in Table 1. We defer the human annotation saving ratio each LLM pair in Appendix C.3.
Evaluation, we apply the same cross-validation procedure to Synthetic Evaluation for fair comparison.
We tested Control Variates Evaluation with finetuning on GRM-2B and Skywork-8B models, which will referred to as GRM-2B (ft) and Skywork-8B (ft) respectively.
Benchmark. We choose LLM evaluation datasets with abundant and trustworthy human annotations. The datasets considered are:
For off-the-shedl evaluators, GPT-4 achieves high saving ra- tio on both benchmarks. However, an 8B reward model like ArmoRM-89 has comparable performance. Using the fine- tuning option Control Variates Evaluation, Skywork-88)
ChatBot Arena (Zheng et al., 2023) contains 33k
7

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
Table I: Averaged human annotation saving ratio across different synthetic evaluators on Chatbot Arena and MT Bench. The averaged human annotation saving ratio is the mean of human annotation saving ratios on L.LM pairs with at least 100 human annotations.
Then, we shift the x--axis of the Human Evaluation as fol- lows. Suppose s is the averaged human annotation sav- ing ratio we tested in Table 1, and (z, y) is a point on the curve of Human Evaluation. Then we shift point (z, y) to (x(1 - s); y). After shifting all points of the Human Eval- uation, we get a new curve, referred to as Human (shifted). According to Proposition 4.1 (3), the ratio of the number of human annotations in Human Evaluation and Control Vari- ates Evaluation should be 1 : (1 s) so that they have the same variance. So ideally, the shifted curve of Human Eval- uation should coincide with the curve of Control Variates Evaluation. We present the bootstrap curves for Skywork- 8B with and without the finetuning procedure on Chatbot Arena Figure 3. The other results are listed in Figure of Appendix.
Chatbot Arena
MT Bench
GRM-2B
10.6%
5.7%
GRM-2B (ft)
17.1%
10.9%
Skywork-8B
8.3%
7.5%
Skywork-8D (ft)
24.8%
12.6%
ArmoRM-8B
12.2%
9.6%
GPT-4
12.2%
11.9%
On all figures, the averaged mean-square error of Control Variates Evaluation converges to near 0, indicating negli- gible evaluation bias. Furthermore, the shifted curve of Control Variates Evaluation overlaps with that of human evaluation. Therefore, the human annotation saving ratio predicts the actual variance reduction of our algorithm al- most perfectly, even if the control variates coefficient is estimated. This means that we can simply compute the hu- man annotation saving ratio from the correlation coefficient, and then we know whether the synthetic evaluator will bring us the desired variance reduction effect when it is to be used in Control Variates Evaluation.
(ft) surpasses the performance of GPT-4 on both bench- marks. With finetune, small model (GRM-2B (ft) can also match outperform GPT-4 in averaged human anno- tation saving. This means that we can save from 10% to 20% human annotations using an casy-to-deploy reward model at nearly no cost. In fact, the most expensive syn- thetic evaluator in our experiment, GPT-4, costs just about cent per annotation. This is negligible comparing to human annotation cost.
Theory matches practice. We empirically justify that the theoretical human annotation saving ratio aligns well with the practical variance reduction ratio. Besides, we verify the claim in (Owen, 2013, Chapter 8.9) that Equation (2) leads to negligible bias.
5.3. Control Variates Evaluation v.s. Synthetic Evaluation
In this section, we compare the error in predicting the win rate between the Control Variates Evaluation and Synthetic Evaluation. The error metric is the mean square error with respect to the ground truth win-rate, which we approximate with the averaged human annotations on all samples of cach head-to- head comparison. For Control Variates Evaluation, we use the averaged mean-square error from the previous section. For Synthetic Evaluation, we average the synthetic annotations on all samples of a fixed LLM pair as the pre- dicted win rate and then calculate the mean square error. We also include the averaged mean-square error of human for convenience of comparison.
First, we measure the estimated mean square error of Human Evaluation and Control Variates Evaluation w.r.t number of human samples for each fixed LLM pair via bootstrapping That is. we repeatedly run the evaluation method 1000 times with a fixed number of human annotations, collect the out- put win rate estimates, and compute the mean-square error, where the ground truth win-rate the averaged human pref- erence on all data of that LLM pair. For Human and Control Variates Evaluation, we run bootstrapping using different numbers of human annotations on different LLM pairs and plot a curve respectively with labels "Human" and "Control Variates" (c.f. Figure 8), in which the y-axis is the averaged mean square error of the evaluation on different LLM pairs, and the --axis is the number of human annotations.
Figure 1 (right) presents the result of finetuned Skywork-88 and Figure 5 presents that of GPT-4, both on Chatbot Arena. Other results are deferred to Figure 7. Although GPT-4 is claimed to be an accurate evaluator (Zheng et al., 2023), still has a significantly high error compared to Control Variates and Human Evaluation. Similarly, even if finetune reward model like Skywork-8B (ft), it also suffers from high error if used in Synthetic Evaluation alone. However, these evaluators can be incorporated into Control Variates Evaluation to achieve much lower evaluation error.
Theoretically, the mean-square error can be decomposed into the square of evaluation bias and the variance. There- fore, the mean-square error curve still effectively reflects the variance reduction tendency as the number of human annotations increases, and when the number approaches infinity, we can extract the bias of the evaluation through the limit of mean square error.
8

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
Table 2: Averaged strong evaluator's sample saving in LLM- as-a-judge using control variates evaluation. evaluator GPT-4.
The strong
10
Weak Evaluator
Chatbot Arena
MT Bench
Averaged MSE
GRM 2B sftreg
22.8%
14.6%
Human Control Variates
10-)
Skywork 8B
13.5%
15.1%
Synthetic
ArmoRM 8B
16.0%
18.8%
Chatbot Arena and nearly 15% saving on MT Bench. This save the cost in LLM-as-a-judge.
10
200
1000
Number of Human Annotations
In addition, we provide the experiment result of Control Variates Evaluation for single response evaluation tasks in Appendix C.4.
Figure 5: Average mean square error versus number of human annotations for GPT-4 evaluator on Chatbot Arena (Zheng et al., 2023). Note that even GPT-4 has high bias if used alone for Synthetic Evaluation.
6. Conclusion
In this work, we propose Control Variates Evaluation to reduce human annotation costs while maintaining unbiased- ness. Our method demonstrates significant savings in human annotations across benchmarks like Chatbot Arena and MT Bench, aligning well with theoretical predictions. This pro- vides a scalable and cost-effective alternative full human evaluation without compromising reliabiliiyy
5.4. How does Finetuning Improve Control Variates Evaluation?
We visualize the averaged human annotation saving ratio before and after finetuning for GRM-2B and Skywork-8B on Chathot-Arena and MT-Bench in Figure 4. For all ex- periments, the finetuning procedure provides at least 5% more saving ratio. Specifically, for Skywork -8B on Chatbot Arena, the saving ratio nearly triples.
We only study the most canonical evaluation of head-to- head win rate between two LLMs, and it is an interesting future direction to explore more nuanced human evaluation metrics and complex evaluation settings, including multi- model ranking and fine-grained assessments. Other future work can focus on improving synthetic feedback through adaptive selection ensembling multiple evaluators.
On the other hand, finetuning indeed introduces additional computation requirement. For instance, the fine-tuning of Skywork-8B model requires four H100 GPUs with 80GB of GRAM. Regarding whether to finetune the evaluator or not, there are two major considerations. The first one the human annotation saving ratio on the pretrained evaluator. If is not satisfactory, finetuning can improve the human annotation saving ratio if finetune dataset is available. The other consideration is the number of future tasks, as this is trade-off between future savings in human annotation cost and the current additional cost of finetuning computation. If there are many future models to evaluate, then finetuning is beneficial because the savings generalize to unseen models.
A Acknowledgement
This work used DeltaAl at NCSA through allocation CIS240294: Advancing Large Language Model Alignment through Hybrid Feedback Integration from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program (Boerner al., 2023), which is supported by U.S. National Science Foundation grants #2 #2138259, #2138286, #2138307, #2137603, and #2138296.
5.5. Other Applications of Control Variates Evaluation
We also thank the ICML 2025 program chair as well as Ojash Neopan for directing us to Prediction Poweeeed Infer- ence (PPI), an important and relevant line of our work.
Control Variates Evaluation can be similarly applied in the LLM-as-a-judge setting. The difference that the human annotator is replaced with strong LLM evaluator, and smaller, cheaper model plays the role of the synthetic evaluator, to save the cost of querying the expensive model.
Impact Statement
We set GPT-4 as the strong evaluator and test the averaged human annotation saving ratio in the scenario of LLM-as- -as- a-judge, as shown in Table A 2B reward model like GRM-2B can achieve over 20% saving of GPT-4 annotation
This work seeks to accelcrate LLM evaluation while pre- serving its unbiasedness. The societal and ethical impact aligns with that of most LLM evaluation research, which has been widely discussed.

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
References
Glasserman, P. Monte Carlo methods in financial engineer- ing. Springer, 2004.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Gu, J., Jiang, X., Shi, Z.., Tan, H., Zhai, X., Xu, C.,, Li, W., Shen, Y., Ma, S., Liu, H., et al. A survey llm-as-a- judge. arXiv preprint arXiv: I. 15594, 2024.
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., and Zmic, T. Prediction poweerd inference. Science, 382 (6671):669-674, 2023a.
Hesterberg, T. C. and Nelson, B. L. Control variates for probability and quantile estimation. Management Science, 44(9):1295-134) 1998.
Angelopoulos, A. N., Duchi, J. C., and Zmic, T. Ppi++: Efficient prediction-pon poweeed inference. arXiv preprint arXiv:2311.0145.3 2023b.
Kemna, A. G. Z. and Vorst, A. C. A pricing method for options based on average asset values. Journal of Banking & Finance, 14(1):113-129, 1990.
Lan, T., Zhang, W., Xu, C., Huang, H., Lin, D., Chen, K., and Mao, X.-I. Criticeval: Evaluating large language model critic. arXiv preprint arXiv:2402. 13764, 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204. 05862, 2022.
Lavenberg, S. and Welch, P. perspective on the use of control variables to increase the efficiency of monte carlo simulations. Management Science, 27(3):322-335, 1981.
Boerner, T. J., Deems, S., Furlani, T. R., Knuth, S. L., and Towns, J. Access: Advancing innovation: Nsf's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Re- search Computing 2023: Computing for the Common Good, pp. 173-176. 2023.
Li, D., Jiang, B., Huang, L.,, Beigi, A., Zhao, C., Tan, Z., Bhattacharjee, A., Jiang, Y., Chen, C., Wu, T., et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.1654 2024a.
Boyeau, P., Angelopoulos, A. N., Yosef, N., Malik, J., and Jordan, M. I. Autoeval done right: Using synthetic data for model evaluation. arXiv preprint arXiv:2403.0740 00000000000 2024.
Li, Y., Ma, J., Ballesteros, M., Benajiba, Y., and Horwood, G. Active evaluation acquisition for efficient IIm bench- marking. arXiv preprint arXiv:2410.05952, 2024b.
Lin, C.-Y. Rouge: package for automatic evaluation of summaries. In Text summarization branches out, pp.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
74 2004.
Liu, C. Y., Zeng, L.,, Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y. Skywork-reward: Bag of tricks for reward modeling in 1lms. arXiv preprint arXiv:2410. 18451, 2024.
Broadie, M. and Glasserman, Risk management and analysis. vol. 1, 1998.
Chaganty, A. T., Mussman, S., and Liang, P. The price of de- biasing automatic metrics in natural language evaluation. arXiv preprint arXiv: arXiv:1807.022 1802.02202 2018
Müller, T., Rousselle, F,, Keller, A., and Novák, J. Neural control variates. ACM Transactions on Graphics (TOG), 39(6):1-19, 2020.
Chiang, W.-L., Zheng, L.,, Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonza- lez, J. E., et al. Chatbot arena: An open platform for evaluating Ilms by human preference. arXiv preprint arXiv:2403.041322 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, B. Length-comtrolled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.044/ 2024.
Owen, A. B. Monte Carlo theory, methods and examples. https: httt:///attteee aatt/arttteeeeeee/eeee su doma i ns hss/m///// mc/ ,2013.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Geffner, T. and Domke, J. Using large ensembles of control variates for variational inference. Advances in Neural Information Processing Systems, 31, 2018.
10

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
Perlitz, Y.. Bandel, E., Gera, A., Arviv, O., Ein-Dor, L., Shnarch, E., Slonim, N., Shhueli-Scheuerr M., and Choshen, L. Efficient benchmarking (of language mod- els). arXiv preprint arXiv:2308. 1696, 2023.
2020. URL id=SkeHuCVEDr.
wetiiiewwweeeeeeeeeeee net/morumt forum?
Zheng. L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances Neural Information Processing Systems, 36: 463955-6623, 2023.
Phan, D., Hoffman, M. D., Dohan, D., Douglas, S., Le, T. A., Parisi, A., Sountsov, P., Sutton, C., Vikram, S., and A Saurous, R. Training chain-of-though via latent- variable inference. Advances in Neural Information Pro- cessing Systems, (6:72819-72885 2023.
Zheng, X., Pang, T., Du, C., Liu, Q. Jiang, J., and Lin, M. Cheating automatic Ilm benchmarks: Null models achieve high win rates. arXiv preprini arXiv:4410.07137, 2024.
Polo, F. M., Weber, L.., Choshen, L.., Sun, Y., Xu, G., and Yurochkin, M. tinybenchmarks: evaluating Ilms with fewer examples. arXiv preprint arXiv:2402.14992, 2024a.
Zhou, J. P., Belardi, C. K., Wu, R., Zhang, T., Gomes, P., Sun, W., and Weinberger, K. Q. On speeding up language model evaluation. arXiv preprint arXiv:2407.0611 2024.
Polo, M., Xu, R., Weber, L., Silva, M., Bhard- waj, O., Choshen, L., de Oliveira, A. F. M., Sun, Y., and Yurochkin, M. Efficient multi-prompt eval- uation of LLMs. In The Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2024b. URL https: htttp:://ppereeviiiee net forum? j20p
Rousselle, F., Jarosz, W., and J.vák, Image-space control variates for rendering. ACM Transactions on Graphics (TOG), 35 2016.
Verma, A. and Hanawal, M. K. Stochastic multi-armed bandits with control variates. Advances in Neural Infor- mation Processing Systems, 34:27592-27603, 2021.
Wang, H., Xiong, W. Xie, T., Zhao, H., and Zhang, T. Inter- pretable preferences via multi-objective reward modeling and mixture-of-experts In EMNLP, 2024.
Yang, R., Ding, R., Lin, Y., Zhang, H., and Zhang. T. Reg- ularizing hidden states enables learning generalizable reward model for llms, arXiv preprint arXiv:2406. 10216, 2024.
Ye, Q., Fu, H., Ren, X., and Jia, R. How predictable are large language model capabilities? case study on BIG-bench. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, December 2023.
Yuan, H. Liu, Y., Wu, S., Zhou, X., and Gu, Q. Mars: Unleashing the power of variance reduction for training large models. arXiv preprint arXiv:2411.10432 2024.
Yuan, W., Neubig, G., and Liu, Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263-27277, 2021.
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. International Conference on Learning Representations,

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
A. Proof of Proposition 4.1
Note that expectations, variances, covariances and correlation coefficients this section are taken under the distribution Uniform(XY) y' ( 2), x² (-)
Proof of unbiasedness
We have
E B un [20v.]]
Ex,y' y/2 [z a (8 [sa)
(Bz, y² x²(3] 1t
En,y' 1/12]
= p (et > -1)
Proof of variance reduction
We have
Var, Varz.yt [2 Varz,g y2 [2 a (8 -2]]
Varz.y y² [2] - 2aCov x. .22 [z, (8 - M2)l + a² V 22 [el
Varz,gt. 2 [2] - 2aCov x.y² :0a 12, 2] + a² Varz.g+ v y [3]
Varz,y w² [&] (a Covz,y' .ya [2, 2]
(Cov [2, 2
+ Varayy u [z]
Varz, y¹ .v² [8]
Varr,y' .2² [2]
x.ov..'' .²² [z, 2]
Varz,gt y [z]
Varz,g y [8]
The equality holds if and only a = Cov pol.
w To further simplify the formula, recall that
(Cov .y2 [z, 2
p² (Corta,'' 22, Varz,s' [=]=] Var y [
Therefore we have
Var ay y 120 2 Var 2e [2] (Cov x. .ya [z, 8
Var .y2 [2]
Var..''.' y² [2] -
p0= parr w [z]
(1 Varr [z].
Cov Cow u.
The optimality point is a*
Var .e
Proof' of human annotation saving
Since all samples are i.i.d., we have
m
1
Var s.y' . [2220a
Var
Ziz
m
m
j=1
1 (1 0² Var 1. [2]
Varz.g [x]
-Va
2
12

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
B. Experiment Details
B.1. Hyperparameters
The Control Variates Evaluation Algorithm I has no hyperparameters except for the optional finctuning procedure. When finetuning Skywork-8B and GRM-2B on Chatbot Arena and MT Bench, we use global batch size 32 and train for 1 epoch. The finetuning of GRM-2B on Chatbot Arena uses learning rate 1e-6, others all use learning rate 3e-6.
To determine the optimal hyperparameters for finetuning, we conduct systematic search over range of learning rates and batch sizes. For instance, when we finetune Skywork-8B on Chatbot Arena, we follow these steps:
(1) We sort the LLM models Chatbot Arena in alphabetical order and select the first model, RMKV-4-Raven-14D as the holdout model split train and dataset.
We tested learning rates in (1 x 1x1007,3xx 1x For each hyperparameter combination, finetune for one epoch and record the final test accuracy.
10 1,100=,3x000
1x10-3,3x1000 10- and batch sizes in (32, 64, 128). 1x10--
(3) The combination yielding the highest final test accuracy is selected as the optimal hyperparameter setting, We use the chosen hyperparameter setting to finetune Skywork-8B on all other holdout models.
The similar procedure applies when we finetune other synthetic evaluators on other benchmarks.
B.2. Hardware
The experiments are run on H100 GPUs. Finetuning Skywork-8B requires 4 GPUs. Finetuning GRM-2B as well as the collection of synthetic annotations can all be done on GPU.
B.3. Prompt Template
We use the GPT-4 annotations for MT-Bench from the Hugging Face repository https: httpss://huggingfaccee co/ httastt///myyyy/ttbbbeeeo.uuuaat htwp://eeeeeueeeeeeteeeeee/eeeee htw.weeewwwwwww////eeeeeeeeee/ee Bapplltggpt__pai__
We follow the prompt template in (Zheng al., 2023, Figure 5, Appendix A) to get GPT-4 annotations in Chatbot Arena. We provide the template in Figure 6 for completeness.
C. Additional Experiment Results
C.1. Bias of Synthetic Evaluation
As described in Section 5.3, we measure the averaged mean square error of Human Evaluation, Synthetic Evaluation and Control Variates Evaluation on different evaluators and datasets, as shown in Figure The Synthetic Evaluation has significantly high bias, while the error of both Human Evaluation and Control Variates Evaluation converge zero.
C.2. Human Annotation Saving Ratio Matches Variance Reduction in Practice
As described in Section 5.2, we measure the averaged mean square error versus number of samples for different evaluators on different datasets. The z-coordinate of curves "Human" and "Control Variates" correspond to the number of human annotations (Zheng al., 2023). The curve "Control Variates (shifted)" is derived by horizontally scaling the Control Variates curve by 1/(1 s), in which s is the averaged human annotation saving ratio in Table 1. The human annotation saving ratio aligns perfectly with the actual variance relationship between Human Evaluation and Control Variates Evaluation.
C.3. Human Annotation Saving Ratio on Each LLM pair
We visualize the human annotation ratio (in percentage) each LLM pair that we use to compute the averaged human annotation saving ratio in Table 1. The results are shown in Figures 9 and 10. For pretrained evaluator, each entry of the matrix the human annotation saving ratio (in percentage) on that LLM pair. For a finetuned evaluator, each entry of the matrix the human annotation saving ratio (in percentage) on the corresponding LLM pair, in which the LLM on the row the left-out LLM, while the LIM on the column used in finetuning. Please refer to Section 5.1 for the details
13

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
Prompt for GPT-4 Annotations in Chatbot Arena
[System]
Please act as an impartial judge and evaluate the quality of the responses provi ded by two Al assistants to the user question displayed below. You should choose the assistant that follows the user's s instructions and answers the user's question better. Your evaluation should consider factors such the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your valuation by comparing the two responses and provide a short explanation. Avoid any position blases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence objective
favor certain names the assistants. possible, After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[3]]" if assistant B is better, and "[[C]]" for a tie.
your evaluation.
[User Question] (question)
[The Start of Assistant A/s Answer]
[answer_a
[The End Assistant A's Answer]
[The Start of Assistant B's Answer]
(answer_b)
[The of Assistant B's Answer]
Figure 6: Prompt template from (Zheng et al., 2023, Figure 5, Appendix A), which is used to get GPT-4 annotations in Chatbot Arena.
of finetuning procedure. Therefore, the matrices for pretrained evaluators are symmetric, while they are asymmetric for finetuned evaluators. The diagonal entries are white and do not have values becuase measuring human annotation saving ratio on identical L.LSs is meaningless.
Note that there some additional white entries with no values when testing GPT-4 the synthetic evaluator. This is because GPT-4 cannot always follow the prompt template, that sometimes cannot extract valid preference out of the output. In case that there are too few samples in an LLM pair, it is likely that we cannot compute a valid human annotation saving ratio.
C.4. Single Response Evaluation
Besides head-to-hed head comparison, Control Variates Evaluation also applies smoothly to other evaluation scenarios, e.g., single response evaluation, where a human gives scores to single LLM generation, instead of giving preference to two LLM generations.
We utilize the validation split of the HelpSteer2 dataset our benchmark. This split consists of 1.04K samples, each containing a prompt, response, and five human-annotated attributes: helpfulness, correctness, coherence, complexity, and verbosity. Each attribute is scored from 0 to 4, with higher scores indicating better performance. Our focus is on the helpfulness attribute, as it is the primary metric that reward models are typically trained to evaluate. We employ the Control Variates Evaluation method to predict the average helpfulness score. The human annotation saving ratio is shown in the Table 3.
14

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
152
Averagen D51
lammnn
Hamer
Taman
larenn
Ava
30-1
Humber Human
Numberd
Humber Hammer
(a) Chatbol Arena, ArmoRM-SB
(b) Chatbot Arena, GRM-2B
(c) Chatbot Arena, Skywork-8B
(d) Chatbot Arena, GPT-4
16.7
Synerasi
Syrebers
Syreres
Sydtrasc
200
540
5000
200
200
620
1000
9X0
L000
200
300
620
5000
(e) MT Bench, ArmoRM-8B
(I) MT Bench, GRM-2B
(g) MT Bench, Skywork-8B
(h) MT Bench, GPT-4
Ha man
MEC
10-1
Carti VNVANN Syotharz
Syrthart
10"
300
of wo Arw.dattiosss
of 14 Arevdin lans
of HaTer Anrentations
Chatbot Arena, GRM 2B (ft)
(i) Bench, GRM-2B (ft)
(k) Chatbot Arena, Skywork-88 (I) MT Bench, Skywork-8B (ft) (ft)
Figure 7: Averaged mean square error of Human Evaluation, Synthetic Evaluation and Control Variates Evaluation on different evaluators and datasets. The Synthetic Evaluation has significantly high bias, while the error of both Human Evaluation and Control Variates Evaluation converge to zero.
Table 3: Averaged human annotation saving ratio across different synthetic evaluators Helpsteer2.
Human Annotation Saving Ratio
GRM-2B
10.3%
Skywork-8B
21.0%
ArmoRM-8B
14.1% 1%
GPT-4o
27.4%
15

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
Corovi Veruza suman (nlited)
Caroel
Contvll
Corpul Numma (aitte)
Purtnn biihhed
Human Linfted)
Age roper 102
Humber Haman Amadatita
Number Hummmn Amnotar am
Number Hummnn Annoaat Dis
Humber Hamen Armiatiira
(a) Chatbot Arena, GRM-2B
(b) Chatbot Arena, ArmoRM-SB (c) Chatbot Arena, Skywork-8B
(d) Chatbot Arena, GPT-4
Cartoll Voriow (ottte)
Courcll virites
Centrll
Centrol Veriow
suman
(imman (02tted)
Iurran (osfed)
Genrgger
233
400
300
1000
26C
900
66c
1000
230
400
800
1000
403
1000
(e) MT Bench, ArmoRM-8B
(I) MT Bench, GRM-2B
(g) MT Bench, Skywork-8B
(h) MT Bench, GPT-4
Itel
Hunsa Carboll Varaom
Hin Carrel viniates
Hurson
Harna Verinn
Centrl
200
433
800
500
200
430
500
1000
230
420
830 wtaaarrr Aww.aaraa...aaaaaooooo
200
1000
200
423 Ilanat.Ammtaaiern
.00
5000
Chatbot Arena, GRM-2B 2B (ft)
(i) MT Bench, GRM-2B (ft)
(k) Chatbot Arena, Skywork-8B (1) MT Bench, Skywork-8B (ft) (ft)
Figure 8: Averaged mean square error versus number of samples for different evaluators on different datasets. The x-
coordinate of curves "Human" and "Control Variates" correspond the number of human annotations (Zheng et al., 2023). The curve "Control Variates (shifted)" is derived by horizontally scaling the Control Variates curve by 1/(1- -s,, in which is the averaged human annotation saving ratio in Table 1. The human annotation saving ratio aligns perfectly with the actual variance relationship between Human Evaluation and Control Variates Evaluation.
16

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
H
20
70 10
2
22
1
Lit
3
23 11
15
1114
34
130
1
30
16
14
57
11
the
de
(a) Chatbot Arena, GRM-2B 2B
(b) Chatbot Arena, GRM-2B (ft)
RAVC Gaveen
apace
agace
daudo
1100
20 27
ED
2
(c) Chatbot Arena, Skywork-8B
(d) Chathot Arena, Skywork-8B (ft)
110044//eenn 140
11A 4..44 148-
0003
20.07
14
masps
PH
14
12 1
P1
coasovi
12
RED
17
15
10
2014
P
10
20.25.00
R
2244 Albeeeee
2035000
2 K
D
P
14 35
130
22
6
37
TO DO BA
m
22
14
12
13
37
VOUE
gesten
(e) Chatbot Arena, ArmoRM-SB
(l) Chatbot Arena, GPT-4
Figure 9: Human annotation saving ratio (in percentage) on each LLM pair for different evaluators on Chatbot Arena. Diagonal entries are white and do not have values because it is meaningless to compute the human annotation saving ratio on two identical LLMss. Non-diagonal white entries in (f) imply an invalid result, because sometimes valid preference cannot be extracted from GPT-4's response.
17

---

Accelerating Unbiased LLM Evaluation via Synthetic Feedback
apace-130
15
37
alpaca
20
12.5
125
gpt-3 3.5turbo
3-5tbbb
10.0
10.0
gn--4
gPt-4
7.5
7.5
lama-130
5.0
famm-13 1.b
17
12
12
16
5.0
2.5
2.5
wcune-13b--1.22
alpacc- 13b
gen Aurbo
Barne 17b
0.0
elpacaa 13b
clasdeev
get Saibbo
lamna 13b
vicuna Libok
0.0
vicuna-
(a) MT Bench, GRM-2B 2B
(b) MT Bench, GRM-2B (ft)
20.0
20.0
atpace 13b
17.5
175
cluudev]
10
11
15.5
claudee-11
13
5
11
15.0
12.5
get-3 S-turo
10
9pt-3.5-turbe S-tubb
gpt-3
25
14
15
20
10.0
10.0
Iame-1 So-
11
Iamm-118
24
12
10
16
vicune-13b-il
14
18
21
11
eipacca- 33b
cleutte-p 2
S-tutbo
vicunte vicums-1ibe122 13be1
alpasca 3bb
clasdin #
get 5 5-turo
Bamns 130
vicuna Libel.2
(c) MT Bench, Skywork-8B
(d) MT Bench, Skywork-8B (f)
alacc
alpoca
daudov1
claude
3.5 turbo
144
11
opt 3.5 turbe
13
10
axt-4
301-4
5.0
5.0
2.5
v1.2
vicunc: vb.11.
spaca1
Cludle-2
met turbo
Lune-1 Vib
vicua 3b
5.0
dipaca
dauddevk
msta Aurbes
got-4
siana 1ook
13b
0.0
(c) MT Bench, ArmoRM-88 M-B8
(f) MT Bench, GPT-4
Figure 10: Human annotation saving ratio (in percentage) on each LLM pair for different evaluators on MT Bench. Diagonal entries are white and do not have values because it meaningless to compute the human annotation saving ratio on two identical LLMs.
18


Reviews:
[{"review_id": "Srw41z7bzi", "reviewer": "Reviewer_1bee", "summary": "The paper proposes Control Variates Evaluation, a method to reduce human annotation costs in evaluating large language models (LLMs) while maintaining unbiased results. By combining synthetic feedback from LLMs, the method achieves variance reduction in win rate estimation. The approach is theoretically grounded in the control variates technique and has been validated on benchmarks like Chatbot Arena and MT Bench, demonstrating human annotation savings. This general and scalable method offers a cost-effective alternative to full human evaluation without compromising reliability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and constructive feedback. We address your comments and questions below.\n> **Theoretical Claim: The general application of Control Variates does not require the sampled $x$ to follow a uniform distribution.**\n\nOur method can be applied directly to the setting where $X$ follows a non-uniform distribution. We assume the prompt $x$ to be sampled uniformly from $X$ because that is what people usually do when evaluating LLMs in practice. We will add a clarification in the camera-ready version.\n\n> **Essential References Not Discussed: [R1] Training Chain-of-Thought via Latent-Variable Inference**  \n\nWe will include this paper in Section 2.3 in the camera-ready version.\n\n> **Weaknesses: Focus primarily on head-to-head win rate estimation**\n\nThere are currently no public datasets available for testing multi-model ranking, and collecting such a dataset might require a significant amount of human effort which might be beyond the scope of our paper. However, we believe that this will be an interesting future effort and currently our theory directly applies to the multi-model ranking setting.\n\nThat said, we conduct an additional experiment in the single response evaluation setting, where a human gives scores to a single LLM generation, instead of giving preference to two LLM generations.\n\nWe utilize the validation split of the HelpSteer2 dataset as our benchmark. This split consists of 1.04K samples, each containing a prompt, a response, and five human-annotated attributes: helpfulness, correctness, coherence, complexity, and verbosity. Each attribute is scored from 0 to 4, with higher scores indicating better performance. Our focus is on the helpfulness attribute, as it is the primary metric that reward models are typically trained to evaluate. We employ the Control Variates Evaluation method to predict the average helpfulness score.\n\nThe human annotation saving ratio is shown in the table below:\n\n|     Model                         | GRM-2B | Skywork-8B | ArmoRM-8B | GPT-4o |\n|------------------------------|--------|------------|-----------|--------|\n| Saving | 10.3%  | 21.0%      | 14.1%     | 27.4%  |\n\nThe result above indicates the perspective of Control Variates Evaluation in single-response evaluation. To our best knowledge, this is the only public dataset with high-quality human annotation for single-response evaluation. We will include this experiment in the camera-ready version.\n\n> **Typos**  \n\nThanks for pointing them out. We will fix them in the final version.\n\n> **Question 1: How to ensure that the responses in the evaluation dataset are out-of-distribution with respect to the fine-tune dataset?**  \n\n\nAs we discussed in Section 5.1, the evaluation dataset is out-of-distribution in the sense that the evaluated model's response is excluded from the training dataset for fine-tuning. In the final version, we will reference Section 5.1 in the paragraph \"(Optional) Synthetic Evaluator Fine-Tuning\" on Page 5 for added clarity.\n\n> **Question 2: Generalize Control Variates to other evaluations beyond head-to-head win rates**  \n\nThis is addressed in the \"Weaknesses\" section above.", "claims_and_evidence": "The claims made in the paper are well-supported by clear and convincing evidence:\n\n- The paper shows the method is unbiased and achieves variance reduction. Experimental results demonstrate that Control Variates Evaluation matches human evaluation accuracy with fewer annotations, and the mean square error converges to near zero, indicating negligible bias.\n\n- The paper establishs the properties of the control variates method and its application to LLM evaluation.\n\n- The paper presents human annotation saving ratios across different evaluators and benchmarks, showing consistent savings. The experimental results validate that the theoretical saving ratios match practical variance reduction, with figures demonstrating the effectiveness across various evaluators and benchmarks.", "methods_and_evaluation_criteria": "The method is theoretically grounded in the control variates technique from statistics, which is a well-established variance reduction method in Monte Carlo sampling. Thus combining human annotations with synthetic feedback in a principled way makes sense for reducing annotation costs while preserving unbiasedness", "theoretical_claims": "Yes, I have thoroughly reviewed the theoretical proofs. The general proofs are correct, as they directly follow from the principles of Control Variates. However, I have one question:\n\n- The general application of Control Variates does not require the sampled $x$ to follow a uniform distribution. Unless, in your design, the uniform distribution is an inherent assumption in the evaluation objective (win rate)—specifically, to estimate the global win rate equally weighted across all prompts, rather than reflecting preferences for a specific distribution. I would appreciate it if the authors could clarify this point.", "experimental_designs_or_analyses": "I have reviewed the experimental sections. The overall experimental designs are well-structured and comprehensive. The experiments are well-aligned with the theoretical results.", "relation_to_broader_scientific_literature": "I believe Control Variates can serve as a valuable technique for LLM evaluation in the future. Although Control Variates is not a new concept, to my knowledge, this is the first time it has been introduced into the field of synthetic LLM evaluation.", "essential_references_not_discussed": "Given the generality of Control Variates, I believe the paper is self-explanatory and does not require further elaboration from other sources. However, it might be beneficial to cite and discuss additional papers that utilize Control Variates for different purposes, such as [R1].\n\n[R1] Training Chain-of-Thought via Latent-Variable Inference", "other_strengths_and_weaknesses": "Strengths:\n- The paper presents a novel application of the control variates technique from statistics to the problem of LLM evaluation. This creative combination of an established statistical method with modern LLM evaluation challenges addresses a gap in the field. The introduction of the human annotation saving ratio as a metric provides a clear way to quantify the method's effectiveness.\n\n- The extensive experimental validation across multiple evaluators and benchmarks effectively demonstrates the robustness and generalizability of the method. The inclusion of both pretrained and fine-tuned synthetic evaluators further enhances the practical relevance of the findings. Particularly, I was surprised to see in Figure 6 how closely the shifted Control Variates aligned with human error.\n\n- The paper is well-structured and clearly written. The figures and tables effectively illustrate the method and results, and the appendix provides additional details for reproducibility.\n\nWeaknesses:\n\n- The method and the experiments focus primarily on head-to-head win rate estimation. The paper does not explore other evaluation metrics or more complex scenarios like multi-model ranking.", "other_comments_or_suggestions": "Typos:\n\n- Line 186: It should be $u_{\\hat{z}}$ instead of $\\hat{u}_{z}$.\n\n- Line 601: It should be $\\frac{1}{n}$ instead of $\\frac{1}{n}(1-\\rho^{2})$", "questions_for_authors": "- Regarding the fine-tuned section, could the authors elaborate on how they ensured that the responses in the evaluation dataset are out-of-distribution with respect to the fine-tune dataset? This is crucial because if the datasets share a similar distribution, the comparison would no longer be fair.\n- Can authors discuss how to generalize Control Variates to other evaluations beyond head-to-head win rates.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZHIQqqcbNU", "reviewer": "Reviewer_yoN8", "summary": "-Paper proposes Control variates evaluation --- the goal being to reduce the cost of LLM evaluations\n\n-It does so using a principled statistical approach that combines human annotations with synthetic feedback (i.e. LLM as a judge). \n\n-Specifically, the synthetic feedback is the control variate to reduce the variance of limited human evals.\n\n-Paper shows the generic & fine-tuned approach reduces the number of human annotations.\n\n---\nUpdate after rebuttal \n\nThank you for the detailed responses.\n\nAdding these to the revised paper would strengthen the paper and ensure clarity.\n\nThat said, I retain my score and my positive assessment of the paper - best of luck :)", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback. We address your        questions and comments below.\n> **Weakness 1 & Comment 2: Only head-to-head evaluation tasks**  \n\nPublic datasets for other evaluation tasks are limited, and collecting such data may require significant human effort, which is beyond our paper's scope. However, we believe that this will be an interesting future effort and currently our theory directly applies to many other tasks, such as single response evaluation, where human scores are given to individual LLM outputs rather than comparing two responses.\n\nTherefore, we conduct an experiment in this setting using validation split of HelpSteer2 as our benchmark. \nThe human annotation saving ratio is shown in the table below:\n\n|     Model                         | GRM-2B | Skywork-8B | ArmoRM-8B | GPT-4o |\n|------------------------------|--------|------------|-----------|--------|\n| Saving | 10.3%  | 21.0%      | 14.1%     | 27.4%  |\n\nTo our best knowledge, this is the only public dataset with high-quality human annotation for single-response evaluation. We will include this experiment in the camera-ready version.\n\n> **Weakness 2 & Question 3: No comparison with other variance reduction methods, e.g. active learning**  \n\nTo our best knowledge, Control Variates Evaluation is the first unbiased LLM evaluation method with variance reduction. There are indeed other methods to reduce human annotations, such as Active Evaluation Acquisition (AEA) [1]. However, AEA might introduce bias to the evaluation because choosing a subset of human annotation data causes distribution shift in the evaluation dataset. In addition, AEA requires training of the neural process, while finetuning is optional in our method.  \n\nWe can also combine active learning and control variates evaluation to further reduce human annotations in LLM evaluation. To be specific, we can first apply active learning to select a representative subset of prompts for evaluation, and then run control variates evaluation on this subset. The downsides of this combination are:  \n1) The evaluation will be biased because strategic sampling of responses causes distribution shift with respect to the original evaluation dataset.  \n2) Active learning–based approaches like AEA [1] require an additional training procedure, which relies on existing human annotations.\n\n[1] Li, Yang, et al. \"Active Evaluation Acquisition for Efficient LLM Benchmarking.\" arXiv preprint arXiv:2410.05952 (2024).\n\n> **Weakness 3 & Comment 1: Assessment of computational costs**  \n\nIn our experiment, fine-tuning reward models, such as the 7B model, can be performed locally using four H100 GPUs with 80GB of GRAM. The evaluation cost for GPT-4o is approximately $0.0035 per annotation. While the exact cost of human annotation is unknown, we believe it is significantly more expensive, by orders of magnitude. We will include this discussion in the final version.\n\n> **Question 1: Minimum number of human annotations needed for the control variate to be reliable**  \n\nTheoretically, the reliability of Control Variates Evaluation is independent of number of human evaluation since it is unbiased. That said, if we want to compare with purely synthetic evaluation in practice, then the minimum number of human annotations required depends on the point at which the variance of Control Variates Evaluation is lower than the square of the synthetic evaluation bias. Since this threshold is influenced by the intrinsic variance of human annotations on a given evaluation dataset, it must be determined empirically. However, as shown in Figures 4 and 5, Control Variates Evaluation with just 200 human annotations already achieves significantly lower error than synthetic evaluation across all experiments. This is a relatively small number compared to the scale of popular LLM benchmarks such as MT Bench and Chatbot Arena.\n\n> **Question 2: Interpretation of Figure 7 and 8**  \n\nThey show the human annotation saving ratio (please refer to Page 4, right column of Line 180) of different LLM pairs. Factors influencing this ratio include the architecture of the response generator LLMs, the datasets used for training the generators, and whether an LLM is distilled from another LLM. Exploring these factors in detail is a focus for our future research.\n\n> **Theoretical Claims: Assumption of the strong correlation between human and synthetic annotations**  \n\nWe would like to clarify that our method works as long as the correlation is non-zero. While a strong correlation is ideal, a weak correlation can still lead to significant human sample savings: we observe from experiments (Table 1) that weak reward models can still achieve satisfactory correlation and thus good human annotation saving in Control Variates Evaluation.\n\n> **Supplementary Material: More details on prompt templates**  \n\nWe will attach the specific prompt template for completeness in the final version.", "claims_and_evidence": "The claims made in the paper are generally well-supported by theoretical analysis and experiments.\n\n- Good theory in Sec 4 and formal proofs in the appendices.\n\n- The unbiasedness is shown empirically and theoretically\n\n- The main claim of human annotation savings is shown across a variety of settings.\n\n------------\n\n- Only doubt is whether the claims are generalizable beyond the head-head settings.", "methods_and_evaluation_criteria": "-Use of control variates is well motivated from the perspective of variance reduction\n\n-Good use of established benchmarks chatbot arena and MT-bench\n\n-Nice evaluation across different sizes of models\n\n-Q: Asked later --- are their other variance reduction approaches that could be baselined against. i.e. is control variates the best?", "theoretical_claims": "-I’m not a theory expert, but the proofs seem correct\n\n-One thing I noted is the assumption of the strong correlation between human and synthetic annotations being important. Perhaps some ablations or analysis where the relationship is weak would be useful", "experimental_designs_or_analyses": "As mentioned above experimental designs and analyses are sound:\n\n- Well structured, uses multiple synthetic evaluators, datasets and fine-tuning vs no fine-tuning\n\n- As mentioned before, and acknowledged by the authors it’s unclear how this generalizes to other eval setups.", "relation_to_broader_scientific_literature": "The paper situates itself well within several research areas: LLM as a judge, efficiency LLM evals and control variates.", "essential_references_not_discussed": "It would be useful for the paper to position a bit better against alternative variance reduction approaches. For instance, one could use active learning to decide which limited set needs human eval and which can remain as synthetic.", "other_strengths_and_weaknesses": "Strengths:\n\n-\tTackles an important problem with a general approach --- variety of evaluators\n\n-\tConvincing empirical results for an important problem\n\n-\tNice theoretical links\n\nWeaknesses:\n\n-\tOnly head-to-head evaluation tasks considered\n\n-\tNo comparison with other variance reduction methods\n\n-\tAssessment of computational costs", "other_comments_or_suggestions": "-It would be useful to add some computational costs, e.g. cost of fine-tuning vs human annotation costs incurred\n\n-Maybe adding an appendix fleshing out how the method could be extended beyond head-head comparison", "questions_for_authors": "-Is there a minimum number of human annotations needed for the control variate to be reliable?\n\n-Figs 7 and 8 show variance for the annotation saving ratio across different LLM pairs? Do you have insights into the characteristics of different LLM pairs and why certain ones are more amenable and have higher savings?\n\n-Is it possible that alternative variance reduction approaches like active learning could fit into this paradigm and how would they differ from the proposed approach", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rDiYCtAhlV", "reviewer": "Reviewer_j7BE", "summary": "This paper introduces Control Variates Evaluation, a novel method for unbiased and cost-efficient evaluation of large language models (LLMs) in head-to-head comparisons. The approach leverages synthetic feedback from LLMs, combined with human annotations, to reduce annotation costs while maintaining evaluation reliability. The authors demonstrate that this method reduces the number of required human annotations by up to 24.8% when synthetic feedback is fine-tuned. Theoretical guarantees for variance reduction are provided, and the method is empirically validated against benchmarks such as Chatbot Arena and MT Bench. Additionally, the paper introduces a human annotation saving ratio metric to predict the potential savings.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback. We address your comments below.\n> **Weakness 1: Beyond head-to-head comparisons**  \n\nOur theory directly applies to many other evaluation tasks, such as single response evaluation, where a human gives scores to a single LLM generation, instead of giving preference to two LLM generations. However, there are limited public datasets available for these tasks, and collecting such a dataset might require a significant amount of human effort which might be beyond the scope of our paper. Nonetheless, we believe that this will be an interesting future effort.\n\nThat said, we conduct an additional experiment in the single response evaluation setting. We utilize the validation split of the HelpSteer2 dataset as our benchmark. This split consists of 1.04K samples, each containing a prompt, a response, and five human-annotated attributes: helpfulness, correctness, coherence, complexity, and verbosity. Each attribute is scored from 0 to 4, with higher scores indicating better performance. Our focus is on the helpfulness attribute, as it is the primary metric that reward models are typically trained to evaluate. We employ the Control Variates Evaluation method to predict the average helpfulness score.\n\nThe human annotation saving ratio is shown in the table below:\n\n|     Model                         | GRM-2B | Skywork-8B | ArmoRM-8B | GPT-4o |\n|------------------------------|--------|------------|-----------|--------|\n| Saving | 10.3%  | 21.0%      | 14.1%     | 27.4%  |\n\nThe result above indicates the perspective of Control Variates Evaluation in single-response evaluation. To our best knowledge, this is the only public dataset with high-quality human annotation for single-response evaluation. We will include this experiment in the camera-ready version.\n\n> **Weakness 2: Theoretical details, bias analysis in Equation (2)**  \n\nThis is discussed in https://artowen.su.domains/mc/Ch-var-basic.pdf, Page 32. We will expand the clarification in Appendix in the final version for completeness.\n\n> **Weakness 3: Assume access to high-quality synthetic feedback**  \n\nOur method is effective as long as there is a non-zero correlation between human and synthetic evaluations. While high-quality synthetic feedback is ideal due to its typically strong correlation, evaluations from a small reward model—though highly biased regarding human preferences—can still yield satisfactory performance in Control Variates Evaluation, as shown in Table 1. Ultimately, we believe the correlation requirement will diminish as AI systems continue to progress.\n\n> **Relation To Broader Scientific Literature: related methods for reducing human annotation costs**  \n\nTo our best knowledge, Control Variates Evaluation is the first unbiased LLM evaluation method with variance reduction. There are indeed other methods to reduce human annotations, such as Active Evaluation Acquisition (AEA) [1]. However, AEA might introduce bias to the evaluation because choosing a subset of human annotation data causes distribution shift in the evaluation dataset. In addition, AEA requires training of the neural process, while finetuning is optional in our method.  \n\nFurthermore, we can combine active learning and control variates evaluation to further reduce human annotations in LLM evaluation. To be specific, we can first apply active learning to select a representative subset of prompts for evaluation, and then run control variates evaluation on this subset. The downsides of this combination are:  \n1) The evaluation will be biased because strategic sampling of responses causes distribution shift with respect to the original evaluation dataset.  \n2) Active learning–based approaches like AEA [1] require an additional training procedure, which relies on existing human annotations.\n\nWe will add this discussion in the final version of our paper.\n\n[1] Li, Yang, et al. \"Active Evaluation Acquisition for Efficient LLM Benchmarking.\" arXiv preprint arXiv:2410.05952 (2024).\n\n\n> **Essential References Not Discussed**  \n\nWe will include these papers in the camera-ready version.\n\n> **Typos**  \n\nWe will change it to \"improve the human annotation saving ratio.\"", "claims_and_evidence": "The claims in the paper are mostly supported by convincing evidence. The authors provide both theoretical analysis and extensive experimental results to validate the effectiveness of the proposed method. Key claims, such as the reduction in human annotations and the alignment between theoretical variance predictions and empirical results, are well-supported. However, some claims about the scalability and generalizability of the method to more complex evaluation tasks (e.g., beyond head-to-head comparisons) are less substantiated and could benefit from further exploration.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are well-suited to the problem at hand. The use of Chatbot Arena and MT Bench as benchmarks ensures the relevance and applicability of the results. The incorporation of synthetic feedback and the focus on reducing human annotation costs align with the goals of scalable and efficient LLM evaluation. However, the paper could expand on how the method might generalize to other evaluation setups, such as multi-model ranking or fine-grained assessments.", "theoretical_claims": "The theoretical claims, particularly those concerning variance reduction using control variates, appear sound. The proofs provided in Section 4.1 are logically structured, and the derivations seem correct at a high level. However, I did not verify all mathematical details rigorously, and some minor steps in the derivations (e.g., bias analysis in Equation 2) could benefit from additional clarification. While the claims are likely correct, their presentation could be more transparent for broader accessibility.", "experimental_designs_or_analyses": "The experimental design is comprehensive and addresses the key questions about the effectiveness of the proposed method. The use of multiple synthetic evaluators (e.g., GPT-4, Skywork-8B) and fine-tuning experiments adds robustness to the findings. The alignment between theoretical savings and empirical results is a strong point. However, the experiments primarily focus on head-to-head comparisons, and it would be valuable to test the method on more diverse evaluation tasks. Additionally, some results (e.g., the saving ratios in Table 1) could be better contextualized to highlight their practical implications.", "relation_to_broader_scientific_literature": "The paper is well-situated within the broader literature on LLM evaluation. It builds on prior work on synthetic feedback (e.g., LLM-as-a-judge) and variance reduction techniques (e.g., control variates in Monte Carlo sampling). \nMoreover, this paper also connect to some famous concepts like critique ability and reward models.\nThe connections to recent benchmarks like Chatbot Arena and MT Bench are appropriate and timely. However, the paper could benefit from a deeper discussion of related methods for reducing human annotation costs, such as active learning or adaptive sampling, to highlight its unique contributions.", "essential_references_not_discussed": "Some related works are not described, like the papers include critique ability concepts\n1. A Survey on LLM-as-a-Judge\n2. From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge\n3. CriticEval: Evaluating Large Language Models as Critic", "other_strengths_and_weaknesses": "Strengths:\n\nThe method is original and addresses a critical bottleneck in LLM evaluation—reducing human annotation costs.\nThe theoretical framework is solid, and the empirical results strongly support the claims.\nThe introduction of the human annotation saving ratio as a predictive metric is a useful and practical contribution.\nWeaknesses:\n\nThe focus is narrow, primarily on head-to-head comparisons, limiting the generalizability of the results.\nSome theoretical details, while likely correct, could be presented more clearly.\nThe paper assumes access to high-quality synthetic feedback, which may not always be feasible in practice.", "other_comments_or_suggestions": "Typos: In Section 5.5, the phrase \"introduce more significant savings\" could be rephrased for clarity.", "questions_for_authors": "No questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes Control Variates Evaluation, a method to reduce human annotation costs in evaluating large language models (LLMs) while maintaining unbiased results. The approach leverages synthetic feedback from LLMs, combined with human annotations, to reduce annotation costs while maintaining evaluation reliability. The method is empirically validated against benchmarks such as Chatbot Arena and MT Bench demonstrating that Control Variates Evaluation matches human evaluation accuracy with fewer annotations, and the mean square error converges to near zero, indicating negligible bias.

Strengths: 
Strong results: The paper presents human annotation saving ratios across different evaluators and benchmarks, showing consistent savings. The experimental results validate that the theoretical saving ratios match practical variance reduction, with figures demonstrating the effectiveness across various evaluators and benchmarks.
The method is theoretically grounded in the control variates technique from statistics, which is a well-established variance reduction method in Monte Carlo sampling. Thus combining human annotations with synthetic feedback in a principled way makes sense for reducing annotation costs while preserving unbiasedness.
The method is original and addresses a critical bottleneck in LLM evaluation—reducing human annotation costs.

Can you please discuss how much this methods differs to PPI++?
Anastasios N. Angelopoulos, John C Duchi, and Tijana Zrnic. PPI++: Efficient PredictionPowered Inference. arXiv preprint arXiv:2311.01453, 2023
Decision: Accept (poster)