Length-Controlled AlpacaEval: A Simple Debiasing of Automatic Evaluators
Authors: Yann Dubois
Percy Liang
Tatsunori Hashimoto
Venue: COLM (2024)
Keywords: evaluation,llm,bias
TL;DR: A simple regression-based method to mitigate the length bias of AlpacaEval.

Abstract: LLM-based auto-annotators have become a key component of the LLM development process due to their cost-effectiveness and scalability compared to human-based evaluation. 
However, these auto-annotators can introduce complex biases that are hard to remove. Even simple, known confounders such as preference for longer outputs remains in existing automated evaluation metrics.
We propose a simple regression analysis approach for controlling biases in auto-evaluations. 
As a real case study, we focus on reducing the length bias of AlpacaEval, a fast and affordable benchmark for instruction-following LLMs that uses LLMs to estimate response quality.
Despite being highly correlated with human preferences, AlpacaEval is known to favor models that generate longer outputs.
We introduce a length-controlled AlpacaEval that aims to answer the counterfactual question: "What would the preference be if the model's and baseline's output had the same length?" 
To achieve this, we first fit a GLM to predict the biased output of interest (auto-annotator preferences) based on the mediators we want to control for (length difference) and other relevant features. 
We then obtain length-controlled preferences by predicting preferences while conditioning the GLM with a zero difference in lengths. 
Length-controlling not only improves the robustness of the metric to manipulations in model verbosity, we also find that it increases the Spearman correlation with LMSYS' Chatbot Arena from 0.94 to 0.98. 
We release \thecode{} and \leaderboard{}.

Published as a conference paper at COLM 2024
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
Yann Dubois & Percy Liang & Tatsunori Hashimoto Stanford University (yanndubs, pliang, thashim)estanford.edt
Abstract
LLM-based auto-annotators have become a key component of the LLM de- velopment process due to their cost-effectiveness and scalability compared human-based evaluation. However, these auto-annotators can introduce complex biases that are hard to remove. Even simple, known confounders such as preference for longer outputs remain in existing automated eval- uation metrics. We propose a simple regression analysis approach for controlling biases in auto-evaluations. As real case study, we focus on reducing the length bias of AlpacaEval, fast and affordable benchmark for chat LLMs that uses LLMs to estimate response quality. Despite being highly correlated with human preferences, AlpacaEval is known to favor models that generate longer outputs. We introduce a length-controlne pacaEval that aims answer the counterfactual question: "What would the preference be the model's and baseline's output had the same length?" To achieve this, we first fit generalized linear model to predict the biased out- put of interest (auto-annotator preferences) based on the mediators we want control for (length difference) and other relevant features. We then obtain length-controle preferences by predicting preferences while conditioning the GLM with zero difference in lengths. proves the robustness of the metric to manipulations in model verbosity, we also find that it increases the Spearman correlation with LMSYS' Chatbot Arena from 0.94 to 0.98. We release the code and resulting leaderboard
Al-
not only im-
Introduction
Chat Arena Spearman correlation
0.35
0.51
0.59
0.63
0.66
0.69
0.83
0.87
0.94
0.94
0.98
Output Length
TuthfulQAA
HellaSwag
GSM-BK
Open LLM
WinoGrande
ARC-C
MMLU
MT-bench
LC AlpacaEval 2.0
AlpacaEval
Figure Length-controlled AlpacaEval increases correlation with Chatbot Arena from 0.94 to 0.98. It is currently the benchmark with the highest correlation with Chatbot Arena.
Developing and improving NLP systems requires reliable, low-cost evaluations that can quantify progress. closed-ended tasks, such as multiple-choice QA, such evaluations are straightforward to implement and trust (Novikova et al., 2017 Yeh et al. 2021). However, such evaluations cannot be applied to extremely open-ended settings such as instruction following for language models. Even neural reference-based evaluation metrics such as BERTscore (Zhang" et al. 2020) face challenges in those settings due to the difficulty of collecting diverse set of references that can cover the space of valid outputs.

---

Published as a conference paper at COLM 2024
Recently, there has been push toward freerence-ffee evaluation methods that leverage pigh--peformance LLMs, e.g. AlpacaEval (Li et al. 2023), MTBencc (Zheng et al. 2023), and WildBench (Lin et al., 2024). While these approaches show high correlation with human annotators, they often do so by exploiting spurious correlations such as the length of the output, the presence of lists, various position biases Li et al. (2023); Zheng et al. (2023);
Koo et al. (003)); Wang et al. (2023) Wu & Aji 2023).
Creating a way to debias automated evaluation metrics would be highly valuable would address the major drawback of LLM-based reference-free evaluations, and enable low-cost, accurate evaluations for developing NLP systems in many open-ended settings. Our work focuses on this challenge of taking an existing automated evaluation metric (e.g. AlpacaEval) and suspected spurious correlate (e.g. length) and producing a debiased metric.
We propose simple, interpretable debiasing strategy for automated evaluation metrics based on basic, regression-basee basedd adjustments for observational causal inference. We view spurious correlates such the length of the response as undesirable mediators Vander- Weele 2015) in a causal graph and use regression-based causal inference Hernán & Robins (2010) techniques to provide simple adjustments to automated evaluations that control for any suspected spurious correlation.
Applying this approach to the popular AlpacaEval benchmark, we show that controlling for length has significant positive effects on automated evaluation. We find that it is more correlated average with LMSYS' Chatbot Arena (Zheng et al. 2023) than both (length- uncontrolled) AlpacaEval and MT-bench, and that it is significantly more robust to gaming the evaluation by increasing the verbosity of the output.
Our contributions are the following:
We propose a simple regression-based debiasing approach for automated evaluation that satisfies several desirable properties for an automatic evaluation metric.
We apply the approach to AlpacaEval, producing Avplc-Eval-LLC that is more robust to length-based spurious correlates.
We show that AlpacaEEvl--.C correlates better with the human evaluations of model rankings (Chatbot Arena) Fig. 1
Background and Problem Setting
Our work relates both the classic literature on reference-free evaluations, as well as more recent developments in automated and human evaluation of chatbots. We describe some of these relevant works below, with some additional exposition on the details of AlpacaEval, which we study more closely in our debiasing experiments.
Reference-free evaluation metrics Reference- free evaluation metrics have long history, including classic methods (Louis & Nenkova 2013) and more recent neural supervised learning methods (Kryscinski et 2000 et 2020 Goyal & Durrett 2020). While this latter class of algorithms has become sufficiently accurate that they match inter-annotator agreement rates, other works have shown that such measurements are heavily confounded by spurious correlations such as perplexity and length (Durmus et al.) 2022). Recently, there zero-shot, reference- evaluation measure (Zheng)
has been push to leverage LLMs
et al. 2023 Dubois et al. 2023 Li et al. 2023 Lin et al. 2024). In the chatbot setting, two well- such metrics AlpacaEval and MI-bench, both of which query an LM (based
upon GPT4) to attempt to assess the quality of a weaker LM's output.
AlpacaEval AlpacaEval is an LLM-based automated evaluation metric - it operates on fixed set of 805 instructions chosen to be representative of user interactions on the Alpaca web demo. For each instruction, both a baseline model b (currently GPT-4 turbo) and the evaluated model produce responses. A GPT-4 turbo-based evaluator then compares the responses head-to-heal and outputs the probability of preferring the evaluated model. A win rate is then computed the expected probability that the auto-evaluator prefers the
2

---

Published as a conference paper at COLM 2024
evaluated model's output the 805 instructions. This win rate serves as performance measure of the evaluated LM chatbot.
development metric for the Alpaca chatbot (Taori et al. 2023) and AlpacaFarm simulator (Dubois et al. 2023). The metric was designed to control for certain biases, such as the order in which the model and baseline were presented, by randomizing their sequence. However, other factors like length and style effects were not controlled for, as the authors found that humans had similar biases on the analyzed data. Subsequent use of AlpacaEval as a leaderboard revealed that these uncontrolled biases could be significantly gamed by AI systems ways that human biases couldn't.
Originally, AlpacaEval was designed
Important to our work is that AlpacaEval has several interpretable properties. As a win rate, its values are in (0%, 100%), it has symmetry baseline swaps, ie, AlpacaEval(b, m) 100% AlpacaEval m, b), i.e., and comparing baseline to itself is AlpacaEval(b, Any posthoc correction to AlpacaEval should maintain these properties, alongside the usual desiderata of being low-cost, accurate, and robust.
50%
Chat Arena The automated approach to pairwise evaluation in AlpacaEval can be viewed as a low-cost approximation to the Chatbot Arena Zheng et al. (2023), which aims to build real-world human evaluations through live interactions. The approach in Chatbot Arena that users are presented with pair of anonymized LMs, and they can send an instruction to both LMs simultaneously. The user receives responses from both LMs and rates the response that is of higher quality. At the end, the head-to-head comparisons are converted to an Elo rating Elo & Sloan (1978) which serves as the model score. As a reminder, the difference Flo ratings between two players can be converted a win rate, and vice versa.
This approach has many desirable properties - it is driven by real users and the dynamic nature of the instructions makes it hard saturate this benchmark. However, this metric can- not be used for model development due to the cost of rumning many live human evaluations.
In the remainder of this work, we will treat the Chatbot Arena as a silver standard that we wish to approximate. Although Chatbot Arena likely still contains biases (e.g. internet users may focus on surface features rather than "hard to measure" capabilities such as factuality), it represents the largest and most ecologically valid human evaluation process today.
Problem statement We define the problem of pairwise evaluation of a language model in the following way. Given an instruction x sampled from distribution p(x), a baseline model generates response and the evaluated model generates a response Zm. A human annotator then produces a preference y {m, b) indicating the model with the better re- sponse. An automated surrogate such as AlpacaEval is a (potentially randomized) predictor (Zz, Em, x) that aims to approximate the corresponding human label ply m m Zm,, x). AlpacaEval's current win rate is winrate(m. (m, b) = 100 [Ex Zm x)], xm,x)]
i.e., the expected pre- dicted preference of human annotators for the model response over the baseline's 's response.
3 Length-Controlled AlpacaEval
major challenge in building automated evaluations Zm, is the spurious correlations problem. Specifically, consider simple example in which there is spurious correlate è (e.g. length) such that heavily relying upon can be predictive of the human label y. The confounder is initially predictive of y but becomes less predictive model builders explicitly begin to optimize against the metric. Adopting this causal view, we ask our motivating question
What would the AlpacaEval metric be, if the outputs of all models had the same length as those of the baseline?
Our goal in this section will be to operationalize this into a simple regression-based estimator. To be precise, we hypothesize that automated evaluation measures f such AlpacaEval return their quality estimates through a combination of direct effects that measure the quality of the model response and indirect effects that are mediated by spurious variables such as
3

---

Published as a conference paper at COLM 2024
the length of outputs. The goal of controlling for the spurious correlates is thus equivalent to controlling these indirect effects. See Fig. for a visual representation.
Model
Other Mediator
Length of Output
Desired interaction
Instruction
Preference
Figure 2:
AlpacaEval predicts the direct effect of the model (green) on the auto-annotators preference (white) when controlling for undesirable mediators (red) and other useful features (gray).
This abstraction leads us to simple approach for bias correction, inspired by methods for estimating Controlled Direct Effect (VanderWeele) 2010), for which simple Generalized Linear Models (GLMs) can provide reasonable estimates.
Length control via regression Our approach will be to estimate the contribution of different components to the AlpacaBval quality judgment:
Model identity Whether an output comes from the baseline model b or the evaluated model m should impact the probability that output wins the pairwise comparison.
Length of output The length of output is known to affect both human and model judg- ments of output quality (Dubois et al., 2023 Singhal et al. 2023), and we expect this to also affect the win probability.
Instruction difficulty Models do not perform uniformly over instructions: the preference of humans will generally depend the instruction. For example, the baseline might be much better for coding tasks than any other tasks. For every instruction we thus want model the difficulty of that task for the baseline. Note that the (baseline) "instruction difficulty" is not caused by "model" but conditioning on it can enhance the precision of estimates in regression analysis by reducing unexplained variability Pearl (2009).
We can obtain length corrected AlpacaEval score in two steps: (i) first, we can fit model to these three attributes, and (ii) then we zero out the "length of output" term to obtain counterfactual estimates of AlpacaEval win rate.
The regression model Motivated by the previous discussion, we will model the AlpacaE- val predictions fZm, Xp, with logistic regression that has 3 terms: model, length, and instruction. We will first present the overall regression formula, explain the details of the featurization, and then describe some naturally appealing properties of our featurization.
(0 ml Zm, Zhy x) ==
len(zm) len (4m - 446
logistic
0m Ob + 4m,b tanh
+
stt((len(z)) (Zm) len eenz)))
Model
Length
Instruction
(1)
The model and instruction terms straightforward they can be viewed the lineai contribution of the model (m, b) and each instruction's difficulty (7) on the baseline win rate. The length term is linear in normalized length feature, where the normalizer standardizes the length to have unit variance and transforms this via a tanh, as differences in lengths should have strong diminishing returns the log odds.
Importantly, this formula fulfills the identity property, i.e., q(v = m Zb, Zp, x), = 0.5, and symmetry property, i.e, q(y 2mz 2b,
1.0 gly =
Zp, Zm, x) of the original
4

---

Published as a conference paper at COLM 2024
win rate. Identity holds the length term is zero due to having no difference in length, while the other two terms are zero as the coefficients are identical. For symmetry note that logistic(x) 1 - logistic(-at and it is clear that swapping m and b flips the sign of the model and instruction terms. For the length term, the same is true as flipping m and negates the length difference, and tanh is odd function. More generally any additive term that is antisymmetric and centered around O would satisfy the cesired properties.
Obtaining length corrected (LC) win rate Using the model from Eq. (1) we answer the counterfactual question of what the automatic evaluation f might be if the length of the evaluated model matched that of the base model, i.e., len(zm) = len(zz). (z)) In this case, the second, length term becomes zero and we obtain the length corrected win rate estimate as
winrate (m,,te b) 100 Ex [logistic(On 0b + (4m (Wm--b))mx]]
vpb))=)]
(2)
Le, we remove the length term from the regression and compute the implied win rate.
Training Training of the regression is simple and uses off-the-shelf libraries for fitting generalized linear models. Since our GLM uses a logit link function, we fit the model in [q using the cross-entropy loss L(0, 4, 4) E xt)p=//...x))(/.......
[ Zm, Z,, x)].
AlpacaEval's leaderboard uses a constant baseline so without loss of generality we can drop bb 4bb which can be absorbed into the corresponding parameters for m. In total, for a leaderboard with M models and N instructions, our GLM contains 3M + N parameters to be estimated from MN examples (fmm m,,,, Pm for each model, Yx for each instruction). This will be overdetermined when M and N are both large, as in the case of AlpacaEval (M > 128, N = 805). However, to ensure our procedure is robust even for small N and M, we use 5-fold cross-validation with L2 regularization on weights to avoid potential overfitting.
The one complexity of our regression is that the instruction difficulty term Yx is shared across models, and so we estimate this separately by first fitting joint regression across all models with the 4 t term fixed to one and using the estimated 7x from this regression.
For the remaining regression coefficients, we simply fit 8, p, and &p on the AlpacaEval predic- tions for each model separately, re-using the already estimated Yx these do not depend on the model being evaluated. Fitting models separately is important as it implies that previously computed metrics won't change when adding new model to the leaderboard.
Finally, we have added an additional weak regularization on Pm,b to prevent an adversary from performing attacks that intentionally truncate sequences that model performs poorly on. In this case, the poor performance of the model would be perfectly correlated with the short length, and the model builder would be able to exploit the length corrections the performance of the model. Adding regularization term makes it so that any model performance issues would be explained by the model terms first, and then any residual effects would be captured by the length effects, as intended. The regularization is weak enough that we empirically found it to not affect non-adversarial models.
boost
Results
We apply our approach to all of AlpacaEval, as this benchmark has known length con- founders, contains a large set of pre-computed LLM-based pairwise comparisons (> 120 models, 805 instructions), and is widely used by the research community. We evaluate our approach on several measures of interest:
Decreasing length gameability: We call a metric length gameable simply prompting the model to be more or less verbose significantly affects the metric outcome. Ideally, length gameability should be low for two reasons. First, we would like evaluations that prioritize the content rather than the style of the answer. Second, the benchmark should not be too dependent on the prompting strategy as users usually think of evaluations of models rather than the entire system, which includes the prompt.
5

---

Published as a conference paper at COLM 2024
Correlation with chatbot arena: If our gameability and robustness metrics represent better capturing human preference, we should improve our correlation with chatbot arena. We measure Spearman rather than Pearson correlation as probabilities are oog-linearly correlated with ELO ratings, rather than linearly.
Robustness and interpretability: Our corrected metric should be robust to simple adver- sarial attacks such as truncation, and be interpretable to users win rate.
We show that AlpacaEval-T.C fulfills all these goals. We release the code for all experiments.
AlpacaEval
Length-controlled AlpacaEval
concise
standard
verbose
concise
standard
verbose
9pt4_1106 1_
22.9
50.0
64.3
41.9
50.0
51.6
Bektral-8k7B-Instructta
13.7
18.3
24.6
23.0
23.7
23.2
gpt4_0613
15.8
23.2
21.6
30.2
33.8
claude-2.1
9.2
15.7
24.4
18.2
25.3
30.3
9pt-3.5-turbo-111
9.2
12.8
19.3
22.0
alpaca-7b
2.0
2.6
2.9
4.5
5.9
6.8
Figure 3; Length-controlled AlpacaEval decreases the sensitivity to prompting the evaluated model for more concise or verbose outputs.
4.1 AlpacaEval-LC decreases length gameability
A good evaluation metric should not be sensitive to length that prompting for longer shorter responses completely changes the metric. To measure gameability we prompted dif- ferent models "Answer with as much detail possible." (verbose) "Be concise as pos- sible while still providing all the necessary information to answer the question." " (concise).
Figure shows that AlpacaEval is highly length gameable. The baseline model poot4_1T@6_previees fluctuates from 22.9% to 64.3% by varying the verbosity instruction in the prompt. Even worse, significant gains are possible by asking weaker models to be verbase, seen with Claude-2.1.
In contrast, the length-controlled AlpacaEval has significantly lower gameability 1oot4_11@___reeiee'ss win rates now only fluctuate from 41.9% to 51.6%), and rankings are generally stable verbosity prompts. Quantitatively, the normalized standard deviation across the three verbosity prompts decreases from 25% to 10% from the length control.
4.2 AvaacLEval-LC increases correlation with Chatbot Arena to 0.98
Our prior experiments demonstrate that length control reduces the high sensitivity to length in Alaa However, our goal is not simply to make metrics that are less sensitive to length, but to produce metrics that are overall more representative of human judgments.
Figure 1 shows that controlling for length increased the Spearman correlation with Chat Arena from 0.94 to 0.98. Of existing benchmarks, this difference is significant enough to make the length-corrected version of AlpacaEval the metric with the highest correlation with Chat Arena which we aware of. Correlations that evaluates at least 25 models from the Chatbot Arena. AlpacaEvaa and AipacaFvvl--LC have 38 such models, MT bench has 34. The bootstrap p-value comparing the correlation with AlapcaEval's correlation is 0.07 and 0.06 compared to MT-bench.
computed every benchmark
Length control generally improves the rankings of proprietary models Figure 4shows leaderboard changes due to our length control approach. We see that proprietary models,

---

Published as a conference paper at COLM 2024
which often generate shorter responses, perform much better on AlpacaEval-LC, and the biggest rank losses are in open-source models that have gone through the RLHF process Ouyang al. (002). Given that AlpacaEval is potential optimization target for open- source language models, these results are consistent with the hypothesis that existing open models had exploited the length bias of AlpacaEval.
Langth
Win Rate
New Win Rate
Win Rate Gain
Rank Gain
gpl4 1106 praview
2049
50.0
50.0
0.0
slaude-3-apus--ee 20u4-20240029
1386
29.0
40.4
11.4
spl4
1966
25.6
88.1
14.8
Owent.8-728-Che.
1649
26.8
38.6
10.1
gpt4, 0514
22.1
18.2
Seaiud---soomee-2000e 20240222
mistral-largen Satrrral--rrg-2400
1962
Samba-Cote v0.2 best-ot-16
1578
27.0
31.5
gol4 0613
1140
15.8
90.2
14.4
20
Snorkel Mistrii-itaaa--- PairRM DPO bos-tost--
2516
34.9
30.0
-4.9
Contectuai-KTT M--Mitral--PirAMM
2521
33.2
29.7
-3.5
pairmm- pairmm-Y1-34B.. YI-34B-Chat
2196
31.2
28.8
-2.4
mistral-medium
21.8
Samba-CoE-v0.2
7D-instruct
oassi-sft-py phtti-----ttt
13b
1774
3.0
-0.5
-12
guanaco-76
1364
2.8
2.9
-0.0
baichhan-113--a 13b-chat
1727
2.0
2.1
0.1
Figure Closed source models often perform better (red) on length AlpacaEval as they are often shorter. The first column shows the length of outputs. The 4th and 5th columns, respectively, show the win rates and rank gains due to LC. The table shows the top 15 and bottom 5 systems in the leaderboard.
4.3 AlpacaEval-LCC is interpretable and robust
100
gpt4_1106 1ee_1116______ 10110_preeveewwee
60
62
65
67
68
claude- 3- o-opss-22u 2004-20220229
50
53
58
62
61
80
gpt4
38
47
50
53
54
55
59
58
61
Qwen1.5 1.een1.5.72e 5-7B-Chhat
52
56
Model
gpt4. 0314
42
46
48
50
51
54
54
Brude-3-sonnet-222...
35
41
45
47
49
50
53
53
mistral-large-2402
46
CoE-
32
39
42
44
46
47
50
50
53
9pt4_ 0613
36
39
43
48
47
Baseline
Figure 5: With our GLM predict what the win rate would be if the baseline any other model from the leaderboard. The x-axis shows the baseline, and the y-axis the model. Both are in the same order.
7

---

Published as a conference paper at COLM 2024
Regularization makes AlpacaEval-LC robust to truncation One potential issue with simple bias corrections is that they may be gamed through white-box adversarial attacks, e.g postprocessing the outputs of models to make them look better on AlpacaBval-LC. One example of such an attack is to truncate all outputs to a few characters, besides those that much better and around the same length as the baseline. A naive GLM fitted on such outputs should naturally predict very high win rates in the counterfactual world where outputs have the same length as the baseline. Indeed, when doing such post-processing to GPT-4 outputs, win rates increase from 3.7 (AlpacaEval 2.0) to 25.9 (AlpacaEval-LC, no regularization). To mitigate such adversarial attacks, our approach includes a regularization term on Pm,b- This decreases the gamed win rate to 12.2 (AlpacaEval-L.C with regularization) while having an imperceptible impact on non-adversarial models.
Interpretability win rate 5 shows that LC win rates can be interpreted similarly to raw win rates. In particular, the baseline always has a win rate of 50% and winrate(m, (m, b) 100% winrate(b, m) 10,, 100%]. This seems very natural but wouldn't hold for most length-correction methods, such as normalizing by length.
More interestingly, a nice property of GLM is that we fit the weights for one baseline, we can predict the win rate between any pair of models on the leaderboard. As result, we can predict the leaderboard for any other baseline as seen in Fig. 5
4.4 Comparisons baselines for length control
Arena Correlation (t)
Gameability (1)
Adversarial Win Rate Gain
(1)
Win Rate
0.94
26%
0.0
Length-Controlled Win Rate
0.98
10%
8.5
Montth-Normalized Win Rate
0.96
15%
3.6
Length-Ballnd BallnceBB Win Rate
0.95
15%
40.8
Figure 6: Length-controlled win rate has the best Arena Correlation and gameability from considered methods, while still being relatively robust to adversarial attacks.
Let's now briefly discuss two other potential family length-correction methods that have been proposed in the community Duong (2024) Galambosi (2024): Teortaxes (2024).
Length-balanced win rate Another common way control some covariates is through stratification. One potential metric, dubbed length-balanced (LB) win rate, would thus be to compute the average win rate stratified on examples where the model outputs are (1) longer and (2) shorter than the baseline Duong (2024). LB satisfies many of the desiderata of length control but has one main downside: robustness.
In particular, stratification relies upon having enough samples within each stratum, oth- erwise the estimates may rapidly become unstable. This can increase variance, e.g., if one model is naturally longer than another, but can also introduce adversarial vulnerabilities.
The first and last rows in Fig. 6) show that length-balanced win rates improve both the length gameability (measured by the normalized standard deviation of win rate across con- stae/ssaadadd /undd//eeee /verbose prompts) and the Chatbot arena correlation. However, this approach is strictly dominated by our length-controlled method - in arena correlation, gameability, and adversarial win rate gains from truncating bad GPT-4 outputs as discussed in Section 4.3
Length-normalized win rate Another option Galambosi 2024); Teortaxes 2024 is to
directly normalize the win rate by function of the length of the model's and baseline's output. We have tried several variations on normalization (e.g. directly dividing by lengths, logistic function of lengths, etc). In our experiments, the function that performed best dividing the raw win rate by a temperature-scaled logistic function of the average difference of lengths. We call this metric length-normalized (LNN) win rate.
8

---

Published as a conference paper at COLM 2024
Figure 6 shows that this simple LN win rate performs surprisingly well on many of the metrics. We chose to present and implement the length-controlled (LC) win rate, as it is more principled (as an estimate of the direct effect), interpretable (as win rate), and performs slightly better on all quantitative metrics except adversarial gameability.
5 Discussion
Auto-annotator
gpt4_1106 prrvveeweview
claude-3-ppus-2224022
mistral--arge-2402
@p14_1100_preview
50.0
50.0
50.0
dhude-3-opus-2224022
40.4
43.3
47.5
mistral-targe-2402
32.7
26.2
45.5
9pt4_0613
30.2
20.5
34.3
gpt-3.5-turbo-1106
19.3
16.7
28.9
Figure 7: Length-controlled win rate has the best Arena Correlation and gameability from considered methods, while still being relatively robust to adversarial attacks.
Other biases Length is well-known bias of automated evaluators of chatbot LLMs but several others have been noted, inchuding bias of models towards their own outputs ZZeng et al. (2023), presence of lists Dubois et al. (2023). While focus more detailed study of length biases here, we note that the same approaches can be applied to other biases by representing them as additional features in the logistic regression.
Additionally, our preliminary explorations of self-annotator biases shows that the effect exists but is often smaller than general model differences. [Fig. shows that the ranking of considered models does not change when using different annotators. In particular, Claude Opus prefers GPT4 Preview, and Mistral Large prefers the former two than itself.
Length-controlling in RLHF Our work is closely related to the recent work that aims to debias (implicit or explicit) reward models used to finetune LLMs with RLHF (Singhal et al. ).023)). For example Shen et al. (2023); Chen et al. (2024) try to train reward model that is uncorrelated to length by making it predict the length at the same time as the reward and disentangle the two. Park al. (2024) extends this intuition to the case of implicit reward models. This type of debiasing would not work out-of-the-boo in typical auto-evaluation settings, e-g. Alaa,aEval, which uses closed source LLM judges rather than training reward model. Our post-hoc debiasing could however be used in the RLHF setting, and we encourage future work look into that.
Limitations Firstly, only tested proposed debiasing mechanism the AlapcaEval benchmark, which uses a set of relatively simple English instructions and a particular prompt for the LLM judge. Secondly, AlpacaBval-LChh is based on the simplifying assumption that you would like to compare the model and the baseline as if they had the same length. Finally, do not (aim to) solve any other issues associated with the use of an LLM-judge Zheng et al. (2023). Despite these limitations, we show that the correlation with chat arena increases significantly, which suggests that AlpacaEval-LCC takes step in the right direction.
Conclusion We propose simple method for mitigating the length bias of LLM-based automatic evaluations, specifically, AlpacaEval. The procedure consists fitting a gener- alized linear model to predict the auto-evaluatons preferences, conditioned on the length of the models' output. We then get the length-controlled preference by predicting what the auto-evaluator would have preferred if the model's output and the baseline's output had the same length. We show that the resulting length-controlled AlpacaEval, has higher correlations with humans, has much less length bias, and is robust (hard to game).

---

Published as a conference paper at COLM 2024
Acknowledgments
First, we note that Balázs Galambosi an author on this paper as shown on the Arxiv version of the paper, but we can't It update the author list here. We thank Xuechen Li, Rohan Taori, and Tianyi Zhang for help maintaining AlpacaEval. We thank Viet Hoang Tran Duong for suggesting to consider length-balanced win rates. We thank the Twitter ML community for emphasizing the need of length-controlled autoevaluations. We thank the community for all the 100+ models they added to AlpacaEval. We thank OpenAl and Together Al for API credits to generate outputs and evaluate models. We gratefully acknowledge the support of Open Philanthropy Project Award. Tatsunori Hashimoto is supported by the Tianqiao and Chrissy Chen Institute. Yann Dubois is supported by a
Scholarship.
References
Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. arXiv preprint arXiv:2402.07313 2024.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
Viet Hoang Tran Duong. Length-balanced a 2.0, 2024, URL https: com/ Tatsu-lab/aappeee http:////.aaaaaaaaaaaa//////////
2115462149
Esin Durmus, Faisal Ladhak, and Tatsunori Hashimoto. Spurious correlations in reference- free evaluation of text generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Compu- fational Linguistics (Volume 1: Long Papers), pp. 1443-1454, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: /1/06/////20222alllooogg acl https://aclanthologgy aon.oo..............o.....o 102
URL
Arpad Elo and Sam Sloan. The rating of chessplayers: Past and present. 1978.
Balazs Galambosi. Advanced lenggth-normalizedd alpacaeval 2.0, 2024. URL https:/s:///gtt 1942201420
Tanya Goyal and Greg Durrett. Evaluating factuality in generation with dependency-level entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020.
Miguel A Hernán and James M Robins. Causal inference, 2010.
Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang, Benchmarking cognitive biases in large language models as evaluators. arXiv preprint arXiv:2309.170122 2023.
Wojeiech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9332-9346, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18853///0 /v1 /1000//mmmmmmmmmmmmmn///0 URL
https://acllanthologr hwwpaatthhoooooooooooeoooo www.orgg//222 2w00g/2220...0 manlp-mainn.. 750
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of 2023.
instruction-followiin models. https: htttps://githine
http://attuulabbalaaca...
Bill Yuchen Lin, Khyathi Chandu, Faeze Brahman, Yuntian Deng, Abhilasha Ravichander; Valentina Pyatkin, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild, 2024. URL spaces/allenai/// WiBnnhchhh
co/
10

---

Published as a conference paper at COLM 2024
Annie Louis and Ani Nenkova. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39())267-300, June 2013. doi: 10.1162/COLI a_00123. htLLhhtts:://ccalaaa
Jr3/
J. Novikova, O. Dušek, A. C. Curry, and V. Rieser. Why we need new evaluation metrics for NLG. In Empirical Methods Natural Language Processing (EMNLP), 2017.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. Xrx preprint (ttt:2403.111599) 2024.
Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009. ISBN 052189560X.
Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. arXiv preprint XrIi:22100.05199 2023.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigat- ing length correlations in rlhf, 2023.
Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L. Hamilton, and Joelle Pineau. Learning an unreferenced metric for online dialogue evaluation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2440-2441, Online, July 2020. Association for Computational Linguistics. doi: 10.18653 http://0.2020/mm/aa/ URL BuRL/ttps://aaaaathooog. www.orgg//22/ acl main.228,
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following Ilama model. https://gi htt.://.....mmmttttaaa///tttaaae
2023.
Teortaxes. Length-normmllzzed alpacaeval 2.0, 2024. URL http:///ww.oomteeettttetete///ee SSatus//77545501777717777777
Tyler VanderWeele. Explanation in causal inference: methods for mediation and interaction. Oxford University Press, 2015.
Tyler J. VanderWeele. Controlled direct and mediated effects: Definition, identification and bounds. Scandinavian Journal of Statistics, 38, 2010. URL https://api. semanticscholar
org/Corpus 12046639
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arv preprint ar arXiv:2305.17926, 2023.
Minghao Wu and Alham Fikri Aji. Style over substance: Evaluation biases for large language models. arXiv preprint ar Xix::3077003025, 2023.
Y. Yeh, M. Eskenazi, and S. Mehri. A comprehensive assessment of dialog evaluation metrics. In The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pp- 15-33, Online, November 2021. Association for Computational Linguistics.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL httptst://ppeeeeiiiiiii// wttp://rrwiiieeeeeooorrrmr/m/mee www..nne/oorumit--seeetttttt
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonza- lez, and Ion Stoica. Judging auddage with mt-bench and chatbot arena. ArXiv, 0bs/2206.05683 2023. URtphtts:: htttp:::/ppp..smanticccchoo semanticscholar httpr//ccccccc..ooooooooooo////a ortg/oorpusD:2259123399999
11


Reviews:
[{"review_id": "YkJha7B6VU", "reviewer": "Reviewer_3Bhq", "summary": "This paper proposes a simple regression model to adapt existing LLM-based LLM-chat evaluation metrics to control for different sources of noise in automated evaluations (demonstrating the method by length-adapting the Alpaca metric). They fit a generalised linear model to predict which of two models is better for a given instruction based on the predictions of the existing evaluator (Alpaca), the relative length of the outputs of the two models, and the instruction/prompt given to elicit the output. After fitting this model, they use it for prediction by ignoring the length term. This results in a model which has better correlations with human performance (0.98 vs the previous best of 0.94) and the authors also consider factors like 'gameability'--how easy is it to tweak your outputs (e.g. through prompt manipulation) to get the scores or 'win-rate' you want and demonstrate that their method performs well for at least one measure of gameability.\n\nOverall, this paper seems like a worthwhile contribution for researchers interested in approximating human preferences for English-language chatbot LMs without a desire for finer grained assessements (e.g. of accuracy, factuality, naturalness, style, grammaticality, etc).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review. We incorporated your feedback, which we believe improved our manuscript!\n\nIn particular, we added a limitation section, which now explicitly states that the paper only considers prompts written in English and that we inherit all the limitations from focusing on a single-metric leaderboard for evaluation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Clearly articulated model and benefits for metrics used on a popular leaderboard. Reasonable evaluation to support the claims.", "reasons_to_reject": "The focus on leaderboards and 'overall quality' metrics oversimplifies the complexity of evaluation. The paper is implicitly about evaluating English-language LLMs but never states this explicitly.", "questions_to_authors": "Requests and observations:\n- I don't think the figure on your first page helps with understanding the problem or your solution, and more properly belongs later in the article.\n- please update your figures so that the colors are distinguishable in grayscale (e.g. when printed black and white or viewed on an e-ink display) and ensure that all text embedded in the figures is legible (e.g. sometimes there is black text on a dark background that is very difficult to read)\n- please convert your tables to native LaTeX so that they can be highlighted while reading.", "extra_scores": {}}, {"review_id": "TBvh01mtNk", "reviewer": "Reviewer_P3wF", "summary": "This work proposes a method for mitigating the length bias of LLM-based evaluations -- AlpacaEval in this case. They fit a generalized linear model to predict the LLM evaluator's preferences, conditioned on the length of the models’ output. The length-controlled preference is then obtained by predicting what the LLM evaluator would have preferred if the model’s output and the baseline’s output had the same length. LC-AlpacaEval is shown to have a higher correlation with humans, reduced length bias, and is less sensitive to length changes.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "​​Thank you for your constructive review and great questions. We clarified and highlighted the following answers to your questions in the updated manuscript.\n\n> Which data exactly is the regression model trained on? \n\n> What were the ranges of M (# of models) and N (# of instructions) used for experimentation? \n\nThe regression model was trained on the entire open AlpacaEval benchmark at the time of submission (805 instructions and >120 models). We clarified that in the updated manuscript. Note that only the (805) instruction parameters are shared across different GLMs. Once the instruction parameters are fitted, our pipeline can be thus seen as training 120 GLMs (one per model), each on 805 examples and each containing 3 parameters. Fitting each GLM takes less than a second. The entire pipeline thus should work well at different scales of models and instructions.\n\n> Have you considered other confounders covered in related work [...] \n\nGreat question! We briefly discussed other confounders in section 5 under ``Other biases'' but did not have the space to properly answer this question. One of the reasons we like the regression analysis approach is that it should be easy to extend it to new confounders by controlling for them (see figure 2). In practice, this would translate to adding a new term/feature in the GLM. To maintain all the properties of win rates discussed in section 4 you would just need to ensure that the added term is antisymmetric and centered around 0. We are thus relatively optimistic about extending our method to new biases.\n\nIn terms of which biases to control for, here are three ones we thought about and why we decided not pursue them in this paper:\n- Order bias: LLM preference depends on the order in which the examples were shown. AlpacaEval avoids this issue by randomizing the order across prompts. \n- Egocentric bias: Figure 7 shows that this bias exists but does not impact the ranking between models (for the considered models). Furthermore, egocentric bias mostly impacts one model out of all the models on a benchmark. We thus decided not to control for it. \n- List bias: Figure 9 in the [AlpacaFarm](https://arxiv.org/abs/2305.14387) paper shows that LLMs prefer outputs that contain lists. We think that it would be an interesting bias to consider but we decided to prioritize length since AlpacaFarm showed that humans prefer lists even more than LLMs, and since it’s harder to quantify the amount of lists in an answer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The work proposes a simple and effective approach for controlling biases in LLM-as-judge evaluations. \n* Shows strong correlations with Chatbot Arena.\n* The paper is well-written, explains the nuances well, and is easy to understand. \n* Flexibility  -- once the weights are fitted for one baseline, win rate between any pair of models on the leaderboard can be predicted.\n* Offers some interesting insights into open-source LLMs:\n> these results are consistent with the hypothesis that existing open models had exploited the length bias of AlpacaEval.", "reasons_to_reject": "Some crucial details are missing: \n* Which data exactly is the regression model trained on? \n* Details of models and instructions is missing. \n* What were the ranges of M (# of models) and N (# of instructions) used for experimentation?", "questions_to_authors": "* Have you considered other confounders covered in related work, such as the length of instructions, specific word choices,  and so on ? Specifically, using Koo et al. (2023)'s taxonomy, the paper addresses Salience bias (length), Egocentric bias (model identity), and an additional challenge of difficulty of instructions. Do you see any other biases that could exist in the framework (e.g. order bias or compassion fade). How could those be addressed, especially when they co-occur with length?\n* It would be beneficial to add recommendations for what improvements LLMs can make (especially while collecting human feedback for alignment) for controlling biases such as this one.", "extra_scores": {}}, {"review_id": "2OuBHhDFCS", "reviewer": "Reviewer_39UA", "summary": "This paper aims at improving AlpacaEval (a framework that uses LLMs to judge instruction-following models' quality, using the metric of \"win rate\" compared to a baseline model) by mitigating the length-bias issue: it is well known that LLM judges have verbose biases (prefer longer outputs). \n\nThe authors propose to use _all the predicted results of LLM judges and several tested models_ to fit a logistic regression model. The features include (1) the models tested (how strong the model is), (2) the instruction used (how difficult the instruction is), and (3) the length of the response. Then the length-controlled win rate can be calculated by the regression model without using the length feature. Specifically, the authors added a weak regularization term on the weight of the length feature to avoid attacks (by deliberately truncating the response). \n\nThe paper has done a thorough analysis on the proposed length-control AlpacaEval. They showed that (1) the new evaluation reduced the chances the model can game the benchmark by being more verbose (through prompting); (2) the correlation with chatbot arena (human evaluation) is the highest among all model-based evaluation; and (3) it is robust to attacks like truncating the model responses. The authors also compared this method to some naive length-controlled evaluation and showed the advantage of the proposed method.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and great questions. We clarified and highlighted the following answers to your questions in the updated manuscript.\n\n> It is also unclear what data (how many models, how many instructions) [...]\n\nThe regression model was trained on the entire open AlpacaEval benchmark at the time of submission. This means 805 instructions and >120 models. We clarified that in the updated manuscript.\nNote that only the (805) instruction parameters are shared across different GLMs. All the other parameters (3 per model) are independent between models. Once the instruction parameters are fitted, our pipeline can be thus seen as training 120 separate GLMs (one per model). Each GLM contains 3 parameters and is trained on 805 examples in less than a second. The entire pipeline thus should work well at different scales of models and instructions.\n\n> From my understanding, every time there is a new model, the evaluation needs to retrain the regressor, which leads to additional overhead and constantly changing numbers.\n\nThis is an important concern that we wanted to avoid. As explained in the previous answer and at the end of section 3: once the instruction parameters are fitted we can fit models separately and \"Fitting models separately is important as it implies that previously computed metrics won't change when adding a new model to the leaderboard'. In summary, we fit 805 parameters on 805*120 data points once, and for each model, we can then fit each of the 3 parameters of the GLM separately.\n\n> How does the number of models affect the performance of the regressor? \n> What happens if you only have a few models evaluated?\n\nAs mentioned above, only the parameters that quantify instruction difficulty are shared, so the question is whether you can evaluate those shared parameters with a few models. First, to put things in perspective, we found that even if you drop the instruction difficulty terms we get 0.96 correlation with ChatBot Arena, so if you only had a few models you would be fine just dropping that term. Second, for every instruction the instruction difficulty is a scalar trained with regularization and cross-validation, we would thus expect to be able to get a useful estimate of it even with a few models being evaluated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The discussed topic is of high impact and of high value in the language model community (model-based evaluation for instruction-following models). The proposed new evaluation mitigated a significant bias in those evaluations -- length biases. \n\n(2) The evaluation of the method is thorough -- the authors demonstrated, from different perspectives, that length-controlled AlpacaEval is better.\n\n(3) The proposed method is principle and novel compared to some of the existing solutions to the length bias problem.", "reasons_to_reject": "(1) The paper's writing could be improved. The description of the regression part (the main method) is especially unclear. For example, the authors did not explain $\\theta_m$, $\\theta_b$, (very easy to be confused with LM parameters), $\\psi_m$, and $\\gamma_x$ in Equation (1). It is also unclear what data (how many models, how many instructions) the authors used to train the regressor, which could be critical details.\n\n(2) From my understanding, every time there is a new model, the evaluation needs to retrain the regressor, which leads to additional overhead and constantly changing numbers. This could be a limitation of the method.", "questions_to_authors": "How does the number of models affect the performance of the regressor? I did not find a lot of details on this, but I imagined all the experiments are done on dozens of models. What happens if you only have a few models evaluated? (could be a realistic scenario when you have a new benchmark that only a few models are available, or even just two!)", "extra_scores": {}}, {"review_id": "Z385GOEWyZ", "reviewer": "Reviewer_suS5", "summary": "This paper describes a methodology for controlling for length bias in the automatic LLM evaluation methodology AlpacaEval. The method involves decomposing AlpacaEval winrates into a logistic regression problem with an explicit output length parameter. The resulting corrected metric has good Chatbot Arena correlation and shows resilience to gameability. It's a simple method that controls for a simple confounder.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and detailed suggestions. We've updated the manuscript to incorporate your suggestions and talk about the limitations that you've highlighted.\n\n> Length bias is a highly interesting phenomenon, likely rooted in a strong correlation between length and quality. I wonder if zeroing this out is actually the \"correct\" way to go about this? \n\nThis is a great question and we hope our work spawns more work in that domain to consider what should be done concerning length bias. We think that this is the right way of answering our motivating question (section 3): \n\n\"What would the AlpacaEval metric be, if the outputs of all models had the same length as those of the baseline?\". \n\nBut we agree that the question itself may not be what you want to answer. We believe that the increase in correlation with ChatBot Arena compared to no length correction, and to other methods (see Figure 6) shows that this is a sensible step in the direction of controlling for length bias. Still, we believe that much more research has to be done to show what is the ideal way of considering length in our evaluation benchmarks.\n\n> Given that the paper solely focuses on AlpacaEval, the utility of this method inherits much of the limitations of AlpacaEval itself [...]\n\nYes, we completely agree and added a limitation section to the updated manuscript that highlights the main limitations of the method including the ones you mentioned.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Good utility:\n\nAutomatic evaluation is a valuable topic. Assuming Chatbot Arena as a gold standard, creating automatic evaluation metrics that best mimics it without human crowdsourcing is useful. The proposed approach in the paper results in a very high correlation. The paper does a good job of convincing the necessity of length correction.\n\nComparisons with alternative methods:\n\nThe methodology is simple, but comparisons are made with other methods of length correction methods. Analysis shows that the proposed method has strong gameability resistance and best Arena correlation.", "reasons_to_reject": "Limited scope:\n\nThe scope of the paper is limited to improving AlpacaEval's winrate-based methodology. There are no experiments to assess the length-controlling method for other automatic evaluators. It would have been interesting to adopt the method to investigate length bias of other metrics. Given that the paper solely focuses on AlpacaEval, the utility of this method inherits much of the limitations of AlpacaEval itself (LLM as judge may be inaccurate to evaluate stronger models or harder queries).", "questions_to_authors": "Length bias is a highly interesting phenomenon, likely rooted to a strong correlation between length and quality. I wonder if zero-ing this out is actually the \"correct\" way to go about this?\n\n\nWriting suggestions:\n\n- Section 4: \"Ideally, length gameability is low\" -> \"Ideally, length gameability should be low\"\n\n- Section 4.1: \"changes the me\" -> \"changes the metric?\"\n\n- Section 4.2: \"We see that proprietary models...\" this point is not immediately clear to me. Proprietary models would also have undergone RLHF, and also may have been optimized for widely used evals. It could be noted which models here are considered \"proprietary\" to make a distinction. I think I understand the general sentiment, but think it would be more appropriate to talk about it as a matter of training data rather than proprietary vs open-source.\n\n- Section 4.3: \"Regulatization makes LCAE\" LCAE -> AE-LC?\n\n- Section 4.3: \"imperceptible impact\" could this be elaborated on?\n\n- minor point, but \"winrate\" \"win rate\" and \"win-rate\" are used throughout the paper.\n\n\n=== comments after response ===\n\nThank you for the response. I'm glad to hear about the updates.\nI believe the paper has a narrow, but solid contribution. I will keep my scores.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper aims to address the length bias issue in LLM-as-a-judge framework, focusing specifically on AlpacaEval. The core idea of the paper is to train a logistic regression model to fit the LLM judge predictions for all tested models using the tested models, instruction difficulty, and length difference with a baseline response as the features. The original alpacaEval winrates are converted into length controlled win-rates by zero-ing out the length difference in the above LR model.  

Pros:
Overall, the reviewers agree that the idea is pretty neat, and the experiments clearly show that this reduces the game-ability of the metric by manipulating the generation's lengths. They also report higher correlation with the ChatBot Area, i.e. human evaluation, benchmark. 

Minor Cons:
The paper only validates their approach on one LLM-as-as-judge framework. But, given the popularity of AlpacaEval, this is a minor criticism. 
The authors should also incorporate the reviewer comments about improving the presentation of the paper.
Decision: Accept