Large Language Models are not Fair Evaluators (2024)

Abstract: In this paper, we uncover a positional bias in the evaluation paradigm of adopting large language models (LLMs), e.g., GPT-4, as a referee to score and compare the quality of responses generated by candidate models. We find that the quality ranking of candidate responses can be easily hacked by simply altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other, e.g., Vicuna-13B could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator. We propose a simple yet effective calibration framework to address our discovered positional bias. To evaluate the effectiveness of our framework, we manually annotate the “win/tie/lose” outcomes of responses from ChatGPT and Vicuna-13B in the Vicuna Benchmark’s question prompt. Extensive experiments demonstrate that our approach successfully alleviates evaluation bias, resulting in closer alignment with human judgments. To facilitate future research on more robust large language model comparison, we integrate the techniques in the paper into an easy-to-use toolkit FairEval, along with the human annotations 1.

heading: 1 Introduction
text: The rapid advancement of Large Language Models (LLMs) (Brown et al., 2020; Chowdhery et al., 2022) has underscored the importance of evaluating their alignment with human intent in generated responses, making it an active field of research. Traditional n-gram metrics like BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), as well as more sophisticated model-based evaluations such as BERTScore (Zhang et al., 2020) and BARTScore (Yuan, Neubig, and Liu, 2021), are insufficient for thoroughly assessing this alignment (He et al., 2023). While human evaluation provides
1https://github.com/i-Eval/FairEval
the most accurate measure of model performance and valuable insights, it can often be costly and time-consuming. As a result, there is a growing demand for automated assessment methods that can consistently align with human judgments while being more efficient and cost-effective.
ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023) have recently demonstrated remarkable performance across various tasks, leading to their widespread use as both the annotators (Peng et al., 2023; Xu et al., 2023) and evaluators (Zheng et al., 2023; Peng et al., 2023; Sun et al., 2023; Zhou et al., 2023; Gao et al., 2023; Wang et al., 2023b; Dubois et al., 2023; Wang et al., 2023a). For example, The evaluation pipeline of Vicuna (Zheng et al., 2023) has gained significant interest and wide usage due to its simplicity and interpretability. It prompts GPT-4 to score and compare candidate responses and provide explanations, making it a valuable tool for evaluation. However, it is unclear how reliable LLMs are as evaluators, as they are known to be sensitive to textual instructions and inputs (Dong et al., 2022; Turpin et al., 2023; Bowman, 2023). This raises questions about the resilience of this paradigm against perturbations, such as the ordering of candidates during scoring,
9440
potentially becoming the Achilles’ Heel that can be easily hacked for unreliable evaluations.
In this paper, we take a sober look at the LLMsas-evaluator paradigm and uncover a significant positional bias. Specifically, we demonstrate that GPT-4 exhibits a preference for the first displayed candidate response by consistently assigning it higher scores, even when the order of candidates is subtly altered. As illustrated in Figure 1, merely swapping the presentation order can reverse evaluation outcomes. This bias is also present in ChatGPT, which typically favors the second response. These findings highlight previously overlooked limitations in the current evaluation paradigm.
To address this issue, we propose three simple yet effective strategies to calibrate positional bias: 1) Multiple Evidence Calibration (MEC): We prompt the model to generate evaluation evidence before assigning scores, leveraging the inherent properties of causal language models for calibration. We also employ ensemble techniques to incorporate multiple evidence calibration results to further stabilize the evaluation. 2) Balanced Position Calibration (BPC): To further reduce positional bias, we evaluate each candidate in both positions across two runs and compute the final score as the average of the two runs. 3) Human In The Loop Calibration (HITLC): We also explore human-inthe-loop evaluation and consider a diversity-based method to get a cue to indicate biased candidates based on the evaluation results of MEC and BPC.
To assess the efficacy of our methods, we manually annotate the “win/tie/lose” outcomes of responses from ChatGPT and Vicuna-13B in the Vicuna benchmark (Zheng et al., 2023), encompassing 80 questions spanning 9 distinct question categories. Our MEC and BPC enhance the evaluation alignment of GPT-4 and ChatGPT by 9.8% and 14.3% accuracy, respectively. Moreover, based on MEC and BPC, our HITLC can further effectively integrate human assistance into the evaluation process. Specifically, with only a 20% human annotation cost, GPT-4 and ChatGPT can achieve comparable or even better annotation alignment with the average human performance, reducing the annotation cost by up to 39%.
In summary, our key contributions are: 1) We reveal that LLMs exhibit severe positional bias, compromising their fairness as evaluators; 2) We develop a calibration framework with three simple yet effective strategies to calibrate the positional bias of
LLMs; 3) We manually annotate the “win/tie/lose” outcomes of responses from ChatGPT and Vicuna13B in the Vicuna benchmark and demonstrate the effectiveness of our proposed approach through experimental results, which show closer alignment with human judgments.

heading: 2 Positional Bias of the LLM Evaluator

heading: 2.1 LLMs as Evaluators
text: Recently, a lot of works (Zheng et al., 2023; Wang et al., 2023a; Sun et al., 2023; Zhou et al., 2023) have been utilizing LLMs such as GPT-4 as evaluators to compare the performance of two AI assistants. Table 1 illustrates a commonly used evaluation template from the Vicuna benchmark (Zheng et al., 2023). The template has three placeholders T (Q,R1, R2), and is used to query the LLM for evaluation. For each testing question q, given two responses r1 and r2 from Assistant 1 and Assistant 2, respectively, the researchers populate these responses into the corresponding slots of the evaluation template to form a prompt: T (Q = q,R1 = r1, R2 = r2). The prompt is then used to query the LLM in order to obtain the comparison result. In this paper, we found that LLM suffers from severe positional bias, i.e., by swapping the slots of the two responses and
querying LLM twice, the evaluator will most likely produce conflicting evaluation results, and the evaluator prefers the response at a certain position.

heading: 2.2 Revealing the Positional Bias
text: In this section, we adopt GPT-4 and ChatGPT as evaluators to analyze the characteristics of positional bias in LLM evaluators. We find that:
LLMs are sensitive to the position of responses. As shown in Table 2, in the evaluation of “Vicuna13B v.s. ChatGPT” and “Vicuna-13B v.s. Alpaca13B”, when the order was changed, LLMs provide different evaluation results, e.g., the win rate of Vicuna-13B extremely differs when Vicuna-13B is evaluated as Assistant 1 and Assistant 2.
To empirically evaluate the sensitivity, we introduced a metric Conflict Rate to measure the sensitivity of the model to response positions quantitatively. Formally, given N examples {(qi, r1i, r2i)}Ni=1, for each example (qi, r1i, r2i), we query the LLM with two prompts T (qi, r1i, r2i) and T (qi, r2i, r1i), and obtain corresponding two evaluation results ERr12i and ER r21 i . Then we calculate the Conflict Rate of the LLM evaluator as follows:
Conflict Rate =
∑N i=1 I(ER
r12 i ̸= ERr21i ) N , (1)
where I(.) is the indicator function. We found that GPT-4 exhibited conflict rates of 46.3% and 5.0%, respectively. In contrast, ChatGPT displayed considerably higher conflict rates, with figures of 82.5% and 52.5%, respectively. These findings indicate that LLMs can be self-conflicting due to the sensitivity of the response order in the template, with stronger models being less influenced by the placement of responses.
LLMs suffer from Positional Bias, i.e., they prefer the response in the specific position. Based
on the same evaluation template T in Table 1, GPT4 tends to favor the response in the first position, while ChatGPT shows a preference for the response in the second position. For example, as illustrated in Table 2, in the comparison “Vicuna-13B v.s. ChatGPT”, GPT-4 yields Win Rates of 51.3% and 23.8% for Vicuna-13B when it is positioned as Assistant 1 and Assistant 2, respectively. Conversely, ChatGPT indicates Win Rates of only 2.5% and up to 82.5% for Vicuna-13B when it is positioned as Assistant 1 and Assistant 2, respectively.
The degree of positional bias varies based on the difference in response quality. We notice that the conflict rate of “Vicuna-13B v.s. Alpaca13B” is much lower than that of “Vicuna-13B v.s. ChatGPT”, suggesting that positional bias may not have the same impact on the assessment of different responses. One potential reason is that there is a significant difference in the quality of responses between Alpaca models and Vicuna models, and positional bias is not strong enough to change the judgment in such a situation. To further investigate this issue, we grouped all the examples based on
the score difference between the two responses. As shown in Figure 2, we found that when the score difference between the two responses is small (e.g., score gap ≤ 1), the evaluation results of GPT-4 are significantly affected by the position of the responses. On the other hand, when the score difference between the two responses is large (e.g., score gap ≥ 3), GPT-4’s evaluation results are relatively stable.

heading: 2.3 Calibrating the Positional Bias
text: We have identified that positional bias can significantly impact the evaluation results of LLMs, making them unfair evaluators. In this section, we propose a calibration framework with three simple yet
effective strategies to alleviate this bias to achieve a more reliable and fair evaluation result.

heading: 2.3.1 Multiple Evidence Calibration
text: Previous studies (Zheng et al., 2023; Wang et al., 2023b) utilize the evaluation template that draws the conclusion first and then makes an explanation, e.g., the template used in Table 1. However, due to the nature of the auto-regressive model, the conclusions generated by the model are not supported by the explanation generated afterward. To this end, as shown in Table 3, we design an evidence calibration (EC) evaluation template TEC(Q,R1, R2) that requires the model to generate the explanation (evaluation evidence) first and then give the score. In this way, the score can be calibrated with the evaluation evidence. To further improve the reliability of the evaluation, rather than generating only a single EC score for each response, we perform a multiple evidence calibration (MEC, Figure 3(a)) that samples k EC scores {S1r1, . . . , Skr1} and {S′1r2, . . . , S ′k r2} for responses r1 and r2, where Sr and S ′ r denotes scores of the response r at the first and second positions, respectively.

heading: 2.3.2 Balanced Position Calibration
text: We further employ a balanced position calibration (BPC) strategy to alleviate the previously identified positional bias of LLMs. As shown in Figure 3(b), for each example (q, r1, r2), BPC additionally creates a query prompt TEC(q, r2, r1) by swapping the position of two responses in the original query prompt TEC(q, r1, r2). Combined with MEC, we can achieve 2k scores {S1r1, . . . , Skr1, . . . , S ′1 r1, . . . , S ′k r1} and {S′1r2, . . . , S ′k r2, . . . , S 1 r2, . . . , S k r2} for r1 and r2,
respectively. The final calibrated scores of two responses (CSr1 and CSr2) are the average of the 2k scores:
CSR = k∑
i=1
SiR + S ′i R
2k ,R = r1, r2 (2)
and we regard the response with the higher average score as the better response.

heading: 2.3.3 Human-in-the-Loop Calibration
text: In addition to the automatic calibration strategies, another interesting question we want to explore is whether Human-In-The-Loop Calibration (HITLC) which performs the cooperation of humans and LLMs as evaluators, could stabilize the evaluation result. The key point of human-in-the-loop calibration is when humans should be involved in the evaluation and calibrate the evaluation result on which LLM evaluators do not perform well.
To target the “when” problem, we introduce a Balanced Position Diversity Entropy (BPDE) score to find examples requiring auxiliary human calibration based on the evaluation results of MEC and BPC. Specifically, as shown in Figure 3(c), we first compute 2k evaluation results {ERi}2ki=1 based on the 2k pairs of scores.
ERi 1≤i≤k =    win,Sir1 > S ′i r2 tie,Sir1 = S ′i r2
lose,Sir1 < S ′i r2
, ER′i 1≤i≤k =    win,S ′i r1 > S i r2 tie,S ′i r1 = S i r2
lose,S ′i r1 < S i r2
,
(3)
and BPDE is defined as the entropy of the evaluation results:
BPDE = ∑
er∈{win,tie,lose} −per log per (4)
per =
∑k i=1I(ERi = er) + I(ER ′ i = er)
2k . (5)
A higher BPDE score indicates that it is more likely the evaluation requires manual correction. A threshold is needed for BPDE as the hyper-parameter to select the top-β most likely biased evaluations. After selection based on the BPDE score, the annotators will evaluate the selected examples and integrate the human annotations based on the majority opinion as described in Section 3.1.

heading: 3 Experiments

heading: 3.1 Human Annotation
text: To assess the effectiveness of our proposed strategies, three of the authors manually annotate the
“win/tie/lose” outcomes of responses from ChatGPT and Vicuna-13B independently in all 80 Vicuna Benchmark questions. All of the annotators are researchers familiar with Artificial Intelligence and are well-equipped to assess the quality of the responses. Following the same template as the original Vicuna, the annotators are instructed to assess the responses provided by Vicuna-13B and ChatGPT from four different perspectives: helpfulness, relevance, accuracy, and level of detail. The responses of Vicuna and ChatGPT are presented to the annotators in random order. The evaluation process for each example took an average of three minutes. The final result is based on the majority opinion among three annotators.

heading: 3.2 Experimental Setup and Metric
text: We use the OpenAI API to conduct our experiments (“gpt-3.5-turbo” for ChatGPT, and “gpt-4” for GPT-4). For the methods that do not need to sample multiple generation results, we set the generated temperature to 0 for deterministic generation results. For the multiple evidence strategy, we set the temperature to 1 and k = 3. We use the accuracy and kappa correlation coefficient (McHugh, 2012) with the final majority of human annotation results to measure the performance of different evaluators and evaluation methods. When calculating the results for methods that do not utilize BPC, we randomize the order of the two responses from the assistants and calculate the average results of 100 runs to ensure stable results.

heading: 3.3 Main Results
text: Table 4 illustrates the performance of different methods on our manually annotated 80 annotated examples. As is shown: 1) There is a good correlation coefficient between the annotations provided by each human annotator and the final voting results. In detail, the average accuracy and the kappa correlation coefficient of human annotations are 71.7% and 0.54, respectively; 2) Overall, GPT-4 achieves higher alignment with human judgments compared with ChatGPT, showing its powerful alignment ability with humans; 3) Compared to the commonly used VANILLA evaluation method, our proposed automatic calibration strategies (i.e., EC, MEC and BPC) significantly enhance the alignment between GPT-4 and ChatGPT with human judgments; For instance, by employing the MEC and BPC calibration strategies, ChatGPT demonstrates a notable improvement in both accuracy and
the kappa correlation coefficient. Specifically, the accuracy is improved by 14.3%, and the kappa correlation coefficient is increased from 0.06 to 0.31; 4) “MEC (k = 3) + BPC (k = 3)” outperforms “MEC (k = 6)”, demonstrating that LLMs are affected by positional bias, and BPC effectively ensures that LLMs serve as fair evaluators; 5) Our proposed HITLC can effectively enhance the alignment between GPT-4 and ChatGPT with human judgments, requiring only a small amount of human labor. For example, by incorporating just 20% (β = 20%) human assistance, ChatGPT attains comparable Human Average accuracy, while reducing the annotation cost from $30 to $18.3, a 39% reduction.2
In conclusion, our proposed calibration methods are simple yet very effective in improving the evaluation performance with LLM as evaluators, while maintaining low costs.
2The minimum hourly wage in the United States is near $7.5, which can be found at https://www.worker. gov/. On average, annotating an example takes 3 minutes, and the Vicuna evaluation benchmark comprises 80 examples in total. Consequently, the cost per annotator amounts to $30.

heading: 4 Analysis

heading: 4.1 Ablation on Evidence Number k and Temperature t
text: In the MEC and BPC strategy, we sample k evaluation results for each query prompt and ensemble them to enhance the evaluation process. We conduct an analysis to examine the influence of the number of evidence k, on the model’s evaluation performance. As illustrated in Figure 4(a), we compared the performance of ChatGPT with different values of k, namely 1, 3, 5, and 7. The model’s performance increases and then tends to be
constant or decreases slightly as k becomes larger. Despite the slight decrease, the enhancement of the model effect by the MCE strategy is still significant, illustrating the stability of the MEC strategy. Consequently, we found that a k value of 3 yields an optimal performance. With this value, the model achieves a notable level of performance while keeping the API cost relatively low. We further investigate the impact of sampling temperature t on evaluation performance. Figure 4(b) illustrates that both low temperature (i.e., 0.2 ) and high temperature (i.e., 1.4) result in sub-optimal evaluation alignment. We believe that low temperature eliminates the randomness of sampling, weakening the effect of MEC, while high temperature compromises the quality of generation results, leading to poor performance. Hence, it is crucial to select an appropriate temperature (e.g., 0.6 or 1.0 in our experiments) for the LLM evaluators.

heading: 4.2 Effectiveness of the BPDE
text: Our HITLC strategy utilizes BPDE score to select examples for human annotations. In order to analyze the efficiency of BPDE score, we compare BPDE with two typical baselines, Random and Vanilla Diversity Entropy, where Random denotes randomly select examples for human annotations, and Vanilla Diversity Entropy is calculated by using only the evaluation results of one position without swapping the position of two responses. To ensure fairness, the total number of evaluation results is 6 for both BPDE and Vanilla Diversity Entropy. As shown in Figure 5: 1) Two Diversity Entropy methods outperform Random, showing the effectiveness of selecting examples based on the diversity entropy; 2) BPDE outperforms Vanilla DE, which shows LLMs are sensitive to position exchange, and the results of BPC can significantly improve the performance of HITLC compared to relying solely on the results of MEC.

heading: 4.3 Generalization on the Pairwise Comparison Evaluation Template
text: To provide a more comprehensive validation of our proposed calibration methods, in addition to the previous SCORING evaluation template that rates each response, we extend our analysis to incorporate the COMPARING evaluation template. This template facilitates a direct comparison between two responses, eschewing explicit scores in its assessment. Specifically, we prompt LLMs to produce results labeled as “Assistant 1”, “Assistant 2”,
Accuracy for Different Thresholds
or “Same”, indicating whether the response from Assistant 1 is better, worse, or equal to that of Assistant 2. As is shown in Table 5: 1) Our proposed methods are applicable to both of these templates, leading to enhanced accuracy and a heightened correlation coefficient for ChatGPT; 2) The significant performance gap (nearly 6% accuracy) between the VANILLA method of two templates, coupled with the high conflict rate, highlights the sensitivity and unreliability of LLMs. However, our methods effectively narrow this performance gap and reduce conflict, showcasing how calibration enhances LLM robustness.

heading: 4.4 Fine-Grained Analysis of Evaluation Quality
text: To further analyze the evaluation capabilities of the model, we perform a fine-grained analysis of the questions by dividing them into 9 categories following Zheng et al. (2023). We calculate the perfor-
mance of different evaluators. As shown in Figure 6, we find that: 1) In certain complex tasks such as common sense, coding, and math, GPT-4 performs significantly better than ChatGPT, highlighting the strength of GPT-4 as a more fair evaluator in these scenarios; 2) Our proposed MEC+BPC strategy demonstrates noticeable improvement in evaluating ChatGPT’s performance on complex tasks, allowing us to obtain satisfactory evaluation results with a low API cost.

heading: 5 Related Work
text: Large Language Models as Evaluators How to achieve reliable evaluations of generative AI assistants is one of the most important problems in AI research. Traditional metrics, like BLEU, ROUGE, BERTScore, or BARTScore fail to measure the alignment of the generated response with human intent (He et al., 2023). Considering the potent capabilities of LLMs, researchers have started utilizing LLMs to evaluate the proficiency of generative models in adhering to human instructions (Zheng et al., 2023; Lu et al., 2023; Li et al., 2023). There are two typical LLM-as-evaluator variations: pointwise evaluation and pair-wise evaluation. Pointwise evaluation scores the responses one by one, while pair-wise evaluation achieves the evaluation results by comparing two responses. Point-wise evaluation is more convenient, because para-wise evaluation may lack scalability when the number of responses increases. However, Pair-wise evaluation can give more accurate evaluation results than point-wise evaluation (Zheng et al., 2023). Specifically, assigning a precise score to a response is challenging as it requires providing evaluators with a detailed and accurate scoring standard that
is difficult to establish. In contrast, just making the comparison is much easier and more stable. In this work, we discover and address the positional bias of LLMs-as-evaluators for pair-wise evaluation. We believe it is essential for the whole community to achieve reliable evaluations.
Bias of Deep Neural Networks Deep Neural Networks have been proven to easily learn biases from the data, which significantly impacts their reliability. Specifically, bias has also been investigated in natural language inference (Gururangan et al., 2018; McCoy, Pavlick, and Linzen, 2019; Belinkov et al., 2019; Liu et al., 2020a,b), question answering (Min et al., 2019), ROC story cloze (Cai, Tu, and Gimpel, 2017; Schwartz et al., 2017), lexical inference (Levy et al., 2015), visual question answering (Goyal et al., 2017), information extraction (Wang et al., 2021, 2022; Song et al., 2023; Xia et al., 2023) and so on. LLMs are pre-trained using a vast amount of data from the internet, making it highly likely for them to learn biases present in those materials. Although the LLMs are already widely adopted as a proxy of human evaluators, the reliability of this paradigm is not well explored. In this paper, we critically examine the LLMs-asevaluator paradigm and uncover a significant positional bias. Furthermore, we propose three simple yet effective methods to calibrate the positional bias to achieve reliable and fair evaluation results.

heading: 6 Conclusion
text: In this paper, we reveal a positional bias of the LLMs-as-evaluators paradigm, i.e., by manipulating the order of candidate responses during evaluation, the quality ranking results can be significantly
influenced. The positional bias seriously affects the reliability of the evaluation. We propose a calibration framework with three simple yet effective strategies to address the discovered bias. We also manually annotate the “win/tie/lose” outcomes of responses from ChatGPT and Vicuna-13B in the Vicuna Benchmark’s question prompt to evaluate the effectiveness of our framework. Extensive experiments demonstrate that our approach successfully alleviates evaluation bias, resulting in closer alignment with human judgments.
Limitation
In this paper, we identified the positional bias of LLM evaluators, which is essential for building reliable LLM evaluators. However, we did not investigate the underlying causes of this bias in LLMs. Future research should delve deeper into this issue.

heading: Acknowledgements
text: This paper is supported by the National Key Research and Development Program of China 2020AAA0106700. The contact author is Zhifang Sui.

heading: Future Work
text: Future research should delve deeper into this issue