CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges
Authors: Haitao Li, Junjie Chen, Qingyao Ai, Zhumin Chu, Yujia Zhou, Qian Dong, Yiqun LIU
Accepted at: acl25

CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges

Haitao Li1, Junjie Chen1, Qingyao Ai1*, Zhumin Chu1, Yujia Zhou1, Qian Dong1, Yiqun Liu1
1Department of Computer Science and Technology, Tsinghua University
liht22@mails.tsinghua.edu.cn

Abstract
The use of large language models (LLMs) as automated evaluation tools to assess the quality of generated natural language, known as “LLMs-as-Judges”, has demonstrated promising capabilities and is rapidly gaining widespread attention. However, when applied to pairwise comparisons of candidate responses, LLM-based evaluators often exhibit selection bias. Specifically, their judgments may become inconsistent when the option positions or ID tokens are swapped, compromising the effectiveness and fairness of the evaluation result. To address this challenge, we introduce CalibraEval, a novel label-free method for mitigating selection bias during inference. Specifically, CalibraEval reformulates debiasing as an optimization task aimed at adjusting observed prediction distributions to align with unbiased prediction distributions. To solve this optimization problem, we propose a non-parametric order-preserving algorithm (NOA). This algorithm leverages the partial order relationships between model prediction distributions, thereby eliminating the need for explicit labels and precise mathematical function modeling. Empirical evaluations of LLMs in multiple representative benchmarks demonstrate that CalibraEval effectively mitigates selection bias and improves performance compared to existing debiasing methods. This work marks a step toward building more robust and unbiased automated evaluation frameworks, paving the way for improved reliability in AI-driven assessments. The code can be found at https://github.com/CSHaitao/CalibraEval.

1 Introduction
In recent years, large language models (LLMs) have attracted widespread attention in both academia and industry (OpenAI, 2023; Zeng et al., 2022; Li et al., 2024). These models achieve significant performance in a wide range of tasks, sometimes even exceeding human capabilities (Dong et al., 2024a). However, evaluating the quality of the texts generated by LLMs is difficult, particularly in subjective tasks such as open-ended story creation and summarization. Traditional n-gram metrics (like BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004)) and semantic-based metrics (such as BERTScore (Zhang et al., 2019) and BARTScore (Yuan et al., 2021)) are insufficient to comprehensively reflect the capabilities of LLMs. Human evaluation, often regarded as the “gold standard”, can measure model performance most accurately and provide valuable feedback, but it is costly and time-consuming. Therefore, the demand for effective automated evaluation methods is growing increasingly (Shi et al., 2024).

Some powerful commercial LLMs, such as GPT-4, have been widely applied to evaluate the quality of texts generated in response to open-ended questions. This paradigm, known as “LLMs-as-Judges”, provides a scalable and transparent alternative to human evaluation of text quality. Within this paradigm, two common methods are pointwise and pairwise evaluations. In pointwise evaluation, LLMs assign scores to individual responses based on specific criteria, while in pairwise comparison, LLMs select the better response between two options. Pointwise evaluation tends to be unstable and susceptible to noise, as subtle differences in

![Illustration of selection bias in LLMs-as-Judges. Selection bias manifests in two aspects: prefers a specific position or prefers a specific token.](page_820_370_563_312.png)

Figure 1: Illustration of selection bias in LLMs-as-Judges. Selection bias manifests in two aspects: prefers a specific position or prefers a specific token.
wording or interpretation may lead to inconsistent results. In contrast, pairwise comparison can better reflect human judgment (Liu et al., 2024; Zheng et al., 2023b), resulting in its widespread application and considerable attention.

Despite the success, LLMs are not perfect evaluators and are believed to exhibit certain biases (Zheng et al., 2023a,b). As shown in Figure 1, when applied to pairwise comparisons of candidate responses, simply changing the positions or the ID tokens may lead to inconsistent evaluation results. Previous studies have classified these biases as position bias (Zheng et al., 2023b; Shi et al., 2024) and token bias (Pezeshkpour and Hruschka, 2023; Raina et al., 2024). Positional bias refers to the tendency of LLMs to favor answers based on their specific positions (e.g., first or last), and token bias indicates that LLMs may assign more probability to certain option ID tokens (e.g., A or B). Given the inherent link between option tokens and their positions, we collectively refer to them as selection bias in this paper.

Addressing selection bias in “LLMs-as-Judges” is crucial for ensuring valid and fair evaluations. However, this task is not trivial, as selection bias is influenced by task-specific characteristics, such as domain and difficulty, as well as the inherent properties of LLMs, such as context window, family characteristics, and model capabilities (Shi et al., 2024; Ye et al., 2024). A straightforward method is to exclude inconsistent judgments or consider them “ties” (Chen et al., 2024; Zheng et al., 2023b). While this approach enhances consistency and reliability, it may lead to a loss of evaluative information. Furthermore, more advanced methods, such as split and merge (Li et al., 2023b) or discussions (Chan et al., 2023; Li et al., 2023a) among multiple agents, have been proposed to improve evaluation effectiveness. However, these approaches typically require multiple rounds of interaction, making them costly and time-consuming, and their effectiveness in mitigating selection bias remains uncertain.

To address these limitations, we propose CalibraEval, a label-free, inference-time method for mitigating selection bias. CalibraEval reformulates the debiasing problem as an optimization task to build a projection function that maps the original prediction distribution to an unbiased distribution. Our optimization objective is based on consistency judgments obtained after swapping option positions and ID tokens. Moreover, we propose a non-parametric order-preserving algorithm (NOA). The NOA narrows the solution space by preserving the partial order relationship between predicted distributions of observed samples. It derives the optimal calibration function by exploiting the relationship between the prediction distributions from different combinations of options. This approach effectively minimizes the reliance on explicit labels and precise mathematical function modeling, enhancing scalability and transferability.

We conduct extensive experiments on representative evaluation benchmarks with various LLMs. The experimental results indicate that CalibraEval outperforms strong baselines in debiasing performance and achieves state-of-the-art results. Furthermore, we validate CalibraEval’s robustness across diverse prompt templates, varied option tokens, and in-context learning scenarios, demonstrating its potential for application in a variety of contexts.

2 Related Work

2.1 LLMs as Judges

Recently, the “LLMs-as-Judges” approach has become a promising alternative to human annotations and is being widely used (Ye et al., 2024). This method utilizes powerful, widely recognized LLMs, such as GPT-4 (OpenAI, 2023), to facilitate automated evaluation, thereby reducing the dependence on manual assessment. Generally speaking, the “LLMs-as-Judges” evaluation approach can be classified into two categories: pointwise (Kim et al., 2023) and pairwise (Lambert et al., 2024; Zhu et al., 2023). Pointwise evaluation involves LLM judges scoring individual responses based on specific criteria. Pairwise comparison requires choosing the better answer from two responses. Pairwise comparison evaluation has gained widespread adoption and particular attention due to its outstanding performance. Wang et al. (Wang et al., 2023) discovered that pairwise comparison methods outperform traditional score-based evaluation approaches in terms of consistency with human assessments. Liu et al. (Liu et al., 2024) observed that pairwise comparisons better reflect human evaluation standards compared to other methods. This advantage may be attributed to the fact that LLMs often utilize pairwise preference or ranking data during the Reinforcement Learning from Human Feedback (RLHF) training phase (Dong et al., 2024b).
2.2 Bias in LLM Judges

Recent research has identified various biases affecting LLM evaluations, including selection bias (Zheng et al., 2023b), position bias (Li et al., 2023b), contextual bias (Zhou et al., 2023), and self-reinforcing bias (Li et al., 2023a). Among these, selection bias has emerged as a particularly critical issue, as it is prevalent across various tasks and affects both open-source and commercial models, significantly impacting their performance. This bias is typically evident in pairwise comparison evaluations: if an LLM evaluation model consistently favors a specific option even after swapping positions or IDs, this indicates the presence of selection bias. Two types of bias may contribute to selection bias: position bias and token bias. However, there is still no consensus on which of these two is the dominant factor (Pezeshkpour and Hruschka, 2023).

Effectively mitigating bias remains an unresolved issue. Scholars are exploring various approaches to identify and reduce biases in LLMs (Shi et al., 2024). Zheng et al. (Zheng et al., 2023a) use prior estimates from partial samples to address selection bias. Furthermore, Liu et al. (Liu et al., 2024) argued that existing calibration techniques aimed at reducing bias are insufficient for calibrating LLM evaluators, even with supervised data. Therefore, mitigating biases in “LLM-as-Judges” is a widespread, significantly impactful, and challenging issue.

3 CalibraEval

3.1 Problem Statement

In this paper, we focus on addressing the selection bias present in “LLMs-as-Judges”. Selection bias refers to the phenomenon where LLMs consistently prefer a specific option during pairwise comparisons, regardless of the content.

To standardize the terminology, we define the following terms: \( t_i \) represent the option ID tokens (e.g., A, B), and \( o_i \) denotes the specific option contents (e.g., Response_x, Response_y). Additionally, let \( I \) represent the input instruction, and \( X_0 \) represent the default connection of option ID tokens and contents, that is, \( X_0 = [(t_1, o_1); (t_2, o_2)] \).

Following previous studies (Zheng et al., 2023a), we assume that when the LLM serves as an evaluator, the observed probability distribution \( P_{ob} \) on \( t_i \) can be decomposed into a combination of the prior distribution \( P_{pr} \) and the debiased distribution \( P_{de} \), i.e.,

\[
P_{ob}(t_i|I, X_0) = f(P_{pr}(t_i|I, X_0), P_{de}(t_i|I, X_0))
\]

where \( f(\cdot) \) is a function that represents the relationship between \( P_{ob}, P_{pr} \) and \( P_{de} \). Accurately estimating the form of \( f(\cdot) \) is challenging. Firstly, the interaction between \( P_{pr} \) and \( P_{de} \) is complex and may not be simply multiplicative or additive. Secondly, the observed probability distributions \( P_{ob} \) may be affected by noise, complicating the identification of the precise form of \( f(\cdot) \). In previous work, Zheng et al. (Zheng et al., 2023a) proposed Pride, which simplify the problem by assuming that \( f(\cdot) \) is a linear multiplication, i.e.,

\[
P_{ob}(t_i|I, X_0) \propto Z_{I,X_0}^{-1} P_{pr}(t_i|I, X_0) \times P_{de}(t_i|I, X_0)
\]

where \( Z_{I,X_0}^{-1} \) is the normalization item. Zheng et al. (Zheng et al., 2023a) select a subset of test samples and then use the average observed probability distributions from different arrangements as the prior estimates \( P_{pr}(t_i) \). The debiasing is then performed using the following equation:

\[
P_{de}(t_i|I, X_0) \propto P_{ob}(t_i|I, X_0)/\bar{P}_{pr}(t_i)
\]

Although Pride is effective, its simplified assumption overlooks the complex relationships between probability distributions, leading to suboptimal performance.

In this paper, considering the complexity of \( f(\cdot) \), we do not attempt to directly create a precise mathematical function of \( f(\cdot) \). Instead, we focus on determining a calibration function \( g(\cdot) \), which can map the observed probabilities to an unbiased probability distribution, i.e.,

\[
P_{de}(t_i|I, X_0) = g(P_{ob}(t_i|I, X_0))
\]

3.2 Optimization Objective

In this section, we reformulate the debiasing problem as an optimization task, with the unbiased probability distribution serving as the optimization objective. Intuitively, an unbiased evaluator should provide consistent judgments even when the option position or ID tokens are swapped. Specifically, in pairwise comparisons, there are four possible combinations of positions and ID tokens:

\[
X_0 = [(t_1, o_1); (t_2, o_2)], X_1 = [(t_2, o_2); (t_1, o_1)]
\]
Figure 2: Four different types of combinations. \( t_1/t_2 \) represents the option IDs (A/B), while \( o_1/o_2 \) denotes the corresponding option contents.

\[
X_2 = [(t_1, o_2); (t_2, o_1)], X_3 = [(t_2, o_1); (t_1, o_2)]
\]
(6)

In Figure 2, we present the relationship among these four combinations. An unbiased evaluator can accurately determine the correct option context, regardless of changes in option orders (Swap Positions) or option ID tokens (Swap Tokens). Suppose that the ground truth is \( o_1 \), the evaluator should satisfy the following conditions:

\[
P_{de}(t_1|I, X_0) = P_{de}(t_1|I, X_1) = P_{de}(t_2|I, X_2)
\]
(7)

\[
P_{de}(t_2|I, X_2) = P_{de}(t_2|I, X_3) = P_{de}(t_1|I, X_0)
\]
(8)

Since Equations (7) and Equations (8) are duals, we only need to select one as the optimization objective. Also, we can simply normalize the original token prediction probabilities, ensuring that the sum of the probabilities for outputs \( t_1 \) and \( t_2 \) equals 100%, i.e.,

\[
P_{de}(t_1|I, X_0) = 1 - P_{de}(t_2|I, X_0)
\]
(9)

With the above reasoning, we formulate the de-biasing problem on \( K \) samples as follows:

\[
\min_{g(\cdot)} \sum_{i=1}^{K} [g(s_0^i) + g(s_2^i) - 1]^2 + [g(s_0^i) - g(s_1^i)]^2 - \lambda [g(s_0^i) - g(s_2^i)]^2
\]
(10)

\[
s.t. s_j^i = P_{ob}(t_1|I, X_j), j = 0, 1, 2, i = 1, ..., K.
\]
(11)

where \( g(\cdot) \) is the mapping function for the probability of token \( t_1 \), \( G \) denotes the solution space of \( g(\cdot) \). \( \lambda \) is a hyper-parameter. For each option ID token, a corresponding mapping function \( g(\cdot) \) is defined. In the following process, we use \( g(\cdot) \) as an example. In Equations (10), the first term ensures consistent judgments when option ID tokens are swapped. The second term aims to maintain consistent judgment when option positions are exchanged. The third term serves as a regularization term, which prevents convergence to the trivial solution \( g(\cdot) = 0.5 \).

3.3 Non-parametric Order-Preserving Algorithm (NOA)

The optimization problem presented in Equation (10) is an NP problem, featuring an extensive solution space \( G \). Furthermore, the absence of explicit labels prevents us from employing supervised methods to determine \( g(\cdot) \).

To address these limitations, we propose a non-parametric order-preserving algorithm called NOA. Non-parametric methods do not rely on specific model assumptions, making them well-suited for handling high-dimensional data or complex functions. NOA searches for the optimal solution by directly evaluating the output of the calibration function, eliminating the need for explicit labels or precise mathematical modeling.

To narrow the solution space \( G \), we assume that the mapping function \( g(\cdot) \) is order-preserving for the same ID token. This assumption, widely and implicitly applied in previous work (Zheng et al., 2023a), rests on the premise that the prior distribution \( P_{pr} \) reflects the LLM’s inherent bias toward certain option ID tokens, which remains conditionally independent of the unbiased probability distribution \( P_{de} \). Intuitively, for a given LLM, the partial order relationship under the same prior bias should remain consistent, meaning higher observed probabilities generally correspond to higher unbiased probabilities for the same ID token.

Specifically, we first collect an estimation set with \( K \) samples. Each sample is processed by swapping ID tokens and swapping positions, resulting in three probabilities \( s_0 \) (default output), \( s_1 \) (swap positions), and \( s_2 \) (swap ID tokens). The probabilities from all samples are combined into a set \( S = \{ s_0^i, s_1^i, s_2^i | i \in 1, ..., K \} \). Then, we sort \( S \) in ascending order to form a sequence \( z_1 \leq z_2 \leq ... \leq z_{M-1} \), where \( M = 3K + 1 \). We then append boundary conditions to the sorted sequence by defining \( z_0 = 0 \) and \( z_M = 1 \), producing the complete sequence \( Z = \{ z_0, z_1, ..., z_{M-1}, z_M \} \).

To optimize the model, we introduce a set of parameters \( d_k \) (\( k = 0, 1, 2, ..., M \)) initialized to the values of \( z_k \). These parameters will be optimized
during the process. Then, we define the mapping function \( g(\cdot) \) using the softmax-like expression:

\[
g(z_k) = \frac{\sum_{i=0}^k exp(d_i)}{\sum_{i=0}^M exp(d_i)}
\] (12)

\( g(\cdot) \) is a discrete mapping function with parameters \( d_k \), which satisfies the constraint of order preservation. We employ gradient descent methods to iteratively update the parameters \( d_k \). The update rule is given by:

\[
d_k^{(new)} = d_k^{(old)} - \gamma \frac{\partial L}{\partial d_k}
\] (13)

where \( \gamma \) is the learning rate, \( L = [g(s_0^i) + g(s_2^i) - 1]^2 + [g(s_0^j) - g(s_1^j)]^2 - \lambda [g(s_0^i) - g(s_2^i)]^2 \). This iterative process allows the parameters to converge toward the optimal values that minimize the loss, thereby reducing the bias in the probability distribution.

For \( \frac{\partial L}{\partial d_k} \), we derive the following equation. The detailed derivation process can be found in Appendix C.

\[
\begin{align*}
\frac{\partial L}{\partial d_k} &= (2 \left[ g(s_0^i) + g(s_2^i) - 1 \right] + 2 \left[ g(s_0^i) - g(s_1^i) \right]) \frac{\partial g(s_0^i)}{\partial d_k} \\
&\quad + (-2 \left[ g(s_0^i) - g(s_1^i) \right]) \frac{\partial g(s_1^i)}{\partial d_k} \\
&\quad + (2 \left[ g(s_0^i) + g(s_2^i) - 1 \right]) \frac{\partial g(s_2^i)}{\partial d_k} \\
&\quad - (2\lambda \left[ g(s_0^i) - g(s_2^i) \right]) \frac{\partial g(s_2^i)}{\partial d_k}
\end{align*}
\] (14)

\[
\frac{\partial g(z_j)}{\partial d_k} = \begin{cases}
-\frac{\sum_{i=0}^j \exp(d_i) \exp(d_k)}{\left( \sum_{i=0}^M \exp(d_i) \right)^2} & (j < k) \\
\frac{\exp(d_k) \left( \sum_{i=0}^M \exp(d_i) - \sum_{i=0}^j \exp(d_i) \right)}{\left( \sum_{i=0}^M \exp(d_i) \right)^2} & (j \geq k)
\end{cases}
\] (15)

We note that there are infinite solutions that satisfy the optimization problem. This is because any constant change in the value of \( d_k \) does not affect the relative values in the exponential terms of Equation (12). To obtain a unique solution, we apply the normalization constraint \( \sum_{i=0}^M d_i = 0 \) after each iteration. The optimization proceeds until a convergence criterion is met, such as the loss function \( L \) reaching a minimum threshold or the parameter updates becoming sufficiently small.

After the solution process converges, we obtain the sample points \( Z = \{z_1, ..., z_{M-1}\} \) and their corresponding calibrated values \( y = \{g(z_1), ..., g(z_{M-1})\} \). For sample points not included in \( Z \), we use existing sample points to learn the continuous calibration function \( g^*(\cdot) \). The goal is to identify a set of non-decreasing piecewise linear functions that minimize the sum of squared deviations between the estimated values and the calibrated values of the samples. Specifically, we fit the calibration values by minimizing the following objective function:

\[
\min \sum_{i=1}^{M-1} w_i (g(z_i) - g^*(z_i))^2
\] (16)

\[
s.t. z_1 \leq z_2 ... \leq z_{M-1}
\] (17)

\[
\sum_{i=1}^{M-1} w_i = 1, w_i \geq 0
\] (18)

The above problem is a weighted least squares quadratic programming problem. We apply the Pool Adjacent Violators Algorithm (PAVA) (Zadora et al., 2014) to derive the continuous calibration function \( g^*(\cdot) \).

It is worth noting that CalibraEval does not require explicit labels and can be executed during inference with minimal computational cost. The calibration function can be calculated after observing all test samples or by utilizing a subset of samples. The entire process of CalibraEval is summarized in Algorithm 1 in the Appendix.

4 EXPERIMENT SETUP

4.1 Datasets and Metrics

We conduct experiments on three representative benchmarks: RewardBench (Lambert et al., 2024), MTBench (Zheng et al., 2023b), and PreferenceBench (Kim et al., 2024). Due to space limitations, their detailed descriptions and statistics are provided in Appendix D.1.

In the evaluation, we primarily utilize reference-free metrics to measure the consistency of model evaluations. We compute Fleiss’s Kappa (Falotico and Quatto, 2015) and intraclass correlation coefficient (ICC) (Bartko, 1966) between the evaluation results obtained after swapping option ID tokens and option positions. We report two specific ICC metrics: ICC(2,k) and ICC(3,k) in this paper.

For the reference-based evaluation, we report the standard deviation of recalls (RStd) and accuracy. Following Zheng et al. (Zheng et al., 2023a), the balance of recalls serves as an effective measure of the extent of selection bias. A greater imbalance in recalls signifies a more pronounced selection bias. In addition, MTBench includes “tie” options assessed by human evaluators. We exclude all “tie”
options when calculating the reference-based metrics. In Appendix D.2, we provide the details of evaluation metrics.

4.2 Baselines
We employ the following methods as our baselines. Since CalibraEval is a label-free method, we do not compare it with supervised methods.

• Debiasing Instruct (DI) is implemented by including the instruction: “Avoid any position bias and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain tokens of the option. Be as objective as possible”.

• Contextual Calibration (CC) (Zhao et al., 2021) involves applying an affine transformation to model outputs in order to calibrate LLM predictions. It estimates the bias for each option tokens by requesting its prediction with a prompt alongside a content-free input, such as “N/A”.

• Domain-context Calibration (DC) (Fei et al., 2023) is designed to minimize label bias in in-context learning. It estimates a contextual prior by using a random in-domain sequence, achieving state-of-the-art results.

• Pride (Zheng et al., 2023a) estimates the model’s prior bias toward option ID token by reorganizing the test samples and then removes this bias using a division operation.

4.3 Implementation Details
We evaluate six models from three LLM-families including: Llama-3-8B (Touvron et al., 2023), Llama-3.1-8B (Touvron et al., 2023), Qwen-14B (Bai et al., 2023), Qwen-72B (Bai et al., 2023), ChatGPT (OpenAI, 2023), and GPT-4o (OpenAI, 2023). The version of ChatGPT used is gpt-3.5-turbo-1106. The estimation set used to derive the calibration function can be constructed either by sampling from the test data or by using the entire test set without the gold labels. For a fair comparison, we opted for the latter approach. In the main experiment, all baselines used the full test data as the prior estimation set. For CC, we use the predefined token “N/A” to replace the option contents, generating content-free input. For DC, we randomly extract words from the task corpus to construct the content-free input. Moreover, we set \( \lambda = 0.5 \) and \( \gamma = 10 \). We employ the batch gradient descent method with a batch size of 32. The optimization process stops when the parameters change range is less than the threshold \( \epsilon \) i.e., \( \sum_{i=1}^N \triangle d_i < \epsilon \). The \( \epsilon \) is set to 0.001. All experiments presented in this paper are conducted on 8 NVIDIA Tesla A100 GPUs. All the prompts used in this paper can be found in Appendix F.

5 Experiment Result

5.1 Main Results
The performance comparison of CalibraEval with baselines is presented in Table 1. Based on the experimental results, we can draw the following conclusions.

• Debiasing Instruct does not consistently lead to improved or more robust performance, as its effectiveness is limited by the instruction-following capabilities of LLMs and the nature of tasks. In some cases, adding debiasing instructions may even result in consistency degradation. Consequently, relying solely on instructions is not a reliable approach for effective debiasing.

• CC and DC are originally designed to mitigate label bias in in-context learning. Therefore, their estimated priors may not accurately reflect the inherent selection bias in LLMs-as-Judges, leading to suboptimal debiasing performance and difficulties in interpretation.

• When applied to lower-capability LLMs, such as Llama-3-8B and Qwen-14B, Pride effectively estimates bias and improves consistency. However, its effectiveness diminishes with more advanced models (e.g., GPT4o). This limitation may arise from the simplified probabilistic relationships employed in Pride.

• CalibraEval consistently improves performance across various LLMs and tasks. On average, CalibraEval shows enhancements over all the baselines. Overall, CalibraEval is a versatile technique applicable to multiple evaluation tasks, delivering stable performance improvement. This also indicates that CalibraEval can effectively reduce selection bias in LLMs-as-judges, leading to more consistent and fair evaluation results.

Table 2 presents the performance of the reference-based metrics. Due to space constraints,
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">RewardBench</th>
    <th colspan="3">MTBench</th>
    <th colspan="3">PreferenceBench</th>
    <th colspan="3">Average</th>
  </tr>
  <tr>
    <th>Kappa</th><th>ICC(2,k)</th><th>ICC(3,k)</th>
    <th>Kappa</th><th>ICC(2,k)</th><th>ICC(3,k)</th>
    <th>Kappa</th><th>ICC(2,k)</th><th>ICC(3,k)</th>
    <th>Kappa</th><th>ICC(2,k)</th><th>ICC(3,k)</th>
  </tr>
  <tr>
    <td>Llama-3-8B</td><td>20.81</td><td>66.24</td><td>71.79</td><td>14.36</td><td>60.96</td><td>73.08</td><td>58.25</td><td>86.23</td><td>86.61</td><td>31.14</td><td>71.14</td><td>77.16</td>
  </tr>
  <tr>
    <td>DI</td><td>19.63</td><td>64.98</td><td>70.87</td><td>15.93</td><td>59.11</td><td>65.00</td><td>39.90</td><td>76.77</td><td>80.87</td><td>25.15</td><td>66.95</td><td>72.25</td>
  </tr>
  <tr>
    <td>CC</td><td>15.49</td><td>58.77</td><td>63.70</td><td>5.60</td><td>39.48</td><td>52.35</td><td>54.84</td><td>83.60</td><td>86.26</td><td>25.31</td><td>60.62</td><td>67.47</td>
  </tr>
  <tr>
    <td>DC</td><td>23.17</td><td>69.04</td><td>72.25</td><td>25.28</td><td>69.63</td><td>72.71</td><td>56.78</td><td>84.88</td><td>85.90</td><td>33.21</td><td>76.58</td><td>78.88</td>
  </tr>
  <tr>
    <td>Pride</td><td>27.65</td><td>72.72</td><td>73.77</td><td>27.38</td><td>72.77</td><td>74.50</td><td>37.01</td><td>85.49</td><td>86.33</td><td>37.35</td><td>76.83</td><td>78.20</td>
  </tr>
  <tr>
    <td>CalibraEval</td><td><b>30.32</b></td><td><b>86.51</b></td><td><b>86.66</b></td><td><b>28.63</b></td><td><b>75.45</b></td><td><b>76.80</b></td><td><b>58.54</b></td><td><b>88.17</b></td><td><b>89.43</b></td><td><b>39.16</b></td><td><b>83.38</b></td><td><b>84.30</b></td>
  </tr>
  <tr>
    <td>Llama-3-1B-8B</td><td>15.02</td><td>68.82</td><td>76.65</td><td>16.91</td><td>62.51</td><td>67.60</td><td>38.73</td><td>74.61</td><td>78.46</td><td>23.55</td><td>68.65</td><td>74.24</td>
  </tr>
  <tr>
    <td>DI</td><td><b>21.59</b></td><td>74.04</td><td>80.00</td><td>15.12</td><td>53.24</td><td>57.08</td><td>36.59</td><td>68.47</td><td>72.14</td><td>24.43</td><td>65.25</td><td>69.74</td>
  </tr>
  <tr>
    <td>CC</td><td>16.89</td><td>59.24</td><td>60.09</td><td>2.59</td><td>32.42</td><td>33.96</td><td>40.80</td><td>76.69</td><td>78.44</td><td>20.09</td><td>56.12</td><td>57.50</td>
  </tr>
  <tr>
    <td>DC</td><td>23.17</td><td>72.89</td><td>76.22</td><td>16.02</td><td>65.86</td><td>68.80</td><td>41.61</td><td>77.23</td><td>78.81</td><td>26.55</td><td>71.68</td><td>74.18</td>
  </tr>
  <tr>
    <td>Pride</td><td>14.63</td><td>66.62</td><td>76.02</td><td>17.98</td><td>63.76</td><td>67.51</td><td>38.58</td><td>76.84</td><td>79.35</td><td>23.73</td><td>69.07</td><td>74.29</td>
  </tr>
  <tr>
    <td>CalibraEval</td><td>20.97</td><td>83.23</td><td><b>86.14</b></td><td><b>19.12</b></td><td><b>66.26</b></td><td><b>69.26</b></td><td><b>41.56</b></td><td><b>81.56</b></td><td><b>82.84</b></td><td><b>27.61</b></td><td><b>74.93</b></td><td><b>79.57</b></td>
  </tr>
  <tr>
    <td>Qwen-14B</td><td>19.69</td><td>63.41</td><td>66.86</td><td>17.53</td><td>54.43</td><td>65.11</td><td>48.94</td><td>84.90</td><td>85.78</td><td>28.72</td><td>67.58</td><td>73.58</td>
  </tr>
  <tr>
    <td>DI</td><td>11.90</td><td>50.46</td><td>53.20</td><td>4.98</td><td>36.83</td><td>46.02</td><td>30.76</td><td>74.41</td><td>82.95</td><td>15.88</td><td>53.90</td><td>60.72</td>
  </tr>
  <tr>
    <td>CC</td><td>-1.62</td><td>-4.71</td><td>-4.93</td><td>-5.48</td><td>18.08</td><td>31.20</td><td>40.35</td><td>73.79</td><td>75.56</td><td>11.08</td><td>29.05</td><td>33.94</td>
  </tr>
  <tr>
    <td>DC</td><td>16.04</td><td>45.10</td><td>45.26</td><td><b>25.51</b></td><td><b>64.37</b></td><td><b>66.46</b></td><td><b>48.15</b></td><td><b>82.29</b></td><td><b>84.52</b></td><td><b>29.90</b></td><td><b>63.92</b></td><td><b>65.41</b></td>
  </tr>
  <tr>
    <td>Pride</td><td>24.73</td><td>67.46</td><td>83.80</td><td>13.00</td><td>51.73</td><td>57.39</td><td>52.79</td><td>90.86</td><td>91.20</td><td>30.17</td><td>70.02</td><td>72.30</td>
  </tr>
  <tr>
    <td>CalibraEval</td><td><b>26.40</b></td><td><b>75.75</b></td><td><b>76.11</b></td><td>17.68</td><td>53.64</td><td>63.43</td><td><b>62.91</b></td><td><b>92.56</b></td><td><b>92.57</b></td><td><b>35.66</b></td><td><b>73.98</b></td><td><b>77.37</b></td>
  </tr>
  <tr>
    <td>Qwen-72B</td><td>78.28</td><td>92.77</td><td>93.13</td><td>71.35</td><td>90.33</td><td>91.16</td><td>82.77</td><td>94.78</td><td>94.90</td><td>77.47</td><td>92.63</td><td>93.06</td>
  </tr>
  <tr>
    <td>DI</td><td>77.12</td><td>92.27</td><td>92.63</td><td>68.17</td><td>89.91</td><td>91.11</td><td>83.51</td><td>95.09</td><td>93.11</td><td>76.27</td><td>92.16</td><td>92.88</td>
  </tr>
  <tr>
    <td>CC</td><td>69.23</td><td>88.71</td><td>89.40</td><td>70.40</td><td>90.29</td><td>90.72</td><td>78.08</td><td>92.63</td><td>93.24</td><td>72.65</td><td>89.54</td><td>91.29</td>
  </tr>
  <tr>
    <td>DC</td><td>66.61</td><td>87.03</td><td>87.90</td><td>64.59</td><td>87.46</td><td>88.45</td><td>74.03</td><td>91.01</td><td>92.01</td><td>68.41</td><td>88.50</td><td>89.45</td>
  </tr>
  <tr>
    <td>Pride</td><td>78.64</td><td>92.99</td><td>93.30</td><td>71.50</td><td>90.54</td><td>91.27</td><td>83.44</td><td>94.89</td><td>95.00</td><td>77.86</td><td>92.81</td><td>93.19</td>
  </tr>
  <tr>
    <td>CalibraEval</td><td><b>82.80</b></td><td><b>95.47</b></td><td><b>95.75</b></td><td><b>71.88</b></td><td><b>95.70</b></td><td><b>96.71</b></td><td><b>85.25</b></td><td><b>97.56</b></td><td><b>97.57</b></td><td><b>79.98</b></td><td><b>96.24</b></td><td><b>96.68</b></td>
  </tr>
  <tr>
    <td>ChatGPT</td><td>20.08</td><td>62.67</td><td>70.75</td><td>37.25</td><td>73.90</td><td>76.92</td><td>64.62</td><td>87.63</td><td>87.81</td><td>40.65</td><td>74.73</td><td>78.49</td>
  </tr>
  <tr>
    <td>DI</td><td>24.52</td><td>70.23</td><td>71.58</td><td>24.33</td><td>66.80</td><td>67.69</td><td>56.00</td><td>82.90</td><td>84.89</td><td>34.95</td><td>73.31</td><td>74.72</td>
  </tr>
  <tr>
    <td>CC</td><td>24.28</td><td>57.25</td><td>58.23</td><td>23.71</td><td>64.06</td><td>71.91</td><td>61.63</td><td>86.18</td><td>86.38</td><td>36.54</td><td>69.16</td><td>72.17</td>
  </tr>
  <tr>
    <td>DC</td><td>27.94</td><td>66.09</td><td>70.16</td><td>16.85</td><td>68.73</td><td>70.73</td><td>51.92</td><td>82.83</td><td>83.38</td><td>33.37</td><td>74.86</td><td>75.93</td>
  </tr>
  <tr>
    <td>Pride</td><td>28.25</td><td>70.38</td><td>73.16</td><td>39.02</td><td>76.61</td><td>76.64</td><td>47.56</td><td>87.82</td><td>87.82</td><td>43.97</td><td>78.18</td><td>79.53</td>
  </tr>
  <tr>
    <td>CalibraEval</td><td><b>32.02</b></td><td><b>77.25</b></td><td><b>77.60</b></td><td><b>39.71</b></td><td><b>79.07</b></td><td><b>79.85</b></td><td><b>55.52</b></td><td><b>87.77</b></td><td><b>87.92</b></td><td><b>45.75</b></td><td><b>81.36</b></td><td><b>81.79</b></td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>82.57</td><td>94.83</td><td>94.89</td><td>72.42</td><td>92.99</td><td>93.20</td><td>79.42</td><td>93.50</td><td>94.11</td><td>78.14</td><td>93.77</td><td>94.07</td>
  </tr>
  <tr>
    <td>DI</td><td>78.57</td><td>93.72</td><td>94.01</td><td><b>74.21</b></td><td>93.35</td><td>93.50</td><td><b>79.97</b></td><td>94.48</td><td>94.93</td><td>77.58</td><td>93.85</td><td>94.15</td>
  </tr>
  <tr>
    <td>CC</td><td>81.38</td><td>94.47</td><td>94.48</td><td>67.10</td><td>90.30</td><td>90.78</td><td>77.56</td><td>92.76</td><td>93.43</td><td>75.35</td><td>92.51</td><td>92.90</td>
  </tr>
  <tr>
    <td>DC</td><td>76.89</td><td>92.28</td><td>92.35</td><td>68.94</td><td>90.94</td><td>91.64</td><td>70.48</td><td>89.30</td><td>90.02</td><td>72.10</td><td>90.84</td><td>91.34</td>
  </tr>
  <tr>
    <td>Pride</td><td>82.53</td><td>94.94</td><td>94.98</td><td>70.34</td><td>92.35</td><td>92.74</td><td>79.74</td><td>93.62</td><td>94.20</td><td>77.54</td><td>93.64</td><td>93.97</td>
  </tr>
  <tr>
    <td>CalibraEval</td><td><b>83.25</b></td><td><b>96.25</b></td><td><b>96.27</b></td><td>72.60</td><td><b>95.04</b></td><td><b>95.20</b></td><td>79.73</td><td><b>97.29</b></td><td><b>97.60</b></td><td><b>78.53</b></td><td><b>96.19</b></td><td><b>96.36</b></td>
  </tr>
</table>

Table 1: Performance comparison between CalibraEval and baselines. We report the Fleiss’ Kappa (%) and Intraclass Correlation Coefficient (%) for each dataset and the averages. The row corresponding to the model name represents the default results without applying any debiasing methods. Best performances are marked bold.

<table>
  <tr>
    <th>Model</th>
    <th>RewardBench<br>Rstd, Acc (%)</th>
    <th>MTBench<br>Rstd, Acc (%)</th>
    <th>Preference Bench<br>Rstd, Acc (%)</th>
  </tr>
  <tr>
    <td>Llama-3-8B</td>
    <td>15.01, 65.79</td>
    <td>16.42, 67.08</td>
    <td>8.34, 83.24</td>
  </tr>
  <tr>
    <td>Pride</td>
    <td>7.51, 66.54</td>
    <td>11.64, 70.63</td>
    <td>4.35, 83.24</td>
  </tr>
  <tr>
    <td>CalibraEval</td>
    <td><b>6.48</b>, <b>68.12</b></td>
    <td><b>5.22</b>, <b>70.63</b></td>
    <td><b>3.42</b>, <b>83.98</b></td>
  </tr>
  <tr>
    <td>Qwen-14B</td>
    <td>11.63, 61.31</td>
    <td>17.24, 65.61</td>
    <td>11.99, 80.65</td>
  </tr>
  <tr>
    <td>Pride</td>
    <td>4.16, 64.09</td>
    <td>16.31, 65.29</td>
    <td>7.38, 83.55</td>
  </tr>
  <tr>
    <td>CalibraEval</td>
    <td><b>2.72</b>, <b>64.35</b></td>
    <td><b>6.64</b>, <b>65.29</b></td>
    <td><b>5.12</b>, <b>83.88</b></td>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>16.79, 65.27</td>
    <td>7.66, 72.67</td>
    <td>3.04, 85.61</td>
  </tr>
  <tr>
    <td>Pride</td>
    <td>8.54, 66.36</td>
    <td>6.01, 72.86</td>
    <td>3.51, 85.68</td>
  </tr>
  <tr>
    <td>CalibraEval</td>
    <td><b>5.51</b>, <b>67.13</b></td>
    <td><b>5.20</b>, <b>72.98</b></td>
    <td><b>2.82</b>, <b>85.98</b></td>
  </tr>
</table>

Table 2: Results of reference-based metrics. ↓ indicates that lower values correspond to better performance.

we only report the experimental results for Llama-3-8B, Qwen-14B, and ChatGPT, while the complete results are available in Appendix E. For a fair comparison, we report the average values of Rstd and Accuracy under the conditions of swapping option positions and option IDs. Across the average performance of the three datasets, CalibraEval consistently achieves lower Rstd and higher accuracy, outperforming other baselines. Surprisingly, although this is not the original intent, CalibraEval frequently improves accuracy. We believe this may indicate that selection bias influences the model’s judgments, leading to reduced accuracy. Therefore, effective bias mitigation methods can enhance the model’s performance in its evaluative role. Additionally, we found that lower Rstd is often associated with higher accuracy. The more pronounced the debiasing effect, the more significant the performance improvement. For example, on RewardBench, ChatGPT’s Rstd decreased from 16.79 to 5.51, while its accuracy increased from 65.27 to 67.13. Overall, CalibraEval not only enhances the reliability of model evaluations but also unlocks the potential for these LLMs to perform optimally in various tasks.

5.2 Robustness Analysis

In this section, we conduct additional experiments to further validate the effectiveness of CalibraEval across diverse scenarios. Due to the high cost of GPT-4o, we opt for Qwen-72B and ChatGPT on the RewardBench for the following experiments. Unless otherwise stated, the ICC for subsequent experiments is ICC(2,k). Due to space limitations, we present the model’s robustness regarding Different ID tokens and the number of in-context learning examples in Appendix E.1 and Appendix E.2.

5.2.1 Different prompt templates

We conduct experiments on four distinct prompt templates (see Appendix F for details). Figure 3 shows performance comparisons on RewardBench. We observed that model outputs without bias correction exhibit low consistency and high variance.
(a) ChatGPT-ICC    (b) ChatGPT-Kappa

(c) Qwen72B-ICC    (d) Qwen72B-Kappa

Figure 3: Performance comparison across different prompt templates.

<table>
  <tr>
    <th>Model</th>
    <th>Kappa</th>
    <th>ICC(2.k)</th>
    <th>ICC(3.k)</th>
    <th>Rstd.</th>
    <th>Acc.</th>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>20.08</td>
    <td>62.67</td>
    <td>70.75</td>
    <td>16.79</td>
    <td>65.27</td>
  </tr>
  <tr>
    <td>w. \( L_1 \)</td>
    <td>26.78</td>
    <td>72.04</td>
    <td>73.66</td>
    <td>10.32</td>
    <td>65.33</td>
  </tr>
  <tr>
    <td>w. \( L_2 \)</td>
    <td>27.38</td>
    <td>72.73</td>
    <td>75.99</td>
    <td>8.64</td>
    <td>66.04</td>
  </tr>
  <tr>
    <td>w. both</td>
    <td><b>32.02</b></td>
    <td><b>77.25</b></td>
    <td><b>77.60</b></td>
    <td><b>5.50</b></td>
    <td><b>67.13</b></td>
  </tr>
  <tr>
    <td>Qwen-72B</td>
    <td>78.33</td>
    <td>92.77</td>
    <td>93.13</td>
    <td>4.17</td>
    <td>87.37</td>
  </tr>
  <tr>
    <td>w. \( L_1 \)</td>
    <td>81.95</td>
    <td>94.94</td>
    <td>95.04</td>
    <td>2.42</td>
    <td>87.78</td>
  </tr>
  <tr>
    <td>w. \( L_2 \)</td>
    <td>81.32</td>
    <td>93.77</td>
    <td>95.52</td>
    <td>2.78</td>
    <td>87.74</td>
  </tr>
  <tr>
    <td>w. both</td>
    <td><b>82.80</b></td>
    <td><b>95.47</b></td>
    <td><b>95.75</b></td>
    <td><b>0.94</b></td>
    <td><b>88.06</b></td>
  </tr>
</table>

Table 3: Ablation study on RewardBench. Best results are marked bold.

While Pride improves consistency, it still exhibited considerable variance. In contrast, CalibraEval demonstrates substantial performance enhancement while maintaining low variance, indicating its consistent effectiveness across different prompt templates.

5.3 Ablation Studies

To better illustrate the rationality and effectiveness of model design, we conduct two ablation experiments. We first analyze the effectiveness of well-defined optimization objectives. Specifically, we consider two variants. The first variant focuses solely on ensuring that the model maintains consistent judgments after swap ID tokens, i.e.,

\[
L_1 = \arg\min_{g \in G} \sum_{i=1}^K |g(s_0^i) + g(s_2^i) - 1|^2 - \lambda |g(s_0^i) - g(s_2^i)|^2
\]

The other variant focuses on ensuring that the model maintains consistent judgments after position exchanges, represented as:

\[
L_2 = \arg\min_{g \in G} \sum_{i=1}^K |g(s_0^i) - g(s_1^i)|^2 - \lambda |g(s_0^i) - 0.5|^2
\]

Since this variant does not involve \( s_2^i \), the regularization term is modified to \( |g(s_0^i) - 0.5|^2 \) to prevent the model from converging to a trivial solution.

Table 3 illustrates the impact of different optimization objectives. Both objectives contribute to the calibration benefits observed. When the model is significantly influenced by position bias, the improvements from \( L_2 \) are more substantial. Conversely, when token bias is more prevalent, \( L_1 \) leads to better improvements. The combination of both objectives, which defines our CalibraEval optimization goal, achieves optimal performance. These experiments validate the effectiveness of our chosen optimization settings.

In Figure 4, we further test the impact of the estimation set size on the performance. We randomly sampled a certain proportion of test data to estimate the calibration function, which is then applied to debias the entire test set. We found that increasing the size of the estimation set can better enhance consistency. Additionally, a smaller estimation set can also effectively support CalibraEval in reducing bias. For ChatGPT, using only 10% of the data resulted in improvements of over 85% compared to the full dataset. Overall, even with limited data, CalibraEval can still produce reliable calibration functions.

(a) ChatGPT    (b) Qwen-72B

Figure 4: Performance of CalibraEval across different estimate set sizes. “Percentage” refers to the proportion of the test set selected for use as the estimation set.

6 Conclusion

In this paper, we propose CalibraEval to mitigate the selection bias present in LLM-as-judges. We reformulate the debiasing problem as an optimization problem and utilize the characteristics of unbiased evaluators as our optimization objectives. Moreover, we propose the non-parametric order-preserving algorithm (NOA) to determine the calibration function. Experiments involving six LLMs across three representative datasets demonstrate that CalibraEval effectively reduces selection bias while enhancing accuracy. We argue that mitigating selection bias is essential for developing more reliable LLM evaluators. In the future, we plan to investigate additional biases in LLM-as-judges applications to create even more robust and trustworthy automated evaluations of large models.
References

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.

John J Bartko. 1966. The intraclass correlation coefficient as a measure of reliability. Psychological reports, 19(1):3–11.

Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201.

Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or llms as the judge? a study on judgement biases. arXiv preprint arXiv:2402.10669.

Qian Dong, Yiding Liu, Qingyao Ai, Zhijing Wu, Haitao Li, Yiqun Liu, Shuaqiang Wang, Dawei Yin, and Shaoping Ma. 2024a. Unsupervised large language model alignment for information retrieval via contrastive feedback. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 48–58, New York, NY, USA. Association for Computing Machinery.

Qian Dong, Yiding Liu, Qingyao Ai, Zhijing Wu, Haitao Li, Yiqun Liu, Shuaqiang Wang, Dawei Yin, and Shaoping Ma. 2024b. Unsupervised large language model alignment for information retrieval via contrastive feedback.

Rosa Falotico and Piero Quatto. 2015. Fleiss’ kappa statistic without paradoxes. Quality & Quantity, 49:463–470.

Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. Mitigating label biases for in-context learning. arXiv preprint arXiv:2305.19148.

Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations.

Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535.

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787.

Haitao Li, Qingyao Ai, Jia Chen, Qian Dong, Zhijing Wu, Yiqun Liu, Chong Chen, and Qi Tian. 2024. Blade: Enhancing black-box large language models with small domain-specific models.

Ruosen Li, Teerth Patel, and Xinya Du. 2023a. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint arXiv:2307.02762.

Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. 2023b. Split and merge: Aligning position biases in large language model based evaluators. arXiv preprint arXiv:2310.01432.

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.

Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulic, Anna Korhonen, and Nigel Collier. 2024. Aligning with human judgement: The role of pairwise preference in large language model evaluators. arXiv preprint arXiv:2403.16950.

OpenAI. 2023. Gpt-4 technical report.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.

Pouya Pezeshkpour and Estevam Hruschka. 2023. Large language models sensitivity to the order of options in multiple-choice questions. arXiv preprint arXiv:2308.11483.

Vyas Raina, Adian Liusie, and Mark Gales. 2024. Is llm-as-a-judge robust? investigating universal adversarial attacks on zero-shot llm assessment. arXiv preprint arXiv:2402.14016.

Lin Shi, Weicheng Ma, and Soroush Vosoughi. 2024. Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms. arXiv preprint arXiv:2406.07791.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators.

Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263–27277.

Grzegorz Zadora, Agnieszka Martyna, Daniel Ramos, and Colin Aitken. 2014. Pool adjacent violators algorithm.

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pages 12697–12706. PMLR.

Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023a. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023b. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. 2023. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. arXiv preprint arXiv:2309.17249.

Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631.
A Limitation

We acknowledge several limitations in this study and aim to address them in future work. First, CalibraEval limits selection bias to position and token swapping scenarios. However, in real-world situations, other types of bias, such as context and trend biases, may also be present. In the future, we will investigate the factors that influence these additional biases and explore methods to mitigate them. Futhermore, CalibraEval relies on pairwise comparisons. Although pairwise comparisons can be extended to listwise comparisons, this increases the time complexity. In the future, we aim to broaden the scope of CalibraEval to accommodate a wider range of scenarios.

B Legal and Ethical Considerations

Our research prioritizes ethical considerations throughout the development process. We ensure that our work does not involve any discriminatory content or personal privacy violations. We take great care in using only publicly available, authorized datasets, and we avoid incorporating any data that could lead to biased or harmful outcomes. All models, datasets, and code related to this research are publicly available, fostering transparency and enabling further research and validation by the community.

C SUPPLEMENTARY PROOF

To compute \( \frac{\partial L}{\partial d_k} \) for the given loss function:

\[
[g(s_0^i)+g(s_2^i)-1]^2+[g(s_0^i)-g(s_1^i)]^2-\lambda[g(s_0^i)-g(s_2^i)]^2 \tag{21}
\]

Applying the chain rule, we have:

\[
\frac{\partial L}{\partial d_k} = \frac{\partial L}{\partial g (s_0^i)} \frac{\partial g (s_0^i)}{\partial d_k} + \frac{\partial L}{\partial g (s_1^i)} \frac{\partial g (s_1^i)}{\partial d_k} + \frac{\partial L}{\partial g (s_2^i)} \frac{\partial g (s_2^i)}{\partial d_k} \tag{22}
\]

For \( g(s_0^i) \):

\[
\frac{\partial L}{\partial g (s_0^i)} = 2 [g (s_0^i) + g (s_2^i) - 1] + 2 [g (s_0^i) - g (s_1^i)] \tag{23}
\]

For \( g(s_1^i) \):

\[
\frac{\partial L}{\partial g (s_1^i)} = -2 [g (s_0^i) - g (s_1^i)] \tag{24}
\]

For \( g(s_2^i) \):

\[
\frac{\partial L}{\partial g (s_2^i)} = 2 [g (s_0^i) + g (s_2^i) - 1] - 2\lambda [g (s_0^i) - g (s_2^i)] \tag{25}
\]

Then, We substitute the derivatives back into the equation 22:

\[
\begin{align*}
\frac{\partial L}{\partial d_k} &= (2 [g (s_0^i) + g (s_2^i) - 1] + 2 [g (s_0^i) - g (s_1^i)]) \frac{\partial g (s_0^i)}{\partial d_k} \\
&\quad + (-2 [g (s_0^i) - g (s_1^i)]) \frac{\partial g (s_1^i)}{\partial d_k} \\
&\quad + (2 [g (s_0^i) + g (s_2^i) - 1] - 2\lambda [g (s_0^i) - g (s_2^i)]) \frac{\partial g (s_2^i)}{\partial d_k}
\end{align*}
\]
(26)

Next, using the quotient rule, the derivative of \( g(z_j) \) with respect to \( d_k \) is:

\[
\frac{\partial g (z_j)}{\partial d_k} = \frac{\partial}{\partial d_k} \left( \frac{\sum_{i=0}^j \exp (d_i)}{\sum_{i=0}^M \exp (d_i)} \right) \tag{27}
\]

For \( j < k \), \( d_k \) affects the denominator:

\[
\frac{\partial g (z_j)}{\partial d_k} = \frac{-\sum_{i=0}^j \exp (d_i) \exp (d_k)}{\left( \sum_{i=0}^M \exp (d_i) \right)^2} \tag{28}
\]

For \( j \geq k \), \( d_k \) affects both the numerator and the denominator:

\[
\frac{\partial g (z_j)}{\partial d_k} = \frac{\exp (d_k) \left( \sum_{i=0}^M \exp (d_i) - \sum_{i=0}^j \exp (d_i) \right)}{\left( \sum_{i=0}^M \exp (d_i) \right)^2} \tag{29}
\]

The final formula is as follows.

\[
\frac{\partial g (z_j)}{\partial d_k} =
\begin{cases}
- \frac{\sum_{i=0}^j \exp (d_i) \exp (d_k)}{\left( \sum_{i=0}^M \exp (d_i) \right)^2} & (j < k) \\
\frac{\exp (d_k) \left( \sum_{i=0}^M \exp (d_i) - \sum_{i=0}^j \exp (d_i) \right)}{\left( \sum_{i=0}^M \exp (d_i) \right)^2} & (j \geq k)
\end{cases}
\]
(30)

D Details of Dataset and Metrics

D.1 Details of Datasets

We conduct experiments on three representative benchmarks: RewardBench, MTBench, and PreferenceBench.

• RewardBench (Lambert et al., 2024) is a benchmark dataset designed for evaluating reward models. It contains 2,985 prompt-choice-rejection trios across four task categories: Chat, Chat Hard, Safety, and Reasoning.

• MTBench (Zheng et al., 2023b) is a multi-turn response dataset. It contains 3,355 expert-level pairwise human preferences for responses, generated by 6 models for 80 MTBench questions.

• PreferenceBench (Kim et al., 2024) is a test set designed to assess the evaluation capabilities of LLMs, comprising 2,000 response pairs (classified as “win” or “lose”) and 200 evaluation criteria.
Table 4 presents the statistical of benchmarks. The average answer length of different options in each dataset is nearly identical, and the distribution of label categories is balanced. This design minimizes the potential influence of other biases on the evaluation results. Overall, the datasets exhibit a well-balanced difficulty distribution and are thoughtfully constructed, ensuring a fair and robust evaluation process.

D.2 Details of Metrics

D.2.1 Reference-Free Metrics

Fleiss’s Kappa is a statistical measure used to assess the reliability of agreement between multiple raters. It is calculated using the formula:

\[
K = \frac{P_o - P_e}{1 - P_e}
\] (31)

where \( P_o \) is the observed agreement among raters. \( P_e \) is the expected agreement by chance. The value of Kappa ranges from -1 to 1, where values closer to 1 indicate strong agreement among raters, values around 0 suggest no agreement beyond chance, and negative values indicate systematic disagreement.

Intraclass Correlation Coefficient (ICC) is a measure of reliability that assesses the consistency or agreement of measurements made by different raters or instruments. In this paper, we report two specific Intraclass Correlation Coefficient (ICC) metrics: ICC(2,k) and ICC(3,k). ICC(2,k) measures the consistency of ratings from multiple raters for the same set of subjects under a random effects model, while ICC(3,k) assesses the consistency of ratings from specific and fixed raters for the same subjects under a fixed effects model. Both are useful for measuring the reliability and consistency of ratings.

The ICC(2,k) is calculated using the formula:

\[
\text{ICC}(2, k) = \frac{\sigma_B^2 - \sigma_W^2}{\sigma_B^2 + (k-1)\sigma_W^2}
\] (32)

The ICC(3,k) is calculated using the formula:

\[
\text{ICC}(3, k) = \frac{\sigma_B^2 - \sigma_W^2}{\sigma_B^2 + k \cdot \sigma_W^2}
\] (33)

where \( \sigma_B^2 \) is the variance between the subjects. \( \sigma_W^2 \) is the variance within the subjects. \( k \) is the number of raters.

D.2.2 Reference-based Metrics

The standard deviation of recalls (RStd) quantifies the variability in recall scores across different evaluations. It is calculated using the formula:

\[
RStd = \sqrt{\frac{1}{N-1} \sum_{i=1}^{N} (R_i - \bar{R})^2}
\] (34)

where \( R_i \) is the recall for the \( i \)-th evaluation. \( \bar{R} \) is the mean recall across all evaluations. \( N \) is the total number of evaluations.

Accuracy is a widely used evaluation metric that measures the overall correctness of a model’s predictions:

\[
\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}
\] (35)

where \( TP \) represents the number of instances that are correctly predicted as positive, while \( TN \) denotes the number of instances that are correctly predicted as negative. Conversely, \( FP \) indicates the number of instances that are incorrectly predicted as positive, and \( FN \) refers to the number of instances that are incorrectly predicted as negative.

E More Evaluation Result

In Table 5, we present the complete results of the reference-based experimental metrics. On average, CalibraEval achieved better improvements in Rstd and accuracy. This indicates that CalibraEval effectively reduces selection bias and enables the model to realize its potential. By mitigating selection bias in the evaluation process, CalibraEval contributes to achieving more accurate and reliable results, paving the way for further advancements in model calibration and evaluation methodologies.

E.1 Different ID tokens

We also conduct experiments using four distinct sets of ID tokens: A/B, a/b, Alice/Bob, and X/Y. Figure 5 illustrates the performance comparison. CalibraEval consistently achieves significant performance improvements with low variance across all tested ID tokens. This highlights its robustness and effectiveness regardless of the specific tokens used. Furthermore, when applied to the highly consistent model Qwen-72B, the improvement of Pride is negligible, while CalibreEval continued to enhance consistency even further.
(a) ChatGPT-ICC        (b) ChatGPT-Kappa

(c) Qwen72B-ICC        (d) Qwen72B-Kappa

Figure 5: Performance comparison across different ID tokens.

E.2 Different number of in-context learning examples

We further investigate the effectiveness of CalibraEval when conducting in-context learning. Specifically, we provided the LLMs with 1-shot, 2-shot, and 3-shot examples, respectively. As shown in Figure 6, CalibraEval remains effective even when examples are provided for in-context learning. We find that as the number of examples increases, the consistency of the model’s judgments also improves. This may be because examples help the model better understand the task, leading to more confident and consistent evaluations. Additionally, we also observed that the effectiveness of calibration methods like Pride and CalibraEval decreases as the number of examples increases. This may be due to these examples introducing new biases, which affect the effectiveness of the calibration. Therefore, we believe that calibration methods have greater potential for application in zero-shot scenarios.

F THE DESIGN OF PROMPT

Table 6 presents all the prompts used in this paper. The default prompt, employed in the main experiments, serves as the foundational basis for assessing model performance. In the robustness experiments, four distinct prompts are utilized to evaluate variations in model responses.

(a) ChatGPT

(b) Qwen-72B

Figure 6: Performance comparison under in-context learning.
Algorithm 1: Calibration process of Cali-braEval

Input: Language model, test samples \( \mathcal{D} \), estimate set size \( K \), threshold \( \epsilon \)
Output: Debiased Prediction \( \mathcal{Y} \)

1 Sample \( K \) estimation samples from the test samples \( \mathcal{D} \)
2 for each sample \( i \in \{1, ..., K\} \) do
3   Generate the probabilities after exchanging option IDs and positions.
4   Obtain the set \( s^i = \{s_0^i, s_1^i, s_2^i\} \)
5 end
6 Combine all sets \( s \) to form a global set
\( S = \bigcup_{i=1}^{K} S^i \)
7 Sort \( S \) in ascending order and append
\( z_0 = 0, z_{3K+1} = 1 \). Obtain the sequence
\( Z = \{z_0, z_1, ..., z_{3K}, z_{3K+1}\} \)
8 Initialize the parameter \( d_k \) as \( z_k \) for each \( k \)
9 while \( \sum_{i=0}^{3K+1} \triangle d_i > \epsilon \) do
10   for \( i = 0 \) to \( 3K + 1 \) do
11     Calculate \( \frac{\partial L}{\partial d_i} \) using Equation (14) and Equation (15)
12     Update the \( d_i \) using Equation (13)
13   end
14 end
15 Standardize \( d_i \) to satisfy \( \sum_{i=0}^{3K+1} d_i = 0 \)
16 Obtain the discrete mapping function \( g(\cdot) \) by using Equation (12)
17 Obtain the continuous calibration function \( g^*(\cdot) \) by solving Equation (16)
18 for \( q \in \mathcal{D} \) do
19   Debias the model prediction with \( g^*(\cdot) \)
20   Add the predicted answer to \( \mathcal{Y} \)
21 end
return Debiased Prediction \( \mathcal{Y} \)
<table>
  <tr>
    <th rowspan="2">Datasets</th>
    <th colspan="3">Average Length</th>
    <th colspan="3">Label num</th>
    <th rowspan="2">Type</th>
    <th rowspan="2">Type_num</th>
  </tr>
  <tr>
    <th>Total_num</th>
    <th>prompt</th>
    <th>answer_a</th>
    <th>answer_b</th>
    <th>first</th>
    <th>second</th>
    <th>tie</th>
  </tr>
  <tr>
    <td>RewardBench</td>
    <td>2985</td>
    <td>1771</td>
    <td>667</td>
    <td>658</td>
    <td>1490</td>
    <td>1495</td>
    <td>0</td>
    <td>Chat<br>Chat_Hard<br>Safety<br>Reasoning</td>
    <td>358<br>456<br>739<br>1432</td>
  </tr>
  <tr>
    <td>MTBench</td>
    <td>3355</td>
    <td>4039</td>
    <td>1524</td>
    <td>1512</td>
    <td>1293</td>
    <td>1282</td>
    <td>780</td>
    <td>Turn1<br>Turn2</td>
    <td>1689<br>1666</td>
  </tr>
  <tr>
    <td>PreferenceBench</td>
    <td>1998</td>
    <td>2485</td>
    <td>886</td>
    <td>893</td>
    <td>980</td>
    <td>1018</td>
    <td>0</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 4: Statistics of benchmark datasets.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">RewardBench</th>
    <th colspan="2">MTBench</th>
    <th colspan="2">PreferenceBench</th>
    <th colspan="2">Average</th>
  </tr>
  <tr>
    <th>Rstd ↓</th>
    <th>Acc.(%)</th>
    <th>Rstd ↓</th>
    <th>Acc.(%)</th>
    <th>Rstd ↓</th>
    <th>Acc.(%)</th>
    <th>Rstd ↓</th>
    <th>Acc.(%)</th>
  </tr>
  <tr>
    <td>Llama-3-8B</td>
    <td>15.01</td>
    <td>65.79</td>
    <td>16.42</td>
    <td>67.08</td>
    <td>3.36</td>
    <td>83.43</td>
    <td>11.60</td>
    <td>72.10</td>
  </tr>
  <tr>
    <td>DI</td>
    <td>15.61</td>
    <td>66.35</td>
    <td>9.42</td>
    <td>66.79</td>
    <td>4.03</td>
    <td>83.45</td>
    <td>9.69</td>
    <td>72.20</td>
  </tr>
  <tr>
    <td>CC</td>
    <td>14.62</td>
    <td>64.52</td>
    <td>8.70</td>
    <td>69.09</td>
    <td>9.47</td>
    <td>82.78</td>
    <td>10.93</td>
    <td>72.13</td>
  </tr>
  <tr>
    <td>DC</td>
    <td>13.79</td>
    <td>66.31</td>
    <td>20.86</td>
    <td>64.60</td>
    <td>2.90</td>
    <td>83.65</td>
    <td>12.52</td>
    <td>71.52</td>
  </tr>
  <tr>
    <td>Pride</td>
    <td>7.51</td>
    <td>66.54</td>
    <td>11.64</td>
    <td>70.63</td>
    <td>4.35</td>
    <td>83.24</td>
    <td>7.83</td>
    <td>73.47</td>
  </tr>
  <tr>
    <td>CalibraEval</td>
    <td><b>6.48</b></td>
    <td><b>68.12</b></td>
    <td><b>5.22</b></td>
    <td><b>70.63</b></td>
    <td><b>2.42</b></td>
    <td><b>83.98</b></td>
    <td><b>5.04</b></td>
    <td><b>74.24</b></td>
  </tr>
  <tr>
    <td>Llama-3-1.8B</td>
    <td>17.93</td>
    <td>64.96</td>
    <td>14.73</td>
    <td>67.58</td>
    <td>6.65</td>
    <td>77.54</td>
    <td>13.10</td>
    <td>70.03</td>
  </tr>
  <tr>
    <td>DI</td>
    <td>12.02</td>
    <td>64.25</td>
    <td>13.72</td>
    <td>67.47</td>
    <td>12.42</td>
    <td>75.14</td>
    <td>12.72</td>
    <td>68.95</td>
  </tr>
  <tr>
    <td>CC</td>
    <td>8.43</td>
    <td>65.39</td>
    <td>6.75</td>
    <td>65.09</td>
    <td>6.31</td>
    <td>77.91</td>
    <td>7.16</td>
    <td>69.49</td>
  </tr>
  <tr>
    <td>DC</td>
    <td>14.54</td>
    <td>66.90</td>
    <td>9.72</td>
    <td>67.74</td>
    <td><b>5.91</b></td>
    <td>77.98</td>
    <td>10.06</td>
    <td>70.87</td>
  </tr>
  <tr>
    <td>Pride</td>
    <td>13.94</td>
    <td>65.90</td>
    <td>12.63</td>
    <td>67.56</td>
    <td>9.47</td>
    <td>77.92</td>
    <td>12.01</td>
    <td>70.46</td>
  </tr>
  <tr>
    <td>CalibraEval</td>
    <td><b>6.88</b></td>
    <td><b>67.11</b></td>
    <td><b>6.67</b></td>
    <td><b>67.86</b></td>
    <td>6.19</td>
    <td><b>78.64</b></td>
    <td><b>6.58</b></td>
    <td><b>71.20</b></td>
  </tr>
  <tr>
    <td>Qwen-14B</td>
    <td>11.63</td>
    <td>63.14</td>
    <td>17.24</td>
    <td>65.61</td>
    <td>11.99</td>
    <td>80.68</td>
    <td>13.62</td>
    <td>69.81</td>
  </tr>
  <tr>
    <td>DI</td>
    <td>9.76</td>
    <td>61.88</td>
    <td>19.09</td>
    <td>62.18</td>
    <td>15.77</td>
    <td>76.14</td>
    <td>14.87</td>
    <td>66.73</td>
  </tr>
  <tr>
    <td>CC</td>
    <td>7.01</td>
    <td>60.21</td>
    <td>26.47</td>
    <td>58.21</td>
    <td>7.03</td>
    <td>78.41</td>
    <td>13.50</td>
    <td>65.61</td>
  </tr>
  <tr>
    <td>DC</td>
    <td>3.02</td>
    <td>62.47</td>
    <td>8.23</td>
    <td>68.48</td>
    <td>10.07</td>
    <td>79.96</td>
    <td>7.11</td>
    <td>70.30</td>
  </tr>
  <tr>
    <td>Pride</td>
    <td>4.18</td>
    <td>64.09</td>
    <td>16.31</td>
    <td>65.29</td>
    <td>7.36</td>
    <td>83.55</td>
    <td>9.28</td>
    <td>70.98</td>
  </tr>
  <tr>
    <td>CalibraEval</td>
    <td><b>2.72</b></td>
    <td><b>64.25</b></td>
    <td><b>6.26</b></td>
    <td><b>68.64</b></td>
    <td><b>5.12</b></td>
    <td><b>83.88</b></td>
    <td><b>4.70</b></td>
    <td><b>72.26</b></td>
  </tr>
  <tr>
    <td>Qwen-72B</td>
    <td>4.01</td>
    <td>87.20</td>
    <td>5.76</td>
    <td><b>81.32</b></td>
    <td>2.54</td>
    <td>90.12</td>
    <td>4.10</td>
    <td>86.21</td>
  </tr>
  <tr>
    <td>DI</td>
    <td>3.65</td>
    <td>86.32</td>
    <td>5.79</td>
    <td>80.82</td>
    <td>0.80</td>
    <td>90.21</td>
    <td>3.41</td>
    <td>85.78</td>
  </tr>
  <tr>
    <td>CC</td>
    <td>7.72</td>
    <td>85.74</td>
    <td>4.72</td>
    <td>81.05</td>
    <td>5.41</td>
    <td>89.30</td>
    <td>5.95</td>
    <td>85.36</td>
  </tr>
  <tr>
    <td>DC</td>
    <td>6.57</td>
    <td>83.87</td>
    <td>6.76</td>
    <td>80.52</td>
    <td>7.04</td>
    <td>88.83</td>
    <td>6.79</td>
    <td>84.41</td>
  </tr>
  <tr>
    <td>Pride</td>
    <td>3.82</td>
    <td>87.25</td>
    <td>5.24</td>
    <td>81.23</td>
    <td>2.27</td>
    <td>90.26</td>
    <td>3.78</td>
    <td>86.25</td>
  </tr>
  <tr>
    <td>CalibraEval</td>
    <td><b>0.94</b></td>
    <td><b>88.06</b></td>
    <td><b>4.99</b></td>
    <td><b>81.25</b></td>
    <td><b>0.69</b></td>
    <td><b>90.71</b></td>
    <td><b>2.21</b></td>
    <td><b>86.67</b></td>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>16.79</td>
    <td>65.27</td>
    <td>7.66</td>
    <td>72.67</td>
    <td>3.04</td>
    <td>85.61</td>
    <td>9.16</td>
    <td>74.70</td>
  </tr>
  <tr>
    <td>DI</td>
    <td>7.22</td>
    <td>65.24</td>
    <td>7.01</td>
    <td>69.84</td>
    <td>9.82</td>
    <td>83.94</td>
    <td>8.02</td>
    <td>73.01</td>
  </tr>
  <tr>
    <td>CC</td>
    <td>7.93</td>
    <td>64.89</td>
    <td>16.31</td>
    <td>70.49</td>
    <td>3.13</td>
    <td>84.83</td>
    <td>9.12</td>
    <td>73.40</td>
  </tr>
  <tr>
    <td>DC</td>
    <td>11.40</td>
    <td>66.89</td>
    <td>20.23</td>
    <td>68.67</td>
    <td>5.81</td>
    <td>82.68</td>
    <td>12.48</td>
    <td>72.75</td>
  </tr>
  <tr>
    <td>Pride</td>
    <td>8.54</td>
    <td>66.36</td>
    <td>6.01</td>
    <td>72.86</td>
    <td>3.51</td>
    <td>85.68</td>
    <td>6.02</td>
    <td>74.97</td>
  </tr>
  <tr>
    <td>CalibraEval</td>
    <td><b>5.51</b></td>
    <td><b>67.13</b></td>
    <td><b>5.20</b></td>
    <td><b>72.98</b></td>
    <td><b>2.82</b></td>
    <td><b>85.98</b></td>
    <td><b>4.51</b></td>
    <td><b>75.36</b></td>
  </tr>
  <tr>
    <td>GPT4o</td>
    <td>1.95</td>
    <td>89.34</td>
    <td>3.23</td>
    <td>82.27</td>
    <td>5.29</td>
    <td>90.44</td>
    <td>3.49</td>
    <td>87.35</td>
  </tr>
  <tr>
    <td>DI</td>
    <td>3.91</td>
    <td>88.64</td>
    <td>2.24</td>
    <td>82.26</td>
    <td>4.49</td>
    <td>90.46</td>
    <td>3.55</td>
    <td>87.12</td>
  </tr>
  <tr>
    <td>CC</td>
    <td><b>0.54</b></td>
    <td>89.11</td>
    <td>4.84</td>
    <td>81.23</td>
    <td>5.79</td>
    <td>90.02</td>
    <td>3.72</td>
    <td>86.79</td>
  </tr>
  <tr>
    <td>DC</td>
    <td>1.84</td>
    <td>87.84</td>
    <td>3.57</td>
    <td>81.93</td>
    <td>5.99</td>
    <td>88.22</td>
    <td>3.80</td>
    <td>86.00</td>
  </tr>
  <tr>
    <td>Pride</td>
    <td>1.68</td>
    <td>89.38</td>
    <td>3.84</td>
    <td>82.10</td>
    <td>5.24</td>
    <td>90.47</td>
    <td>3.60</td>
    <td>87.32</td>
  </tr>
  <tr>
    <td>CalibraEval</td>
    <td>1.42</td>
    <td><b>89.54</b></td>
    <td><b>2.89</b></td>
    <td><b>82.29</b></td>
    <td><b>4.29</b></td>
    <td><b>90.49</b></td>
    <td><b>2.87</b></td>
    <td><b>87.44</b></td>
  </tr>
</table>

Table 5: The complete results of reference-based metrics. We report the Standard Deviation of Recalls (RStd) and Accuracy (Acc.), with the best results highlighted in bold. ↓ indicates that lower values correspond to better performance.
<table>
  <tr>
    <th>Default Prompt.</th>
    <td>Given a question and two answers. Determine which one better answers the question. You only need to output A or B directly to indicate which answer is better.</td>
  </tr>
  <tr>
    <th>Prompt Variant One.</th>
    <td>Please evaluate the quality of the responses to the question displayed below. Don’t provide your explanation, only output your final verdict by strictly following this format: A if assistant A is better, B if assistant B is better.</td>
  </tr>
  <tr>
    <th>Prompt Variant Two.</th>
    <td>You are an advanced evaluator, and your task is to assess which response addresses the inquiry more effectively. Output A if response A is better, or B if response B is better.</td>
  </tr>
  <tr>
    <th>Prompt Variant Three.</th>
    <td>Below is a query along with two different responses generated by AI assistants. Your task is to determine which response provides a more accurate and helpful answer to the question posed. Don’t provide your explanation. Simply output A if response A is more effective, or B if response B is more effective.</td>
  </tr>
</table>

Table 6: Different prompt templates used in this paper

Reviewer paper summaries: LLMs are not perfect judges since they exhibit certain biases, such as position bias and token bias. To address these two types of biases in LLMs-as-Judges, this paper introduces CalibraEval, a label-free, inference-time method. CalibraEval targets at debiasing pair-wise evaluation, and formulates it as an optimization task that maps the observed prediction distribution from LLMs to an unbiased distribution.

Specifically, CalibraEval first define three probabilities of default output, swapping position of pairs of responses, and swapping ID tokens before the responses. Then given a set of samples and associated probabilities, they sort the probabilities and optimize a set of trainable {d_k} parameters associated with each probability. Finally they learn a continuous calibration function from discrete parameters.

This method achieves best evaluation consistency in most settings compared to baseline method (table 1), and achieves best performance when compared to gold reference (table 2).
Summary of strengths: -	It is interesting to formulate the debiasing problem as an optimization task, and the proposed optimization objective is shown effective with the ablation study.
-	CalibraEval effectively enhances the consistency of model evaluation and also marginally improves the accuracy of model evaluation based on reference.
-	The paper is well written and clear in general.
Summary of weaknesses: -	This paper argues that CalibraEval only requires minimal computational cost but does not discuss the cost carefully nor how does the computational cost increases with the prior estimation set size. Compared to baseline methods, CalibraEval requires more cost. 
-	The designed optimization objective only applies to pair-wise evaluation, and it might be difficult to extend to other scenarios.
-	The main results are presented by experimenting with same prior estimation set size instead of controlling for computational cost. Figure 4 shows that CalibraEval can produce reliable calibration functions with a limited subset of all data. It is not clear how this result would compare to baseline methods with a similar computational cost.
Comments, suggestions and typos: - Discuss the computational cost of CalibraEval to help understand its effectiveness.
Overall assessment scores: 4
Metareview score: 4
Metareview: This paper introduces CalibraEval, an inference-time debiasing method that tries to mitigate selection bias in LLMs-as-Judges settings. CalibraEval frames debiasing as an optimization task that adjusts the observed prediction distribution of LLMs to match an “unbiased reference” through a non-parametric order-preserving algorithm (NOA). The method focuses two types of biases in pairwise evaluation settings, position bias and token identifier bias. Evaluation of the debiasing method across three benchmarks in six LLMs from three model families show that CalibraEval consistently improves evaluation consistency compared to existing debiasing approaches.
Summary of reasons to publish: - Novel and effective idea to cast LLM-as-Judge debiasing as an optimization problem
- Experimental results demonstrate consistent improvements across model families
- Clearly an effective method that would be useful across LLM-as-Judge settings
- Authors have addressed major concerns around computational and time costs, and provided additional results that support the effectiveness of their method