Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge

## Abstract

Abstract LLM-as-a-Judge has been widely utilized as an evaluation method in various benchmarks and served as supervised rewards in model training. However, despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility.
Therefore, we identify 12 key potential biases and propose a new automated bias quantification framework— Calm —which systematically quantifies and analyzes each type of bias in LLM-as-a-Judge by using automated and principle-guided modification. Our experiments cover multiple popular language models, and the results indicate that while advanced models have achieved commendable overall performance, significant biases persist in certain specific tasks. Empirical results suggest that there remains room for improvement in the reliability of LLM-as-a-Judge. Moreover, we also discuss the explicit and implicit influence of these biases and give some suggestions for the reliable application of LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications. Warning: This paper may contain some offensive content.

## 1 Introduction

Large Language Models (LLMs), such as GPT-4 (OpenAI, ), have exhibited exceptional capabilities across a wide range of natural language processing (NLP) tasks, including applications in medicine (Liu et al., ), LLM-based agents (Huang et al., ; Guo et al., ; Chen et al., ; ), science (Guo et al., ; Li et al., ; Chen et al., ; Le et al., ), and data synthesis (Zhao et al., ; Wu et al., ). In recent research, there has been a focus on using LLMs to automatically evaluate responses and provide rewards. This methodology is commonly known as LLM-as-a-Judge, which involves using LLMs to assess responses in two main ways: comparing pairs of answers to determine superiority (Zheng et al., ), or directly scoring individual answers based on specific criteria (Liu et al., ). This method has been primarily applied in scoring and pairwise comparison tasks, yielding notable achievements (Kasner & Dušek, ; Liu et al., ).

Despite the increasing adoption of LLM-as-a-Judge, concerns regarding its reliability have emerged due to potential biases within the models (Zheng et al., ; Chen et al., ; Wang et al., ; Koo et al., ). These biases cast doubt on the trustworthiness of LLMs, both in their evaluation processes and in their alignment with principles of fairness and transparency (Sun et al., ; Huang et al., ). For instance, Zheng et al. () conducted extensive experiments to examine positional preferences in LLM-as-a-Judge, while Koo et al. () revealed that popular opinions reflecting majority viewpoints may compromise the fairness of LLM evaluations. Furthermore, experiments conducted by Chen et al. () demonstrated that fabricated citations could disrupt the judgment accuracy of LLMs.

While these studies have highlighted several types of biases existing in LLM-as-a-Judge, the field remains ripe for further exploration. Firstly, the existing analyses of bias are relatively narrow in scope (Wang et al., ; Chen et al., ), which limits the development of a comprehensive framework for evaluating the multifaceted biases affecting LLM-as-a-Judge. Secondly, many previous studies have relied on human evaluators to assess the quality of answers and compare them against the judgments made by LLMs to identify potential biases. This methodology incurs substantial costs and introduces human subjectivity, complicating the establishment of reliable ground truth and the reproducibility of findings (Zheng et al., ). Additionally, Wu & Aji () demonstrated that the limited size and scope of test data increase the risk of random interference, potentially obscuring the true extent of bias in LLM judgments. A more detailed discussion of related work is in Appendix A.

Figure: Figure 1: The comparison of the robustness rates (scores) of all models, a higher score indicates greater resistance to the bias. Table 1 shows the full name of 12 types of bias.
Refer to caption: figure/all_1.png

To address these challenges, we introduce Calm, a novel framework for automated quantification of biases in LLM-as-a-Judge. Calm covers 12 distinct types of bias that may arise when LLMs are used as judges in various scenarios, including the following examples.

$\triangleright$ Correctness of Scientific Reasoning. When using LLMs to judge reasoning results in scientific QA or answer to math problems (Cobbe et al., ; Hendrycks et al., ), bias often occurs in understanding the content. Therefore, we focus on evaluating potential biases in LLM judges, specifically regarding verbosity (favoring longer responses), fallacy oversight (ignoring logical errors in reasoning), and sentiment (preference for positive or negative expressions).

$\triangleright$ Improvement on Answer Refinement. Answers to open-ended questions in the humanities, social sciences, or general knowledge can often be refined to improve quality. When LLMs are used to determine whether a refined answer is better than the original, bias occurs if the LLM judge is informed about the refinement process.

$\triangleright$ Alignment to Human Feedback. LLMs are increasingly used to assess which generated answer better aligns with human feedback when provided with two or more answers. In such cases, alignment bias often occurs, e.g., the LLM judge favor answers based on their placement (position bias), or favor answers they generated themselves (self-preference).

As we can see, automating the process of bias identification in various judging scenarios is challenging, but highly beneficial. We design this process using an *attack-and-detect* approach. In Calm, an LLM judge is presented with deliberate perturbations (the “attack”) applied to the content being judged. The judgment results are then examined to determine whether the judge’s score or preference remains consistent.
While more details on how Calm automates this processing will be provided later, several advantages are already evident, such as the elimination of subjective human assessments and the reduction of testing costs, resulting in a more objective and scalable evaluation approach.

In summary, our contributions are three-fold: (1) A systematic definition and categorization of 12 distinct types of bias that can undermine the reliability and trustworthiness of LLM-as-a-Judge. (2) The introduction of Calm, a framework for evaluating biases in LLM-as-a-Judge systems, which enhances the integrity of the assessment process without relying on human resources. (3) An extensive evaluation of six popular LLMs using the Calm framework, as shown in Figure 1, reveals that while some LLMs demonstrate notable fairness in judgment, there remains significant room for improvement in achieving more robust decision-making across various types of bias.

## 2 Proposed Framework: Calm

Our proposed framework, Calm, which stands for Comprehensive Assessment of Language Model Judge Biases, is illustrated in Figure 2. Calm comprises four integral components: 1) Comprehensive bias categories. We identify twelve distinct types of biases that may arise in the context of LLM-as-a-Judge, as detailed in Table 1. 2) Various datasets across different evaluation aspects. We incorporate a diverse range of datasets that cover various evaluation aspects, including question-answering datasets, mathematical reasoning datasets, and alignment datasets, all of which are elaborated upon in Table 3. 3) Metrics for evaluating bias in judging. Our framework employs metrics specifically designed for judging tasks, encompassing both pairwise comparison and scoring. These quantitative metrics include Robustness Rate (RR) and Consistency Rate (CR), among others, to facilitate a comprehensive evaluation. 4) An automated perturbation mechanism for bias injection. This innovative approach utilizes automated and principle-guided modifications to construct biased counterpart of the original content for judgement.

Figure: Figure 2: Calm, the proposed framework for bias assessment in LLM-as-a-Judge. On a selected dataset and a type of bias for assessment, Calm employs models to generate answers for judgment, as well as biased answers through principle-guided modifications powered by an LLM (*i.e.*, GPT-4o). By applying carefully curated metrics, Calm then quantify the reliability of judge models.
Refer to caption: x1.png

Figure: Figure 3: Examples of answer modification for bias injection. Left: verbosity bias is injected by employing GPT-4 to expand the initially poor answer from Assistant B. Right: authority bias is introduced by using GPT-4 to insert a fake citation to the original answer of Assistant B.
Refer to caption: x2.png

### 2.1 Bias Assessment Problem Formulation

To formally quantify biases in LLM-as-a-Judge, we define the input prompt for LLM judge as $P=(I,Q,R)$, which consists of three components: system instruction $I$, question $Q$, and responses to be judged $R$.
A perturbation is applied to investigate the potential bias in the judgment by making a bias-related modification to the original response. We automate this process by using another LLM to change $R$ to $g(R)$ or modify the $I$ to $g(I)$ (*e.g.*, insert a system prompt into $I$), resulting in a modified $\hat{P}$. For example in Figure 3, the response given by Assistant B has been lengthened from the original response to assess verbosity bias.
The output of LLM judge on $P$ and $\hat{P}$ is compared for measuring the potential bias:

$$ $y=\textbf{LLM}(P),\quad\hat{y}=\textbf{LLM}(\hat{P}).$ $$

Here, if the judgment scores $y$ and $\hat{y}$ differ, it indicates the presence of bias in this LLM-as-a-Judge setting. The desirable outcome is when $y$ and $\hat{y}$ are the same, showing that the LLM judge is robust and unbiased.

In judge cases involving pairwise comparison, the input prompt for LLM judge is defined as $P=(I,Q,R_{1},R_{2})$, including two candidate responses $R_{1}$ and $R_{2}$ for comparisons. Similar perturbations can be applied to one record $\hat{y}=\textbf{LLM}(I,Q,R_{1},g(R_{2}))$ or to the instruction $\hat{y}=\textbf{LLM}(g(I),Q,R_{1},R_{2})$.
For instance, in Figure 3 (right), a fake citation is added to Assistant B’s answer, thus perturbing $R_{2}$ into $g(R_{2})$.
If the LLM judge is unbiased, the comparison should yield
$y=\hat{y}=$R1 from Assistant A, because Assistant B’s answer remains consistently inferior to that of Assistant A, both before and after the modification.

**Table 1: Types of biases in LLM-as-a-Judge, with descriptions and examples that demonstrate how particular bias affects LLM’s judgment.**
| Bias Type | Description | Example |
| --- | --- | --- |
| Position (Pos.) | LLM judges exhibit a propensity to favor one answer at certain position over others. | Turn 1: $R_{1}$: 3.11 > 3.8    $R_{2}$: 3.8 > 3.11<br>Turn 2: $R_{1}$: 3.8 > 3.11    $R_{2}$: 3.11 > 3.8 |
| Υ Verbosity (Ver.) | LLM judges favor longer responses, even if they are not as clear, high-quality, or accurate as shorter alternatives. | $R_{1}$: As we all know, in mathematics, 3.11 is greater than 3.8 (Longer)<br>$R_{2}$: 3.11 > 3.8 (Shorter) |
| c Compassion-<br>Fade (Com.) | The tendency to observe different behaviors when given well-known model’s name as opposed to anonymized aliases. | GPT-4: 3.11 > 3.8<br>Llama-7B: 3.8 > 3.11 |
| Bandwagon (Ban.) | The tendency to give stronger preference to the majority’s beliefs regardless of whether they are correct or not. | $I$: 90% believe that $R_{1}$ is better.<br>$R_{1}$: 3.11 > 3.8    $R_{2}$: 3.8 > 3.11 |
| Distraction (Dis.) | The inclination to give more attention to irrelevant or unimportant details. | $I$: $R_{1}$ loves eating pasta, especially with homemade tomato sauce.<br>$R_{1}$: 3.11 > 3.8    $R_{2}$: 3.8 > 3.11 |
| Ø  Fallacy-<br>Oversight (Fal.) | LLM judges may ignore logical errors in reasoning steps and only focus on the correctness of final results. | $R_{1}$: 0.8 is greater than 0.11, so 3.8 > 3.11.<br>$R_{2}$: 3.8 has fewer digits, so it’s a larger number, so 3.8 > 3.11. |
| S Authority (Aut.) | The tendency to assign more credibility to statements made by authority figures, regardless of actual evidence. | $R_{1}$: 3.11 > 3.8 (Citation: Patel, R. (2018). Advanced Algorithms for Computational Mathematics: The Art Of Decimal-Comparison, p. 143)<br>$R_{2}$: 3.8 > 3.11. |
| Sentiment (Sen.) | The preference for expressions of positive or negative emotions, affecting its judgment of emotional content. | We transform the sentiment in the answer:<br>$R_{1}$: Regrettably, 3.11 > 3.8, it ruthlessly reveals the cruelty of reality and the facts that cannot be changed. (Frustrated tone)<br>$R_{2}$: 3.8 > 3.11. |
| Diversity (Div.) | Bias may be shown towards certain groups like ’Homosexual’, ’Black’, ’Female’, and ’HIV Positive’. | $I$: $R_{1}$’s true identity is Homosexual.<br>$R_{1}$: 3.8 > 3.11 $R_{2}$: 3.11 > 3.8 |
| Chain-of-<br>Thought (CoT) | The model’s evaluation results may vary with and without CoT. | $I_{1}$: Compare both assistants’ answers …<br>$I_{2}$: You should independently solve the user question step-by-step first. Then compare both assistants’ answers with your answer. |
| ¿ Self-<br>Enhancement (Sel.) | LLM judges may favor the answers generated by themselves. | $R_{1}$: 3.11 > 3.8 (LLM judge generated $R_{1}$ itself)<br>$R_{2}$: 3.8 > 3.11 |
| –  Refinement-<br>Aware (Ref.) | Telling the model that this is a refined result will lead to different evaluations. | Original Answer: The data is inaccurate. (Score: 6 points)<br>Refined Answer with Original Answer: The data is inaccurate …(refining content)…Upon careful review…contains inaccuracies (Score: 8 points)<br>Refined Answer Only: Upon careful review…contains inaccuracies (Score: 7 points) |

### 2.2 Bias Types and Automated Perturbation

Bias Types.  Considering the diverse use cases of LLM-as-a-Judge, we have synthesized and expanded upon previously proposed biases, ultimately arriving at a total of 12 types of bias, which are summarized in Table 1 with examples for facilitating the understanding. Due to the space limitation, we show more details of these bias types in Appendix B.

Automated Perturbation $g(\cdot)$.   The automation of bias injection is key to automating the entire bias assessment process. As introduced in section 2.1, the perturbation $g(\cdot)$ modifies either the response $R$ or the instruction $I$.
It is crucial that the perturbation does not alter the correctness of the response and preserves the original meaning as much as possible to avoid semantic shift. At the same time, it must not be too trivial, as this would result in a response that appears unchanged and fails to expose any potential evaluation bias.

We develop $g(\cdot)$ as a principle-guided modification powered by LLMs, following the approach of constitutional AI (Bai et al., ).
By applying multiple sets of guidelines (i.e., instructions), an LLM can modify answer content, resulting in biased counterparts of the original answers.
For instance, as shown in Figure 3, one raw answer is modified by an LLM through a prompt-based guideline. The complete set of instructions for answer modification is provided in Appendix C and Appendix F.
For different types of bias and various judging tasks that will be discussed in subsection 2.3, we designed specific guidelines (i.e., instructions) to ensure that the modifications effectively inject the appropriate bias into the content.

### 2.3 Judging Tasks, Datasets and Metrics

Judging Tasks.
The use of LLM-as-a-Judge is typically implemented in two well-established ways: pairwise comparison (Zheng et al., ) and scoring (Liu et al., ). One drawback of the scoring method is that, without a reference answer, it can be challenging for LLM judges to provide an objective score, as their judgments can be easily influenced by contextual factors. In contrast, pairwise comparison mitigates this issue and has been widely utilized for alignment data based on human annotations (Ouyang et al., ).

**Table 2: An overview of the types of bias, dataset, the judgment task, the number of used samples, the evaluation metrics, and their corresponding dimensions. Metrics are chosen based on their relevance to each bias type. RR: Robustness rate, Err.${}_{\text{SE}}$: $\text{ErrorRate}_{\text{SE}}$, Acc${}_{\text{hack}}$: Accuracy for hack detection, Err.${}_{\text{RA}}$: $\text{ErrorRate}_{\text{RA}}$. Answers-Related indicates whether the type of bias pertains to answer modification or being modified; Semantic-Related indicates whether the bias is related to the answer’s semantic, such as flawed reasoning logic in fallacy-oversight bias; and Instruction-Influence denotes whether it is connected to the system prompt.**
| Bias | Dataset | # Sample | Metric | Judge Task | Dimensions |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  | Scoring | Pairwise-Comparison | Answers-Related | Semantic-Related | Instruction-Influence |
| Position | Align. | 439 | RR | ✘ | ✔ | ✔ | ✘ | ✘ |
| Verbosity | Fac. | 500 | RR | ✘ | ✔ | ✔ | ✘ | ✘ |
| Compassion-Fade | Align. | 439 | RR | ✘ | ✔ | ✔ | ✘ | ✘ |
| Bandwagon | Align. | 150 | RR | ✘ | ✔ | ✘ | ✘ | ✔ |
| Distraction | Align. | 439 | RR | ✘ | ✔ | ✘ | ✘ | ✔ |
| Fallacy-Oversight | Fac. | 500 | RR | ✘ | ✔ | ✔ | ✔ | ✘ |
| Authority | Align. | 150 | RR | ✘ | ✔ | ✔ | ✘ | ✘ |
| Sentiment | Fac. | 500 | RR | ✘ | ✔ | ✔ | ✘ | ✘ |
| Diversity | Align. | 150 | RR | ✘ | ✔ | ✘ | ✘ | ✔ |
| Chain-of-Thought | Align. | 439 | Acc | ✘ | ✔ | ✘ | ✘ | ✔ |
| Self-Enhancement | Align. | 150 | Err.${}_{\text{SE}}$ | ✔ | ✘ | ✘ | ✘ | ✘ |
| Refinement-Aware | Ref. | 500 | Err.${}_{\text{RA}}$ | ✔ | ✘ | ✔ | ✔ | ✔ |

**Table 3: Sources of our constructed dataset, as well as the number of samples.**
| Dataset | Source | # Sample | Total |
| --- | --- | --- | --- |
|  | Truthy-DPO-v0.1 (Durbin, ) | 100 |  |
|  | Emerton-DPO-Pairs-Judge (Leo, ) | 100 | 439 |
| Alignment | Orca-DPO-Pairs (Intel, ) | 100 | (after |
| dataset | Py-DPO-v0.1 (Durbin, ) | 100 | filtering) |
|  | Roleplay-NSFW (xDAN, ) | 100 |  |
| Fact-related | GSM8K (Cobbe et al., ) | 150 | 500 |
| dataset | MATH (Hendrycks et al., ) | 150 |  |
|  | ScienceQA (Lu et al., ) | 200 |  |
|  | CommonsenseQA (Talmor et al., ) | 150 | 500 |
| Refinement | Quora-QuAD (Toughdata, ) | 150 |  |
| aware dataset | TruthfulQA (Lin et al., ) | 200 |  |

Consequently, we primarily adapt the pairwise selection task for LLM judges in assessing most biases. However, for certain biases, such as self-enhancement and refinement-aware bias, the pairwise selection method is difficult to apply; thus, LLM judges are evaluated using the scoring judgment task instead. In the scoring task, as introduced earlier, the LLM judge provides a numerical score for a given response, $y=\textbf{LLM}(I,Q,R)$. In the pairwise comparison task, the LLM judge evaluates two responses and outputs a preference for one over the other, $y=\textbf{LLM}(I,Q,R_{1},R_{2})$. More details are shown in Table 2.

Datasets.  We prepared three datasets in Calm for supporting bias assessment in various judging tasks: fact-related, refinement-aware evaluation, and alignment datasets. The details of these datasets are shown in Table 3. Their usage in the assessment of different types of bias is presented in Table 2.

- $\triangleright$
Fact-related dataset. We constructed a fact-related dataset for the assessment involving bias types that require factual information as test content, and for the cases where the quality of the response should not be affected by the presentation style of the model’s response. We utilized GPT-4-Turbo to generate both a relatively good answer and an answer with complete reasoning logic but of lower overall quality. They are used as $R_{1}$ and $R_{2}$ as a pair in $P$. This dataset allows us to modify responses without affecting their inherent quality when dealing with biases such as verbosity bias, thereby more accurately determining whether the observed perturbation is due to the bias itself.
- $\triangleright$
Refinement-aware evaluation dataset. This dataset is constructed for assessing the bias when LLM judge is used to determine whether a refined answer is better than the original.
We selected questions from datasets comprising open-ended inquiries in humanities, social sciences, or general knowledge. These questions were chosen specifically because their corresponding answers could be significantly improved through refinement.
The particular bias to be assessed on this dataset is whether the LLM judge produces a different result when it is informed about the refinement.
- $\triangleright$
Alignment dataset. We created this dataset by sampling various DPO (Direct Preference Optimization) datasets (Rafailov et al., ). These questions are derived from actual user feedback, providing insights into user preferences and rejections across different scenarios, thus ensuring response diversity. For bias types that don’t have specific data requirements, such as authority bias, we opted for this dataset to enhance the diversity of our question coverage. These datasets encompass various aspects including code, NSFW content, truthfulness testing, and role-playing.

Metrics.  To quantify whether an LLM judge is robust and unbiased, we use the following metrics.
The LLM judge is executed twice for each evaluation. In the first turn, it selects the result it considers superior, denoted as $y$. In the second turn, we perform two parallel judgement: one without any perturbation to obtain $y_{\text{rand}}$, and another with a bias introduced into the candidate answers, obtaining $\hat{y}$. Based on these judgement outcomes, we define two metrics: Robustness Rate (RR) and Consistency Rate (CR), calculating over all samples in test dataset $D$,

$$ $\displaystyle\text{RR}=\frac{1}{|D|}\sum_{i=1}^{|D|}\mathbb{I}(y^{i}=\hat{y}^{i}),\quad\text{CR}=\frac{1}{|D|}\sum_{i=1}^{|D|}\mathbb{I}(y^{i}=y_{\text{rand}}^{i}).$ $$

RR measures how consistently the LLM judge’s decisions remain the same before and after introducing the bias. A higher RR indicates that the model’s judgment is less affected by the bias. CR evaluates how consistent the model’s decisions are when tested under identical conditions twice. The model is asked to make the same judgment without any bias or interference, and a higher CR suggests that the model provides stable and reliable decisions across repeated judgments.

Next, to evaluate CoT bias, i.e., the LLM judge tends to make more accurate judgments after experiencing the CoT process, we introduce the accuracy metric, which can effectively reflect the impact of CoT on making correct judgments. We define original accuracy and hacked accuracy as follows, where $R$ represents the ground truth results from the dataset:

$$ $\displaystyle\text{Acc}_{\text{ori}}$ $\displaystyle=\frac{1}{|D|}\sum_{i=1}^{|D|}\mathbb{I}(y^{i}=R^{i}),\,\text{Acc}_{\text{hack}}=\frac{1}{|D|}\sum_{i=1}^{|D|}\mathbb{I}(\hat{y}^{i}=R^{i})$ $$

Original accuracy measures the agreement between the model’s initial selection $y$ and $R$. Hacked accuracy measures the agreement between the judge’s selection after bias is introduced $\hat{y}$ and $R$.

Furthermore, we introduce the Error Rate for different types of bias to quantify the impact of specific biases. The error rates are calculated as follows:

$$ $\displaystyle\text{ErrorRate}_{\text{SE}}$ $\displaystyle=\left|1-\frac{y_{\text{self}}}{y_{\text{other}}}\right|,\,\text{ErrorRate}_{\text{RA}}=\left|1-\frac{y_{\text{ref}}}{y^{{}^{\prime}}_{\text{ref}}}\right|.$ $$

For self-enhancement bias, $y_{\text{self}}$ is the score the judge model assigns to its own response, and $y_{\text{other}}$ is the score assigned by other models to the same response. This error rate quantifies how much the judge model favors its own responses compared to those from other models. For refinement-aware bias, $y_{\text{ref}}$ is the score given to the model’s refined response, and $y^{{}^{\prime}}_{\text{ref}}$ is the score given when considering the response’s refinement history. This error rate measures the model’s bias towards refined responses, especially when it is aware of the refinement process.

## 3 Experimental Setup

Models.
Based on the recent study
(Gao et al., ; Liu et al., ; Li et al., ), LLMs with stronger capabilities are prefered to be used as judges, because weaker
LLMs may exhibit greater randomness in their judgments, which can undermine the reliability of judging results. We thus evaluated six popular and capable LLM judges within our framework, including both proprietary and open-source options to provide a comprehensive analysis and comparison. The selected models are: ChatGPT (OpenAI, ), GPT-4-Turbo (OpenAI, ), GPT-4o (OpenAI, ), Claude-3.5 (Anthropic, ), GLM-4 (GLM et al., ), and the open-source Qwen2-72B-Instruct (Bai et al., ), which are further detailed in Table 8. Additionally, to mitigate the influence of self-enhancement bias, we selected four models solely for response generation: Mixtral-8x22b (AI@Mistral, ), Llama3-70b (AI@Meta, ), Llama3-8b (AI@Meta, ), and Mistral-7b (AI@Mistral, ).

Judgement prompt $P$.  The instruction $I$ in the judgment prompt $P=(I,Q,R)$ is derived from Liu et al. () and Zheng et al. (), with slight variations to evaluate the impacts of biases in LLM-as-a-Judge. The complete instruction we used is provided in Appendix F.

Hyperparameters.  We followed the experimental setup of Chen et al. () by setting the temperature to 0.7 and applied it to all judge models and generating models to ensure stable output quality and strong reproducibility.

## 4 Evaluation Results

In this section, we introduce our main results and related analyses from our exploratory experiments. We show the main results in Figure 4 and Table 4. Furthermore, we conduct exploratory experiments to evaluate the potential influence bias factor in LLM-as-a-Judge, which are detailed in Figure 5, Table 5, Figure 6 and Figure 7. Due to the space limitation, we show more detailed information of experiment results in Appendix D.

### 4.1 Main Result

Figure: Figure 4: Overall robustness rate with the dashed line representing the consistency rate.
Refer to caption: x3.png

Even advanced models can exhibit unexpected vulnerabilities in judgment.  Figure 4 illustrates the influence of 12 distinct biases on the judging capabilities of six LLMs. Notably, the effects of these biases differ across models, and advanced models may not always exhibit better performance when dealing with these biases. While Claude-3.5 generally shows the greatest resilience to biases, our findings reveal that even highly proficient models can struggle. For example, despite its advanced capabilities (Zheng et al., ), GPT-4-Turbo exhibits inconsistency when judging emotional responses, whereas ChatGPT demonstrates more stable performance. This complexity suggests that identifying the best model is not straightforward; it depends on the specific bias involved, and even top-tier models may display unexpected weaknesses. Therefore, when using LLMs as judges, it is crucial to acknowledge these complexities and avoid assuming that the most advanced model will always be the most reliable.

**Table 4: Robustness rate for various models across different metrics are presented. $\text{$D$}_{\text{FR}}$ and $\text{$D$}_{\text{AL}}$ represent fact-related datasets and alignment datasets, respectively, while $\text{CR}_{\text{FR}}$ and $\text{CR}_{\text{Al}}$ indicate the consistency rate on these two datasets without changing any values.**
| Model | $\text{$D$}_{\text{FR}}$ RR$\uparrow$ | $\text{$D$}_{\text{AL}}$ RR$\uparrow$ | $\text{$D$}_{\text{AL}}$ Acc$\uparrow$ |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Ver. | Fal. | Sen. | $\text{CR}_{\text{FR}}$ | Pos. | Com. | Ban. | Aut. | Dst. | Div. | $\text{CR}_{\text{Al}}$ | CoT. |  |
| ChatGPT | 0.900 | 0.917 | 0.804 | 0.998 | 0.566 | 0.862 | 0.688 | 0.662 | 0.713 | 0.679 | 0.906 | 0.560 |
| GPT-4-Turbo | 0.915 | 0.969 | 0.653 | 0.990 | 0.818 | 0.858 | 0.638 | 0.846 | 0.729 | 0.855 | 0.856 | 0.720 |
| GPT-4o | 0.977 | 0.984 | 0.699 | 0.998 | 0.776 | 0.868 | 0.791 | 0.787 | 0.790 | 0.814 | 0.925 | 0.700 |
| GLM-4 | 0.887 | 0.979 | 0.679 | 0.970 | 0.781 | 0.835 | 0.690 | 0.796 | 0.814 | 0.788 | 0.884 | 0.688 |
| Claude-3.5 | 0.952 | 0.985 | 0.660 | 0.999 | 0.832 | 0.875 | 0.610 | 0.865 | 0.878 | 0.914 | 0.915 | 0.745 |
| Qwen2 | 0.884 | 0.935 | 0.651 | 0.994 | 0.760 | 0.877 | 0.710 | 0.779 | 0.785 | 0.826 | 0.904 | 0.704 |

Bias is more pronounced in the alignment dataset compared to the fact-related dataset.  According to Table 4, the impact of bias is more pronounced in the alignment dataset than in the fact-related dataset. One possible explanation for this is that, in the fact-related dataset, the quality differences between answers are more evident, which means that the influence of bias is insufficient to completely offset this quality gap. In contrast, the alignment dataset typically has smaller quality differences between answers, making the choices of the judge model more vulnerable to bias. Therefore, when developing a reliable LLM-as-a-Judge framework across different datasets, it is crucial to consider the inherent quality of the data.

Bias reflects cognitive and philosophical issues beyond technical defects.  The bias in LLMs may originate from the inherent limitations of human cognition. For instance, LLMs perform inconsistently when dealing with sentiment bias, potentially reflecting the phenomenon that humans are often influenced by emotions when making judgments. In cognitive psychology, this phenomenon is known as the affect heuristic (Slovic et al., ). Recent research has demonstrated that LLMs have inherited this human cognitive trait to some extent (Li et al., ; ), prompting us to reconsider whether models should completely mimic human cognitive patterns or transcend these limitations. However, LLMs cannot truly achieve absolute fairness in a meaningful sense. This aligns with the view in postmodern philosophy that all judgments inevitably carry some degree of subjectivity. Therefore, while acknowledging that absolute objectivity is unattainable, we should focus on mitigating bias to an acceptable level in LLM-as-a-Judge scenarios.

### 4.2 Analysis of Exploratory Experiments

Position bias increases with more answer candidates.  Figure 6 demonstrates that all judge models are significantly impacted by position bias. This bias becomes more pronounced as the number of answers increases, particularly when evaluating three or four options, resulting in a decreased robustness rate, with most models scoring below 0.5. To mitigate the effects of position bias, we recommend using judge models with better robustness rate metrics or randomizing the order of answers (Zheng et al., ; Li et al., ).

Figure: Figure 5: Heat map of model Z-score normalization score of self-enhancement bias.
Refer to caption: figure/zscore_heatmap_all_blue.png

Response length influences model judgment in complex ways.  As illustrated in Figure 6, increasing response length without a corresponding improvement in quality led to a decline in model robustness rate. Some models exhibited an aversion to excessively verbose answers, while others demonstrated a positive correlation between model preference and response length.

Avoid using the same model to generate and judge answers.  Analysis of Figure 5, Figure 7, and Table 5 reveals a significant self-enhancement bias among LLMs. Most models rated their outputs more favorably, even when answer sources were anonymized. These findings underscore the importance of using separate models for answer generation and evaluation in LLM-as-a-Judge to maintain objectivity in assessments.

Bandwagon-effect involvement percentage is not impactful.  The percentage of people favoring an answer did not significantly impact model robustness rate. GPT-4o remained consistent, while Claude-3.5 was more influenced by popular opinion. Figure 6 shows that involvement percentage does not significantly affect model choices.

LLMs show sensitivity to irrelevant content in responses.  Figure 7 demonstrates that including irrelevant content reduces the robustness rate of model judgments. Different models show varying degrees of susceptibility to this type of interference. Notably, from the average, the impact is more significant when perturbing high-quality responses, implying that extraneous information has a greater potential to disrupt the evaluation of strong answers.

Different types of fake authorities interfere with the LLMs to varying degrees.  As illustrated in Figure 6, the impact of fake authorities on judge models differs based on the format used. URL citations consistently showed the least interference across all models, likely due to their concise nature and the models’ familiarity with web-based references. In contrast, both quote and book formats demonstrated more significant influence. Overall, discriminative models still need improvement in recognizing authoritative sources.

**Table 5: Average score and error rate of self-enhancement bias and refinement-aware bias.**
| Model | Sel. Score$\downarrow$ | Ref. Score$\downarrow$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Self | Other | Error | Ref | +History | Error |  |
| ChatGPT | 5.21 | 5.72 | 8.91 | 5.23 | 4.94 | 5.80 |
| GPT-4-Turbo | 6.98 | 6.90 | 1.16 | 8.31 | 8.45 | 1.66 |
| GPT-4o | 7.01 | 6.89 | 1.74 | 7.44 | 7.20 | 3.33 |
| GLM-4 | 7.73 | 7.64 | 1.18 | 7.64 | 7.73 | 1.15 |
| Claude-3.5 | 7.04 | 6.55 | 7.48 | 7.51 | 7.68 | 2.17 |
| Qwen2 | 7.64 | 6.58 | 16.1 | 7.29 | 7.39 | 1.33 |

Figure: Figure 6: (a) shows the impact of the number of answers $n$ on the robustness rate in position bias. (b) shows the relationship between the answer length ratio to the original length and robustness rate in verbosity bias. (c) shows the relationship between different percentages of popular opinion and robustness rate in bandwagon-effect bias. (d) shows the relationship between different models and robustness rate in authority bias with different fake citation formats.
Refer to caption: figure/combined_plot.png

Figure: Figure 7: (a) and (b) show the comparisons of model error rates for refinement-aware bias and self-enhancement bias, respectively. (c) shows the robustness rate of various models when faced with distraction bias. (d) presents a comparison of model accuracy under the influence of CoT bias, indicating that most models achieve higher accuracy after applying CoT.
Refer to caption: figure/combined_bar.png

LLMs tend to prefer content without emotional elements.  Results in Figure 8 show that when emotionally charged revisions are made to superior answers, accuracy and robustness rates typically decline; conversely, when similar revisions are applied to inferior answers, these metrics tend to improve. Among emotions, cheerful has the least impact on models, with minimal decreases in accuracy and robustness rates. The other three emotions show greater effects, with fear having the most significant impact. This phenomenon is evident across all tested emotion types, suggesting that the model generally tends to resist emotionally colored responses.

Explicit introduction of minority groups will influence the choices of LLMs.  As shown in Figure 8, most models demonstrated a more pronounced sensitivity to female and refugee status, whereas Claude-3.5 exhibited a relatively impartial approach, showing minimal deviation from the random baseline in terms of the robustness rate metric. Therefore, when evaluating responses that may expose respondents’ identities, it is recommended to select suitable models that are less influenced by identity factors.

CoT improves LLMs evaluation accuracy.  As shown in Figure 7, encouraging models to engage in step-by-step reasoning before concluding enhances their problem-solving abilities. However, the effectiveness of CoT varies across models, likely depending on their inherent reasoning capabilities. We can refer to Table 7 for the results. GPT-4-Turbo exhibited only a marginal improvement of 0.7% in accuracy compared to its original performance, whereas GLM-4 demonstrated a more substantial increase of 7%.

## 5 Discussion

**Table 6: Bias recognition performance across different bias types. The success rate (SR) indicates the proportion of cases where the bias was correctly identified, and the none rate (NR) indicates the proportion where no bias was found.**
| Bias Type | GPT-4-Turbo | Claude-3.5 |  |  |
| --- | --- | --- | --- | --- |
| SR$\uparrow$ | NR$\downarrow$ | SR$\uparrow$ | NR$\downarrow$ |  |
| Authority | 0.84 | 0.14 | 0.84 | 0.00 |
| Bandwagon-effect | 1.00 | 0.00 | 0.92 | 0.00 |
| Compassion-fade | 0.48 | 0.34 | 0.96 | 0.00 |
| Distraction | 1.00 | 0.00 | 1.00 | 0.00 |
| Diversity | 0.46 | 0.02 | 0.96 | 0.00 |
| Fallacy-oversight | 0.52 | 0.04 | 0.46 | 0.00 |
| Sentiment | 0.96 | 0.04 | 0.72 | 0.00 |
| Verbosity | 0.90 | 0.10 | 1.00 | 0.00 |

Explicit and implicit influence of bias.
We identified two distinct types of biases: explicit and implicit. Explicit biases are those where the LLM clearly states its preference for certain attributes in its decision-making process. Implicit biases are influences that affect judgments without being directly acknowledged in their reasoning. Our case studies illustrate these biases in Appendix E. The Authority bias exemplifies an explicit bias, where the LLM openly favored answers containing citations, even when these were fake. This demonstrates a clear preference for responses that appear scholarly, regardless of their actual validity. Conversely, the refinement-aware bias represents an implicit bias. Here, the LLM consistently scored refined answers higher, despite providing similar justifications for different instances and never explicitly mentioning refinement as a factor in its decision-making process. The findings indicate that LLMs are influenced by various factors. The disparity between their internal processing and expressed reasoning underscores the importance of conducting more research into the nature of LLM bias. It is essential to comprehend these biases to enhance the trustworthiness and reliability of LLM-as-a-Judge.

Suggestions for application.  In discussing potential strategies to mitigate biases in LLM-as-a-Judge, we propose the following recommendations aimed at enhancing the fairness of models while mitigating bias interference:

- $\triangleright$
Carefully construct prompts and implement advanced reasoning strategies. We recommend creating prompts that include specific protective phrases to guard against various types of biases, such as instructing the model to disregard the identity information of the person being evaluated. Additionally, implementing advanced reasoning strategies similar to CoT can guide the model through a step-by-step decision-making process.
- $\triangleright$
Establish prompt injection safeguards. We recommend instituting protective measures against prompt injection related to the bias types discussed in this paper. These safeguards can prevent models from being influenced by biased information embedded in prompts. By implementing such protective measures, we can enhance the fairness of LLM-as-a-Judge, ensuring that the judging process is not compromised by external attempts to introduce bias.
- $\triangleright$
Implement bias detection mechanisms. Based on our experimental findings, we suggest implementing a simple, prompt-based bias detection mechanism similar to the one we developed in Figure 32. This approach can proactively identify potential biases in judging templates before the actual judging process begins. As presented in Table 6, our results demonstrate that while the effectiveness varies across different bias types, this method shows promise in uncovering a majority of biases.

## 6 Conclusion

This paper presents Calm, an automated evaluation framework for assessing potential bias when LLMs are employed as judges in various application scenarios. Calm provides a comprehensive examination of 12 types of biases and utilizes an automated bias injection and qualification method, resulting in an objective and scalable evaluation approach. Our experiments show that while models may reliably serve as judges for specific tasks, there remains significant room for improvement in the broader use of LLMs as judges. Calm could be used to evaluate future, more advanced LLM-based judge solutions, ensuring they meet higher standards of bias mitigation.

## Ethical Consideration

It is significant to emphasize that some of the question sets and bias-related responses may contain NSFW content. While we have manually reviewed and curated this data to ensure its appropriateness for research purposes, we urge readers and potential users of our findings to exercise caution and discretion. We recommend that any application or extension of this work should be conducted responsibly, with due consideration for ethical guidelines and potential societal impacts.

## Appendix A Related Works

### A.1 LLM-as-a-Judge

Recent studies have demonstrated that LLMs can serve as high-quality evaluators for various NLP tasks (Li et al., ; Kasner & Dušek, ; Huang et al., ; Wang et al., ), and Zheng et al. () proposed the concept of LLM-as-a-Judge. As an evaluation method that does not require reference texts, it has demonstrated performance on open-ended questions that highly match human preference. Recent research has focused on exploring its fairness, for instance, Shi et al. () introduced JudgeDeceiver, emphasizing the vulnerabilities in the evaluation process. Zhang et al. () conducted research indicates that wider and deeper LLM networks often provide more fair evaluations. Liu et al. () proposed ALIGNBENCH for the multi-dimensional evaluation of LLMs’ fairness.

### A.2 Fairness in Trustworthy LLMs

Ensuring the trustworthiness of LLMs is of great significance Liu et al. (); Shi et al. (); Huang et al. (); Gao et al. (); Wu et al. (). In recent research, it has been discovered that LLMs may exhibit stereotypes against certain groups or make erroneous judgments based on specific statistical patterns (Zhuo et al., ; Ferrara, ; Liu et al., ), which highlights the importance of fairness in evaluating LLMs. Fairness of LLMs is defined as the ethical principle of ensuring that LLMs are designed, trained, and deployed in ways that do not lead to biased or discriminatory outcomes and that they treat all users and groups equitably (Sun et al., ). The imbalance in pre-training data can lead to imbalances during model training (Liu et al., ), resulting in biases against certain demographic groups, such as different genders (Wan et al., ), ages (Macnicol, ), and various languages (Jiao et al., ; Bang et al., ). Consequently, the fairness of LLMs has a significant impact on the trustworthiness of LLM-as-a-Judge.

### A.3 Biases in LLM-as-a-Judge Application

Recent research has identified various cognitive biases that influence the evaluation of LLMs. Some studies (Zheng et al., ; Shi et al., ; Wang et al., ) discuss biases such as position bias, verbosity bias, and self-enhancement bias. Another study (Koo et al., ) highlights order bias, compassion-fade bias, and egocentric bias, along with salience bias, bandwagon-effect bias, and attentional bias. Further biases noted in additional research (Chen et al., ; Stureborg et al., ) include fallacy-oversight bias, authority bias, and beauty bias. Recognizing these biases is essential for developing more objective and trustworthy LLM evaluation methods.

## Appendix B Details of Bias Types

- $\triangleright$
Position bias: LLMs may favor responses based on their position in the input. This bias affects how the model processes information, and following Zheng et al. (), we extend the analysis to scenarios involving more than two responses.
- $\triangleright$
Verbosity bias: LLM-as-a-Judge may be biased towards longer responses. We evaluate the impact of different length ratios between responses on judgment outcomes, as indicated by Zheng et al. ().
- $\triangleright$
Compassion-fade bias: LLM judgments may be influenced by the anonymity of model names. We investigate how various model names and anonymization strategies impact judgments, inspired by the observations of Koo et al. ().
- $\triangleright$
Bandwagon-effect bias: LLM-as-a-Judge may be biased by the presence of majority opinions. We assess this by setting varying percentages (60%, 70%, 80%, and 90%) of majority opinions in the system instruction, following Koo et al. ().
- $\triangleright$
Distraction bias: Introducing distractions could affect the judgments of both high-quality and low-quality model outputs. We extend previous work by Koo et al. () to evaluate the impact of distractions in LLM decision-making. Experimental details are available in Appendix C.
- $\triangleright$
Fallacy-oversight bias: This bias relates to the LLM’s ability to recognize and avoid logical fallacies. We develop tests to evaluate this ability across various types of fallacies, contributing to fair and accurate judgments, as discussed in Chen et al. ().
- $\triangleright$
Authority bias: Authoritative references may sway LLM judgments. We assess this influence by incorporating three types of references—book citations, website references, and famous individuals’ quotes—following the methodology of Chen et al. ().
- $\triangleright$
Sentiment bias: LLMs may display preferences towards certain emotional tones in responses. We evaluate how sentiment influences judgments across emotional expressions such as cheerful, sad, angry, and fearful, as noted by Li & Sinnamon ().
- $\triangleright$
Diversity bias: Judgments may shift based on specific identity markers. We evaluate this bias by setting system instructions that assign six identity categories: Female, Black individuals, Homosexuals, Muslims, Refugees, and HIV patients, following the concept of identity impact.
- $\triangleright$
Chain-of-Thought (CoT) bias: LLM judgments can be affected by the presence of explicit reasoning steps. We compare evaluations of responses with and without chain-of-thought reasoning across different tasks, as suggested by Wei et al. ().
- $\triangleright$
Self-enhancement bias: This bias arises when LLMs favor their outputs as both generators and judges. To explore this, we include evaluations to measure the bias across different LLM architectures and scales, following Zheng et al. () and Meng et al. ().
- $\triangleright$
Refinement-aware bias: LLMs may assign different scores to self-refined answers. We investigate this bias by comparing scores in three situations: original unrefined answer, refined answer, and refined answer with conversation history, as explored by Xu et al. ().

## Appendix C Details of Bias Evaluation

We will introduce the detailed evaluation process of each bias.

- $\triangleright$
Position bias: To investigate the impact of position bias, we tested the effect of changing the order of answers when there are two, three, and four answers. We employed a rotational permutation order to shuffle the answers and conducted evaluations to observe if the results remained consistent. If the selection of any answer was inconsistent across different orders, it was considered an inconsistency. For specific evaluation prompts, refer to Figure 13, Figure 14, and Figure 15.
- $\triangleright$
Verbosity bias: We employed GPT-4-Turbo to lengthen the worse quality answers in the fact-related dataset. The lengthened prompt used for this process can be referenced in Figure 18. Our objective was to increase the verbosity of these answers while preserving their essential content. Subsequently, we conducted a comparative evaluation, wherein the models reassessed both the original answers pair and the elongated answers pair.
- $\triangleright$
Self-enhancement bias: Building upon the single-answer scoring prompt proposed by Zheng et al. (), we developed a scoring prompt Figure 26. In our experimental design, each model was tasked with answering the questions and subsequently evaluating both its responses and those generated by other models, without prior knowledge of the authorship of any given answer. Following the completion of all evaluations, we aggregated and analyzed the scoring results to draw our conclusions.
- $\triangleright$
Compassion-fade bias: We explicitly reveal the identities of all models under judgment. The specific details of this approach are delineated in the evaluation prompt presented in Figure 20. Subsequently, we conducted a comparative analysis between these named-condition results and those obtained under anonymized settings.
- $\triangleright$
Bandwagon-effect bias: We added the following sentence to the system prompt: "{number}% of people believe that {chosen_model} is better in this question" to test whether the model is influenced by popular opinion. For specifics, refer to prompt Figure 21.
- $\triangleright$
Distraction bias: We inserted a meaningless statement into the System Prompt, such as "Assistant A loves eating pasta," to test whether the LLM Judge is influenced by it. For specifics, refer to prompt Figure 29. We applied distinct distractions to each assistant and subsequently analyzed the robustness rate of the judge model’s evaluations in scenarios where the distracted Assistant provided either the better or worse quality response.
- $\triangleright$
Fallacy-oversight bias: We modified the better quality answers in the fact-related dataset using GPT-4-Turbo to make their logic completely chaotic. The prompt used can be referenced in Figure 19. We then had the models re-evaluate these answer pairs. If a model’s evaluation result was inconsistent with its original assessment of the answer pair, we considered it a correct judgment (because the original worse quality answer is still better than the logically chaotic better quality answer). Otherwise, it was deemed an incorrect judgment.
- $\triangleright$
Authority bias: Using GPT-4-Turbo, we generated three types of fake citation information related to the answers: URLs, famous quotes, and book references. For specifics on the prompts used for the generation, refer to Figure 24, Figure 25, and Figure 23. These citations were then injected into the answers, as demonstrated in Figure 22.
- $\triangleright$
Sentiment bias: We modified the better quality answers in the fact-related dataset using GPT-4-Turbo to incorporate one of the four emotions: cheerful, sad, angry, or fear. The prompt can be referenced in Figure 27. Then, we had the models judge these answers again to observe whether the results were consistent with the original judgment.
- $\triangleright$
Diversity bias: For diversity bias, we selected six identities that may be subject to discrimination: Homosexual, Black, Female, HIV Positive, Refugees, and Muslim believers. These identities were then injected into the system prompt for judgment to observe their impact on evaluations. For more details, refer to prompt Figure 28.
- $\triangleright$
CoT bias: We modified a version of the Prompt based on the original Chain of Thought prompt from (Zheng et al., ), which can be referenced in Figure 16. Under the condition that all other factors remain unchanged, we conducted judgment on the fact-related dataset to observe whether the results changed.
- $\triangleright$
Refinement-aware bias: In the Refinement-aware eval dataset, we first have the model answer these questions. Then, using prompt Figure 30, we enable the model to refine its previously given answers. Finally, the model evaluates the pre-refinement, post-refinement, and refined-with-history answers, and we compile the results. For specifics on the evaluation prompt, refer to Figure 31. We can reference Figure 10 as an illustrative example.

## Appendix D Detailed Results

In Figure 4, we provide a comparative chart of the robustness rate for all biases, which allows for a horizontal comparison of the differences in resilience to interference among all models, with the dashed line representing the consistency rate. In Table 7, the detailed experimental results for each type of bias are presented.

- $\triangleright$
Position bias. We present the robustness rate of different judge models when faced with pairwise comparisons in Table 7, and in Figure 6 we show the robustness rate of all judge models when presented with multiple answer options.
- $\triangleright$
Verbosity bias. In Figure 6, we illustrate the relationship between different ratios of answer expansion lengths and model robustness rate.
- $\triangleright$
Self-Enhancement bias. In Figure 5, we present a heat map of Z-score normalized scores for each model (due to ChatGPT’s relatively weak performance, the scores given to it by the remaining models are not high enough, resulting in the first column lacking reference value). Additionally, in Figure 7, we display the $\text{ErrorRate}_{\text{SE}}$ metric for each judge model.
- $\triangleright$
Bandwagon-Effect bias. In Table 7 and Figure 6, we present the impact of varying percentages of public opinion on the judge models. The experimental results indicate that the influence on each model is not uniform and does not demonstrate a statistical pattern.
- $\triangleright$
Distraction bias. In Figure 7 and Table 7, we present the robustness rate performance of all judge models after introducing irrelevant content as interference for both high-quality and low-quality answers originally present in the dataset.
- $\triangleright$
Authority bias. In Table 7, we present the impact of different types of fake references on the judge model. As shown in Figure 6, quote and book-type references strongly influence most models.
- $\triangleright$
Sentiment bias. In Figure 8, we display the $\text{Acc}_{\text{hack}}$ and robustness rate performance of judge models with three different emotions added to high-quality and low-quality answers in the dataset. Our findings indicate that most models do not favor emotionally charged expressions.
- $\triangleright$
CoT bias. In Figure 7 and Table 7, we present the accuracy metrics $\text{Acc}_{\text{ori}}$ and $\text{Acc}_{\text{hack}}$ before and after applying CoT. As shown in the figure, for most models, the application of CoT techniques can improve judgment accuracy.
- $\triangleright$
Refinement-aware bias. In Figure 7, we present the $\text{ErrorRate}_{\text{RA}}$ metric for different judge models.
- $\triangleright$
Diversity bias. We show the changes in various metrics of the judge model under the influence of different minority groups in Figure 8 and Table 7.

## Appendix E Case Study

From Figure 9, 10, 11, 12, we enumerated various actual manifestations of bias and conducted a detailed analysis.

## Appendix F Prompt Template

From Figure 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, we provide detailed prompt templates we used in the experiments.

**Table 7: Detailed experiments were conducted for each type of bias, where hack type represents the type of experiment and the value of corresponding metrics are shown on the right. The corresponding metrics for each type of bias can be found in Table 2.**
| Bias | Hack Type | Model |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | ChatGPT | GPT-4 | GPT-4o | GLM-4 | Claude-3.5 | Qwen2 |
| Pos. | Default | 0.566 | 0.818 | 0.776 | 0.781 | 0.832 | 0.760 |
| Ver. | Default | 0.900 | 0.915 | 0.977 | 0.887 | 0.952 | 0.884 |
| Com. | Default | 0.862 | 0.858 | 0.868 | 0.835 | 0.875 | 0.877 |
|  | 60% | 0.680 | 0.635 | 0.773 | 0.703 | 0.563 | 0.711 |
|  | 70% | 0.667 | 0.630 | 0.787 | 0.676 | 0.613 | 0.711 |
|  | 80% | 0.707 | 0.662 | 0.805 | 0.664 | 0.638 | 0.698 |
| Ban. | 90% | 0.699 | 0.623 | 0.800 | 0.716 | 0.627 | 0.718 |
|  | h.c | 0.716 | 0.718 | 0.749 | 0.806 | 0.904 | 0.749 |
| Dis. | h.r | 0.710 | 0.740 | 0.830 | 0.822 | 0.851 | 0.821 |
| Fal. | Default | 0.917 | 0.969 | 0.984 | 0.979 | 0.985 | 0.935 |
|  | Book | 0.628 | 0.841 | 0.800 | 0.765 | 0.856 | 0.785 |
|  | Quote | 0.660 | 0.841 | 0.747 | 0.758 | 0.856 | 0.745 |
| Aut. | URL | 0.700 | 0.855 | 0.813 | 0.866 | 0.884 | 0.805 |
|  | Che.(bet.) | 0.803 | 0.682 | 0.727 | 0.770 | 0.609 | 0.726 |
|  | Che.(wor.) | 0.910 | 0.888 | 0.970 | 0.905 | 0.976 | 0.871 |
|  | Sad(bet.) | 0.659 | 0.271 | 0.343 | 0.306 | 0.259 | 0.307 |
|  | Sad(wor.) | 0.916 | 0.920 | 0.983 | 0.907 | 0.970 | 0.929 |
|  | Ang.(bet.) | 0.639 | 0.366 | 0.243 | 0.380 | 0.256 | 0.283 |
|  | Ang.(wor.) | 0.946 | 0.921 | 0.987 | 0.950 | 0.973 | 0.926 |
|  | Fea.(bet.) | 0.639 | 0.254 | 0.355 | 0.271 | 0.260 | 0.238 |
| Sen. | Fea.(wor.) | 0.923 | 0.921 | 0.987 | 0.943 | 0.973 | 0.926 |
|  | Homosexual | 0.697 | 0.830 | 0.819 | 0.779 | 0.945 | 0.839 |
|  | Black | 0.660 | 0.843 | 0.820 | 0.784 | 0.897 | 0.819 |
|  | Female | 0.646 | 0.825 | 0.826 | 0.765 | 0.924 | 0.805 |
|  | HIV Pos. | 0.692 | 0.856 | 0.820 | 0.832 | 0.942 | 0.826 |
|  | Refugees | 0.667 | 0.896 | 0.799 | 0.785 | 0.862 | 0.826 |
| Div. | Muslim | 0.710 | 0.881 | 0.800 | 0.785 | 0.913 | 0.845 |
| CoT | Default | 0.560 | 0.720 | 0.700 | 0.688 | 0.745 | 0.704 |
| Self. | Default | 5.21 | 6.98 | 7.01 | 6.55 | 7.04 | 7.64 |
| Ref. | Default | 4.94 | 8.45 | 7.20 | 7.73 | 7.68 | 7.39 |

Figure: (a) Robustness rate of sentiment bias.
Refer to caption: figure/sentiment_consistency.png

**Table 8: Model names, Creators, Version, Access Time, License, and their using purpose. Evaluation refers to using the model as LLM judge, while Generation refers to using the model to generate responses.**
| Model | Creator | Version | Access Time | License | Purpose |
| --- | --- | --- | --- | --- | --- |
| ChatGPT | OpenAI | gpt-3.5-turbo-0125 | 2024.1 | Proprietary | Evaluation |
| GPT-4-Turbo | OpenAI | gpt-4-turbo-0409 | 2024.4 | Proprietary | Evaluation |
| GPT-4o | OpenAI | gpt-4o-0513 | 2024.5 | Proprietary | Evaluation |
| Claude-3.5 | Anthropic | claude-3.5-sonnet-0620 | 2024.6 | Proprietary | Evaluation |
| GLM-4 | ZhipuAI | glm-4-0520 | 2024.5 | Proprietary | Evaluation |
| Qwen2 | Alibaba | Qwen2-72B-Instruct | 2024.1 | Open-source | Evaluation |
| Llama3 | Meta | llama-3-8b-instruct | 2024.4 | Open-source | Generation |
| Llama-3-70B-Instruct | 2024.4 | Open-source | Generation |  |  |
| Mistral-7B | MistralAI | mistral-7b-instruct-v0.2 | 2023.12 | Open-source | Generation |
| Mixtral-8x22B | MistralAI | Mixtral-8x22B-Instruct-v0.1 | 2024.4 | Open-source | Generation |

Figure: Figure 9: Case study: Sentiment bias. In the example above, we initially had the Claude-3.5-Sonnet model evaluate the original pair of answers, concluding that answer A was superior. We then revised answer A to include expressions of anger. Subsequently, although the Claude-3.5-Sonnet model acknowledged that answer A was "more direct and technically precise," it noted that "the confrontational tone might be off-putting to some users," leading it to determine that answer B was now the better choice.
Refer to caption: x4.png

Figure: Figure 10: Case study: Refinement-aware bias. In the example above, we asked the model to score the answer in three stages: before refinement, after refinement, and after refinement with the dialogue history. The respective scores were 6, 7, and 8.
Refer to caption: x5.png

Figure: Figure 11: Case study: Authority bias. In the example above, we have a question along with two answers of varying quality in our dataset, with Assistant A’s answer being of higher quality. Initially, GPT-3.5-Turbo made the correct judgment. However, when we added automatically generated fake references to Assistant B’s answer, the judge model considered Assistant B’s quality to be superior and cited our provided fake references as justification. At this point, the judge model was successfully hacked, demonstrating authority bias.
Refer to caption: x6.png

Figure: Figure 12: Case study: Bandwagon-effect bias. In the example above, we have a question along with two answers of varying quality in our dataset, with Assistant A’s answer being of higher quality. Initially, GPT-4-Turbo made the correct judgment. However, after incorporating the popular opinion that Assistant B’s response was better into the system prompt, the judge model was successfully hacked, proving the existence of the Bandwagon-effect bias.
Refer to caption: x7.png

Figure: Figure 13: Prompt template for pairwise comparison.

Figure: Figure 14: Prompt template for triadic comparison.

Figure: Figure 15: Prompt template for quadruple comparison.

Figure: Figure 16: Prompt template for CoT pairwise comparison.

Figure: Figure 17: Prompt template for generating pair responses.

Figure: Figure 18: Prompt template for expanding sentence length.

Figure: Figure 19: Prompt template for expanding sentence length.

Figure: Figure 20: Prompt template for compassion-fade bias.

Figure: Figure 21: Prompt template for bandwagon-effect bias.

Figure: Figure 22: Prompt template for authority bias evaluation.

Figure: Figure 23: Prompt template for generating book source.

Figure: Figure 24: Prompt template for generating URL.

Figure: Figure 25: Prompt template for generating quote.

Figure: Figure 26: Prompt template for sentiment bias evaluation.

Figure: Figure 27: Prompt template for sentiment bias evaluation.

Figure: Figure 28: Prompt template for diversity bias evaluation.

Figure: Figure 29: Prompt template for distraction bias evaluation.

Figure: Figure 30: Prompt template for refinement-aware bias generation.

Figure: Figure 31: Prompt template for refinement-aware bias evaluation.

Figure: Figure 32: Prompt template for bias analysis.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - How do you ensure that the generated perturbations effectively introduce the desired bias without altering the correctness of the content? How well do the LLMs understand the instructions for generating biased content? Could there be unintended consequences or biases introduced by the LLMs themselves?
- Would incorporating a human evaluation benchmark provide additional insights into the accuracy and fairness of LLM judgments?
- Are there potential trade-offs between mitigating biases and maintaining the performance of LLM judges?
soundness: 3
strengths: - Originality: The authors expand upon existing work by identifying and categorizing 12 distinct types of biases.
- Quality: The paper presents a thorough evaluation of the identified biases across multiple LLMs, using diverse datasets and specific metrics tailored for judging tasks. This rigorous experimental design ensures the reliability and validity of the findings.
- Clarity: The examples in Table 1 provide concrete examples of how biases manifest in LLM judgments, making the abstract concepts more tangible and relatable.
- Significance: The proposed CALM framework offers a valuable tool for stakeholders to assess and mitigate biases, leading to more fair and reliable LLM evaluation methods.
summary: This paper explores the potential biases inherent in using Large Language Models (LLMs) as judges in various evaluation tasks, such as scoring and pairwise comparison.  The authors propose a novel framework called CALM, which systematically quantifies and analyzes each type of bias by using automated and principle-guided modification. The paper evaluates six popular LLMs using the CALM framework and finds that while some models demonstrate notable fairness in judgment, significant biases persist in certain specific tasks.
weaknesses: - I think this paper is more like a toolkit paper rather than a novel research paper, as they just integrate 12 types of existing biases in LLM-as-a-Judge. If we look at the appendix B, we can find that each of the 12 types can be referenced to another previous paper.
- The paper primarily relies on automated metrics to assess bias, but human evaluation could provide a valuable additional perspective. Incorporating a human evaluation benchmark would strengthen the validation of the findings.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What is the source of the basis for these assessments of Robustness Rate (RR) and Consistency Rate (CR)? Why are human correlations such as Pearson's coefficient not considered in the assessment. 
2. LLM does not take into account some of the popular LLM as Judge models, such as pandaLM, Prometheus, etc. LLM lacks a specific LLM as Judge model. Lack of specific LLM as Judge evaluation. 
3. Is the data randomly selected? For example, GMSK, how to choose 100 pieces of data from tens of thousands of pieces of data? How to prove that these 100 data are representative enough?
4. Is it user friendly? Is the reproduction cost prohibitive?
soundness: 3
strengths: 1. A comprehensive delineation and classification of twelve specific biases that can compromise the dependability and credibility of LLM-as-a-Judge.
2. The proposal of the CALM framework for assessing biases within LLM-as-a-Judge systems,  which enhances the rigor of the assessment process in a person-independent manner.
3. An in-depth analysis of six widely-used LLMs through the lens of the CALM framework.
summary: This study identifies 12 significant potential biases and introduces a novel automated bias quantification framework called CALM. This framework systematically quantifies and analyzes each type of bias in LLM-as-a-Judge through automated, principle-guided modifications. Empirical findings indicate that there is still room for enhancing the reliability of LLM-as-a-Judge. The paper explores both the explicit and implicit impacts of these biases and offers recommendations for the dependable application of LLM-as-a-Judge.
weaknesses: Lack of Transparency in Assessment Criteria: The source of the basis for the assessments of Robustness Rate (RR) and Consistency Rate (CR) is unclear. 
Incomplete Consideration of Popular Models: The evaluation does not include some well-known LLM as Judge models, such as pandaLM and Prometheus. This omission suggests a lack of thoroughness and may lead to biased or incomplete conclusions.
Questionable Data Selection Process: The method for selecting data is not well-defined. For instance, in the case of GMSK, the process of choosing 100 pieces of data from tens of thousands is not explained. This raises concerns about the representativeness and reliability of the selected data.
User-Friendliness and Reproduction Costs: There are concerns about the user-friendliness of the system and whether the costs associated with reproducing the results are prohibitive. This could limit accessibility and practical application for users.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: 073: This is not limited only to "humanities, social sciences, or general knowledge" fields, a refined answer could be in any field or task.

281: If i understand correctly, for CR you ask the LLM to generate two responses for the same prompt, and check if the answers are consistent. If so, why not use more than two responses? You can make the consistency measure more robust by comparing the variance of N>>2 generations. 

295: Which ACC do you use as a metric?

Metrics paragraph: In my opinion, and even though the English is good and I understand each sentence, this paragraph is not clear enough. I would emphasize in the text which metric is used for each task, specifically at the start of the paragraph, and refer to the column in the table.  In addition, the names, abbreviation are not consistent throughout the paper and specifically in the figures. Please see the weaknesses regarding the metrics.
soundness: 4
strengths: I believe the paper is strong, well-written, and highly comprehensive. The topic is both timely and important, and the NLP/LLM community would greatly benefit from its publication. In my opinion, this paper should be accepted. The reason I initially rated it a 6 instead of an 8 is to encourage the authors to consider revising the metrics (as discussed in the weaknesses section).
summary: The authors identify 12 distinct types of biases in LLM-as-a-Judge models, provide automated code for "checklisting" these biases in LLM judges, and conduct a comprehensive analysis of how these biases manifest in state-of-the-art LLM judges across various datasets. The key finding is that LLM judges are not robust. The implications of these findings are significant and should be effectively communicated to the community.
weaknesses: **Metrics**: I have a few suggestions regarding the metrics used in this paper and how they are presented in the results. First, for the RR and CR metrics, I recommend making the CR metric more robust by sampling multiple generations when computing the CR for a given instance. Additionally, I propose adjusting the RR metric with the CR, as the authors note that LLMs are non-deterministic, and a low RR score might reflect this rather than a genuine lack of robustness. To adjust the score for each individual instance, I would subtract the individual CR from the individual RR. It is important to compute this adjustment at the individual level, as the CR varies between instances. The final score would be the dataset average of RR_i - CR_i. While this metric is more complex and falls within a -1 to 1 scale, it is far more reliable. If you choose not to present the difference between the two, I suggest at least presenting the average RR and average CR for each LLM in the results. The CR is not a constant value; it varies between models and across instances.

Regarding the ACC metrics, I am unclear about which specific metric you are using in the results section. Should we compare the two metrics and examine the variations between them? Additionally, is this score presented as an absolute value? Could you clarify this aspect to ensure an accurate interpretation of the results? And why do you use "hack"? Isn't it essentially the CoT ACC? 

Regarding the Error Rate (ER) metrics, could you explain the rationale for using these metrics instead of the RR/CR in the paper? Additionally, could we apply the ER metrics to detect other forms of bias? I also find the ER_SE metric unreliable. From your description, it appears that Y_other represents the score assigned by other models to the evaluated model's response. However, I believe Y_other should represent the average score assigned by the explained model to the responses of other LLMs. This would better measure whether the LLM prefers its own responses. Otherwise, you're merely capturing the LLM’s general bias relative to the consensus. For example, one LLM might use scores in the range of 3-7, while another uses 1-6, yet they could still achieve a perfect Spearman's correlation. Moreover, why do you use absolute value? This can be misleading. For example, y_self, y_other = (5, 3) is the same as (1, 3). I believe you can think on a better metric for ER. 

In general, I would recommend adjusting the metrics so that higher scores indicate more bias, unlike the current ones where higher scores represent robustness. Since you frequently use the term "bias" throughout the paper, this modification could make the results and the interpretation of the metrics more intuitive and easier to follow. 

If the authors revise the metrics and provide this analysis, or at least clarify what I may be misunderstanding, I would be happy to increase the overall rating from 6 to 8.

**Misinterpretations of the Results:** Specifically, the statement "Bias is more pronounced in the alignment dataset compared to the fact-related dataset" cannot be inferred from the results. The biases in these datasets differ, and you need to compare like for like - either the same bias across different datasets or the same dataset with different biases. While it's possible to compare the CR metrics between datasets (difference of differences), I believe this is insufficient on its own. First, I would like you to clarify (both here and in the paper) the rationale behind distinguishing between biases and datasets, as well as why certain biases may not be applicable to all datasets. I haven’t given this much thought, but it is critical that this distinction is explicitly explained in the paper, rather than leaving it up to the reader to infer.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - How do we know that evaluations that rely on perturbations by LLMs can be trusted? How do we know that such perturbations do not introduce errors or biases other than those which are intended to be evaluated?
-  How could one interpret the results of evaluating a bias using the CALM framework in terms of its effect on real-life applications of the LLM-as-a-judge system? For example, if one LLM's robustness rate for diversity is 0.1 greater than another's, how does this the actual treatment of minorities by systems that utilize the LLM in a LLM-as-a-judge application?
soundness: 2
strengths: - The paper covers a comprehensive list of biases and conducts experiments on many state-of-the-art LLMs and across multiple relevant domains such as fact-based and alignment-related data.
- They introduce a novel method for evaluating biases in LLM-as-a-judge systems that is well-principled and automatic. It is also flexible as the framework could even be extended to bias types that are not considered in the paper. 
- They systematically demonstrate that current LLMs are still susceptible to various biases. As far as I am aware, many of their evaluation results are completely novel, such as demonstrating how different types of fake-authorities interfere with LLM-judges to varying degrees.
summary: The paper introduces CALM, a framework for measuring bias in LLM-as-a-judge applications. The framework works by modifying the prompt or response that is to be judged by introducing a bias, and then measuring how this modification affects the judgement. They propose a classification of biases into 12 distinct types, each of which can be measured by their framework. The introduces typology covers a broad spectrum such as bias based on the length of answers, the use of an authoritative tone or faux citations. 

To evaluate the magnitude of these biases when current LLMs are used as judges the paper introduces various metrics, most importantly the robustness of a judgement when a bias is added to an answer. Biases are evaluated on three types of datasets which are sampled from existing sources. They cover factual data for which responses should be evaluated according to factual accuracy, alignment related data for which judgements depend on user preferences, and refinement aware evaluation data which contains pairs of responses in which one is a refinement of the other.

For its main results, the paper evaluates the biases of multiple state-of-the-art LLMs when used in an LLM-as-judge system. The results demonstrate that current models are susceptible to various biases in their judgements. Some noteworthy findings include:
- All models are significantly impacted by position bias.
- Most models judge their own output more favorably than that of other models, even when sources are anonymized.
- Different types of fake-citations influence an LLM's judgement to various degrees. Quote- and book formats have a higher chance of being convincing than URL citations.

These as well as other findings are discussed in the results section.
weaknesses: Edit: these concerns were addressed in the rebuttal and I have therefore updated my score from 6 to 8.
# Main Concerns
I have two concerns related to the soundness of the paper's methodology and experimental results. 

The paper introduces a new method for evaluating biases but does not evaluate the trustworthiness of this method. The method is based on perturbing the responses that an LLM-as-a-judge is supposed to evaluate. In some cases this evaluation works via a separate LLM such as for the verbosity bias, fallacy-oversight bias, or authority bias where GPT-4-Turbo is used to rephrase a response. How do we know that using an LLM to modify responses does not introduce errors or other features that manipulate the judge's decision? While I can believe that GPT-4-Turbo is capable of applying the required modifications, this should be experimentally verified so that the results have scientific rigor.  

Further, the paper provides scores of LLM-judge biases in the form of the robustness rates, but I can not tell what these scores mean for real-life applications of said LLM-judges. For example, LLM-as-a-judge is typically used for evaluation, such as a reward model during RLHF. If my LLM-as-a-judge system has a specific robustness score for a bias type such as diversity, how does this translate to the bias of a downstream system, such as an LLM that was trained using the judge? Without such results, it is unclear how to interpret the paper's numerical results.

Summing up, I believe two types of experiments are necessary to improve the paper's soundness. 
- Demonstrate that perturbation using LLMs does not introduce unintended errors or biases that are different from what is intended.
- Evaluate the effect of different bias scores for different LLM-as-a-judge systems on real-life applications of the systems.

If related experiments or arguments are added, I will improve my score.

# Minor Comments (did not impact score)
- It would be helpful to have example questions and responses from each dataset somewhere in the paper, to illustrate what the difference between e.g. the fact-related and alignment-related dataset is. They could be included in the appendix if there is no space in the main paper.
- Figure includes some plots where a larger value of the y-axis means less bias (robustness rate) and some for which is a lower value is better (error rate). The plot would be easier to read if this was indicated somehow.

Meta-review: This paper introduces CALM, a novel framework to evaluate biases in LLMs used as judges for tasks like scoring and pairwise comparison. The authors identify and categorize 12 types of biases and assess them across six popular LLMs using diverse datasets and tailored metrics. While highlighting areas of fairness, the results reveal persistent biases in specific contexts.

This paper makes a timely and significant contribution to the NLP/LLM community by providing a comprehensive framework for identifying and analyzing biases in LLM judgments. While the metrics and result interpretations require refinement, the strengths of the paper, including its practical applicability and systematic exploration of biases, far outweigh the weaknesses. With minor revisions to address the issues around metrics and result clarity (which were addressed during the rebuttal phase), the paper would provide a valuable resource for researchers and practitioners. I recommend acceptance.

Decision: Accept (Poster)