Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge

Authors: Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, Xiangliang Zhang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM, LLM-as-a-Judge, trustworthy LLM, evaluation

Abstract:
LLM-as-a-Judge has been widely utilized as an evaluation method in various benchmarks and served as supervised rewards in model training. However, despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility. 
Therefore, we identify 12 key potential biases and propose a new automated bias quantification framework—CALM—which systematically quantifies and analyzes each type of bias in LLM-as-a-Judge by using automated and principle-guided modification. Our experiments cover multiple popular language models, and the results indicate that while advanced models have achieved commendable overall performance, significant biases persist in certain specific tasks. Empirical results suggest that there remains room for improvement in the reliability of LLM-as-a-Judge. Moreover, we also discuss the explicit and implicit influence of these biases and give some suggestions for the reliable application of LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications.

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 8

Reviews and discussion:
<thread>
creation date:
1729519118294

thread_type:
official_review

<thread_object>
creator: reviewer_jwiq

<content>
summary:
The paper introduces CALM, a framework for measuring bias in LLM-as-a-judge applications. The framework works by modifying the prompt or response that is to be judged by introducing a bias, and then measuring how this modification affects the judgement. They propose a classification of biases into 12 distinct types, each of which can be measured by their framework. The introduces typology covers a broad spectrum such as bias based on the length of answers, the use of an authoritative tone or faux citations. 

To evaluate the magnitude of these biases when current LLMs are used as judges the paper introduces various metrics, most importantly the robustness of a judgement when a bias is added to an answer. Biases are evaluated on three types of datasets which are sampled from existing sources. They cover factual data for which responses should be evaluated according to factual accuracy, alignment related data for which judgements depend on user preferences, and refinement aware evaluation data which contains pairs of responses in which one is a refinement of the other.

For its main results, the paper evaluates the biases of multiple state-of-the-art LLMs when used in an LLM-as-judge system. The results demonstrate that current models are susceptible to various biases in their judgements. Some noteworthy findings include:
- All models are significantly impacted by position bias.
- Most models judge their own output more favorably than that of other models, even when sources are anonymized.
- Different types of fake-citations influence an LLM's judgement to various degrees. Quote- and book formats have a higher chance of being convincing than URL citations.

These as well as other findings are discussed in the results section.

strengths:
- The paper covers a comprehensive list of biases and conducts experiments on many state-of-the-art LLMs and across multiple relevant domains such as fact-based and alignment-related data.
- They introduce a novel method for evaluating biases in LLM-as-a-judge systems that is well-principled and automatic. It is also flexible as the framework could even be extended to bias types that are not considered in the paper. 
- They systematically demonstrate that current LLMs are still susceptible to various biases. As far as I am aware, many of their evaluation results are completely novel, such as demonstrating how different types of fake-authorities interfere with LLM-judges to varying degrees.

weaknesses:
Edit: these concerns were addressed in the rebuttal and I have therefore updated my score from 6 to 8.
# Main Concerns
I have two concerns related to the soundness of the paper's methodology and experimental results. 

The paper introduces a new method for evaluating biases but does not evaluate the trustworthiness of this method. The method is based on perturbing the responses that an LLM-as-a-judge is supposed to evaluate. In some cases this evaluation works via a separate LLM such as for the verbosity bias, fallacy-oversight bias, or authority bias where GPT-4-Turbo is used to rephrase a response. How do we know that using an LLM to modify responses does not introduce errors or other features that manipulate the judge's decision? While I can believe that GPT-4-Turbo is capable of applying the required modifications, this should be experimentally verified so that the results have scientific rigor.  

Further, the paper provides scores of LLM-judge biases in the form of the robustness rates, but I can not tell what these scores mean for real-life applications of said LLM-judges. For example, LLM-as-a-judge is typically used for evaluation, such as a reward model during RLHF. If my LLM-as-a-judge system has a specific robustness score for a bias type such as diversity, how does this translate to the bias of a downstream system, such as an LLM that was trained using the judge? Without such results, it is unclear how to interpret the paper's numerical results.

Summing up, I believe two types of experiments are necessary to improve the paper's soundness. 
- Demonstrate that perturbation using LLMs does not introduce unintended errors or biases that are different from what is intended.
- Evaluate the effect of different bias scores for different LLM-as-a-judge systems on real-life applications of the systems.

If related experiments or arguments are added, I will improve my score.

# Minor Comments (did not impact score)
- It would be helpful to have example questions and responses from each dataset somewhere in the paper, to illustrate what the difference between e.g. the fact-related and alignment-related dataset is. They could be included in the appendix if there is no space in the main paper.
- Figure includes some plots where a larger value of the y-axis means less bias (robustness rate) and some for which is a lower value is better (error rate). The plot would be easier to read if this was indicated somehow.

questions:
- How do we know that evaluations that rely on perturbations by LLMs can be trusted? How do we know that such perturbations do not introduce errors or biases other than those which are intended to be evaluated?
-  How could one interpret the results of evaluating a bias using the CALM framework in terms of its effect on real-life applications of the LLM-as-a-judge system? For example, if one LLM's robustness rate for diversity is 0.1 greater than another's, how does this the actual treatment of minorities by systems that utilize the LLM in a LLM-as-a-judge application?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729800270558

thread_type:
official_review

<thread_object>
creator: reviewer_6m1m

<content>
summary:
The authors identify 12 distinct types of biases in LLM-as-a-Judge models, provide automated code for "checklisting" these biases in LLM judges, and conduct a comprehensive analysis of how these biases manifest in state-of-the-art LLM judges across various datasets. The key finding is that LLM judges are not robust. The implications of these findings are significant and should be effectively communicated to the community.

strengths:
I believe the paper is strong, well-written, and highly comprehensive. The topic is both timely and important, and the NLP/LLM community would greatly benefit from its publication. In my opinion, this paper should be accepted. The reason I initially rated it a 6 instead of an 8 is to encourage the authors to consider revising the metrics (as discussed in the weaknesses section).

weaknesses:
**Metrics**: I have a few suggestions regarding the metrics used in this paper and how they are presented in the results. First, for the RR and CR metrics, I recommend making the CR metric more robust by sampling multiple generations when computing the CR for a given instance. Additionally, I propose adjusting the RR metric with the CR, as the authors note that LLMs are non-deterministic, and a low RR score might reflect this rather than a genuine lack of robustness. To adjust the score for each individual instance, I would subtract the individual CR from the individual RR. It is important to compute this adjustment at the individual level, as the CR varies between instances. The final score would be the dataset average of RR_i - CR_i. While this metric is more complex and falls within a -1 to 1 scale, it is far more reliable. If you choose not to present the difference between the two, I suggest at least presenting the average RR and average CR for each LLM in the results. The CR is not a constant value; it varies between models and across instances.

Regarding the ACC metrics, I am unclear about which specific metric you are using in the results section. Should we compare the two metrics and examine the variations between them? Additionally, is this score presented as an absolute value? Could you clarify this aspect to ensure an accurate interpretation of the results? And why do you use "hack"? Isn't it essentially the CoT ACC? 

Regarding the Error Rate (ER) metrics, could you explain the rationale for using these metrics instead of the RR/CR in the paper? Additionally, could we apply the ER metrics to detect other forms of bias? I also find the ER_SE metric unreliable. From your description, it appears that Y_other represents the score assigned by other models to the evaluated model's response. However, I believe Y_other should represent the average score assigned by the explained model to the responses of other LLMs. This would better measure whether the LLM prefers its own responses. Otherwise, you're merely capturing the LLM’s general bias relative to the consensus. For example, one LLM might use scores in the range of 3-7, while another uses 1-6, yet they could still achieve a perfect Spearman's correlation. Moreover, why do you use absolute value? This can be misleading. For example, y_self, y_other = (5, 3) is the same as (1, 3). I believe you can think on a better metric for ER. 

In general, I would recommend adjusting the metrics so that higher scores indicate more bias, unlike the current ones where higher scores represent robustness. Since you frequently use the term "bias" throughout the paper, this modification could make the results and the interpretation of the metrics more intuitive and easier to follow. 

If the authors revise the metrics and provide this analysis, or at least clarify what I may be misunderstanding, I would be happy to increase the overall rating from 6 to 8.

**Misinterpretations of the Results:** Specifically, the statement "Bias is more pronounced in the alignment dataset compared to the fact-related dataset" cannot be inferred from the results. The biases in these datasets differ, and you need to compare like for like - either the same bias across different datasets or the same dataset with different biases. While it's possible to compare the CR metrics between datasets (difference of differences), I believe this is insufficient on its own. First, I would like you to clarify (both here and in the paper) the rationale behind distinguishing between biases and datasets, as well as why certain biases may not be applicable to all datasets. I haven’t given this much thought, but it is critical that this distinction is explicitly explained in the paper, rather than leaving it up to the reader to infer.

questions:
073: This is not limited only to "humanities, social sciences, or general knowledge" fields, a refined answer could be in any field or task.

281: If i understand correctly, for CR you ask the LLM to generate two responses for the same prompt, and check if the answers are consistent. If so, why not use more than two responses? You can make the consistency measure more robust by comparing the variance of N>>2 generations. 

295: Which ACC do you use as a metric?

Metrics paragraph: In my opinion, and even though the English is good and I understand each sentence, this paragraph is not clear enough. I would emphasize in the text which metric is used for each task, specifically at the start of the paragraph, and refer to the column in the table.  In addition, the names, abbreviation are not consistent throughout the paper and specifically in the figures. Please see the weaknesses regarding the metrics.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730047418278

thread_type:
official_review

<thread_object>
creator: reviewer_uzkd

<content>
summary:
This study identifies 12 significant potential biases and introduces a novel automated bias quantification framework called CALM. This framework systematically quantifies and analyzes each type of bias in LLM-as-a-Judge through automated, principle-guided modifications. Empirical findings indicate that there is still room for enhancing the reliability of LLM-as-a-Judge. The paper explores both the explicit and implicit impacts of these biases and offers recommendations for the dependable application of LLM-as-a-Judge.

strengths:
1. A comprehensive delineation and classification of twelve specific biases that can compromise the dependability and credibility of LLM-as-a-Judge.
2. The proposal of the CALM framework for assessing biases within LLM-as-a-Judge systems,  which enhances the rigor of the assessment process in a person-independent manner.
3. An in-depth analysis of six widely-used LLMs through the lens of the CALM framework.

weaknesses:
Lack of Transparency in Assessment Criteria: The source of the basis for the assessments of Robustness Rate (RR) and Consistency Rate (CR) is unclear. 
Incomplete Consideration of Popular Models: The evaluation does not include some well-known LLM as Judge models, such as pandaLM and Prometheus. This omission suggests a lack of thoroughness and may lead to biased or incomplete conclusions.
Questionable Data Selection Process: The method for selecting data is not well-defined. For instance, in the case of GMSK, the process of choosing 100 pieces of data from tens of thousands is not explained. This raises concerns about the representativeness and reliability of the selected data.
User-Friendliness and Reproduction Costs: There are concerns about the user-friendliness of the system and whether the costs associated with reproducing the results are prohibitive. This could limit accessibility and practical application for users.

questions:
1. What is the source of the basis for these assessments of Robustness Rate (RR) and Consistency Rate (CR)? Why are human correlations such as Pearson's coefficient not considered in the assessment. 
2. LLM does not take into account some of the popular LLM as Judge models, such as pandaLM, Prometheus, etc. LLM lacks a specific LLM as Judge model. Lack of specific LLM as Judge evaluation. 
3. Is the data randomly selected? For example, GMSK, how to choose 100 pieces of data from tens of thousands of pieces of data? How to prove that these 100 data are representative enough?
4. Is it user friendly? Is the reproduction cost prohibitive?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730170107051

thread_type:
official_review

<thread_object>
creator: reviewer_tmem

<content>
summary:
This paper explores the potential biases inherent in using Large Language Models (LLMs) as judges in various evaluation tasks, such as scoring and pairwise comparison.  The authors propose a novel framework called CALM, which systematically quantifies and analyzes each type of bias by using automated and principle-guided modification. The paper evaluates six popular LLMs using the CALM framework and finds that while some models demonstrate notable fairness in judgment, significant biases persist in certain specific tasks.

strengths:
- Originality: The authors expand upon existing work by identifying and categorizing 12 distinct types of biases.
- Quality: The paper presents a thorough evaluation of the identified biases across multiple LLMs, using diverse datasets and specific metrics tailored for judging tasks. This rigorous experimental design ensures the reliability and validity of the findings.
- Clarity: The examples in Table 1 provide concrete examples of how biases manifest in LLM judgments, making the abstract concepts more tangible and relatable.
- Significance: The proposed CALM framework offers a valuable tool for stakeholders to assess and mitigate biases, leading to more fair and reliable LLM evaluation methods.

weaknesses:
- I think this paper is more like a toolkit paper rather than a novel research paper, as they just integrate 12 types of existing biases in LLM-as-a-Judge. If we look at the appendix B, we can find that each of the 12 types can be referenced to another previous paper.
- The paper primarily relies on automated metrics to assess bias, but human evaluation could provide a valuable additional perspective. Incorporating a human evaluation benchmark would strengthen the validation of the findings.

questions:
- How do you ensure that the generated perturbations effectively introduce the desired bias without altering the correctness of the content? How well do the LLMs understand the instructions for generating biased content? Could there be unintended consequences or biases introduced by the LLMs themselves?
- Would incorporating a human evaluation benchmark provide additional insights into the accuracy and fairness of LLM judgments?
- Are there potential trade-offs between mitigating biases and maintaining the performance of LLM judges?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces CALM, a novel framework to evaluate biases in LLMs used as judges for tasks like scoring and pairwise comparison. The authors identify and categorize 12 types of biases and assess them across six popular LLMs using diverse datasets and tailored metrics. While highlighting areas of fairness, the results reveal persistent biases in specific contexts.

This paper makes a timely and significant contribution to the NLP/LLM community by providing a comprehensive framework for identifying and analyzing biases in LLM judgments. While the metrics and result interpretations require refinement, the strengths of the paper, including its practical applicability and systematic exploration of biases, far outweigh the weaknesses. With minor revisions to address the issues around metrics and result clarity (which were addressed during the rebuttal phase), the paper would provide a valuable resource for researchers and practitioners. I recommend acceptance.