Title: Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Evaluation

Bias Mitigation

Reasoning-based Methods
Abstract: LLM-as-a-Judge has emerged as a promising tool for automatically evaluating generated outputs, but its reliability is often undermined by potential biases in judgment. Existing efforts to mitigate these biases face key limitations: in-context learning-based methods fail to address rooted biases due to the evaluator’s limited capacity for self-reflection, whereas fine-tuning is not applicable to all evaluator types, especially closed-source models. To address this challenge, we introduce the **R**easoning-based **B**ias **D**etector (RBD), which is a plug-in module that identifies biased evaluations and generates structured reasoning to guide evaluator self-correction. Rather than modifying the evaluator itself, RBD operates externally and engages in an iterative process of bias detection and feedback-driven revision. To support its development, we design a complete pipeline consisting of biased dataset construction, supervision collection, distilled reasoning-based fine-tuning of RBD, and integration with LLM evaluators. We fine-tune four sizes of RBD models, ranging from 1.5B to 14B, and observe consistent performance improvements across all scales. Experimental results on 4 bias types—verbosity, position, bandwagon, and sentiment—evaluated using 8 LLM evaluators demonstrate RBD’s strong effectiveness. For example, the RBD-8B model improves evaluation accuracy by an average of 18.5% and consistency by 10.9%, and surpasses prompting-based baselines and fine-tuned judges by 12.8% and 17.2%, respectively. These results highlight RBD’s effectiveness and scalability. Additional experiments further demonstrate its strong generalization across biases and domains, as well as its efficiency.


Peer reviews:
comments: SUMMARY:
This paper introduces the Reasoning-based Bias Detector (RBD), a plug-in module designed to identify and mitigate biases in LLM-as-a-Judge evaluations. The key contributions include: (1) a novel framework that detects four types of structural biases (verbosity, position, bandwagon, and sentiment) through structured reasoning, (2) an end-to-end pipeline for constructing biased datasets and training RBD models across different scales (1.5B to 14B parameters), and (3) empirical validation showing RBD improves evaluation accuracy by up to 18.5% and consistency by 10.9% across 8 LLM evaluators. The approach operates externally without modifying evaluators, making it applicable to both open- and closed-source models.

STRENGTHS AND WEAKNESSES:
### Strengths

1. The technical execution is solid, with rigorous experiments across multiple bias types and model scales. The iterative feedback mechanism (Algorithm 1) is well-designed.
2. The paper is clearly written, with good visualizations (e.g., Figure 1, 3) and detailed methodology in Sections 3-4.


### Weaknesses

1. **Limited evaluation scope:** The bias mitigation may negatively impact performance in non-bias scenarios (e.g., when verbosity correlates with quality). The paper does not test whether RBD's corrections introduce new errors in unbiased evaluations. The evaluation is confined to similar task distributions (GSM8K, Arena, ScienceQA). There's no evidence that RBD generalizes to substantially different domains (e.g., creative writing evaluation) where bias patterns may differ.
2. **Out-of-domain evaluation:** the biased evaluation are trained and tested in the in-domain settings. Whether the methods works well in the out-of-domain biased evaluation setup are questionable.

QUESTIONS:
None

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the critical problem of bias in LLM-as-a-Judge evaluations by introducing the Reasoning-based Bias Detector (RBD), a novel external module that identifies biased evaluations and generates structured reasoning to guide evaluator self-correction. Unlike existing approaches that rely on in-context learning or fine-tuning the evaluator itself, RBD operates as a plug-in companion module that can work with both open-source and closed-source LLM evaluators. The authors develop a comprehensive framework spanning bias dataset construction, reasoning corpus generation, and model training, focusing on four representative structural biases: verbosity, position, bandwagon, and sentiment bias.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper presents a novel approach to bias mitigation in LLM evaluation. While previous work has focused on prompting strategies or fine-tuning evaluators directly, the external reasoning-based feedback mechanism represents a creative departure from existing paradigms. The modular design allows integration with any LLM evaluator without requiring access to model weights or architecture modifications, which is valuable for closed-source models. \n\nThe computational overhead of the iterative RBD process, while analyzed, could become substantial in large-scale evaluation scenarios. The reported 6.6 seconds per example with additional inference costs may limit practical applicability in high-throughput settings.\n\nQUESTIONS:\nHow does RBD performance scale when dealing with combinations of more than two bias types simultaneously, and what are the theoretical or practical limits to the number of bias types the framework can handle effectively?\n\nLIMITATIONS:\nWhile the work aims to improve evaluation fairness, the reasoning generation capabilities could potentially be misused to manipulate evaluations in adversarial settings. Discussion of safeguards or detection mechanisms for such misuse would be valuable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a reasoning-based bias detector (RBD), an external module that interacts with an LLM-based evaluator to catch various biases in the evaluation. The RBD is an LLM fine-tuned on a curated dataset for detecting bias distilled from a teacher LRM. Extensive experiments show that RBD is an effective tool for mitigating bias by LLM-based evaluators.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-organized and reads well.\n2. The paper tackles an important issue: bias in LLM-based evaluators. The proposed method is conceptually simple and highly effective.\n3. The evaluation is extremely thorough.\n\nWeaknesses:\n1. A number of important design decisions are under-motivated and missing ablations, such as providing the evaluator's name or a specific bias type to consider. \n2. Some experiments are missing key details which makes the results difficult to interpret. For example, the authors measure accuracy but it's not clear if that's accuracy computed on $\\mathcal{D}$ or $\\mathcal{D}_{bias}$.\n3. Section 4 is fairly notation heavy. Consider including a figure of the data collection and RBD training stage (that's more technical than Figure 1).\n\nQUESTIONS:\n1. Why is the RBD given with the evaluator model's name? How much does this contribute to the success of RBD? What if I want to try a new evaluator that is unknown (e.g., more recent) to the RBD? \n2. Why is the bias type provided instead of letting the RBD select from possible biases? In real-world scenarios, the bias type is not known ahead of time. Does this mean that I need to run interactions between the evaluator and RBD for each bias type? How would I aggregate these into a single judgment? How is this handled in the combined bias scenario (Figures 8 and 9)? \n3. What is being measured in Figure 4, the accuracy of the LLM evaluator post-mitigation technique or the accuracy of the mitigation technique at identifying biases? What are these comparison methods? What dataset is it on? Is accuracy computed on $\\mathcal{D}$ or $\\mathcal{D}_{bias}$? What is the x-axis, the size of the RBD model? If so, does the accuracy of the comparison model change with the size of the RBD?\n4. In Figure 6, why is overconfidence > 1.0? Isn't it a percentage?\n5. In the latency analysis, are you reporting end-to-end latency? Does it take into account the generations by the evaluator model or just the RBD? How many iterations occur between the evaluator and RBD?\n6. Can you provide an end-to-end example that explicitly shows the interactions between the evaluator and the RBD?\n7. I hope that you consider releasing your biased datasets. That alone would be a significant contribution of this work.\n\nIf the authors are willing to publicly release the biased datasets and incorporate sufficient experimental details (e.g., see questions 3, 4, and 5 above) into the manuscript, I'd be willing to raise my score to a 4. I'd consider raising my score to a 5 if the authors could also provide ablations backing their design choices (e.g., including the evaluator's name in the prompt to the RBD) .\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLLM judges are becoming very important today (as they are replacing human annotators) due to their improved performance on many benchmark tasks. However, since they are important business impact tools (cost reduction), they need to become more reliable and trustworthy and more research needs to happen to understand various biases they may internally utilize to make their decisions. The paper presents an interesting plugin based system that works for both closed and open weights models for debiasing the given biases noted in the paper. It uses reasoning (structured response) as a feedback mechanism to update the prompt to give the evaluator a chance to look at the possible biases before responding to prompt (this creates a dynamic approach to model steering with additional bias specific reasoning trace to help the evaluator model make an (unbiased) judgment. The paper is a solid effort that comprises of bias problem formulation, dataset creation methodology, generation of reasoning traces, finetuning (distilling) multiple size models with reasoning traces (generated using a powerful reasoning model) and extensive evaluations of multiple LLM evaluators that show significant performance improvements by addition of the RBD models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The plugin in approach makes the method very practical and allows the application to any closed source of open-weights model judge system. \n- Robustness of the RBD approach: Authors show scalable improvement in performance when using the RBD models with 1.4B to 14B params. This suggests the robustness of the method.\n- The paper uses Reasoning as an intervention mechanism for providing an LLM judge with an explicit, structured reasoning trace about a potential bias can causally guide it toward \"self-correction\". This premise positions reasoning as an active, interventional tool. The framework assumes that the LLM judge can process the RBD's feedback, understand the critique and consequently override its initial flawed judgment in favor of a structurally principled judgment. This theme aligns with a body of work that leverages explicit reasoning to enhance model performance. The \"reasoning\" is presented as a window into a cognitive process (to help mitigate the biases), making the correction mechanism appear interpretable.\n- Strong results are reported in the paper with the 8B model shown to improve the eval accuracy by a good margin! \n- the studies on consistency, stubbornness, overconfidence and generalization (limited) make the work very comprehensive towards understanding the merits of this RBD based approach. \n- the dataset is controlled for correctness that is used for training the RBD models. \n\nWeaknesses:\n- The plugin based model adds another additional model to run with increased resource requirements, increased cost and increased latency (6.6s on avg. which makes this process somewhat slow for real time use cases) \n- The dataset is synthetic and the bias injection method is simple and somewhat rule based. This is the most important weakness of the research warranting different evaluation setup (the model can learn to reverse the rule when it detects that instead of any generalization for real bias detection). The high accuracy improvements also raise the possibility that the system is detecting the surface form patterns and this may not hold when faced with more diverse and subtle biases present in harder datasets.\n- To address the above concern, authors should have used external datasets/benchmarks for testing the evaluator models like CoBBLEr that evaluate some overlapping biases (verbosity, bandwagon). \n- The plugin based approach is promising and scalable, however comparing evaluator Fine-tuning in a similar fashion may be another equally promising solution that the authors didn't explore (mentioning that one can't finetune closed models or that the open-weights models would require huge amount of data (what about PEFT methods that work will small data)?\n- The paper's reliance on reasoning rests on the assumption that the LLM judge's reasoning is `faithful'. This claim is currently contested, given the extensive body of research questioning the faithfulness of CoT and other elicited reasoning methods.\n\nQUESTIONS:\n(please see the weaknesses noted above)\n- authors should have used external datasets/benchmarks for testing the evaluator models like CoBBLEr that evaluate some overlapping biases (verbosity, bandwagon). IMO, external evaluation is critical to understand the generalization capacity of this RBD approach. Can the authors try to add this evaluation to the paper? Also, explore testing with other benchmarks like LLMBar and JudgeBench? \n- Also, can authors have trained on say 3 biases and tested on the fourth to better understand the generalization?\n\nLIMITATIONS:\n- research around exploring how to optimize the feedback loop will help with making the iterative judgment process more efficient", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}