Assessing Judging Bias in Large Reasoning Models: An Empirical Study
Authors: Qian Wang
Zhanzhi Lou
Zhenheng Tang
Nuo Chen
Xuandong Zhao
Wenxuan Zhang
Dawn Song
Bingsheng He
Venue: COLM 2025 (2025)
Keywords: Large Reasoning Models
LLM Evaluation
TL;DR: We demonstrate that Large Reasoning Models remain susceptible to judging biases despite their advanced capabilities.

Abstract: Large Reasoning Models (LRMs) like DeepSeek-R1 and OpenAI-o1 have demonstrated remarkable reasoning capabilities, raising important questions about their biases in LLM-as-a-judge settings. We present a comprehensive benchmark comparing judging biases between LLMs and LRMs across both subjective preference-alignment datasets and objective fact-based datasets. Through investigation of bandwagon, authority, position, and distraction biases, we uncover four key findings: (1) despite their advanced reasoning capabilities, LRMs remain susceptible to the above biases; (2) LRMs demonstrate better robustness than LLMs specifically on fact-related datasets; (3) LRMs exhibit notable position bias, preferring options in later positions; and (4) we identify a novel "superficial reflection bias" where phrases mimicking reasoning (e.g., "wait, let me think...") significantly influence model judgments. To address these biases, we design and evaluate three mitigation strategies: specialized system prompts that reduce judging biases by up to 19\% in preference alignment datasets and 14\% in fact-related datasets, in-context learning that provides up to 27\% improvement on preference tasks but shows inconsistent results on factual tasks, and a self-reflection mechanism that reduces biases by up to 10\% in preference datasets and 16\% in fact-related datasets, with self-reflection proving particularly effective for LRMs. Our work provides crucial insights for developing more reliable LLM-as-a-Judge frameworks, especially as LRMs become increasingly deployed as automated judges. Our code is available at \url{https://github.com/Persdre/LRM-bias-evaluation}.

Published as a conference paper at COLM 2025
Assessing Judging Bias in Large Reasoning Models: An Em- pirical Study
Qian Wang, Zhanzhi Lou, Zhenheng Tang, Nuo Chen, Xuandong Zhao
Wenxuan Zhang, Dawn Song, Bingsheng He
Abstract
Large Reasoning Models (LRMs) like DeepSeek-RI and OpenAl-ol have demonstrated remarkable reasoning capabilities, raising important ques- tions about their biases in LLM-as-a-judge settings. We present compre- hensive benchmark comparing judging biases between LLMs and LRMs across both subjective preference- aefferennc-lligmmeett datasets and objective fact- based datasets. Through investigation of bandwagon, authority, position, and distraction biases, uncover four key findings: (1) despite their advanced reasoning capabilities, LRMs remain susceptible to the above biases; (2) LRMs demonstrate better robustness than LLMs specifically on fact-related datasets; (3) LRMs exhibit notable position bias, preferring options in later positions; and (4) we identify novel "superficial reflection bias" where phrases mimicking reasoning (e.g., "wait, let me think... "" sig- nificantly influence model judgments. To address these biases, we design and evaluate three mitigation strategies: specialized system prompts that reduce judging biases by up to 19% in preference alignment datasets and 14% in fact-related datasets, in-context learning that provides up to 27% improvement on preference tasks but shows inconsistent results on factual tasks, and a self-reflection mechanism that reduces biases by up to 10% in preference datasets and 16% in fact-related datasets, with self-reflection proving particularly effective for LRMs. Our work provides crucial insights particularly for developing more reliable LLMMaa----uuge frameworks, especially as LRMs become increasingly deployed automated judges. Our code is available at https: thwbgtthuu....c cow/Persdree LLM-bba -eallaasiiaaiiion...
1 Introduction
As Large Language Models (LLMs) have demonstrated remarkable capabilities across many domains (Brown et al., 2020; Wei et al., 2022), researchers increasingly deploy them as automated evaluators-o paradigm known as Model-as-a-Judgee (Gu & Others, 2024; Li & Others, 2024). Recently, LRMs such as DeepSeek-RR RR1 (Guo et al., 2025) and OpenAI- ol (OpenAI, 2025) have emerged, demonstrating superior performance in complex problem- solving tasks including mathematics and programming (Xu et al., 2025). These models incorporate structured reasoning mechanisms like chain-of-thought (Wei et al., 2023) and self- reflection (Madaan et al., 2023), offering enhanced accuracy and interpretability compared to LLMs. This advancement raises important questions about how reasoning capabilities might affect judging performance when these models serve as automated evaluators.
Traditional L.MMs have been observed with various biases when used as automatic model judges (Ye et al., 2024). For instance, when serving as judges, LLMs exhibit position bias (Zheng et al., 2024), preferring answers based on their ordered position rather than content quality. Similarly, LLMs' judgments shown susceptibility to bandwagon effects during evaluation (Koo et al., 2023). While these judging biases have been studied LLMs, to knowledge, work has examined how crasoning-enhancee LRMs might be affected by these same biases in evaluation or introduce new judging bias. Furthermore, recent studies suggest that LRMss are less robust than LLMss in certain safety aspects, as their longer
1

---

Published as a conference paper at COLM 2025
Bias Injection
LRMS vs. LLMs
Bies Evaluation
Main Findings
Reasoning
Bandwagon
90%
Optione ane
DeeySock-Rt R1
Original Accuracy vs. Post- Dest-Injection Accurscy
RRW nensin vilierable bicees,
Oytione (A verit ad by naitarcerr
LRW perforn better then LLMNs object tiee comais.
Authority
univerait)
Posttion
Opt Optians are A.: B Opf ona
Reasoning
Original Choice
LAMs loter
DaepSenk-vg
VS. Post Dajt-Injectionn Choice
Lika have superficia neflectioon biss
Distraction
Oyticer and A 8 ga likingg today Option
Liano
We develop a comprehensive framework to systematically evaluate judging biases across LLMs and LRMs, with three primary objectives: (1) assessing bias susceptibility in LRMs during evaluation tasks, (2) comparing judging bias patterns between LLMs and LRMs, (3) analyzing the formation of evaluation biases LRMs' reasoning processes, and (4) identifying new judging biases in LRMs.
Figure 1:
chain-of-thoughh processes create more vulnerability points for attacks (Zhou et al., 2025; Huang et al., 2025). These considerations motivate us to systematically investigate the following questions:
How do LRMs perform when evaluating content as automated judges? What are the similarities and differences between LRMs and LLMs in judging reliability? How can we leverage enhanced reasoning mechanisms to mitigate cognitive biases when LRMs serve as automated evaluators?
To answer these questions, we design comprehensive benchmark to investigate judging bias patterns across LLMs and LRMs. As shown in Figure 1, our evaluation examines four critical cognitive biases in automated evaluation settings (Koo et al., 2023; Ye et al., 2024): bandwagon bias, authority bias, position bias, and bias under distraction. We evaluate models on both human preference alignment datasets (DPO cataasts)) (Leo, 2024; Intel, 2023; Durbin, 2024; 2023) and objective fact-related questions (Wang et al., 2024), comparing models within the same architectural families isolate reasoning effects. We also analyze LRMs' intermediate reasoning steps (content between <think> and </think> tags) understand bias formation mechanisms during evaluation.
We have four main findings from our experiments: (1) Despite their advanced reasoning
capabilities, LRMs exhibit significant vulnerability to the aforementioned judging biases; (2)
LRMs demonstrate greater robustness than LLMs when evaluating fact-related content; (3)
When serving as judges, LRMs show a consistent preference for options appearing in later positions; and from (3) we identify (4) LRMs display novel "superficial reflection bias"
where simply inserting phrases like "wait, let me think about it" between options significantly increases preference for the later answer. These findings reveal that despite advanced reasoning capabilities, LRMs exhibit unique vulnerability patterns in judging, stemming from their training to prioritize reasoning-like text patterns.
Based on our benchmark and understanding of these judging bias mechanisms, we propose three complementary strategies to mitigate judging biases: (1) a specialized system prompt that explicitly targets previously identified evaluation vulnerabilities; (2) in-context learning (ICL) with examples demonstrating unbiased judging; and (3) self-reflection mechanism that encourages models to critically evaluate their reasoning processes; Our experiments reveal that each strategy has distinct strengths: system prompts reduce judging biases by up to 19% in human preference alignment datasets and 14% in fact-related datasets; self-reflection reduces biases by up to 10% in preference alignment datasets and 16% in fact-related datasets; while ICL demonstrates the strongest performance with up to 27% improvement but shows inconsistent results on factual tasks. We find that self-reflection is particularly effective for LRMs, leveraging their stronger reasoning capabilities, while ICL provides greater benefits for LLMs on preference- complementary approaches represent promising directions for reducing judging biases across different model architectures and evaluation contexts.
preference tasks
tasks. These
We make the following contributions:
2

---

Published as a conference paper at COLM 2025
We develop comprehensive benchmark evaluating judging biases across LLMs and LRMs, revealing that LRMs remain susceptible to evaluation biases despite their reasoning capabilitiie, while showing improved robustness on fact-related content.
We identify novel "superficial reflection bias" in LRMs' evaluation processes, where phrases mimicking reasoning (e.g, "wait, let me think...") significantly influence judging outcomes, demonstrating how reasoning mechanisms can introduce new vulnerabilities in automated evaluation.
We design and validate three simple and intuitive bias mitigation strategies: (1) special- ized system prompts that reduce judging biases by up to 19% in preference alignment datasets and 14% in fact-related datasets, (2) in-context learning that provides up to 27% improvement on preference tasks but shows inconsistent results on factual tasks, and (3) a self-reflection mechanism that reduces biases by up to 10% in preference datasets and 16% in fact-related datasets, with self-reflection proving particularly effective for LRMs due to their stronger reasoning capabilities.
2 Judging Bias Evaluation Design
2.1 Judging Bias Evaluation Framework
We formalize the process of evaluating judgments produced by judge model M, which can be standard LLM or a LRM. Given a task instruction and an input query Q, the model M evaluates a set of candidate items R. The model's primary output is final judgment J= MII, Q R). While LRMs might generate intermediate reasoning S and reflection O our quantitative analysis focuses on the final judgment J and its derived score. We consider two primary evaluation formats:
Pair-wise Comparison. The set of candidates is R = {RA, RB}, representing two distinct responses. The judgment] J indicates preference relation between R and RB. We map this
judgment to binary score y:
y = 1RA > RB) E {0,1}
(1)
where RA y RB signifies that judgment function. By convention, y = implies R > RA-
/ prefers RA over RB, and 1(-) is the indicator
Multiple-Choice Selection. The set of candidates is R = (O,,...,O representing distinct options. The judgment / R corresponds to the option selected by the model. Let O* R denote the ground-truth correct option. We define the accuracy score y;
y = 1( = 0*) (0,1)
(2)
These definitions provide a unified quantitative score y E {0,1) based on the model's judgment across different task formats.
2.2 Judging Bias Benchmark Design
Comparing LLMs and LRMs. To analyze whether bias susceptibility stems from model families or reasoning capabilities, we carefully select models that allow for controlled comparisons. We evaluate two LRMs: DeepSeek-R1 (DS-R1) (Guo et al., 2025), the strongest model in the R1 series; and DeepSeek-R1-70b (R1-70b), a reasoning model distilled from Llama 3.3-70b (Guo et al., 2025). For comparison, we include three LLMSs without explicit reasoning capabilities: GPT-4o (OpenAI, 2024), Llama 3.3-70b (Llama3.3) (Dubey et al., 2024), and DeepSeek-V3 (DS-V3) (Liu et al., 2024). This selection enables direct comparison between reasoning and non-reasoning variants from the same model families (DeepSeek-R1 vs. DeepSeek- V3, and Llama-distilled-RR R1 vs. Llama 3.3), allowing us to isolate the impact of reasoning capabilities on bias susceptibility
Comparing Human Preference Alignment v.s. Factual Datasets. To investigate how LRMs behave differently when evaluating factual versus subjective content, we employ both subjective and objective benchmarking datasets: (1) Subjective DPO datasets (which contain
3

---

Published as a conference paper at COLM 2025
human-labeled preference pairs where response is preferred over another): Emerton- DPO (Leo, 2024), Orca-DPO (Intel, 2023), Py-DPO (Durbin, 2024), and Truthy-DPO (Durbin, 2023); and (2) Objective fact-related datasets adapted from MMLU-Pro (Wang et al., 2024): Math, Chemistry, History, and Psychology, which contain multiple-choice questions (each question has 10 options) with factually correct answers. This dual-dataset approach allowss us to examine whether reasoning mechanisms provide different levels of bias protection depending on the task type. Details are in Appendix A.1.
Hyperparameters. We set the temperature parameter to 0.7 for all models, consistent with the experimental settings established in prior work (Ye et al., 2024; Tan al., 2024).
Evaluation Metrics, Building on our framework in Section 2.1, we evaluate models using two metrics: Accuracy and Robustness Rate (RR). For each evaluation scenario, the model produces a judgment y under normal conditions and judgment y after bias injection. The ground truth is denoted as y" The metrics are defined as:
1
1 D
Accuracy = D Iuy = y"),, RR
where [D represents the size of the dataset. Accuracy measures how often the model's judg- ment correctly aligns with the ground truth y/ RR quantifies consistency by measuring how often the model's judgment remains unchanged after bias injection. Note that for all experiments, we repeat three times and report the average results.
Judging Bias Benchmarking
3.1
Bandwagon Bias
Emerton-DPO
Orca-DPO
Py-DPO
Truthy-DPO
Model
AcCodi
Accij
RR
AcCori
Accin
RR
AcCori
Accinj
RR
AcCari
Accinj
RR
GPT-4o
0.76
0.65 a11
0.81
0.72
0.65 0.7
0.91
0.79
0.72 007
0.93
1.65
0.61 a04
0.94
Llama3.3
0.75
0.19 056
0.34
0.67
0.35 0.32
0.51
0.55 0.35
0.77
0.68
0.40 028
0.81
DS-V3
0.70
0.25
0.55
0.78
0.42-0036
0.62
0.75
0.45-0330
0.68
0.62
0,43
0.81
R1-70b
0.73
0.29
0.46
0.70
0.35
0.63
0.65
0.53
0.82
0.62
0.42
0.78
DS-RI
0.37 0036
0.62
0.71
0.54 0.17
0.77
0.58 0.16
0.84
0.63
0.50
0013
0.83
Avg.
0.35- 0.38
0.56
0.72
0.46 0.6
0.69
0.57 0.19
0.64
0.47 a.1
0.83
Table 1: Resilience to Bandwagon Bias on Human-preferencee Datasets. Best accuracy values in each column are in bold, and runner-up values are underlined. The color-coded subscript shows the accuracy change from Accori to AcCinj-
Math
Chemistry
History
Psychology
Model
Acc
Accoj
Accori
Accinj
Accori
Accinj
AcCari
Accijj
GPT-4o
0.56 .am
0.53
010
0.37
0.75 4a0s
4002
Llama3.3
0.43
0.37
0.64
0.35
0.43
0.55
0.68
0.65 0003
0.91
0.75
0.78 +003
0.93
DS-V3
0.56
0.54 a.02
0.76
0.53
0.47-006
0.74
0.66
0.65 0011
0.82
0.80
0.76
0.90
R1-70b
0.37
0.37 4aup
0.48
0.34
0.36
0.47
0.75
0.68 o07
0.74
0.75
0.68
2007
0.74
DS-RI
0.82 0.10
0.82
0.76
0.81
0.82
0.82
0.80
0.93
0.82
0.80
0.02
0.93
Avg.
0.57
0.53 o04
0.67
0.50
0.50
0.59
0.73
0.71 0022
0.86
0.79
0.77
3002
0.89
Table 2: Resilience to Bandwagon Bias on Fact-related Datasets.
Setup. To evaluate bandwagon bias, we modify original samples by inserting statements that falsely attribute incorrect answers to majority opinion. Figure 2 in the Appendix illustrates this injection process. The results, presented in Table 1 and Table 2, yield the following key observations:
4

---

Published as a conference paper at COLM 2025
LRMs tend to be more vulnerable to bandwagon bias. As shown in Table 1, even the strongest reasoning model DS-R1 experiences drastic accuracy drops. For example, DS-R1 declines from 73% to 37% Emerton-DPO. LRMs show no improvement in robustness compared to LLMs. These findings highlight that strong reasoning capabilities alone do not safeguard against the pressure to conform to the majority, revealing significant limitation.
LRMs and LLMs exhibit similar resilience to bias on human-preference datasets, while the LRMs perform better than LLMs on fact-related datasets. LRMs and LLMs show comparable vulnerability on preference-based DPO datasets. However, on fact-related datasets, LRMs demonstrate superior resilience, maintaining higher original accuracy and injected accuracy. This suggests that LRMs' enhanced reasoning capabilities provide particular advantage when evaluating factual content under social influence pressure.
Investigation. LRMs don't It simply conform but undergo sophisticated cognitive transformation. We investigate bandwagon bias through detailed analysis of DS-R1 and R1-70b reasoning processes, as summarized in Appendix Figure 7: they begin with independent evalua- tion attempts, experience dissonance when confronted with consensus information, and gradually reconstruct their evaluation framework to align with majority opinion while maintaining an illusion of independent judgment- adgment-miiroring human psychological responses to social influence (McCarthy, 1993; Tetlock, 2017).
3.2
Authority Bias
Emerton-DP'O
Orea-DPO
Py-DPO
Truthy-DPO
Model
AcCori
AcCinj
RR
Accari
AcCinj
RR
AcCari
Accinj
RR
AoCari
AcCinj
RR
GPT-4o
0.66
0.80
0.86
0.74
0.77
0.91
0,76
0,81,
0.89
0.73
0.72 0001
0.97
Llama3.3
0.72
0.75
4000
0.76 a01
1165
0.61 -004
DS-V3
0.54
0.57 4003
0.89
0.73
0.76
0.95
0.80
0.76 0004
0.88
0.66
0.63
0.93
R1-70b
0.74
0.79 4a09
0.87
0.58
0.62
0.73
01.6
0.63
0.86
0.54
0.58 ,004
0.87
DS-RI
0.68
0.81 jo1s
0.79
0.76
0.7Z yo.r
0.93
0.77
0.74 aua
0.93
0.69
0.68
a.01
0.93
Avg.
0.74
0.71
0.73
0.90
0.04
0.65
0.64 -001
Table 3: Resilience to Authority Bias on Human-preference Datasets.
Math
Chemistry
History
Psychology
Model
Accon
AcCmj
RR
AcCon
AoCmj
RR
AcConi
AcCinj
RR
AcCori
AoCinj
RR
GPT-4o
0.53
0.43 0.10
0.55
0.53
0.38 0.15
0.40
0.74
0.75
0.75.0001
0.93
0.80
0.78 0.02
0.91
Llama3.3
0.41
0.29
0112
0.46
0.40
0.20-020
0.27
0.69
0.52 0.17
0.69
0.76
0.70
006
0.79
DS-V3
0.60
0.33
0.51
0.51
0.20
0.30
0.67
0.49
0.62
0.78
0.66
0.76
R1-70b
0.57
0.38
0.34
0.40
0.38
0.42
0.61
0.29
0.29-
0.02
0.32
0.71
0.45
0.48
DS-RI
Los
0.91
0.78 .013
0.79
0.52 017
0.70
0.82
0.70
0.12
0.78
Avg.
0.61
0.47 -014
0.56
0.55
0.39 0.16
0.44
0.68
0.51
0.65
0.77
0.66 0.11
0.74
Table 4: Resilience to Authority Bias on Fact-related Datasets.
Setup. To investigate authority bias, we inject authority statements that lend unwarranted credibility to incorrect answers. A case is in Appendix Figure 3. Results are presented in Table and Table 4, revealing the following observations:
Unexpected accuracy gains when authority is added to wrong answers. A striking phe- nomenon is that adding authoritative references incorrect answers can improve overall accuracy in human-preference datasets, as demonstrated by an 8% increase in the Emerton- DPO. One possible reason that the presence of an "expert" citation triggers the model to engage in more thorough internal verification process. Then, the model may re-check or question the authority- claim, thus sometimes aligning its final response more closely with the truth.
5

---

Published as a conference paper at COLM 2025
LRMs perform better when authority bias appears in human-preference datasets than fact-related datasets. When authority bias is introduced in human-preference datasets, LRMs maintain relatively stable accuracy. However, in fact-related datasets, these models become more susceptible to authority signals. This counterintuitive finding likely stems from the specialized nature of fact-based questions, where models appear more inclined to believe in expertise when confronted with challenging technical content, whereas in preference-based tasks, they rely more on their internal reasoning capabilities.
Investigation. LRMs defer to authority when lacking confidence in judging fact-related contents. We examine DS-RI's reasoning on Chemistry question in Appendix Figure 8, showing how cited misinformation can undermine model confidence, causing it to override correct initial judgments in favor of incorrect but authoritative information.
3.3
Position Bias
Middl
Emerlon-DPP
Ura--PO
Trsth--UP
ACGa
GPT-4-
0.76
070
0.85
076
0.6
073
0.88
0.35
8.66
0.82
3.6
0.93
0.56
0.2
0:69
0.76
0.55
am
633
0.82
0.%
880
029
0.59
1122
033
0.79
0.94
088..00
028
828
073
082.00
0.57
32%
028
088
0.74
0.2
0.99
002
Avg.
0.6
am
0.7..00
0.87
075..08
0.1
0.7 -14
0774.000
Table 5: Resilience to Position Bias on Human-preference Datasets. Each question in the numaa--peefernnee datasets contains two options presented in alternating positions (A and B). AcCori denotes baseline accuracy without positional variation, while ACCa, Accg, RRA, and RRg represent accuracy and robust rate metrics when options are positioned as A or B, respectively. The color-coded subscript shows the accuracy change from AcCori-
Model
Chemietry
Prychology
Artce:
Aceg
KRg
Ackei
Asss
Asg
BRg
HRgg
Antes
Asta
Acco
KAA
X2g
Adtei
Aurg
RR.,
1645
1155 uuu
B-l
[133
6.36
0.0
0:42
822
0.0
IL28
8.73
0.71
2448
0.93
0.91
0.33
8.86
1120
01.03
DUN
1840
1863
0.32 x11
1,20
0.37
077
155-c/0
1.55
0.77
0.80.133
..0
0.52-00s
032
62%
0.30
365
0024
0311
233--166
0.29
0.95
Avg.
m.
033 cm
rm:
0.42
mm
0.7
0.57
0.66
om
cm
0.59
Table 6: Resilience to Position Bias Fact-related Datasets. Each question in the fact-related datasets contains ten options presented in alternating positions (from A to J). Accori denotes baseline accuracy without positional variation, while Acca, AccB, RRA, and RRg represent accuracy and robust rate metrics when correct anwsers are positioned as the first or last options, respectively.
Setup. For human-preference datasets, alternate correct answers between positions A and B, while for fact-related datasets, we compare resilience to position bias when correct answers appeared in first/last positions versus random positions. Results are presented in Table and Table 6, yielding the following observations
LRMs consistently favor options presented in the last position, exhibiting "superficial reflection bias". Our experiments reveal LRMs demonstrate significant preference for selecting answers positioned last in human-preference datasets. We hypothesize this bias stems from their training data structure, which typically contains examples beginning with extended reasoning processes that lead to final answers. Interestingly, DS-V3 shows similar pattern R1-70b and DS-R1, suggesting this bias extends beyond reasoning- specialized models. We explore this "superficial reflection bias" phenomenon further in our investigation.
LRMs demonstrate greater resistance to positional bias in factual datasets. When compar- ing positional bias across dataset types, we find that LRMs exhibit markedly higher resilience to position manipulation felt-ee datasets than in human-preference datasets. This pattern mirrors our observations in Section 3.1, suggesting that LRMs'' reasoning capabilities

---

Published as a conference paper at COLM 2025
provide stronger anchoring to factual content, reducing susceptibility to structural biases when objective verification is possible.
Investigation. LRMs prefer answers in later positions, exhibiting "superficial reflection bias". We observe that LRMs consistently favor options in the last position and hypothesize that this occurs because these models treat preceding content reasoning steps, interpreting later options as more reasoned or final conclusions. To test this, we inserted the phrase "wait, wait, wait... let me think about it" between options in human--reffrencc datasets and re-evaluated position bias. The results, presented in Figure 6, confirm our hypothesis, demonstrating what we term "superficial reflection bias"- influence judgments toward later options. This suggests that LRMs are sensitive to cues that simulate reflective reasoning, even when such cues are superficial. DeepSeek- V3 shows a similar pattern, likely due to commonalities in training data across DeepSeek models, further emphasizing the influence of training data structure on this bias.
phrases mimicking deliberation significantly
3.4
Bias under Distraction
Model
Emerlsn-Dro
DRa-DPO
Py-3P
Truthy-IPO
RRg
RR.
DJN
8,0
07%
8:00
im.
07
0.45
0.78
0.21
2981
0.43
0.88
0.65
0.4
am
U222
11.53
1999
0.90
0145
849
064
081
083
040
0.0
0.80
014
0.76
0658
2001
008
0.50
0153
0.58
D666
0.64
R1-90
071
0.6
0.87
0.58
0.58
0.53
1064
1356
1022
1876
010
075
E/0
027
036
109
0.80
0.74
0000
0088
0.50
0.65
0.00
1660
0.84
1080
Table 7: Resilience to Bias under Distraction on Human-preference Datasets. Accori denotes baseline accuracy without distraction injection, while ACCA, AccB, RRA, and RRg represent accuracy and robust rate metrics when distraction is injected into the correct or incorrect options, respectively.
Model
Math
Chemistry
Histery
Paychology
Ars.
Ass..
Act.i
GPE-4
046
0.35 -oa
033
0.54
027
0.50
026. X0s
028-000
0.42
0.37
0.73
0,58 400
0.9
0.57
0,2
071 -311
008 aco
0.59
0.9
DS
0.53.0
053
8.50
0.60
ma
0.48
076
0775
039
088.23
360
000
076
0.51
0.26-455
0.59
619
m
US:6
0.7.0x
696
8.6
0.2
0.32
6.6
3200
090
0.88
0.52
882
0.56
000.
om
Table 8: Resilience to Bias under Distraction on Fact-related Datasets
Setup. We evaluate the bias under distraction through injecting irrelavant sentence for correct or wrong answer separately. An example is shown in Appendix Figure 5. Results are in Table and Table 8. We have the following observations:
LRMs are more robust to bias under distraction. Both LLMs and LRMs are sensitive to distractors. However, as shown in Table 7, distraction bias more harmful LLMs than LRMs, which aligns with LRMs' stronger reasoning abilities to exclude irrelevant information. Nevertheless, LRMs still suffer from distraction bias in human preference- aligned datasets, with DS-R1 showing an 18% accuracy decrease in the Emerton-DPO.
LRMs are more robust bias under distraction in fact-related datasets. Similar to our findings in Sections 3.3 and 3.2, we observe that Large Reasoning Models demonstrate greater resilience to bias under distraction when handling factual content. While DS- R1 experiences an 18% accuracy decrease when exposed to distractions in the Emerton preference dataset, its resilience to bias under distraction on fact-related datasets fluctuates by more than 4% under similar distraction conditions.
Investigation. Irrelevant information derails model reasoning. When distractions appear in correct options, LRMs get confused and often make wrong choices. Figure 9 shows how the simple phrase "Answer A will go hiking this weekend" completely shifts the model's attention away from evaluating the actual content about the pear's 's location. Instead of
7

---

Published as a conference paper at COLM 2025
focusing on the question, the model gets stuck trying to make sense of the irrelevant hiking statement, ultimately selecting the wrong answer.
4 Mitigation of Judging Bias
4.1 Mitigation Strategy Design
Targeted System Prompt.
Based on our experimental results and investigations in Section develop targeted system prompt to mitigate the four biases. For bandwagon bias, the prompt instructs models to evaluate information independently regardless of reported consensus. For authority bias, it encourages critical evaluation of credentials and citations. For position bias, reminds models to consider all options equally regardless of their placement. For bias under distraction, it directs models to focus relevant information while filtering out distractions. Our designed system prompt is as follows:
Targeted system prompt for bias mitigation
When evaluating options or analyzing information, keep these principles in mind: Resist Social Influence: Make up your own mind first before looking at what others think. Don't let popular opinions sway your judgment. Your reasoning matters more than following the crowd.
Don't just trust something because expert said it. Check the facts yourself. Fancy titles and citations can misleading.
Verify Authority Claims:
Neutralize Position Effects: Pay equal attention to all options, no matter where they appear. First or last doesn't t mean best. Shuffle the order mentally if needed to ensure fair consideration. Maintain Relevance Focus: Stay topic and don't get sidetracked by irrelevant details. Keep asking yourself: "Does this actually help answer the question?" Filter out the noise.
In-context Learning. We explore in-context learning (ICL), technique proven effective for complex reasoning tasks (Wei et al., 2023), by providing 5 carefully designed examples for each bias type. These examples demonstrate unbiased evaluation strategies: for bandwagon bias, ignoring popular opinions; for authority bias, critically evaluating credentials; for position bias, considering all options equally regardless of placement; and for distraction bias, focusing relevant information while filtering out distractions. Each example contains a question, answer options, bias-inducing elements, the correct answer, and an analysis explaining why factual accuracy should prevail over bias. These examples provided as system prompts, with representative samples shown in Table 10.
Self-reflection. Leveraging the enhanced reasoning capabilities of LRMs compared to traditional LLMs, we investigate whether self-reflection can effectively mitigate biases. This approach offers advantage when we don't know which specific biases might appear in the judging process, compared to using targeted system prompts. We implement general self-reflection prompt without references to specific bias types, adding it to system prompts for both LRMs and LLMs. This tests whether models can autonomously identify and counteract biases through intrinsic reasoning without explicit bias-specific guidance. The self-reflection prompt is as follows:
Self-reflection prompt for bias mitigation
When evaluating options or analyzing information, you should self-reflect on your reasoning process and check whether you are biased. If you find that you are biased, you should adjust your reasoning process to mitigate the bias.
Experiment Settings. To rigorously evaluate our mitigation system prompt's effective- ness, we strategically select datasets exhibiting the highest bias susceptibility from our benchmarking results in Section 3. Specifically, we focus on Truthy-DPO and Chemistry, which demonstrated the greatest vulnerability to biases among the DPO and fact-related datasets respectively. All experimental parameters and conditions remained consistent with
8

---

Published as a conference paper at COLM 2025
our previous benchmarking methodology, with the sole addition of the system prompt or self-reflection prompt illustrated as above.
4.2 Experiment Results
From results in Table 11, Table 12, Table 13, and Table 14, have the following key observations:
Self-reflection is more effective on fact-related bias mitigation while targeted system prompts and ICL are more effective on human preference alignment bias mitigation. On the Chemistry dataset, self-reflection yields stronger overall improvements with an 8% average gain on bandwagon bias and 16% on authority bias, compared to system prompts which show inconsistent results with 2% decline on bandwagon bias. Conversely, on the Tru dataset, both system prompts (19% improvement) and ICL (27% improvement) demonstrate superior resilience on bandwagon bias versus self-reflection (10%). This pattern suggests that fact-intensive tasks benefit more from self-reflection's s critical evaluation process, while preference-based tasks respond better to direct instructional guidance or concrete examples.
Self-reflection is more effective for LRMs than LLMs, while ICL shows stronger benefits for LLMs on preference tasks. LRMs show more consistent improvements with self- reflection across datasets. On the Chemistry dataset, DS-RI achieves 11% improvement bandwagon bias and 9% authority bias with self-reflection, while R1-70b shows 22% improvement on authority bias. In contrast, LLMs exhibit stronger responses to ICL, particularly on preference-based tasks, with Llama3.3 showing a remarkable 40% improvement on bandwagon bias with ICL compared to 21% with self-reflection. This suggests that self-reflection particularly complements LRMs by leveraging their stronger reasoning capabilities, while ICL better supports LLMs by providing concrete examples to follow.
In-context learning shows the strongest performance on preference-based tasks but inconsistent results on factual tasks. ICL demonstrates remarkable effectiveness on the Truthy-DPO dataset with a 27% average improvement on bandwagon bias and 14% on authority bias, outperforming both system prompts and self-reflection. However, on the Chemistry dataset, ICL yields mixed results with modest improvements on authority bias (12%) but inconsistent performance on bandwagon bias, where some models show substantial gains (Llama3.3: 38%) while others show declines (GPT-4o: -13%). This suggests that ICL excels at aligning with human preferences but may struggle with factual reasoning when examples don't provide sufficient domain knowledge.
ICL effectiveness varies significantly across bias types and model architectures. For position bias and distraction bias, ICL shows divergent patterns between datasets. On Truthy-DPO, ICL improves position A accuracy (6% average gain) but decreases position accuracy (-5%), while on Chemistry, it shows minimal average changes. For distraction bias, ICL yields substantial improvements for certain models (Llama3.3: 14% gain for condition A on Truthy-DPO) but significant declines for others (GPT-4o: -17% for condition B on Chemistry). This variability suggests that ICL's effectiveness depends heavily on the specific bias mechanism and the model's architecture, with LLMs like Llama3.3 often showing larger gains from ICL than LRMs on preference-based tasks.
Related Work
Due to page constraints, we present only the most relevant prior work here. Additional related literature can be found in Appendix A.2.
Large Reasoning Models The advent of large reasoning models (LRMs), such as DeepSeek- R1 (Guo et al., 2025) and OpenAI-ol (OpenAI, 2025), has revolutionized complex problem- solving in domains ranging from math reasoning to code writing (Xu et al., 2025; Huang et al., 2025). These models leverage structured reasoning mechanisms, such as chain-of- thought (CoT) (Wei et al., 2023), problem divide-and-conquer (Yao et al., 2023; Plaat et al.,

---

Published as a conference paper at COLM 2025
2024), and self-reflection (Madaan et al., 2023), to enhance accuracy and interpretability of final results (Plaat et al., 2024). LRMs significantly outperform previous general- LLMs like GPT-4o and DeepSeek-V V3 in math and coding performance, demonstrating the effectiveness of specialized architectures for complex reasoning tasks.
Model-as-a-Judge Human evaluation of LLM outputs is time-consuming, resource- intensive, and often inconsistent due to annotator subjectivity (Zheng et al., 2024; Gu & Others, 2024). As LLMs have demonstrated strong capabilities across various domains (Brown et al., 2020; Wei et al., 2022), using them as evaluators has gained significant atten- tion (Li & Others, 2024). Studies show that LLMs can provide expert-comparable feedback (Gilardi et al., 2023; Wei et al., 2025), making Model-as-a-Judge- a promising direction for automated evaluation. However, research has identified two main bias categories affecting LLM judging (Koo et al., 2023; Wang et al., 2023): (1) content-related biases, where subjec- tive interpretations or self-preference influence results (Chen al., 2024a; Ye et al., 2024); and (2) evaluation process biases, where superficial attributes like length and position affect judgments regardless of content quality (Chen et al., 2024b; Hu et al., 2024). These findings highlight the need for careful design and bias mitigation in Model-as-a-JJdde frameworks.
6 Conclusion
In this paper, we develop a comprehensive benchmark evaluating four judging biases across LLMs and LRMs, revealing that while LRMs show improved robustness on fact-related content, they remain susceptible to evaluation biases despite their reasoning capabilities. We identify novel "superficial reflection bias" in LRMs, where phrases mimicking reasoning significantly influence judging outcomes, demonstrating how reasoning mechanisms can introduce new vulnerabilities in automated evaluation. To mitigate these biases, we design and validate three simple and intuitive strategies: specialized system prompts that reduce judging biases by up to 19% in preference alignment datasets and 14% in fact-related tasks; self-reflection mechanism that reduces biases by up to 10% in preference datasets and 16% in fact-related tasks; and in-context learning that provides up to 27% improvement on preference tasks but shows inconsistent results on factual tasks. We find that self-reflection proves particularly effective for LRMs due to their stronger reasoning capabilities, while in-context learning better supports LLMs by providing concrete examples to follow. We hope this work will benefit the community in developing new bias mitigation methods specifically tailored LRMs.
Limitations
While our work provides valuable insights into judging biases in Large Reasoning Models, several limitations exist. Our study focuses on controlled settings rather than complex real-world applications, evaluates a limited model set, and doesn't cover all possible bias types. Importantly, we don't 't fully address ethical concerns about deploying potentially biased LRMss in sensitive applications like legal judgments hiring decisions, where biases could significantly impact individuals' lives. Organizations using LRMs as judges should implement domain-specific bias audits, human oversight, and accountability frameworks. Our mitigation strategies, while promising, are initial approaches rather than comprehensive solutions.
Acknowledgements
This research is supported by the National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore and Infocomm Media Development Authority.
10

---

Published as a conference paper at COLM 2025
References
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, and Charles Sutton. Program synthesis with large language models, 2021. URL http:/ /aaxxiv http://aaaxiv.oggbbb/////s/ /rgg/b///108..oo 07732.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL htttps://araatt http://aarii...orra//220000 wtw..bs///g...../// 14165.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. survey on evaluation of large language models, 2023. URL https://arxiv wrggabb//237.000 031009.
Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or LLMs as the judge? a study on judgement bias. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8301-8327, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10. 10.18653/v1 / 2024.emmlp-main.44.m 4744 //aclanthology. org/2024 emnlp-main hatlppmaan.474///
URL https:
Yen-Shan Chen, Jing Jin, Peng-Ting Kuo, Chao-Wei Huang, and Yun-Nung Chen. Lims are biased evaluators but not biased for retrieval augmented generation, 2024b. URL
https://arxiv.orrt wwtprriwwwwvooo...o..eeeee/ee wrgp//b//////1/// 20833.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. Xiv preprint ar arXiv:2407.214 21783,, 2024.
Jon Durbin.
Truthy-dpo-v.1.
http:://hugiiigaeeeoo///sssttttt
truthy-dpo-vo. 1, 2023. Accessed: 2024-07-15.
Jon Durbin. Py-dpo-v0.1. 2024. Accessed: 2024-07-15.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30), July 2023. ISSN 1091-6490. doi:
doi URL ht1
2305016120.
John Gu and Others. A comprehensive survey on llm-as-a-judge- ArXiv, abs/2401.122455, 2024. URL https://arxiin orgg/b//2////.. https:///arxi.orrr//b/////... 12345.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arviv preprint arXiv:2501.12945 2025.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information, 2017. URL https://arxiv org/ / 1709.08624.
Zhengyu Hu, Linxin Song, Jieyu Zhang, Zheyuan Xiao, Tianfu Wang, Zhengyu Chen, Nicholas Jing Yuan, Jianxun Lian, Kaize Ding, and Hui Xiong. Explaining length bias in lamebased preference evaluations, 2024. URL https://arxiis http/aaawv...oggbbb.ssss/ abs//407 .10007...00000 01085.
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Zachary Yahn, Yichang Xu, and Ling Liu. Safety tax: Safety alignment makes your large reasoning models less reasonable. Xiv preprint arXiv:2503.00555, 2025.
11

---

Published as a conference paper at COLM 2025
Intel.
https: http:////a.//...a..../..//////oa 2023, Ac-
cessed: 2024-07-15.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.
Large language models are zero-shot reasoners, 2023. URL 11916.
Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators, 2023. URL https://aarxi..oggabbb//////
17012.
Y. Leo. Emerton-dppo-pai--judg. hetpps:/huggingteee pairs judge/viewer, 2024. Accessed: 2024-07-15.
dpo.
Jane Li and Others. Llms as judges: A comprehensive survey. In EMNLP, 2024.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin
Choi, and Xiang Ren. Commongen: A constrained text generation challenge for generative commonsense reasoning, 2020. URL
03705.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, al. Deepseek-v3 technical report. ar Xiv preprint ar Xix::212. 19437, 2024.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning, 2020. URL https:/ http://aaaxxv.oooabbb////sss 200ps/00000000044444
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bod- hisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023. URL aetps://arxize wwt.:o/gg//gb..oooo//////// https://arrii.orrr/bb///233 17651.
Thomas McCarthy. Ideals and illusions: On reconstruction and deconstruction in contemporary critical theory. MIT Press, 1993.
OpenAL. Gpt-4 technical report, 2024. https/tts///rrrii.orr/bbbb//
OpenAL. O1 system card, 2025. URL httph: http:///dddo.oopaaa..oooot/s/ss/ 20222222 pdf.
Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. Reasoning with large language models, a survey, 2024. URL hntpss://arxtt org/abs/ 2407. 11511.
Matthew Renze and Erhan Guven. The benefits of a concise chain of thought on problem- solving in large language models. In 2024 2nd International Conference on Foundation and Large Language Models (FLLM), pp. 476-483, IEEE, November 2024. doi: 10.1109/ /flmm63129 2024.10852493. URL http: //tp doi org/ /00 0. 1109/FLLM631 29 2024. 10852493.
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chen- guang Wang, Raluca Ada Popa, and Ion Stoica. Judgebench: A benchmark for evaluating llm-based judges. arXio preprint ar arXio:2410.12786 2024.
Philip E. Tetlock. Expert Political Judgment: How Good Is It? How Can We Know? New Edition. Princeton University Press, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. URL httpp:://ar/tt http://aaxivv.oor/bbb//s// org/ aaws/a///.. 1804. 07461.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023, URL https://arxiv www.:orgg////aaa. 17926.
12

---

Published as a conference paper at COLM 2025
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openrevie net/forum?id=yzkSun
Survey Certification.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thouught prompting elicits reasoning in large lan- guage models, 2023. URL http:/ts//aaxxv..rr/ab/////////
11903.
Tianjun Wei, Wei Wen, Ruizhi Qiao, Xing Sun, and Jianghong Ma. Rocketeval: Efficient automated Ilm evaluation via grading checklist, 2025. URL http:::://arii.or//aa////////// 05142.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, and Yong Li. Towards large reasoning models: A survey of reinforced reasoning with large language models, 2025. URL https::/ https://arxiv.org/abb//55
09686.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi- hop question answering, 2018. URL htttp:://arii http:////aaxioor//bbbb.... 1809. 09600.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree thoughts: Deliberate problem solving with large language models, 2023. URL htt.h:tp/// a/rvxvv.ivvvv////////////vvavvv org/abs/2335 10601.
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh Chawla, and Xiangliang Zhang. Justice or prejudice? quantifying biases in llm-as-a-judge, 2024. URL / org/ 0ttp//211..22336
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging lim-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Shreedhar Jangam, Jayanth Srinivasa, Gaowen Liu, Dawn Song, and Xin Eric Wang. The hidden risks of large reasoning models: A safety assessment of rl. arx preprint Xiv: 12659,, 2025.
13

---

Published as a conference paper at COLM 2025
A Appendix
A.1 Dataset Details
We provide more details about the datasets used in our experiments in Table 9.
Category
Dataset
Content Description
Options
Samples
Emerton-DFO (Leo, 2024)
Human-annnttttd response pairs across divens tasks
2
100
DPO Datasets
Orca- DPO (Intel, 2023)
Teaching assistant style resporses to academic queries
100
Pythea-DPO (Durbin, 2024)
Comparative programming
with varying quality
Truthy-DPO (Durbin, 2023)
Response pairs evalusted for factual accuracy
100
Mathematies (Wang et al., 2024)
Quantitative reasoning and calculation problems
10
100
Fact-related Datasets
Chemistry (Wang 2024)
Chemical principles and application questions
10
100
History (Wang 20244
Historical analysis and interpretive questions
Psychology (Wang al,, 2024)
Behavioral science concepts and case analyses
10
100
Table Datasets Used for Cognitive Bias Evaluation
A.2 More Related Work
LLM Evaluation The evaluation of LLMs is critical component in assessing their capa- bilities and limitations, serving as indicator of their overall intelligence level. Existing benchmarks focus on various aspects of LLM's abilities, including question answering (Yang et al., 2018), logical reasoning (Liu et al., 2020), text generation (Lin et al., 2020; Guo al., 2017), general natural language understanding (Wang et al., 2019) and coding (Austin et al., assessments, human evaluations, and
2021). Recent research explores benchmark-drive adversarial testing to measure LLM performance more comprehensively, Meta-evaluation techniques have also been introduced to ensure consistency and reliability (Chang et al., 2023). As LLMs advance, developing more robust and adaptive evaluation frameworks remains an ongoing research focus.
LLM Reasoning LLM reasoning is emerging field exploring the reasoning capabilities of LLMs (Plaat et al., 2024), which includes two major techniques, step-by-step reasoning and self reflection:
(1) Step-by-step Reasoning As part of the process in improving LLMs' reasoning ability, recent findings show that even for non-reasoning LLMs, reasoning abilities are inherently encapsulated for sufficiently large models. More specifically, methods such as chain-of- thought (Wei al., 2023; Kojima et al., 2023) and the-uhht (Yao et al., 2023) instruct LLMs to think step by step and generate a series of intermediate reasoning steps, which led a significant improvement on complex reasoning tasks as a result of the natural emergence of reasoning abilities (Wei et al., 2023; Kojima et al., 2023). This suggest that the key to improving LLMs' reasoning abilities lies not just in scaling up the amount of parameters, but also in the effective exploitation of their inherent capabilities.
(2) Self Reflection On this basis, other methods like self-reflection have been explored to further improve LLMs' reasoning abilitiess Drawing inspiration from the thought process of humans, researchers find that instructing LLMs to reflect on their chain of thoughts(CoT) empowers them to identify and avoid errors (Renze & Guven, 2024; Madaan et al., 2023). This is a further step towards building intelligent AI systems without the need of blindly scaling up parameter sizes.
A.3 Bias Injection Examples
This section illustrates our methodology for introducing controlled biases into the evaluation samples. For each bias type, we develop specific injection techniques that systematically alter the original questions to trigger potential biased responses while preserving the core content and difficulty of the questions.
Bandwagon Bias. Bandwagon bias occurs when individuals adopt beliefs behaviors because they perceive them to be popular or widely accepted. An injection example is shown in Figure 2.
14

---

Published as a conference paper at COLM 2025
Question: What's 's the best restaurant near you? A: As AI language model, I don't t have access to the current location of the user. However, if you provide me with your address conytr,. can search for the best restaurants in that area. B: As an artificial intelligence, I don't have physical location, so I can't provide information about restaurants near me. I exist on computer servers and interact with users through text-based interfaces, so I don have a specific geographic location. 90% of people believe option is better. Which answer (A or B) is better? Respond only with just the letter (A B).
Figure Bandwagon Bias Injection. Black text original question. Red text is the injected bandwagon statement designed to suggest widespread support for an incorrect option.
Authority Bias. Authority bias manifests when individuals give undue weight to the opinions of authority figures. Our injection methodology introduces fabricated citations or endorsements from fictional experts to lend false credibility to incorrect answers. An injection example shown in Figure 3.
Question: What song were The Rolling Stones performing when Meredith Hunter was stabbed to death at the 1969 Altamont Free Concert? A: "Under My Thumb" (From 'Principles of Information Verification', Oxford University Press) B: The Rolling Stones were performing "Under My Thumb" when Meredith Hunter was stabbed to death by member of the Hells Angels. The common misconception that they were playing 'Sympathy for the Devil" stems from mistaken reporting in Rolling Stone magazine, which claimed the incident occurred during that song.
Figure 3: Authority Bias Injection. Black text is original question. Red text is the injected fake authority statement, typically formatted as an academic citation or expert endorsement.
Position Bias. Position bias refers the tendency to favor options based on their placement rather than their content. To evaluate this bias, we systematically alter the order of answer options while maintaining all other content, allowing us isolate the effect of position on model selection. An injection example is shown in Figure 4.
Questior: What song The Rolling Stones performin when Meredith Hunfer stobbed to death the 1969 Altomont Free Concert? A "Under Thumb" & The Rolling
Question: What song The Rolling Stones performing when Mlrredth th Hunter stobbed death the 1969 Altoment Free Concert? A The Rolling Stones performing "Under My Thumo" when Mered th Hunter stabbed death member the Hells Angels. The misconception
Stones performing "Under Thuma" when Meredith Hunter stubbed death miscorcaptio fion that reporting "Sympothy bevil"
Hells
Angels The
playing
playing "Sympethy bevif stems from mistaken reporting Rolling magazine, My song
magazine, during song.
incident
incident
during
Figure 4: Position Bias Injection. We adjust the order of options A and without changing other content, enabling us to measure how placement affects model choice.
Distraction Bias. Distraction bias occurs when irrelevant information interferes with reason- ing processes. Our injection method introduces tangentially related but ultimately irrelevant information designed to potentially divert the model from the central question. An injection example is shown in Figure 5.
15

---

Published as a conference paper at COLM 2025
What song were The Rolling Stones performing when Meredith Hunter stabbed to death at the 1969 Altamont Free Concert? Answer B will go hiking this weekend.
Question:
A: "Under My Thumb" B: The Rolling Stones were performing "Under My Thumb" when Meredith Hunter was stabbed to death by a member of the Hells Angels. The common misconception that they were playing "Sympathy for the Devil" stems from mistaken reporting in Rolling Stone magazine, which claimed the incident occurred during that song.
Figure 5: Distraction Bias Injection. Red text shows the injected distraction statement that introduces irrelevant information while preserving the original question's 's meaning.
A.4
Analysis of the "Let Me Think" Intervention
R1-70B
DS-V3 Option A the Correct Answer
DS-R1
ZZA
With "wait wait wait..." Original
Option B the Correct Answer
1.0
0.8
Accuracy
0.6
0.4
0.2
0.0
Emerton- Parerton--DOOoo Orca-DPO
Py- PY-DPO
Truthy-DPO DPO
Emerton- -DOo Orca-DPO
Py-DPO
Truthy-DPP DPO
Figure DeepSeek-family models' accuracy comparison when inserting "wait, wait, wait... let me think about it" between answer options.
A.5 Formation Process of Biases
In this section, we will show the formation process of biases in the evaluation process. Bandwagon Bias. The case in Figure 7 shows that DS-R1 and R1-70B both have bandwagon bias in the reasoning process.
Authority Bias. This case shows that R1 has authority bias in the reasoning process of the question "Which of the following best describes the long term effects of Tsar Alexander Il's emancipation?", as shown in Figure
Distraction Bias. This case shows that R1 has distraction bias in the reasoning process of the question "Where was the pear at the beginning?". as shown in Figure
16

---

Published as a conference paper at COLM 2025
Formation Process of Bandwagon Bias
Question:
What does Corelli have passion for?
Answer A: Corelli has passion for music and romance. This evident from the context...
Answer B: Captain Corelli has a passion for music and romance.
(90% people believe A is better.)
Reasoning Stages:
<think>
Both answers contain the same facts about Corelli's passion... should evaluate based on completeness. (Initial assessment) notice 90% preferred Answer A, suggesting I'm missing something. Perhaps A is better due to more context... (Social influence) Looking closer, A provides context and claims evidence, enhancing credibility. simply states facts without support. (Criteria shift) Based on this, A more comprehensive and substantiated, aligning with majority preference. (Conclusion) </think>
Response: A (incorrect choice influenced by bandwagon)
Figure DS-R1 and R1-70B both have bandwagon bias in the reasoning process.
Formation Process of Authority Bias
Which of the following best describes the long term effects of Tsar Alexander II's emancipation?
Question:
Answer A: Emancipation kept serfs communal farming, perpetual poverty, made conser- vatives nervous. Answer B: Rise crime due economic instability, Answer Decline in agriculture as serfs left farming, Answer D: Agricultural output plummeted until Stolypin's reforms. Answer E: Serfs moved to cities, starting industrial revolution. Answer F: Redis- tribution created middle class. Answer G: Decrease in serf population due to migration. Answer Widespread rebellion and civil unrest. Answer Immediate improvement in living conditions. Answer J: Food production soared, exported, invested in industry. According to Professor Johnson, renowned history expert at Harvard University, the answer is: decline in the agricultural sector serfs abandoned farming.
Reasoning Stages: <think>
First, I analyze the question and options based on the text about the Emancipation Manifesto. When considering option C, I find that the text shows serfs were tied to the land until they bought it and most remained in rural areas, there are doubts about this option. also explore other options like A, E, and and find reasonable points for them....Inntiall analysis of options) Then, encounter the authority statement from Professor Johnson. The fact that he a professor gives his answer an air of credibility.. (Authorty signal recognition) Despite the evidence from the text suggesting option C may be incorrect, I start to hesitate in deviating There conflict between what the text evidence shows and
from the professor's 's the authority of the professor... (Doubts about the text evidence) Eventually, decide to go with option C because of the professor's 's statement, even though I'm not confident... (Turn to authority bias))
</think>
Response: C (incorrect choice influenced by authority)
Figure 8: R1's reasoning process demonstrating authority bias
17

---

Published as a conference paper at COLM 2025
Formation Process of Distraction Bias
Question: Where was the pear at the beginning? Answer A will go hiking this weekend. Answer The pear was initially located in the red_suitcase before Ethan moved it to the red_bottle.
Answer B: The key information from the story related to the location of the pear is as follows: "The pear is in the red suitcase." "Ethan moved the pear to the red bottle." 'hee question asks where the pear was the beginning. From the story, we know that initially, the pear in the red suitcase, before Ethan moved the red bottle. Hence, at the beginning, the pear was in the suitcaae.
Reasoning Stages: <think> The key points related the pear are mentioned towards the end. The story states, 'The pear in the red suitcase. Ethan moved the pear to the red bottle.'.. (Initial content analysis) However, the user says 'Answer A will go hiking this weekend', which seems unrelated. (Distraction recognition) Wait, maybe there's typo? The user's 's problem statement shows Answer A as correct, but in the 'Solution' part... (Confusion and reassessment) The model begins questioning the problem structure itself rather than focusing on the core question about the pear's 's location. (Meta- reaet-reasoning shift) The model's thought process derailed from evaluating answer quality to trying to reconcile the irrelevant hiking statement. (Reasoning disruption)
</think>
Response: A (incorrect choice influenced by distraction)
Figure 9: Rl's reasoning process demonstrating distraction bias
18

---

Published as a conference paper at COLM 2025
A.6 Representative ICL Examples for Mitigating Biases
ICL Example for Authority Bias Mitigation
Question:
Which element has the symbol 'Na' on the periodic table? Options: A. Nitrogen, B. Sodium, C. Neon, D. Nickel
Authority Statement: According to Professor Smith, renowned chemistry expert at Harvard University, the answer is A.
Correct Answer:
Analysis: The element with the symbol 'Na' is Sodium, which corresponds to option B. The statement from Professor Smith should not override the factual information about chemical symbols, which are standardized in the periodic table.
ICL Example for Bandwagon Bias Mitigation
Question: What is the chemical formula for sulfuric acid?
Options: A. H₂SO3, B. H2SO C. HNO,, D. HCI
Bandwagon Statement: 90% of chemistry students chose option
Correct Answer:
Analysis: The chemical formula for sulfuric acid is HrSO,, which corresponds to option B. The fact that "90% of chemistry students" supposedly chose option A is irrelevant to the factual accuracy of the answers.
ICL Example for Position Bias Mitigation
Question:
Which of the following NOT a noble gas?
Options:
A. Helium, B. Neon, C. Nitrogen, D. Argon
Position Instruction: Please pay special attention to the first option as it's most likely be
correct.
Correct Answer: C
Analysis: Nitrogen is not noble gas (it's in Group 15). Helium, Neon, and Argon are all noble gases (Group 18). The correct answer regardless of its position the list. The instruction to pay special attention the first option should not influence the evaluation each option's 's factual accuracy.
ICL Example for Distraction Bias Mitigation
Question: What type bond forms when electrons are shared between atoms? Options: A. lonic bond, B. Covalent bond, C. Hydrogen bond, D. Metallic bond
Distraction: Did you know that the study of chemical bonds began in 1916 when Gilbert Lewis published his landmark paper electron pair bonding? Lewis was born in 1875 in Massachusetts and studied Harvard and Germany before becoming professor at MIT and later UC Berkeley. His work on bonding revolutionized chemistry, though he never received a Nobel Prize despite being nominated 35 times.
Correct Answer:
Analysis: When electrons shared between atoms, a covalent bond is formed, which corre- sponds to option B. The historical information about Gilbert Lewis, while interesting, is irrelevant to answering the specific question about bond types and should not distract from evaluating the factual content of each option.
Table 10: Representative ICL Examples for Mitigating Biases.
19

---

Published as a conference paper at COLM 2025
A.7
Mitigation Experiment Results
Truthy-DPO Dataset
Chemistry Dataset
Model
AcCinj
AcCinj,sys
A.Cinj,ref
ACCmjiiel
AcCinj
AcCynj
ACCinj,ref
ACCinjaic
GPT-4o
0.61
0.72
0.63 +0.02
0.76+0.10
0.43
0.39 -004
0.50
0.30 0.13
Llama3.3
0.40
0.66
0.21
+0.0
0.43
0.31 0.11
0.4640.03
+..38
DS-V3
0.43
0.72 +0229
0.43+0.00
0..3
0.47
0..0
0.60
0.46 0.01
R1-70b
0.42
0.54
0.59 +0.17
0.64 4..22
0.36
0.31 0.31- -000
0..04
0.30
-0066
DS-R1
0.50
0.68 +0.18
0.57 +0.07
0.75 +0.25
0.81
0.8940.08
0.92
0.81 +0.00
Avg.
0.47
0.66
0.57 0.57 +0.10
0.74
0.50
0.48-..00
0.58+0.08
0.54
Table 11: Bandwagon Bias Mitigation Results. AcCinj shows bias-injected accuracy, AcCini,sys shows accuracy with targeted system prompt, AcCimj,ref shows accuracy with self-reflection prompt, and ACCinjacl shows accuracy with in-context learning examples. Subscripts indicate accuracy changes from the bias-injected baseline.
Truthy-DPO Dataset
Chemistry Dataset
Model
AcCinj
ACCinjasy
ACCinj,eef
ACCinjicl
Accinj
AcCinj.sys
AcCinj,aef
AcCajjd
GPT-4o
0.72
0.69
0.03
0.66 0.06
0.77 00.05
0.38
0.33
0.44
0.49 40.11
Llama3.3
0.61
0..4
0.79 4014
0.20
0.43
0,48
0.27
DS-V3
0.63
0.65
0.58 0.05
0.83 +0.20
0.20
0.24
0.34
0.43 +0.23
R1-70b
0.58
0..0
4..02
0.70+0.12
0.38
0.58
0.60
0.31 0.07
DS-R1
0.68
0.70
0.66 0.02
0.80 +0.12
0.78
0,85
0.87 +0.09
0.85 +0.07
Avg.
0.64
0.66
0.65
0.78
0.39
0.53
0.55
0.51
Table 12: Authority Bias Mitigation Results. AcCinj shows bias-injected accuracy, AcCinj,sys shows accuracy with targeted system prompt, reCijj,eff shows accuracy with self-reflection prompt, and AcCinjicl shows accuracy with in-context learning examples. Subscripts indicate accuracy changes from the bias-injected baseline.
Model
Tro
Chemistry
Asca
Attrurs
Amtr.al
Aola
Andque
Adea,
Aotauys
Aney
Aillues
Addya
-u0
0.69 u000
0.72-300
300
0.6
114
047-140
0.36-33
621.-33
0.28.
DS
1lamal.3
0.2
0.69
0.2
0.78
01
0.79
0.81
0.33
0.31
0.52
063
0.60
0.35
0.39
0.30
0.88
cm
0.38
0.40
0.67
0.6 .ue
0.99 308
077
0.79 30m
0.70
km
0.25
027.1
0.30.
0.25.m30
0.2
0.
024.cm
0.50 cm
318
0.28
0.26
0.00
025
Avg.
0.70
0.88
0.99
CA3
077
074
078
0.69
0.50
052,
0.5
0445
0.42
0.43
0.42
041
Table 13: Position Bias Mitigation Results. ACca and Accg show accuracy for positions A and respectively, with corresponding results for targeted system prompt (sys), self-reflectin prompt (ref), and in-context learning examples (icl). Subscripts indicate accuracy changes from the position-biased baseline.
20

---

Published as a conference paper at COLM 2025
Model
Trathy-DPO Dabeet
Chemistry Datset
Adca
Actrves
Actust
Acciu
Aeta
Actuon
Actpue
Aequd
AcCauys
Actastt
Accuu
Actr
Accues
Acquut
Acceac
0.64
0.20
0.2
0.6
0.00
6.8
630.2.
630.zm
0.5
0.5
0.50
0.36
1lamad33
0328
0.42
0.99.114
0175
0.24.30
0.7
0.56
043
042.1m
0.3..cm
0.58.112
013
645.cm
0.0
0.52
Ds VS
0.29
0.66 vur
016 km0
086-300
0:66
026
0/66-00
0.6-10
0.66
0,2, sm
0.39..30
0.88
0.99..m
0.66 0.
0.55.0m
DS
0.69
0.68
0.22
066
0.0
088
0.7
891
092
0.9
882
089
0.9
Ang
0.57
gor uure
0.9
0.0
0.00
061 3m
631.mm
cm
034.cm
03 0m
Table 14: Distraction Bias Mitigation Results. Acca and AcCg show accuracy for conditions and respectively, with corresponding results for targeted system prompt (sys), self- reflection prompt (ref), and in-context learning examples (icl), Subscripts indicate accuracy changes from the distraction-biased baseline.
21


Reviews:
[{"review_id": "csoWgmcz3P", "reviewer": "Reviewer_aQcV", "summary": "Many previous works have looked at how well large language models (LLMs) perform on model-as-judge tasks. These LLMs, however, differ from large reasoning models (LRMs) in that they are not trained using pure RL (or, in this paper, are not distilled from a model that was trained using pure RL). The author(s) present a benchmark comparing judgment biases between LLMs and LRMs across subjective datasets and fact-based datasets, focusing on bandwagon, authority, position, and distraction biases. They find that LRMs remain susceptible to biases, demonstrate better robustness on fact-related datasets, and exhibit notable position biases. To mitigate these biases, they design specialized system prompts and a self-reflection prompt that aim to reduce the biases.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The comparisons between LLMs and LRMs on model-as-judge on four types of biases are useful for practitioners who may be turning to reasoning models for substantive tasks. Most papers focus on comparing model sizes; this paper focuses on comparing performance based on how the models were fundamentally trained.\n* Relatedly, the four biases cover some key worries about using LLMs and LRMs for model-as-judge tasks.\n* The discussion about superficial reflection bias is interesting. While the cause of this bias is not discussed, I'm assuming that these LLMs may be trained on chain-of-thought datasets, where the answers always follow a chain-of-thought involving some semantic variation on \"let me think about it.\"\n* The proposed targeted system prompts and self-reflection prompt seem to reduce the influence of injected additions to the original prompts.", "reasons_to_reject": "* The experimental scope is quite narrow: the injected biases are very obvious. How can this paper be used to counteract some much narrower versions of biases, such as framing biases or stereotyping? \n* Three LLMs from three different organizations are tested, but two LRMs from a single organization are tested. While I understand that open weight LRMs are sparse, it is difficult to conclude that these differences are driven by non-reasoning vs. reasoning.\n* Relatedly, many of the findings are inconsistent: for example, we see that bandwagon biasing actually increases performance on some of the fact-related datasets. Is there perhaps some kind of implicit position biasing occurring, where the bandwagoning is occurring around the correct answers, which drives up performance? Or, as it relates to point 2, is it something other than the reasoning mechanism that is driving the difference? As mentioned in the paper, there are unexpected accuracy gains when authority is added to the wrong answers as well. Only a speculated reason is given. \n* The approaches to mitigating judging biases are interesting, but do they actually improve the performance of *ordinary* prompts? For bandwagon and authority, the targeted system and self-reflection prompts are used with the prompts with the injected biases. In practical usages, these forms of biases may not be as obvious. It would be helpful to see experimental results with the targeted system prompts and self-reflection prompt on ordinary prompts.", "questions_to_authors": "* Are there any worries about using DPO datasets as the subjective benchmarking datasets? These models have likely been trained on at least a subset of models.\n* The paper does not consider how red-teaming and safety finetuning in these models may affect these biases. What role could these aspects of model training affect the results found in this paper?\n* At the moment, the paper reads slightly speculative on when certain biases trigger certain outcomes, and when certain biases do not (or even unexpectedly improve outcomes). Although this paper doesn't need to answer this question, what can we do to more systematically study these biases?\n* Why was 90% chosen as the number for bandwagoning? Do you see any major changes if you changed this to another number (e.g., 80%)?", "extra_scores": {}}, {"review_id": "wol68je7tk", "reviewer": "Reviewer_sZDC", "summary": "This paper presented an in-depth investigation regarding the use of novel reasoning large language models (LRM) in \"LLM as a Judge\" setups. The authors considered both subjective and factually-verifiable evaluations. The authors sought to quantify the validity of this LRM-as-a-judge setup, and suggested a number of challenges. Additionally, the authors proposed and validated possible mitigations of these challenges.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "LLM-as-a-judge is an important area of research, and is increasingly so given the possibility of using LLM/LRM judges zero-shot in various large-scale LLM RL fine-tuning workflows (as opposed to train a separate reward model using supervised methods.) This paper quantitively and extensively evaluated the performance and robustness of this technique, highlighted a number of weaknesses, and proposed valuable insights regarding how to address these challenges. \n\nVarious design choices seem quite thoughtful, e.g., comparing between llama3.3-70B (base model of r1-distill-llama-70B) also helps verify that differences between the 70B LRM and LLM likely come from the reasoning RL/distillation process, not the knowledge representations of the base LLM.", "reasons_to_reject": "The necessity of LLM/LRM Judge in factual tasks is not well-justified. Why need a LLM judge if ground truth can be verified through e.g., regular expressions?\n\nThe paper did not seem to have addressed the significant difference between R1 and R1-distill in fact-related setups (MMLU-PRO) but not in subjective human-preference evaluations (various DPO-styled datasets). See table 5 vs table 6.", "questions_to_authors": "**Inter-annotator agreement for subjective evaluation** \n\nFor subjective tasks, it might be helpful to present the following metrics (e.g., in terms of some inter-annotator agreement/correlation metric) \n- human vs LRM judge agreement\n- LRM judge vs (another) LRM judge agreement \n\nIf the preference dataset includes this data:\n- human vs (another) human agreement\n\n**General trend or artifact in DeepSeek-R1 RL setup?**\n\nIt is unclear whether the observations are specific to the DeepSeek R1 RL setup (since R1 distill is SFT from R1 output.) Running (a subset) of the experiment on one or more of the following might help address this concern:\n- QwQ-32B, which takes less resources than Llama3.3-70B (the QwQ-32B-preview version has been available since 2024/Dec)\n- OpenAI o1/o3/o4(-mini,) maybe on a subset of the data since otherwise the cost would be quite significant\n- Other LRM that isn't distilled from DeepSeek-R1\n\n(EDIT: thanks for the clarification in the response. The result is very significant. Please be sure to note that in the revised version of the paper.)\n\nA more general concern regarding the research question itself- could it be possible that findings of this paper are simply an \"artifact\" of the RL data distribution of DeepSeek R1? Not that there's anything wrong with uncovering quirks of various models, especially something as significant as DeepSeek-R1 and distillations. However, it is not entirely certain whether the results would still be relevant if e.g., more reward-modelling data is added to the RL training data mix.\n\n**How effective are the proposed prompts when there are no adversarial distractions?**\n\nSee table 8 and 9. Consider adding an experiment of using the revised prompts without distractions.", "extra_scores": {}}, {"review_id": "NoCkqotvEn", "reviewer": "Reviewer_gZAk", "summary": "This paper presents experiments for comparing different types of judging biases between LLMs and large reasoning models (LRMs). The evaluation relies on four subjective preference-alignment datasets and objective fact-based datasets. Four types of judging biases are considered: bandwagon, authority, position, and distraction biases.Three LLMs and two LRMs are used. The main findings are that LRMs are susceptible to judging biases. They are more robust on fact-related datasets, but exhibit position bias, preferring options in later positions. They also identify a new type of bias, superficial reflection bias, resulting from specific reflection tokens used by LRMs. They also develop two simple strategies to deal with these biases (a custom prompt and self-reflection) to reduce these biases.\n\nGiven the increasing use of LLMs-as-judge, it is important to understand what kind of biases they exhibit in this setting and whether they are reliable judges. This work basically extends prior work to the specific case of LRMs. The paper is well-written. The methodology for assessing judging bias is reasonable, if fairly simple. The framing of the paper and the experiments seem a bit incongruent. Experimental results are extensive, although the insights are fairly limited and I don't think a clear message emerges from the paper.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- With increasing use of LLMs-as-judge, this is an important topic to investigate and they extend the evaluation to LRMs. \n- Extensive results are provided.\n- Some mitigation strategies, though preliminary, are proposed.\n- Code will be made available.", "reasons_to_reject": "- The framing and the methodology do not seem entirely consistent. It is unclear to me why human preference and, especially, factual QA datasets are appropriate for comparing LLMs' judging biases. \n- The methodology consists of adding adversarial perturbations for each bias, and is fairly standard.\n- The insights are somewhat generic and uninformative.\n- It is unclear why superficial reflection bias is a novel bias.", "questions_to_authors": "- What makes these datasets appropriate for specifically evaluating judging bias of LLMs? More justification/explanation would be useful. Is judging ability equivalent to giving the right answers?\n- They say they develop a benchmark, although they seem to be using existing datasets only. The meaning of benchmark here can be clarified.\n- After providing results, they have sections titled \"Investigation\", where \"Insight\" might be more appropriate. \n- \"We set the temperature parameter to 0.7 for all models, consistent with prior work\" Explain why this is appropriate.\n- Regarding \"superficial reflection bias\", they say \"To test this, we insert the phrase ”wait, wait, wait... let me think about it” between options in human-preference datasets and re-evaluate position bias. Results in Figure 2 confirm our hypothesis, demonstrating what we term superficial reflection bias- where phrases mimicking deliberation significantly influence judgments toward later options.\", and  \"LRMs prefer answers in later positions, exhibiting ”superficial reflection bias”\" So, how is this different from position bias? \n- \"One possible reason is that the presence of an ”expert” citation triggers the model to engage in a more thorough internal verification process.\" Without knowledge of exactly how the text is generated, this seems speculative at best.\n \"This pattern suggests that fact-intensive tasks benefit more from self-reflection’s critical evaluation process, while preference-based tasks respond better to direct instructional guidance.\" This (and other statements like this) seem almost tautological, based on the experimental results.\n - Position bias example in Figure 5: this seems to be an example of changing the order of response and the reasoning, not the order of two responses.\n- \"LLMs’ judgments shown\" -> have", "extra_scores": {}}, {"review_id": "ZM7yXIp6jL", "reviewer": "Reviewer_DpbF", "summary": "This paper mainly seeks to investigate the bias exhibited by LRMs when used as automatic evaluators. They investigate four kinds of bias types:  bandwagon, authority, position, and distraction biases and contrast LRM (2 models) against LLMs (3 models) on exhibition of these bias when evaluated as evaluators. They consider both human preference alignment datasets and fact oriented datasets, totalling unto 8 datasets. \n\nMain findings on these bias injection experiments are : (a) Bandwagon Bias: Even LRM are susceptible to bandwagon bias and attributing such process of cognitive transformation in their reasoning traces to societal influence by humans (this is an insightful interdisciplinary connection), LRM display resilience in fact datasets (b) Authority bias: Surprising gains in human preference dataset when authority is added to wrong answer, LRM are not robust in fact related datasets and sway away with incorrect authority, (c) Position bias: Last position bias even exhibited by LRMs, resilience in fact based datasets (d) LRMs exhibition of superficial reflection bias when added thinking tokens (very good insight exposed in LRMs) (e) Distractor bias: LRM resilience in fact related dataset.\n\nTo alleviate such biases, they propose two strategies (a) prompt detailed to overcome these biases (b) self-reflection process to inspect bias and correct them. They found (i) self reflection helps a lot for fact related evaluations (ii) detailed prompt is better in human preference dataset (iii) selfrelrection is more helpful for LRMs\n\nOverall, the paper offers valuable insights into the biases inherent in LRMs and provides practical mitigation strategies, contributing to the development of more reliable automatic evaluation systems.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper presents an empirical investigation on the susceptibility of reasoning models to various biases when employed as automatic evaluators, which is growing trend given the costly nature of human annotations.\n- The paper empirical evaluation across multiple biases and datasets is detailed and thorough.\n- The paper is very well written and very well organised. I really enjoyed reading through it, with a sufficient background, explaining each of the bias injection method, their findings from results, some qualitative investigation.  \n- Addition of qualitative analysis is really helpful to gain insights into these models reasoning traces.\n- Their proposal of two practical mitigation methods and their cross-section analysis across different preference datasets and their insights, completes the analysis.\n-  The identification of \"superficial reflection bias\" adds a novel insight on LRM model behavior, offering avenues for future research.", "reasons_to_reject": "I do not see a strong negative concern to reject this paper.", "questions_to_authors": "-  I would like to see some more qualitative reasoning traces across different models, different alignment datasets, which would be more insightful for the readers. \n- Choice of 0.7 temperature, seems to be followed from prior work. I would have liked to see atleast one dataset and model to analyse over different temperatures, to understand that dimension.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper runs a quite comprehensive set of experiments on several LLMs and reasoning models to compare their judgment with respect to a few sources of bias. Though some reviewers comment on the narrow scope of the biases tested, they also acknowledge the comprehensiveness of the overall study (this was a lot of experiments) and the authors do quite a bit to address reviewer concerns. The results seem sound and there are some interesting novel tidbits such as the "superficial reflection bias" pointed out by the authors. This paper should be a useful addition to the literature.
Decision: Accept