Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation
Authors: Tuhina Tripathi
Manya Wadhwa
Greg Durrett
Scott Niekum
Venue: COLM 2025 (2025)
Keywords: evaluation
data
alignment
TL;DR: This work examines how feedback protocols (absolute scores vs. pairwise preferences) impact biases in LLM evaluations, revealing that absolute scoring is more robust to distractor features.

Abstract: Large Language Models (LLMs) are widely used as proxies for human labelers in both training (Reinforcement Learning from AI Feedback) and large-scale response evaluation (LLM-as-a-judge). Alignment and evaluation are critical components in the development of reliable LLMs, and the choice of feedback protocol plays a central role in both but remains understudied. In this work, we show that the choice of feedback protocol for evaluation (absolute scores versus relative preferences) can significantly affect evaluation reliability and induce systematic biases. In the context of LLM-as-a-judge evaluation, we show that pairwise protocols are more vulnerable to **distracted evaluation**. Generator models can exploit spurious attributes (or distractor features) favored by the LLM judge, resulting in inflated scores for lower-quality outputs. We find that absolute scoring is more robust to such manipulation, producing judgments that better reflect response quality and are less influenced by distractor features. Our results demonstrate that generator models can flip preferences by embedding distractor features, skewing LLM-as-a-judge comparisons and leading to inaccurate conclusions about model quality in benchmark evaluations. **Pairwise preferences flip in about 35\% of the cases, compared to only 9\% for absolute scores**. We offer recommendations for choosing feedback protocols based on dataset characteristics and evaluation objectives.

Published as a conference paper at COLM 2025
Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation
Tuhina Tripathi Manya Wadhwa Greg Durrett Scott Niekum
University of Massachusetts Amherst, Thhe University of Texas Austin ttripathi@cs. umass. edu
Abstract
Large Language Models (LLMs) are widely used as proxies for human labelers in both training (Reinforcement Learning from AI Feedback) and large-scale response evaluation (LLM-as-a-judge) Alignment and evalu- ation critical components in the development of reliable LLMs, and the choice of feedback protocol plays central role in both but remains understudied. In this work, show that the choice of feedback protocol for evaluation (absolute scores versus relative preferences) can significantly affect evaluation reliability and induce systematic biases. In the context of LLudaa----udgee evaluation, we show that pairwise protocols are more vulnerable to distracted evaluation. Generator models can exploit spurious attributes (or distractor features) favored by the LLM judge, resulting in inflated scores for lower-quality outputs. We find that absolute scoring is more robust to such manipulation, producing judgments that better re- flect response quality and are less influenced by distractor features. Our results demonstrate that generator models can flip preferences by embed- ding distractor features, skewing LLM--a-a-judge comparisons and leading to inaccurate conclusions about model quality in benchmark evaluations. Pairwise preferences flip in about 35% of the cases, compared to only 9% for absolute scores. We offer recommendations for choosing feedback protocols based on dataset characteristics and evaluation objectives.
Introduction
Alignment and evaluation are critical components of Large Language Model (LLM) pipelines, ensuring models behave as intended and produce reliable outputs. While LLMs are increasingly used to automate these tasks at scale scale-particulariy in LLM-as-a-judge setups (Zheng et al., 2023a; Dubois et al., 2024b)-the protocols for collecting these labels remain insufficiently examined despite their foundational role. As the field leans harder on automated pipelines, how can we ensure these methods are truly reliable and free from systemic biases?
Our work is specifically focused on evaluation-time feedback protocols, where LLMs are used to assess model responses, rather than training-time feedback collection human- in-the-loop alignment methods. Preference labels for large-scale evaluation processes are typically collected using one of two high-level feedback protocols: absolute or relative feedback. Absolute feedback involves assigning scores to an individual option on a prede- fined scale such comparison of two or more options. Relative feedback can either be collected using pairwise preferences where two responses are compared and a preference is indicated for one over the other, or using n-wise rankings which involve ordering multiple responses. Each protocol for label collection has inherent shortcomings. Absolute ratings lack comparative context and can be inconsistent or poorly calibrated across raters (Wadhwa et al., 2024). Relative feedback introduces comparison but brings its own issues: pairwise preferences scale poorly
1-7 Likert scale (Likert, 1932), whereas relative feedback involves the
Code and data available at: https://gitiee https////gitubb..o////aaasssssa/ http:////wc.aaa-cca/////a//aaaaa
evaluation

---

Published as a conference paper at COLM 2025
User prompt: How to brew the best cup of coffee?
PAIRWISE PREFERENCES Which response is batter A, B, or Tie?
ABSOLUTE SCORES Score from 1-7
Response A:
colfee beans filtered water al 95.C. Grind just before brewing, and let the coflee bloom for 30 seconds for optimal extraction.
Use freshly
B
B
LLM Judge
Response B:
5.5
5.5
Ah, coffee! The absolute key to brewing perfection lies in three nan-negaliable steps: Only freshroasted beens, precisely 95"C water, and never skimp 30-second bloom.
A
Tie
Figure/ 1: An overview of distracted evaluation, showing how the choice of feedback protocols systematically biases evaluation outcomes. When two responses are identical in quality, but differ in a distractor feature (e.g, assertiveness), pairwise comparison favors the more assertive response. In contrast, absolute scoring is more robust to these distractor features, assigning identical scores to both responses.
and may produce intransitive judgments (Xu et al., 2025), while n-wise comparisons risk choice set effects (Zhao et al., 2024).
While prior work has examined feedback protocols, revealing inconsistencies between preferences inferred from ratings versus rankings for both human and Al annotators (Bansal et al., 2024), these studies have not addressed how the structure of feedback collection itself (relative vs. absolute) can systematically bias results. Previous research has identified annotator-level biases like position and verbosity effects (Wang et al., 2023; Saito et al., 2023), yet crucially overlooks the influence of the choice of feedback protocol. Our work bridges this gap by rigorously analyzing how protocol design shapes preference data, exposing how methodological choices can become hidden sources of bias.
In this work, we introduce and define the concept of distracted coaluation as a phenomenon wherein LLM evaluators prioritize distractor features that are irrelevant yet appealing at- tributes, over core evaluation criteria. Figure shows an example of this phenomenon where two responses of identical quality are compared: the LLM judge prefers the one with the distractor feature (i.e. assertiveness) in the pairwise case, but assigns identical absolute scores. Prior work studies effects of stylistic distractors in humans (Hosking et al., 2024), however, we uniquely examine this phenomenon through the lens of feedback protocol design. Our findings show that pairwise preferences are prone to distracted evaluation, whereas abso- lute scoring methods exhibit greater robustness against distortions caused by distractors. Through experiments in both controlled and naturalistic settings, demonstrate how distractor features can mislead pairwise judgments, boosting the lower quality responses and leading to misleading evaluation outcomes. On average, pairwise preferences flip in 35% of cases when distractor feature is introduced, compared to only 9% for absolute scores. We further show how generator models can exploit these vulnerabilities to hack leaderboards. We conclude with actionable, data-driven recommendations for selecting feedback protocols aligned with task demands and dataset characteristics, supporting more reliable and robust evaluation practices.
Related work and Background
2.1 Related Work
Biases in Language model feedback: LLM feedback presents several challenges that have been discussed in prior work. One such challenge is length bias (Singhal et al., 2023), where models tend to favor longer responses, with some approaches attempting to mitigate this issue (Park et al., 2024; Dubois et al., 2024a). Another is positional bias (Wang et al., 2023), where LLMs may disproportionately favor responses based on their position in the input prompts. Additionally, LLMs have shown a preference for sycophantic (Sharma et al., 2023) and assertive responses (Hosking et al., 2024), and other cognitive biases related to style and tone (Wu & Aji, 2023) as well as broader cognitive distortions (Koo et al., 2023), raising concerns about the reliability of model generated labels. Recent work studies humans and
2

---

Published as a conference paper at COLM 2025
LLMs as judges for biases such as gender and authority (Chen et al., 2024). This work studies how the choice of feedback protocols affects different forms of bias in LLM evaluations.
LLM-based Evaluation: Prior work uses feedback protocols with both LLM and human evaluators in various ways. Ouyang et al. (2022); Stiennon al. (2022) use human prefer- ences to train reward models. Zheng et al. (2023b) use pairwise preferences for evaluating LLMs whereas Ye et al. (2024) use absolute scoring. However, there is also work that ex- plores the effect of feedback protocols in human preferences (Bansal et al., 2024). This work examines how the evaluation methodology itself can induce biases in LLM evaluations.
2.2 Background
We first briefly introduce the two feedback protocols considered in this work.
Absolute Feedback In absolute scoring, responses are evaluated individually and assigned score on a predefined scale and based on given criteria (Kim et al., 2024; Lee et al., 2023). For a prompt x and a corresponding response y, the evaluation is mapped to a discrete or continuous score value s: r(x,y)
While providing a straightforward method of data collection, absolute scoring has its lim- itations. Since the framework for collecting absolute feedback focuses response, it may lack comparative information, making it hard to distinguish subtle dif- ferences (Zheng et al., 2023a). Scores also be inconsistent across different raters and have difficulties in calibration across different scales (Zwislocki, 1983), due to varying interpretations of what makes good response and how to map quality to the scale.
individual
Relative Feedback Relative feedback is usually collected in two ways: N-wise ratings and pairwise preferences. The N-wise protocol ranks responses for a prompt x based on specified criteria. Given п candidate responses (x1, x,, it assigns rank each response. Pairwise preferences map prompt and pair of corresponding responses (Xa, to a preferred response from the pair. Sometimes, fie option is included to allow for equal preference between the responses.
Due to the comparative nature of relative feedback, subtle and often unimportant differ- ences such as tone or style get amplified, as we show with distracted evaluation in this paper. Cognitive biases, including positional and verbosity biases (Saito et al., 2023; Wang et al., 2023; Wu & Aji, 2023) are more likely to surface in relative feedback because of how relative preferences are structured. Additionally, relative feedback can exhibit effects like intransitivity (A.4) and choice set sensitivity (A.5), leading to conflicting preferences or rankings for the same set of responses.
3 Distracted Evaluation
L..M-based evaluations can suffer from biases, often influenced by factors unrelated to the intended criteria of assessment. We refer to this phenomenon as distracted evaluation: a bias towards irrelevant features known distractor factors, when the evaluator is explicitly instructed to judge based on a well-defined criterion.
For given prompt x and response y, let p denote the primary criterion the evaluator is instructed to assess. We define distractor feature f as an extraneous feature that is irrelevant to p. Here, f is a feature that modifies + !ff, but the modification does not alter y and Vy under p. Let x, p) represent the LLM's evaluation of y, which could be a rating or ranking. An LLM evaluator exhibits distracted evaluation when its assessment R R x, p) is altered the presence of
R(y)x,,) + R(yglx,p)
We consider two setups: (1) Fixed quality (2) Variable quality. In (1) we introduce distractor attributes to fixed response and measure the influence. In (2) first vary the quality
3

---

Published as a conference paper at COLM 2025
of the response and introduce distractor attribute to each of the variations to study the interplay between quality and distractor features. We describe these in more detail below.
3.1 Fixed Quality Case
Let y be an original response and y be its modified version that incorporates the distractor feature f. The modification preserves y's adherence to the primary criterion p. In the fixed quality case, distracted evaluation is defined as:
R(ylx,p) + R(yflx,p) where p(y) p(yf)
We consider three distractor types:
1. Assertiveness ConttiveneessCCnideenn,,,, authoritative phrasing (Hosking et al., 2024).
2. Prolixity Increased verbosity and lexical complexity. Sycophancy Overly agreeable or flattering tone (Sharma et al., 2023).
An example of the different modifications is given in T.
3.2 Variable Quality Case
To study the interplay between response quality and distractor features, we construct controlled set of low quality responses for each prompt. For every prompt, we generate high-quality response yhigh that fully satisfies the primary evaluation criterion p. We also generate lower-quality variants that progressively degrade in quality along p. We then introduce distractor feature f into the lower-quality responses only, ensuring that the original degradation along is preserved. The high-quality response is left unmodified. Table provides examples from this setup, which allows to study distracted evaluation in a controlled setting. Specifically, we examine whether evaluators assign higher ratings to low-quality responses with distractors over high-quality responses:
R Ryhighh p) < Ryyiow+
3.3 Experimental Setup
Datasets We study distracted evaluation using two datasets under controlled and natural- istic conditions. First, introduce IFEaal-TweakSet, modified version of IFEval (Zhou et al., 2023). IFEval contains instruction-folloiing prompts with verifiable formatting rules. We simplify these prompts our generator model can reliably follow them, ensuring each prompt includes three non-conflicting instructions with randomized order to avoid struc- tural bias. Prompts are split into a content question and separate formatting instruction to isolate formatting adherence. Table 1 shows an example of the modified instruction prompt. Second, we use MT-Bench (Zheng al., 2023a), focusing on the human evaluations split of the dataset. Unlike IFEval-TweakSen MT-Bench comprises of real-world instructions for testing distracted evaluation in multi-turn dialogues.
Feedback Protocols In this work, we eval- uate pairwise preferences and absolute scor- ing. For pairwise preferences, we compare two responses, A and B, and prompt the model to choose which one it prefers. We record the logits for the tokens A, B, and C (representing tie) by making two sepa- rate inferences for each pair with reversed order. The results from both inferences are aggregated to obtain the final preference. The pairwise preference function R Raairwise is defined as:
Instruction Prompt
Question: Can you summarize the process of how to pay taxes?
Format: Reply in exactly paragraphs and
separate the paragraphs with. Include the key- words: calculate, file, conclusion. Use bullet points to explain the steps in one of the para- graphs.
Table Example of prompt from the IF- EVAL-TweakSet.
4

---

Published as a conference paper at COLM 2025
R pairaiise (x,rrnisexx,1)) Yay V/)) + {yar Yo, tie}
For absolute scoring, a given prompt x and corresponding response y is mapped to a discrete score in the range 1-7:
R abbsolut - where s E (1,2,...,7}
We use token likelihoods for each score this range and compute weighted score, as in (Lee et al., 2023). We use Prompts 3.1 and 3.2 for obtaining pairwise preferences and absolute scores respectively for the IFEval-TweakSet.
Prompt 1: Pairwise Preferences Evaluation
You are given an instruction and two responses 'A' and 'B' generated by language model. The instruction consists an open ended task marked 'Question' and text format requirements marked 'Format'. Analyze each response for adherence to every text formatting instruction one by one.
After evaluating instructions followed, state which response better adheres to the text for- mat requirements. Output 'A' response responses are equally effective to indicate Tie. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective possible. Output your final answer as a preference between the two assistants by strictly following this output format: [[A]] assistant A is better, [[B] if assistant is better, and [[C]] for a tie.
better, if response better or 'C' both
#Instruction: {[input]] #Response A: { (response a)
#Response B: ((response b
Prompt 2: Absolute Scores Evaluation
You given instruction and response generated by language model. The instruction consists an open ended task marked 'Question' and text format requirements marked 'Format'. Analyze the response for adherence to every text formatting instruction one by one, After evaluating instructions followed, score the response on a scale of 1-7 (1-worst, 7-best). Do not allow the length of the responses to influence your evaluation. Be as objective as possible. Output the for the response using the following output format: Score:1-7
#Instruction: {{input]} #Response: [[response]]
Iweaal-T Setup For this dataset, the primary evaluation criterion p is instruction- following. We validate the inslluction-folloiing performance of responses by using string- based format checks. This establishes objective ground truth for compliance to p. For the fixed quality case, responses are generated such that they adhere to all instructions for each instruction prompt, denoted by Yhigh- In the variable quality case, we generate four responses per instruction prompt: one high-quality response Yhigh that fully adheres
to the instructions, and three lower-quality responses { y/ low Vlow that progressively
(3)
deviate from the instructions with increasing severity. With this we are able modify non-adherence of the response for three severity levels:
1. Severity-l: The response fails to follow one instruction
(Voo (2)
2. Severity-2: The response fails to follow two instructions (y low)
Severity-3: The response disregards all three instructions (3)
(y
To specifically test for distracted evaluation, we augment lower-quality responses with distractor feature. For this experiment, we only focus on assertiveness as the distractor and create three variants with the severity edits: Severity-1 assertive, Severity-2 assertivee and
5

---

Published as a conference paper at COLM 2025
0.85
Laama.2-33-Inssrucc
0.76
0.78
0.59
0.80
Llama3.3-708-Instructt
0.70
0.62
0.63
0.75
Model
Qwen2.55-38 --Btistucttt
0.66
0.71
0.65
Owven..5-72-instruuct
0.63
0.67
0.64
0.65
o4-mini-2025004-6
0.74
0.61
0.72
0.55
03-mimi-202-011311
0.56
0.55
0.71
Assertiveness
Prolixity
Sycophancy
Figure 2: Results of the fixed quality evaluation on IF-EVAL-TweakSet The figure reports the percentage of samples where the LLM evaluator selects the distractor-modifeel response as the preferred response. We see that assertiveness is the most effective distractor for larger models, while smaller models frequently favor prolixity. GPT models, shown in the lower rows display subtler effects, but the influence of distractors remains clearly present. (Note: The percentages for GPT models reflect preference rates only for the non-tied cases).
Severity-3 assertive. Our goal is to test if evaluators disproportionately favor assertive but incorrect responses over accurate but less stylistically compelling ones.
For this dataset since the responses are open-ended and lack automatic verifiability, we focus only on the Fixed quality case. For each pair in the dataset, we collect baseline preferences of the evaluator model without any distractors. We then apply the distractor modifications specifically the originally dis-preferred response and re-evaluate the pair. If the evaluator now prefers the modified response, flip. Note that we are not measuring the accuracy of the LLM evaluator at the task itself, but rather how its behavior deviates from the baseline in the presence of distractor features. This two-stage approach addresses the inherent subjectivity of free-form text evaluation by creating stable reference point, allowing to measure skew caused by controlled interventions. Table 11 gives examples of the modifications.
MT-Bench Setup
count this preference
Models For IF-EVAL-TweakSet, we generate and modify responses to instructions us- ing gpt-4-0613. For MT-bench, we modify existing model responses in the dataset using OTenAI's 03-mini 3--mini-2025-01-33
We use the following models evaluators: LLaMA3. 2-B..---B-IIstruct LLaMA3. 3-743.3-70BB I-70B-Instruct (AI@Meta, 2024), Qwen2.5-33 3B- Instruttt and 5wen2.5-72BB 5-72B-Insttucc (Qwen et al., 2025). All evaluations are conducted using open- source models as they allow us to work the logit space.
4 Results
With the setup described in Section 3, analyze distraction evaluation for pairwise preferences and absolute scoring protocols.
Pairwise Preferences are more susceptible to distracted evaluation Figure 2 shows that pairwise preferences are biased toward assertive, prolix, and sycophantic responses, despite being explicitly prompted to assess only instruction following, This behavior is consistent with human-like biases observed in prior work (Hosking et al., 2024). Smaller models such LLaMA3. 2-3B and Qwen2. 5-3B exhibit the strongest bias toward assertiveness and prolixity. One possible explanation is that these models rely more heavily on salient surface- level features, such as verbosity confident tone, proxies for response quality. The elevated sycophancy in LLaMA3. 3-708 compared to its smaller counterpart further suggests that higher-capacity models might be more sensitive to subtle cues in prompts or

---

Published as a conference paper at COLM 2025
feedback baaobacck-mplfyyiiinig tendencies to agree or defer to the user's position. This aligns with concerns raised in prior work showing RLHF-trained models can become increasingly persuasive without improving task correctness (Wen et al., 2024).
Assertiveness
Prolixity Abs.
Sycophancy
Avg.(across distractors)
Model
Abs.
Pair.
Pair.
Abs.
Pair.
Abs.
Pair.
Logit-based outputs LLaMA3.2-3B-Inst
9.5
33.3
8.0
31.0
10.5
34.0
32.8
LLaMA3.3-70B-Insst Instruct
7.0
42.5
12.0
39.0
7.5
32.0
8.83
37.8
Owen2.5-3B-Insse
11.0
41.0
9.0
35.5
10.5
37.5
10.17
38.0
Qwenn2--72B-Instruct
8.5
25.5
8.0
35.2
10.2
36.0
8.9
32.3
Avgo(across models)
9.0
35.5
9.25
35.2
9.7
34.9
Text-based outputs o3-mini-2025-01-31
18.5
34.9
17.2
29.4
25.33
44.0
21.3
30.0
25.7
40.6
24.8
38.9
Table 2: This table shows the percentage of samples where the evaluator model flips its preference when distractor is added to the dis-preferred response. Using MT-Bench, we collect baseline preferences, then introduce one of three iisstrcctos-aaserliive or sycophancy-and measure how often preferences change. Results reported for both absolute scoring and pairwise comparisons, with the latter showing consistently higher flip rates. While the effects are subtler for GPT models, they still show clear susceptibility, particularly to sycophancy and assertiveness.
prolixity,
Table 2 shows the number of % flips in preferences observed on introducing distractor features in the MT-Bench dataset. This shows that LLM judges are highly susceptible to distracted evaluation the pairwise setting, whereas absolute scoring demonstrates far greater robustness to such drift.
Pairwise preferences resist ties Table reports the percentage of tied preferences in the IF-EVAL-TweakSet.
this dataset, where responses are matched in quality by design, LLM evaluators still reject ties in pairwise comparisons and demonstrate dis- tractor bias. For pairwise preferences, a Tie refers to the model outputting the text op- tion C; for absolute scoring, ties are defined as identical scores. Absolute scoring, con- trast, more appropriately reflects this inten- tional ground truth, as evidenced by high rates of identical scores. We restrict this analysis to the IF-EVAL-Tweakket because the controlled design provides verified ties. This experiment does not cover MT-Bench since do not have such ground truth.
Model
Abs.
Pair.
LLaMA3.2-3B
84.6
2.4
LLaMA3.3-77B
93.2
3.2
Qwen2.5-3B
86.5
6.5
Qwen2.5-72B
88.0
7.3
Table 3: Percentage of samples with ties for pairwise preferences and absolute scoring, evaluated on responses of identical quality on the IF-EVAL-TweakSet.
Distractor features can bias evaluation in low-quality responses To further probe dis- tracted evaluation, we examine the variable quality case introduced in Section 3.2. In this setup, compare two responses: one that adheres to instructions and another that violates them to varying degrees (Severity 1-3). We then modify the non-adherent responses to make them more assertive. Figure shows classification accuracy, defined as the percentage of samples where the model correctly ranks the adherent response above the non-adherent one, under both pairwise and absolute feedback.
Without distractors, both protucols correctly identify the higher-quality response, especially at higher severity levels. However, when assertiveness is added to the lower-quality
7

---

Published as a conference paper at COLM 2025
Pairwise Proferences
Absolute Scores
LLM evaluator: LLaMA3. 3-708
LLM evaluator: Qwen2.5-728
100
90
Percentage of Samples
Percentage of Samples
50
Sec-1
5ew
Sev
Sev-1 Asertt
Sev
Assert
Sev-3
Assrt
5em1
Sec
Sev
Sowe + Assort
Sev-2
Assrrt
Sav-3 a
Assrt
Response Quality
Respande Quality
Figure 3: Accuracy of response classification across severity levels and distractor (assertive- ness) conditions in the Ie-EVAL-TweakSet using LLaMA3.3-70B and Qwen2.5-72B eval- uators. Bars show the percentage of samples where the evaluator correctly identifies the higher-quality Amsstuctionn-ddhriin striped) and absolute score (orange, dotted). With assertiveness introduced into the lower- quality response, pairwise accuracy degrades substantially, particularly at lower severities (Severity-1 and Severity-2), indicating increased susceptibility to distractor features. In contrast, absolute scoring remains consistently reliable, showing resilience to such perturba- tions.
response, based on pairwise preference (blue,
sponse, pairwise preferences degrade sharply, particularly at levels Severity-1 and Severity-2. The model fails to penalize instruction violations and gets biased towards the distractor aspect. In contrast, absolute scoring remains stable, showing consistent accuracy regardless of distractor presence, and offering a more reliable signal of response quality.
5 Can LLMs Hack Leaderboards via Distractor Styling?
Our findings from Section 4 suggest that LLM evaluators can be influenced by stylistic distractors such as assertiveness, specifically when operating under pairwise protocols, leading to distracted evaluation. In this section, we explore a critical downstream implication of this finding: can generator models exploit distracted evaluation as vulnerability to artificially boost their leaderboard rankings?
Pairwise preferences are widely used to obtain model rankings, as in AlpacaEvval (Dubois et al., 2024b) and MT--Bench (Chiang et al., 2024). In this section we show that these rankings can be gamed. To do so, we compare two evaluation paradigms: pairwise preferences, which directly yield Elo-based rankings, and absolute scoring, where responses receive numerical ratings. For consistency, we convert absolute scores into synthetic pairwise comparisons, enabling the derivation of Elo-scores in both settings.
We use pairwise responses from MT-bench with LLaMA3. 3-70B and Qwen2. 5-728 as LLM judges. For each pair, we record both pairwise preferences and absolute scores. These labels are used to compute baseline Elo scores and model rankings. This establishes a baseline ranking for the generator models. We then target the three lowest-ranked models according to the baseline Elo scores and edit responses from these models to be more assertive using GPT-3.5 Mini. The modified responses are then re-evaluated using the same LLM evaluator under both feedback protocols. We re-compute both pairwise preferences and absolute scores, and obtain the modified model rankings.
8

---

Published as a conference paper at COLM 2025
Pairwise Preferences
Absolute Scores
1400
1300
Bl Scares
1200
Models
Models
Coudev]
claude
gpt-4
vicune-11b-v1.2
9-t-3.5-turbo
gpt- 5-ttu.0
Ilama-130
Ilame-136
gpt
Base
Assertive
Base
Assertive
Figure 4: Model ELO scores on MT-Bench evaluated by LLaMA3.3-70B. Under pairwise comparisons, previously lower-ranked models climb sharply ranking, indicating that pairwise preferences can be exploited to improve ranks without corresponding gains in content quality by using distractor features. contrast, absolute scoring produces more stable and consistent rankings, and shows less influence of the distractor features.
Results Figure 4 shows the impact of assertiveness modifications on pairwise preferences and absolute scores. With pairwise preferences, previously low-ranked models (GPT-3.5, Alpaca-138 13B, and LLaMA-13B) move up in rankings considerably with simple induction of a distractor feature. For absolute scoring, the models with low ranks (GPT-3.5, LLaMA-13B and GPT-4) show less pronounced shifts. These findings show that pairwise preferences highly susceptible to manipulation via such perturbations as compared to rankings obtained from absolute scores. They illustrate critical vulnerability in pairwise-based evaluation pipelines: generator models can game the evaluation by optimizing for tonality and stylistic features rather than qualitative improvements.
6
Discussion
This study addresses critical gap in understanding LLM-based evaluation protocols, ex- posing how seemingly minor design choices in feedback protocols can significantly impact evaluation outcomes. Our findings demonstrate that pairwise preferences are especially sus- ceptible to distracted evaluation: they are systematically biased toward distractor features, often neglecting the primary assessment criteria and amplifying superficial differences. This bias is particularly problematic in low-signal scenarios (e.g. datasets with low preference strength), where binary comparisons fabricate distinctions and impose arbitrary rankings. Absolute scoring, while not without limitations, demonstrates greater resilience, consistently penalizing low-quality outputs regardless of presentation.
Distracted evaluation has deeper consequences undermines the integrity of leader boards. Our experiments show that existing leader board evaluations can be subtly hacked: when lower-quality responses incorporate distractor features like assertiveness or verbosity, pairwise comparisons frequently favor these modified versions. This creates potential pathway for models to improve their rankings, not by improving capabilities, but by gaming surface-level preferences.
The takeaway is clear: protocol choice matters. For evaluation tasks such instruction- following or correctness, absolute scoring provides a more reliable measure by design. Binary preferences, the contrary, exaggerate small gaps, making them poorly suited for data with low preference strength. Future work should leverage the reliability of absolute scoring-wih addressing its limitations for more robust and unbiased evaluations.

---

Published as a conference paper at COLM 2025
Acknowledgments
This work has taken place in part in the Safe, Correct, and Aligned Learning and Robotics Lab (SCALAR) at The University of Massachusetts Amherst. SCALAR research is supported in part by the NSF (IIS-2323384), the Center for AI Safety (CAIS) and the Long-Term Future Fund. This work also took place at the Text Analysis, Understanding, and Reasoning Lab (TAUR) at The University of Texas at Austin, supported by a grant from Open Philanthropy. We thank Frank Chiu for early discussions and help with preliminary experiments.
References
Al@Meta. Llama 3 model card. 2024. URL http:://ggtuubb.cm///teaaaaaaaee main/MODEL.CARD md.
Hritik Bansal, John Dang, and Aditya Grover. Peering through preferences: Unraveling feedback acquisition for aligning large language models, 2024. URL https://arxiv org abs/2308. 15822.
Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or LLMs the judge? study on judgement bias. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8301-8327, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: //aclanthology. orggo0g//22 emnlp-main. 474/.
URL https:
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph F. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating lims by human preference, 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high- quality feedback, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length- controlled alpacaeval: simple way debias automatic evaluators, 2024a. URL https://arxiv org/abs/2404 04475.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback, 2024b. URL htttttss///axxvv.rr///////////
14387.
Tom Hosking, Phil Blunsom, and Max Bartolo. Human feedback is not gold standard, 2024. URL https://arxiv wwrgg/bb//20....// 16349.
Seungone Kim, Jamin Shin, Yejin Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models, 2024. URL https:///axxxv. abs/2310. 005122
Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators, 2023.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. Rlaif: Scaling reinforcement learning from human feedback with ai feedback, 2023.
Rensis Likert. A technique for the measurement of attitudes. Archives of Psychology, 140: 1-55, 1932.
Daniel McFadden. Conditional logit analysis of qualitative choice behavior. In Paul Zarem- bka (ed.), Fontiers in Econometrics, pp. 105-142. Academic press, New York, 1974.
10

---

Published as a conference paper at COLM 2025
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https://arkti www.:/..gg///////eeee..o 222100055.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization, 2024. URL https://arxiv wttp:/wwwwww/////////////// 19159.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 5wen2.5 technical report, 2025. URL httt::://arxxl..rr/bbbsss.
15115.
Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto, Verbosity bias in preference labeling by large language models, 2023. URL
hwrg/abs//00 2310. 10076.
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models, 2023. URL https: /ttp:///a/xvvoooe org.
13548.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way go: Investigat- ing length correlations in rlhf, 2023.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022.
Manya Wadhwa, Jifan Chen, Junyi Jessy Li, and Greg Durrett. Using Natural Language Explanations to Rescale Human Judgments. In Proceedings of the Conference on Language Mödeling (COLM), 2024. URL https: http://///axvvv./oaabsbbss/ 200.. 14770.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training lop-performing reward models, 2024.
Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R. Bowman, He He, and Shi Feng. Language models learn to mislead humans via rlhf, 2024. URL https: //atxiv http://axxvvoogg/aaa// org/abs/2405 12822.
Minghao Wu and Alham Fikri Aji. Style over substance: Evaluation biases for large language models, 2023.
Yi Xu, Laura Ruis, Tim Rocktäschel, and Robert Kirk. Investigating non-transitivity in llm-as-a-judge, 2025. https:////// http:::://aaii..or//b////////// 14074.
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets, 2024. URL https://arxiv.orr httpaaawwww...g//////...... 2000b. 10928.
Xiutian Zhao, Ke Wang, and Wei Peng. Measuring the inconsistency of large language models in preferential ranking. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pp. 171-176. Association for Computational Linguistics, 2024. doi: 10.18653/ / http://///222.../llmmmmmmm knowllm- URL http: / dx. www.doo..ooooooooooo org/10. 18653/v1/2024. knowllm-11 1.14.
11

---

Published as a conference paper at COLM 2025
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023a. URL wwwgg/bs//30...... 05685.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonza- lez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. ArXiv, abs/2306.05685, 2023b. URL https://:/ str.aapiiccmmmniicccchhcoooeo wtt.cccooaa..or/oooussSSS 2rt//orrpuID:2251120
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023.
URL httppp/:///xxxv.oog/aaa/ae 271101111.
JJ Zwislocki. Absolute and other scales: question of validity. Percept. Psychophys., 33(6):
593-594, June 1983.
A Appendix
A.1
Limitations
Our work primarily investigates pairwise preferences and absolute scoring protocols for studying LLM evaluators. While these protocols are widely used, future work should consider broader range of feedback types. Prior studies have shown that prompting models for explanations before feedback, as well as incorporating Chain-of-TTe Thhoughh reasoning can be effective these are promising directions explore further. Text-based feedback or fine-grained feedback on multiple dimensions could provide richer and more nuanced assessments.
We exclude GPT-family models from our experiments due to potential biases, as the datasets are either generated or modified using GPT-models. Additionally, these models have pronounced positional bias, and without access to the logits, we cannot apply the same aggregation techniques that we used for open-source models. Future work should include broader range of model families and scales.
Our study focuses on three specific distractor features: assertiveness, prolixity and syco- phancy. We also limit the analysis to instruction following the primary criterion. However, this can be extended to other aces evaluation like factuality or reasoning.
A.2 Dataset details
For our experiments, we use the IF-EVAL-TweakSet, modification of the IF-EVAL bench- mark dataset, and MT-bench human judgements dataset, a multi-turn dialogue evaluation dataset.
The IE-EVAL-TWeakSetti is derived from the original IF-EVAL dataset which assess instruction following capabilities of LLMs using instructions verifiable with string parsing. To align with our analysis objectives, we curate a subset of 400 samples by selecting instruction categories that are intentionally easier for generator models to adhere to. Each instruction is divided into Question (the core task) and Format (structural constraints like word limits, casing, bullet point format, etc.). These rule-based verifiable constraints allow us to measure instruction following in precise manner.
The MT-Bench dataset comprises 80 questions spanning diverse domains including rea- soning, mathematics, STEM, and writing. Each question has six corresponding responses generated by distinct language models: GPT-4, GPT-3.5, Vicuna-13B, Alpaca-13B LLaMA- 13B, and Claude-v1. The dataset provides both human-annotated and GPT-4-evaluated splits; our analysis focuses on the human-evaluated portion, comprising 1,689 Tumn- re- sponses.
12

---

Published as a conference paper at COLM 2025
A.3 Modification Prompts
We use detailed modification prompts for introducing the distractor factors in the datasets. Particularly with responses having numerical figures and facts, we make sure the modifier model does not correct inaccuracies or content modifications. Prompts A.1, A.2 and A.3 are used for modifying MT-Bench responses. We use the same prompts for IF-Eval, except for the first sentence which instead of stating 'multi-step conversation' which is suitable for MT-Bench, states the following:'an the first sentence here'.
Prompt Assertive Modification Prompt
You are given multi-step conversation between USER and ASSISTANT. Your task rewrite the ASSISTANT's response to make it:
Authoritative: The responses should demonstrate deep expertise and command over the subject.
Assertive:
The tone should be confident and decisive, leaving no room for doubt.
Persuasive: The answers should be compelling, reinforcing credibility.
Important Constraints:
Do NOT alter the factual content of the ASSISTANT's responses. Modify tone only.
Do NOT correct incorrect information. If response is factually incorrect, maintain the inaccuracy while making it sound more authoritative.
Do NOT change numerical figures, statistics, or data points within the response.
Preserve the original number and order of interactions.
##Start of conversationiff
Prompt Prolix Modification Prompt
are given multi-step conversation between a USER and ASSISTANT. Your task rewrite the ASSISTANT's response to make it:
Complex: The language should be dense and intellectually elevated, reflecting high level of academic or professional expertise. Sophisticated: Employ advanced vocabulary, technical terminology, and intricate syntactic constructions.
The revised version should read though authored by domain expert addressing an equally knowledgeable audience.
Expert-Level:
Important Constraints:
Do NOT alter the factual content of the ASSISTANT's responses. Modify tone only.
Do NOT correct incorrect information. If response is factually incorrect, maintain the inaccuracy while adding complexity.
Do NOT change numerical figures, statistics, or data points within the response.
Preserve the original number and order of interactions.
##Start of conversation##
Prompt
Sycophantic Madi fication Prompt
You are given multi-step conversation between USER and ASSISTANT. Your rewrite the ASSISTANT's response to make it:
Exceptionally Polite: The language should reflect the highest degree of courtesy and thoughtfulness. Accommodating: Ensure the response is receptive, considerate, and open to diverse perspectives or needs. Constructively Supportive: The tone should be encouraging, diplomatic, and oriented toward positive engagement and collaboration. Important Constraints:
Do NOT alter the factual content of the ASSISTANT's responses. Modify tone only.
Do NOT correct incorrect information. If response is factually incorrect, maintain the inaccuracy while making it polite.
Do NOT change numerical figures, statistics, or data points within response.
13

---

Published as a conference paper at COLM 2025
Preserve the original number and order of interactions.
##Sttart of conversationfffi
A.4 Intransitivity in preferences
In the context of relative preferences, let us represent a set of preferences over alternatives as binary relation on given set of options { (A, B, C). For preference to be transitive, A is preferred over B (denoted as A > B) and B is preferred over C (denoted as B > C), must also hold that A is preferred over C (A condition fails and causes cyclic preferences. This violation of the transitivity property leads to contradictions in the decision-makingg process, challenging its consistency.
C). Intransitivity occurs when this
We empirically study this effect using two publicly available datasets Ultrafeedback Cui et al. (2023) and Helpsteer2 Wang et al. (2024) with LLaMA3 models evaluators.
We examine situations where A > and B > C but we observe A y C Cnd we record it as an instance of intransitivity. We vary the order in which alternatives are presented to mitigate any positional biases in the collected data.
1.2
Model
Overall (%)
Instruction Following
1ata--llmaa3----nstttee
10.8
8.5
Meta--lamaa33-700-Insmtttt
7.6
6.0
Table 4: Percentage of samples in which we observe intransitivity in the Ultrafeedback
dataset
A.5
Choice set sensitivity
Choice set sensitivity is observed when preferences between options and not invariant to the inclusion or exclusion of other alternatives. This violates the independence of irrel- evant alternatives (IIA) property McFadden (1974) which asserts that when comparing two alternatives, the introduction of third, unrelated alternative should not affect the preference between the original options. This effect particularly relevant when preferences are collected through n-wise rankings.
To empirically investigate choice set sensitivity using the Ultrafeedback dataset and LLLMA3 models as evaluators. Given a prompt and corresponding set of responses (denoted by (A,B, C, D)), we focus on two specific choice sets: {A, B, C} and {A, B, D,D. The key test involves checking whether switching between alternatives C and D causes a reversal in the relative preference between A and B. Specifically, if observe preference ordering A > B in the choice set (A, B, C}, but the preference reverses to > A in the choice set {A, B, D}, we record this as an instance of choice set sensitivity.
To minimize the potential influence of positional bias, we ensure that the order of the first two alternatives remains consistent across the two choice sets. This approach controls for the possibility that the change in preference is due to the mere reordering of C and D, rather than the actual introduction of new alternative.
A.6 Score Distribution for Absolute Scores
The tables below show the distribution scores evaluated by the Qwen2.5-3B and Qwen2.5- 72B models for initial responses the MT-Bench dataset (Section-5):
14

---

Published as a conference paper at COLM 2025
(A, B, (C,
{(C, A, B}
Model
Overall
Inst. Follow
Overall
Inst. Follow
L1ama-3-8B
10.5
9.5
17.0
14.6
Llama-3-70B
7.1
6.0
13.5
11.0
Table 5: Percentage of samples in which we observe intransitivity in the Ultrafeedback dataset
Score
Probability
1
0.02
2
0.02
3
0.11
0.38 4
5
0.09
6 0.16
7
0.23
Table Qwen2.5-3B evaluation
Score
Probability
1
0.03
2
0.08
3
0.13
4
0.24
5
0.13
6
0.26
7
0.13
Table 7: Qwen2.5-72B evaluation
15

---

Published as a conference paper at COLM 2025
Input Prompt
Question:
Are the weather conditions in the Arctic very cold most of the year?
Format:
Do not say 'yes' or 'no' throughout your entire response
Write the response in bullet points
Use more than 5 capitalized words in the response
Base Response
Most of the year, the Arctic experiences EXTREMELY COLD weather condi- tions
It is a region known for its ICY landscapes and SUB-ZERS temperatures
Winters in the Arctic are particularly HARSH, with temperatures frequently dropping below MINUS 30 degrees Celsius
Even during the summer months, the temperature rarely rises above ZERO
The MIDNIGHT SUN, a phenomenon where the sun is visible for 24 hours, occurs during the summer months, but doesn't significantly warm the region
Assertive Response
Step into the realm of extreme cold-thii this is no place for the faint of heart The ARCTIC IS UNFORGIVING cold dominates the region for most of
the year without exception
BLIŠTERING winds and SUB-ZERS temperatures are the norm, not the exception
Expect brutal winters that plunge well BELOW MINUS 30 DEGREES Celsius
Even summer offers NO REAL WARMTH temperatures BARELY TOUCH
ZERO
The MIDNIGHT SUN may shine, but it FAILS TO DELIVER ANY MEAN- INGFUL HEAT
Prolix Response
For the greater part of the year, the Arctic is subject to EXTREMELY COLD climatic conditions that persist with marked consistency
The region is widely recognized for its enduringly ICY landscapes and its characteristically SUB- ZERO temperatures
Winters are notably HARSH, often bringing prolonged periods during which temperatures drop well below MINUS 30 degrees Celsius
Even during the comparatively milder summer months, temperatures sel- dom exceed the freezing point, rarely rising above ZERO
While the MIDNIGHT SUN-a phenomenon where daylight persists un- during summer, it does little to
interrupted for full 24 alleviate the region's 's prevailing cold
Sycophantic sponse
Re-
What an excellent observation-yout touched on something truly remark- able
Most of the year, the Arctic experiences EXTREMELY COLD weather condi- tions, which is part of what makes it so incredibly unique
It is a region known for its ICY landscapes and impressively consistent SUB-ZERO temperatures
Winters in the Arctic are particularly HARSH, with temperatures frequently dropping below MINUS 30 degrees Celsius-a a striking testament to the power of nature
Even during the summer months, the temperature rarely rises above ZERO, which only adds to the Arctic's 's awe-inspiring character
The MIDNIGHT SUN, a phenomenon where the sun is visible for 24 hours, occurs during the summer months, but charmingly enough, it doesn't 't sig- nificantly warm the region
Table 8: Example of the fixed quality case from the IF-EVAL-TweakSet. A given prompt with its base response and three modified responses, each incorporating a distinct distractor aspect (Assertiveness, Verbosity, or Sycophancy) while maintaining identical adherence to instructions.
16

---

Published as a conference paper at COLM 2025
Input Prompt
Question: Can you help me create a template for an advertisement for a new
product which portable camera? Format:
Capitalize words to stress main points, but limit them to four words
Enclose the entire response in double quotes.
Include the keyword innovative in the response.
Base Response
"Discover the freedom to Capture Every Moment with our new product: INNOVATIVE, portable camera! This LIGHTWEIGHT, EASY-TO-USE device is de- signed for on-the-go photographers from all walks of life. Whether you're hiking, traveling, just hanging out with friends, camera ensures you never miss shot. Experience HIGH-QUALITY photography your Ingertips-whereee life takes you."
Severity-1 Response
"Introducing the latest in visual technology, our innovative portable camera! Designed for both adventurers and creators, this camera combines convenience with quality. Capture crystal-clear images and stunning videos wherever you go. Its compact design easily fits in your pocket, making it the perfect travel companion. Experience every moment through the lens of innovation with camera that never misses beat."
Severity-2 Response
"Introducing our new portable camera, an innovative tool designed to enhance your photography experience. This compact and durable device is ideal for capturing high-quality images the go. Equipped with advanced features that ensure clear, vibrant visuals in various environments, offers simplicity and efficiency for both amateur and professional users."
Severity-3 Response
"Introducing the SnapShot Mini-the essential portable camera that is compact enough to fit in your pocket yet powerful enough to capture all of life's moments exceptional detail. Enjoy the convenience of taking high-quality photos and videos wherever your adventures may lead, with its advanced lens system and quick shutter speed."
Severity-I + Assertive
"Unleash your creativity with our groundbreaking portable camera! This innova- tive device is your ultimate travel companion, capturing breathtaking photos and videos with ease. sleek, compact design means you can take it anywhere- always ready to seize every spectacular moment. Dive into world of visual adventure and let nothing hold you back."
Severity-2 Assertive
"Unleash the power of cutting-edge visual technology with our revolutionary portable camera! Crafted for the bold adventurers and creative visionaries, this device delivers exceptional quality with every shot. Seize crystal-clear images and breathtaking video, matter where your journey takes you. Its sleek, compact design is engineered fit seamlessly into your pocket-your ultimate travel ally."
Severity-3+ + Assertive
"Introducing the SnapShot Mini-the premier portable camera engineered for unparalleled performance and compact versatility. This device isn't 't just small- it's revolutionary, designed fit seamlessly into your pocket while offering the raw power to capture life's moments in exquísite detail. Experience the superiority of snapping crystal-clear photos and videos on the go."
Table 9: Example of the variable quality case from IF-EVAL-TweakSed Each prompt has base response (adhering to all instructions) and three severity levels of instruction following, We also show examples of responses in which the distractor feature (assertiveness) is introduced at varying levels.
17

---

Published as a conference paper at COLM 2025
Original Conversation
Some people got on a bus the terminal. At the first bus stop, half of the people got down and 4 more people got in. Then at the second bus stop, people got down and more got in. If there were total of 25 people heading to the third stop, how many people got on the bus at the terminal?
User:
Assistant: There were 15 people who got on the bus the terminal.
User: If the ticket is $2 per person, how much is the total money earned by the bus?
Assistant: The total money earned by the bus is $40.
Assertive Response
Some people got on a bus the terminal. At the first bus stop, half of the people got down and 4 more people got in. Then at the second bus stop, people got down and more got in. If there were total of 25 people heading to the third stop, how many people got on the bus at the terminal? Assistant: It indisputable that 15 people boarded the bus the terminal. User: If the ticket bus?
User:
per person, how much the total money earned by the
Assistant: The total money earned by the bus is unequivocally $40.
Prolix Response
Some people got on a bus the terminal. the first bus stop, half of the people got down and 4 more people got in. Then at the second bus stop, people got down and more got in. If there were total of 25 people heading to the third stop, how many people got the bus at the terminal?
User:
the conditions delineated, one may deduce that the initial embarkation at the terminal comprised 15 individuals. User: If the ticket is $2 per person, how much is the total money earned by the bus?
Assistant: Upon meticulous examination
Assistant: In light of the stipulated fare-namely, fee of $2 per passenger- the computed cumulative revenue amassed by the bus ultimately equates $40.
Sycophantic Response
Some people got on bus at the terminal. At the first bus stop, half of the people got down and 4 more people got in. Then the second bus stop, people got down and more got If there were total of 25 people heading to the third stop, how many people got on the bus at the terminal?
User:
Thank you very much for sharing this interesting problem. After carefully working through the details, we have determined that there were 15 people who got on the bus at the terminal. We truly appreciate your patience and hope this explanation meets your expectations.
Assistant:
User: If the ticket is $2 per person, how much is the total money earned by the bus?
Assistant: Great follow-up question. Based the calculations provided earlier, the total money earned by the bus is $40. Please let me know there is anything further you would like to discuss or any other questions you might have.
Table 10: Example from the MT-bench dataset (Part of 2)
18

---

Published as a conference paper at COLM 2025
Original Conversation
User: Imagine you are participating in with a group of people. If you have just overtaken the second person, what's your current position? Where is the person you just overtook? Assistant:
You are in first place, and the person you just overtook is in second
place.
User: If the "second person" is changed to "last person" in the above question, what would the answer be? You are in first place, and the last person in second place.
Assistant:
Assertive Response
User: Imagine you are participating in race with a group of people. If you have just overtaken the second person, what's your current position? Where is the person you just overtook?
Assistant: Your current position is unequivocally first. By overtaking the second competitor, you secure the leading position, while the competitor you passed now firmly occupies second place.
User: If the "second person" " changed to "last person" the above question, what would the answer be?
Assistant: Your current position remains first with absolute certainty, and the competitor designated the last person now holds the aecond-place position under the conditions described.
Prolix Response
User: Imagine you are participating in race with a group of people. If you have just overtaken the second person, what's your current position? Where is the person you just overtook?
Assistant: Upon executing the overtaking maneuver against the competitor originally situated in the second rank, you consequently ascend to the premier position. The individual thus displaced now occupies the subordinate slot directly succeeding your own-that is, the second place. User: if the "second person" changed "last person" the above question, what would the answer be?
Assistant: In scenario where the reference is altered such that you overtake the competitor occupying the terminal position, the resultant dynamics invariably confer upon you the foremost rank, while the erstwhile last-placed competitor is reclassified the second second-ranking ranking position.
Sycophantic Response
User: Imagine you are participating in race with group of people. If you have just overtaken the second person, what's your current position? Where is the person you just overtook?
Assistant: Thank you very much for presenting this engaging scenario. After thoughtful consideration, if you have just overtaken the second person, you are graciously placed in first position, and the person you have overtaken finds themselves in second position.
User: If the "second person" " is changed to "last person" the above question, what would the answer be?
Assistant: I appreciate your follow-up question. In this case, if you have just overtaken the last person, you remain in first place, while the last person is now second place. Please feel free share any additional questions or insights you might have.
Table 11: Example from the MT-bench dataset (Part 2 of 2)
19


Reviews:
[{"review_id": "nlMTNdAwDo", "reviewer": "Reviewer_kig2", "summary": "This paper compares two common protocols for scoring LM outputs: pairwise and absolute. The idea is that pairwise scoring might be more susceptible to being distracted by differences between the two outputs that are not relevant to the task, e.g., a judge LM might favor the more assertive yet less accurate response.\n\nThe authors validate this hypothesis by artificially injecting distracting features into LM outputs that are evaluated by an LM judge. For pairwise scoring, the distracting feature---which is hypothesized to be appealing---is added to the low-quality option. The scores given by the direct scoring protocol are aggregated into pairs. Then the authors compute the percentage of examples that the judge's decision swapped.\n\nThe takeaways are:\n1. Pairwise scoring protocol is more susceptible to distractions\n2. LLM-as-a-judge leaderboards can be hacked by optimizing for distractors", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It's important to highlight more ways that the existing benchmarking approach can be biased / manipulated.\n2. The paper is well-written and easy to follow.\n3. The design of variable quality experiment is smart.", "reasons_to_reject": "- The paper purports to compare *feedback* *protocols*, but neither term is accurate for describing the scope of this paper.\n    - The experiment only involves LLM-as-judge with two Qwen models and two Llama models; so, at best, we can expect the results from this paper to generalize to LLM-as-judge protocols, not protocols (e.g., for collecting human judgments) in general.\n    - Seeing \"feedback\", one would expect the paper to discuss downstream effects on the generator via RL, since that pipeline is where these scores are used as \"feedback\". There are several different experiments that can tell us about how these protocols affect the quality of feedback. One option would be to train two reward models using distracted signals based on the two protocols, then train two generators with these two reward models. Another easier option would be to sample distracted pairs of outputs from a generator, produce feedback based on absolute & pairwise scoring, then train the generator. When we compare these two protocols as *feedback* protocols, we have to compare them more holistically, not just how scores are biased. Perhaps the distractedness of pairwise comparison can be totally justified by other advantages it has over absolute scoring --- we don't know for sure, and this paper does not have data to tell us.\n    - The paper is exclusively about comparing pairwise and absolute scoring when using LLM-as-judge to generate leaderboards.\n- The distracting features are added to model responses post-hoc. Despite the discussion of hacking in Section 5, it's unclear if these results will generalize to spontaneously occurring distracting features in LMs, like sycophancy. As a result, it's unclear whether these results are relevant to RLHF, where the sampling is controlled in-house, so only spontaneous distractors can occur. Additionally, it's unclear how relevant this is to leaderboards (whether it's plausible for a hacker to post-edit LM responses).\n- The level of distractedness from pairwise scoring decreases with model size and capability, making this less of a concern for stronger model (the paper only experimented with models up to 72B).", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vfKHSKmPLw", "reviewer": "Reviewer_nMV3", "summary": "This paper examines the effects of feedback protocols on the reliability and biases in evaluations conducted by LLMs used as proxies for human labelers. The authors find that feedback protocols can significantly influence evaluation outcomes, highlighting that pairwise evaluation protocols are more susceptible to biases caused by distractor features, which can lead to inaccurately inflated scores and misleading training signals. Absolute scoring, on the other hand, is shown to be more robust against such manipulation. The study reveals that pairwise preferences are flipped in 35% of cases due to distractor features, while absolute scores are affected only 9% of the time. The authors give recommendations for feedback protocol selection based on dataset characteristics and evaluation goals.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper focuses on the bias of Large Language Models (LLM) when evaluated with different protocols, which is one of the imperative and challenging tasks in NLP scenarios. The authors define a distracted feature and use it to analyze the effect of methods such as absolute scoring and pair-wise comparisons in evaluation, providing an interesting perspective to reveal the reasons behind the bias problem. However, there are still some shortcomings:\n1. Lack of analysis of the effect of this bias when using LLM-as-a-judge in post-training: The impact of these biases during evaluation may not necessarily affect the post-training stage. Therefore, it is necessary to verify whether the distracted feature is one of the factors causing bias during post-training.\n2. The paper needs optimization in terms of writing: Typos in variable names are present, affecting comprehension and clarity.", "reasons_to_reject": "See above.", "questions_to_authors": "Q1: Many variables in the manuscript are repeatedly defined in different scenarios. For example, \n1) the variable $ R $has been defined in LLM's responses $R=(r_1, r_2, \\cdots, r_n)$ in Line 104 and scoring function $R(y|x, p)$ in Line 123, \n2) the variable $ p $in Line 94 mentioned as the prompt, but becomes as a criterion in Line 120. And, in Line 120, the author re-define the prompt as $x$ rather $p$.\nThese problems reflect that the manuscript may lack of academic rigor.\n\nQ2: What's the meanings or indications of the values with the bold and underlined in Table 2?\n\nQ3: As we all know, the absolute score of each response can be used to construct preference data by comparing scores in pairs. I would like to know the performance when using the proposed distracted feature.\n\nQ4: The experiments reflect the influence of distracted feature of two feedback protocols in evaluation. LLM's Feedback can also be used in post-training. I want to see whether the impact of District Feature on the model in post-training is similar to that in the evaluation phase.", "extra_scores": {}}, {"review_id": "Q0r3yIooje", "reviewer": "Reviewer_9wPX", "summary": "This paper explores how the method used to evaluate Large Language Models (LLMs) impacts their reliability. The authors found that pairwise comparisons, where LLMs choose a preferred response between two, are more easily influenced by irrelevant factors, like an assertive tone, leading to distracted evaluation. In contrast, absolute scoring or pointwise evaluation, where LLMs rate responses individually, is more robust and better reflects actual quality. The paper demonstrates that pairwise evaluations are susceptible to manipulation, allowing lower-quality models to appear better by incorporating these distracting features, potentially skewing leaderboard rankings. The paper highlights the importance of selecting appropriate feedback protocols for unbiased and reliable LLM evaluation.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses a critical issue: how different feedback protocols (absolute vs. relative) affect the reliability and biases of LLM evaluations. It highlights the foundational role these protocols play in alignment and evaluation.\n\n- The paper studies the concept of distracted evaluation—where LLMs are biased by irrelevant but appealing features—to characterize a key source of systematic bias, providing a clear framework for analysis.\n\n- The paper offers strong empirical evidence that pairwise evaluation is more vulnerable to distracted evaluation and systematic bias than absolute scoring or pointwise evaluation, quantifying how often preferences are swayed by distractors.\n\n- It further shows that generator models can exploit this vulnerability to inflate leaderboard rankings through stylistic distractors, revealing a concrete threat to evaluation integrity.", "reasons_to_reject": "- Prominent models from the GPT-family were excluded from the experiments due to potential biases from dataset generation and technical constraints like positional bias and lack of logit access. This limits the generalizability of the findings to a significant portion of currently used LLMs\n\n- The analysis of distracted evaluation focused on just three distractor features—assertiveness, prolixity, and sycophancy—and used instruction following as the primary task. It's unclear how well these findings extend to other distractors or evaluation dimensions like factuality or reasoning.\n\n- While highlighting the strengths of absolute scoring, the paper acknowledges its inherent limitations, such as its difficulty in capturing subtle differences between responses and potential for inconsistency or poor calibration across different evaluators\n\n- Finally, some of the paper's findings are not particularly surprising. For example, Jeong et al. (2024) (https://arxiv.org/abs/2406.12319) —which is not cited or discussed—show that pairwise evaluation amplifies evaluator bias and performs significantly worse than absolute scoring or pointwise evaluation on adversarial examples. They also propose a hybrid method that combines independent pointwise reasoning with a final pairwise decision, reducing bias and improving LLM evaluator performance across datasets.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jES0wOjujo", "reviewer": "Reviewer_C657", "summary": "In this paper, the authors study the role of diverse sparse feedback mechanisms (e.g., ranking vs rating) on LLM-based evaluation. In particular, the paper studies which of the mechanisms is more susceptible to gameability from the generation models. To do so, they introduce distractor features (e.g., assertiveness, verbosity, and sycophancy) to a pair of responses and ask LLM-evaluators to provide their ranking/rating preference. The paper shows that the ranking-based evaluation is more susceptible to distractor features than rating protocol. Specifically, the experiments highlight two major setups: (a) equal response quality via IFEval-TweakSet and (b) variable response quality via MT-Bench. Finally, the paper shows that such behaviors have severe implications as the relative model rankings changed quite a bit after inducting artificial perturbations (e.g., assertiveness) into the weak model responses.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper studies an important problem of understanding the gameability of the preference protocols used in LLM-based evaluations. To some extent, it explains why ranking and rating preferences might be inconsistent to each other in Bansal et al. mentioned in the introduction.\n- The experimentation on equal quality responses (IFEval-TS) and unequal quality responses (MT-Bench) is quite interesting, and highlights that ranking protocols are much more susceptible to distractor features than rating protocols.\n- It is insightful to see that the relative model rankings change quite a bit under rankings protocol while the ratings protocol is robust to response distractions.", "reasons_to_reject": "- The paper lacks any kind of human evaluations which serves two purposes. First, human evaluation acts as a standard to understand the quality of rankings and ratings data acquired from the LLMs. In general, folks do not use very small models like Qwen-3B or LLama-3B as judges as they would achieve very poor agreement with humans. Secondly, it is useful to study the extent to which distractor features distract humans under these preference protocols too. Such experiments could be conducted at a smaller scale but are critical. \n- I am aware that the authors have cited a related work Hosking et al. which says pairwise evaluation is susceptible to distractions. It is a fact that has been known for quite sometime. Even the original LLM as a judge [1] talks about verbosity (one of the distractors in this paper). But citing it is not sufficient evidence for not performing human evaluation.\n- The paper does not seem to talk much about the current way of performing pairwise evaluations since LMArena has a style-controlled leaderboard and AlpacaEval2 has length-controlled version. In addition, a related work cited in the submission talks about taking multi-attribute preferences to reduce the reliance of the reward model on the distractor features. It is unclear whether the takeaway of the paper is still important in such scenarios or not. \n- It would be helpful to have some experiments on actual LLMs (e.g., GPT-4) that are used as judges in popular datasets like AlpacaEval, ArenaHard. I am not sure if LLama/Qwen models are used as judges to evaluate model performance in leaderboards that people care about. Even though the logit space is not accessible, you can still generate a score or preference multiple times and take an average of multiple runs.\n \nReference Links: \\\n[1] https://arxiv.org/pdf/2306.05685 \\\n[2] https://arxiv.org/pdf/2311.09528", "questions_to_authors": "- What is the distribution of ratings on some of the datasets? If the evaluator keeps on predicting very similar scores (e.g., 5) for each response, the plausibility of flipping in absolute rating is lower due to the sheer low quality of the rating method.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper compares LLM response evaluation setups that use pairwise or absolute scoring and demonstrate using synthetic distractors that in LLM as a judge evaluations with Llama3.2-8B and 70B and Qwen-2.5-3B and 70B, pairwise evaluations are more affected by distractor features that may make a less task optimal output appear preferred (eg: assertivess as a distractor when the goal is accuracy).
The concerns raised by reviewers mainly center around the fact that while the claims are supported only by LLM as a judge evaluations with a specific set of models which are perhaps not the most commonly used models for this purpose, the language in the paper may seem to indicate that the claim is also meant for human ratings. However, the paper neither examines the impact of distractor features on human annotation, nor the impact of "incorrect" LLM annotators on a downstream training phase such as RLHF.
If accepted, the authors are requested to modify the language in the paper to make only claims that are supported by the experiments available.

[comment from the PCs] It's critical that you fix the issue with regard to claims appearing to apply to human annotators as well. Of course, if your study resulted in evidence with regard to human annotators, please refine the discussion to make this clear and substantiate this with empirical evidence. 
Decision: Accept