AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset
Authors: Bingxiang He
Wenbin Zhang
Jiaxi Song
Cheng Qian
Zixuan Fu
Bowen Sun
Ning Ding
Haiwen Hong
Longtao Huang
Hui Xue
Ganqu Cui
Wanxiang Che
Zhiyuan Liu
Maosong Sun
Venue: COLM 2025 (2025)
Keywords: Large Language Models
Direct Preference Optimization
Preference Learning
TL;DR: We propose \textbf{AIR}, a framework to systematically dissect preference datasets into three core components—\textbf{A}nnotations, \textbf{I}nstructions, and \textbf{R}esponse Pairs—and quantify their alignment impact.

Abstract: Preference learning is critical for aligning large language models (LLMs) with human values, yet its success hinges on high-quality datasets comprising three core components: Preference \textbf{A}nnotations, \textbf{I}nstructions, and \textbf{R}esponse Pairs. Current approaches conflate these components, obscuring their individual impacts and hindering systematic optimization. In this work, we propose \textbf{AIR}, a component-wise analysis framework that systematically isolates and optimizes each component while evaluating their synergistic effects. Through rigorous experimentation, AIR reveals actionable principles: annotation simplicity (point-wise generative scoring), instruction inference stability (variance-based filtering across LLMs), and response pair quality (moderate margins + high absolute scores). When combined, these principles yield +5.3 average gains over baseline method, even with only 14k high-quality pairs. Our work shifts preference dataset design from ad hoc scaling to component-aware optimization, offering a blueprint for efficient, reproducible alignment.

Published as a conference paper at COLM 2025
AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset
Bingxiang Hel Wenbin Zhang? Jiaxi Song", Cheng Qian', Zixuan Ful, Bowen Sun! Ning Ding', Haiwen Hong", Longtao Huang", Hui Xue Ganqu Cui Wanxiang Che 2t Zhiyuan Liu', Maosong Sun!
"Tsinghua University, Harrbi Institute of Technology, Shanghai Al Lab +University of Illinois Ürbana-Champajjn
$ Allibaaa Group
hebx24emails. tsinghua, edu. en, cuiganquepjlab. .rr.guoopplabbo............ org.cn (wenbinzhang, @iir www.r.uiituee.crrr...mm
Abstract
Preference learning is critical for aligning large language models (LLMs) with human values, yet its success hinges on high-quality datasets com- prising three core components: Preference Annotations, Instructions, and Response Pairs. Current approaches conflate these components, obscur- ing their individual impacts and hindering systematic optimization. In this work, we propose AIR, component- -isie analysis framework that systematically isolates and optimizes each component while evaluating their synergistic effects. Through rigorous experimentation, AIR reveals actionable principles: annotation simplicity (point-wise generative scoring), instruction inference stability satriance-based filtering across LLMs), and response pair quality (moderate margins + high absolute scores). When combined, these principles yield +5.3 average gains over baseline method, even with only 14k high-quality pairs. Our work shifts preference dataset design from ad hoc scaling to component-aware optimization, offering a blueprint for efficient, reproducible alignment.
1 Introduction
The alignment of large language models (LLMs) with human preferences has become cornerstone of modern LLM development, achieved through methods like reinforcement learning from human feedback (RLHF; Christiano et al., 2017; Ziegler et al., 2019) and direct preference optimization (DPO; Rafailov al., 2023). While these techniques differ in relies on reward modeling and policy optimization (e.g., PPO; Schulman et al., 2017), while DPO bypasses explicit reward learning-ther share a critical dependency: high-quality preference datasets. Whether aligning models for correctness in mathematical reasoning (Shao et al., 2024) or real- usability in instruction following (Dubois et al., 2024; Lin et al., 2024), the success in powering state-of-the-art models (Dubey et al., 2024; Achiam et al., 2023) has spurred efforts to scale preference dataset construction (Cui et al., 2023; Wang et al., 2024; Xu et al., 2024), yet the principles governing dataset quality remain poorly understood, creating critical bottleneck in alignment research.
their mechanisms-RLin
Preference datasets are built on three core components: Annotations, Instructions, and Response Pairs. The construction of preference dataset begins with defining instructions (input prompts that define tasks), which may be synthetic (Xu al., 2024) or sourced from real human interactions (Zhao et al., 2024). Next, responses are generated- either from models being aligned (on-policy) (Dubey et al., 2024) or external models (off-policy) (Cui et al., 2023). Finally, annotations are collected via human judgments (Bai et al., 2022), AI- based scoring (Cui et al., 2023), or hybrid methods. Some works use pairwise comparisons
"Equal contribtuion
I *Corresponding Author.

---

Published as a conference paper at COLM 2025
Moderate Margin
55
Generative RM
Hybrid Mixing
wwopppicc Laft wody
6/8
Aneatcher
Point- wise Scriig fore spone plation
Avg. Perf. on all Benchmarke
50
Higher Mixing
Hybrid
Score
Var:
beror Mosera Speasmn
Margin
Model "ypee
Point-
Moderate Margin
Score
grain
Arnot ertinns
smartay
Besión
Pretegt
45-
Vanilla
Pref.
Scoting
Wise
45.60
Re seggonnt
Data
36.96
44.61
Preference Learning
dnrsdy
Highhs Score 28
42.36
ail
40
10.37
Instructions
Broadard
Salection
Nambbar
Torns
Resporde Variance
Score Varionca
2g0
35
Bosed Selection
Multi- Turn
A
Score Var.
30
1.5
Initial LLM
AIR Analy Framework
Figure 1: Left: The AIR framework dissects preference learning into three core components: Annotations, Instructions, and Response Pairs, with the outermost layer highlighting empirically validated optimal design principles. Right: Additive improvement across all benchmarks, resulted by stepwise integrating principles for better annotations, response pairs and instructions with 14k preference pairs.
(Dubey et al., 2024), while others first assign scalar scores via classffier-based (Xu et al., 2024) or generative reward models (RM) (Cui et al., 2023), then construct pairs from score gaps.
However, prior works tend to conflate the impact of these components rather than analyzing them systematically (Yasunaga et al., 2024; Amini al., 2024). For example, improvements attributed to "better annotations" might instead stem from unmeasured changes in instruc- tion diversity or response quality (Gu al., 2024). This opacity limits reproducibility and hinders targeted optimization. This conflation obscures fundamental question: How annotations, instructions, and response pairs independently shape dataset efficacy? Without isolating these components, progress in dataset design remains ad hoc, prioritizing scalability over interpretable principles.
To address this gap, propose AIR (Annotations, Instructions, and Response Pairs), a component-wise analysis framework to systematically dissect preference dataset con- struction, shown in Figure 1 (left). By independently varying each component while controlling others (e.g., fixing instructions and responses to isolate annotation effects), we quantify how design choices in one component propagate to downstream alignment perfor- mance. Our goal is to identify generalizable principles for preference dataset design and provide actionable insights for practitioners balancing quality, cost, and scalability.
We conduct controlled experiments to isolate the impact of dataset components using open-source models. Starting with strong LLM, we generate responses from a diverse pool of LLMs, following a modified UltraFeedback pipeline (Cui et al., 2023). Instructions are sourced from different public datasets, with preferences scored (0-9) via LLMs using multiple annotation strategies. To focus on dataset design, we apply vanilla DPO (Rafailov et al., 2023), avoiding algorithmic confounders. Aligned models are evaluated across commonly used benchmarks spanning instruction following, reasoning, coding, math, and real-world usability, ensuring robust evaluation alignment-critical capabilities.
Our systematic analysis yields three empirically validated principles:
Simplify annotation with generative scoring: Replace pairwise comparisons or fine- grained guidelines with generative reward models using point-wise evaluatinn single responses via greedy decoding. This streamlined approach outperforms classifier- based RMs and complex multi-sample aggregation, prioritizing holistic quality over checklist compliance.
Prioritize low-variance instruction selection: Select instructions with low variance in scores across responses from different LLMs-a proxy for fine-grained preference distinctions outtnttions-oupprfooonn prior LLM-based methods. Additionally, incorporating multi- turn context offers marginal benefits (MT-Bench Turn 2) but no broad improvement.
2

---

Published as a conference paper at COLM 2025
Optimize response pairs for quality and diversity: Curate pairs with moderate score margins (A = 2 or 3) to balance preference clarity and dataset coverage, high absolute scores (2 8) to ensure both responses are high-quality, and hybrid mix of on-policy (base model) and off-policy (external model) responses. This triad maximizes contrastive learning while avoiding noisy trivial comparisons.
When stepwise integrating our empirically validated principles principles-generativye generative streamlined an- notation (+1.78), variance-based instruction selection (+1.07), and optimized response pairs (+2.46)-we observe a cumulative +5.31 average gain over baseline (w. vanilla preference data) across all benchmarks under controlled variable experiments, as shown Figure (right). This additive improvement confirms that each AIR component independently con- tributes alignment performance while synergizing for compounded gains, validating our framework's systematic design.
2 Related Work
Preference Learning and Alignment. Post-training alignment via preference learning has become pivotal for refining LLMs. Early work (Christiano et al., 2017; Ziegler et al., 2019) established RLHF as a framework for training reward models and optimizing policies with algorithms like PPO (Schulman et al., 2017). Subsequent innovations simplified the pipeline, notably DPO (Rafailov et al., 2023), which bypasses explicit reward modeling. These methods have driven progress in summarization (Stiennon et al., 2020), instruction following (Ouyang et al., 2022), and frontier LLM development (Team et al., 2023; Achiam et al., 2023; Dubey et al., 2024), but all hinge on the quality of preference datasets persistent bottleneck.
Components of Preference Datasets and Design Insights. Preference datasets are shaped by three core components: Annotations, Instructions, and Response Pairs. Early datasets like Anthropic HH (Bai et al., 2022) relied on human labor for all components, limiting scalability. Recent work automates these elements: UltraFeedback (Cui et al., 2023) generates responses from LLM pools and annotates preferences with GPT-4, while Magpie-DPO (Xu et al., 2024) leverages the chat template to sample on-policy responses and annotates them via classifier- based RMs. Recent efforts have begun dissecting these components. For anno- tation, Yasunaga et al. (2024) propose probability-base scoring, and Lambert et al. (2024a) observe minimal performance variation across generative LLM-as-a-Judde annotators. For response pair construction, Yu et al. (2025) emphasize long rejected responses, Amini et al. (2024) weight pairs by score margins, while Khaki et al. (2024); Wu al. (2024) leverage reward distributions. For instruction selection, Kim et al. (2024); Lu al. (2023) identifies the importance of prompt difficulty filtering, He et al. (2025) highlights the role of data similarity. Broader analyses by Ivison al. (2024) link dataset quality, algorithmss, and reward models to downstream performance. However, these works address components in isolation or conflate dataset design with training algorithms, offering fragmented insights. Our work bridges this gap by proposing component-wise analysis framework to disentangle and optimize each component, identifying actionable principles for high-quality preference datasets construction.
3 Preliminary
3.1 Preference Learning
Preference learning aligns language models with human preferences by leveraging pairwise comparisons, most commonly operationalized through the Bradley-Terry (BT) framework (Bradley & Terry, 1952). The BT framework models the probability that response yw is preferred over yi given instruction as:
p(y > yılx) (x, -r" r' (x,x)))),
(1)
3

---

Published as a conference paper at COLM 2025
where is the logistic function, and r* (x represents the reward score of response y.
Modern alignment methods build on this framework in two key paradigms: (1) RLHF: Trains reward model (x y) to approximate r" (x(, then optimizes policies via RL algorithms like PPO (Schulman et al., 2017); (2) RL-free: Bypasses explicit reward modeling by reparameterizing the BT model using the policy 7g and reference model Te like DPO (Rafailov et al., 2023):
T (ylx)
yx = -BBlly Bllgg
(2)
zel (y|x)*
yielding the DPO objective:
Tt To(ywlx) Tee Yy x)
B log To (y(l)))) The (y:
DPO E (xxy log (Blog
(3)
3.2 AIR Framework
Both paradigms in Section 3.1 assume access to high-quality preference datasets (f)
{(((1, yi ,y yj m However, existing preference learning paradigms treat the dataset as monolithic, conflating three critical components:
Annotations: How to assign scores (i) to (x()),(()) pairs. We analyze the effect of annotator model types, annotation strategies for prompt design, and scoring methods.
Instructions: How select high-quality x(). We investigate the score variance and number of context turns compared to prior LLM-based methods.
Response Pairs: How to construct (yw, yı) from scored triples. We study relative score margins, absolute score thresholds, and on/ ooff-policy mixing.
Our AIR Framework (Annotations, Instructions, Response Pairs) systematically isolates each component's impact on alignment performance, shown in Figure 1 (Left), enabling: (1) Diagnosis: Identify whether dataset weaknesses stem from noisy annotations, poor instructions, or uninformative responses; (2) Optimization: Prioritize improvements to the most impactful component. While validated in LLM alignment, AIR generalizes any task requiring preference data-from multimodal learning to human-Al collaboration.
Experiments and Results
We systematically analyze the impact of preference dataset components using our AIR framework. Following the experimental setup (Section 4.1), we conduct controlled experi- ments isolating each component: Annotations (Section 4.2), Instructions (Section 4.3), and Response Pairs (Section 4.4). We then evaluate their combined impact (Section 4.5) and validate robustness via cross-verification with alternative models and datasets (Section 4.6).
4.1 Experimental Setup
We conduct controlled experiments using open-source models isolate the impact of dataset components following a modified UltraFeedback pipeline (Cui et al., 2023).
Base Model and Response Generation. We start with Llama-3.--Tuluu-SS-SST- (Lambert et al., 2024a), strong instruction-tuned model, and generate responses using a diverse pool of 17 open-source LLMss (model list refers to Appendix A.2). These include on-policy responses (from the base model) and off-policy responses from models like Llama, Mixtral, and Qwen2 family.
Instruction Sources and Annotation. Instructions are sourced from GPT--4-generated ShareGPT and UltraFeedback, detailed in Appendix A.1.1. We annotate response quality (0-9 scores) using Llama-3.1770B-Instruus (Dubey et al., 2024) or Dwen225-72B-IInstrus (Qwen et al., 2025), sampling a subset of responses including 4 on-policy responses using high temperature and 4 off-policy responses per instruction to balance cost and diversity.
4

---

Published as a conference paper at COLM 2025
53
Téle 3 (SFT)
Tulo (SPT)
Single-Aug
Rised
Single-Prob
Generative RM
Single Dasic
Score
31. 24
243
25-
30
15-
MT-
Wikl
Eurus
Live
Arena
Avenage
MT-
Wild
Eurus
Live Bench
Arena
Average
Hard
Bench
Suite
Figure 2: Performance across different benchmarks, comparing different annotator model types (left) and different scoring methods (right). The average score shows the effectiveness of generative RM and Single-Basie Basic greedy decoding. Scores normalized to 100-point scale for averaging.
Training Protocol. For all experiments, we sample total number of 30k preference pairs and apply vanilla DPO (Rafailov et al., 2023) for epoch to isolate dataset effects, avoiding complex RL algorithms, More hyperparameters are in Appendix B.1.
Evaluation. We evaluate aligned models across 6 benchmarks: MT-Bench (Zheng et al., 2023b), ArenaHard (Li et al., 2024b), AlpacaEval 2.0 (Dubois et al., 2024), WildBench (V2) (Lin et al., 2024), Eurus evaluation suite (Yuan et al., 2024), and LiveBench (White et al., 2024), cov- ering various domain among coding, math, reasoning, knowledge, Instruction-following and chat benchmarking. This ensures robustness across alignment-criticc capabilities. Benchmark details and evaluation metrics are provided Appendix B.2.
Final Dataset Statistics. We also provide detailed dataset statistics, including distributions of response length and annotation score across LLMs, detailed in Appendix A.4.
4.2 Preference Annotation
We analyze three critical dimensions of preference annotation: (1) annotator model types, (2) annotation strategies for prompt design, and (3) scoring methods. Our experiments reveal that:
Takeaway 1: A generative RM with point-wise scoring (evaluating single responses directly, avoiding pairwise com comparisons or complex guidelines) and greedy decoding achieves superior alignment performance, prioritizing simplicity and efficiency over intricate aggregation methods.
4.2.1 Annotator Model Type: Classifier-Based RM us. Generative RM
We compare a state-of-the-art! classifier-based RM (Skywork-Reward-Geene --rar-----aa------a-00 RewardBench: 94.3) with generative RM Laama-331-70B-Instruuee RewardBench: 84.0). Both models score 4 responses per ShareGPT instruction using the point-wise scoring prompt template in Appendix A.3.2. Finally we construct preference pairs for vanilla DPÒ training.
a generative RM achieves +1.4 higher alignment performance on average across benchmarks, as shown in Figure (left), outperforming the SOTA classifier-bases RM on all benchmarks. This suggests that RewardBench may undervalue generative RMs' generalization, and next token prediction (generative RM) better captures holistic response quality.
Results. Despite lower RewardBench score, Slama-31-70B-Instruue
4.2.2 Annotation Strategies: Prompt Design for Response Scoring
Compared classifier-based RM, generative RM flexible when annotating the scores, as can dynamically adjust its annotation prompt. Here we evaluate annotation strategies for
Which is SOTA RewardBench of experimental timeframe.

---

Published as a conference paper at COLM 2025
Annotation Strategy
MT-Bench
WildBench
Eurus
LiveBench
ArenaHard
Avg.
Titlu-3-BB- 3-B---3-BBSFTTTT (Baseline)
6.38
12.50
55.77
24.3
13.8
34.03
Single-Basic
7.20
26.80
58.27
26.4
33.8
43.45
Pair-Basic
6.73
25.49
57.86
24.8
31.9
41.47
Pair-Guided
6.79
25.68
57.61
23,5
31.8
41.30
Pair-Explained
6.88
25,55
57.90
23.8
32.7
41.75
Pair-Guided-Explain
6.78
25,90
57.64
22.9
31.9
41.23
Pair-Guided-Explaanee-TO 6.60
24.56
57.68
23.8
29.6
40.33
Table 1: Impact of 6 annotation strategies on DPO performance, Blue indicates improve- ments over the baseline, and orange degradations; color depth reflects the approximate relative magnitude of performance change. All strategies ensure identical instruction sets and pair counts used for DPO training. Scores are normalized to 100-point scale for averaging. FG denotes Fine-Grained.
generative RM, varying the response count (point-wise pairwise scoring), guideline com- plexity (none vs. coarse vs. fine-grained), and explanation requirements (presence/ absence of reasoning). Details about each strategy and prompt template can refer to Appendix A.3.2,
Setup. Using Lllam-3.1--2 3.1--..1--7--me Inn-rustt we score 4 responses per ShareGPT instruction across all strategies, mitigating position bias via reversed-order averaging (Zheng et al., 2023a). For fine-grained guideline setting, task-specific guidelines are generated by LLMs. Annotation strategies range from minimal (Single-Bar Basic: score one response) to complex (Pair-Guided- Aeplained-Fine-Graiiee
pairwise scoring with fine-grained guidelines and explanations). Finally we construct preference pairs for vanilla DPO training.
Results. The Single-Basic annotation strategy outperforms all alternatives, achieving a top average score of 43.45 (Table 1), surpassing Pair-Basic and the most complex fine-grained guideline approaches by +1.98 and +3.12, respectively. Key insights:
Pairwise comparisons lag behind, likely due to context overload from dual-response evaluation.
Fine-grained guidelines harm performance (10.9), indicating the artifacts in the checklist limit the judge range.
Explanations/ coarse guidelines offer negligible gains ( 0.28) over simpler baselines.
These results suggest that minimalist prompt, leveraging annotators' innate judgment, better align with real-world preference signals than complex protocols. The trend toward intri- cate annotation workflows may inadvertently introduce noise, advocating for streamlined designs.
4.2.3 Scoring Methods: Aggregation Techniques for Reliable Annotations
While recent work proposes aggregation methods like multi-sample averaging (Yu et al., 2025) or probability-based weighting (Yasunaga al., 2024) improve annotation reliability, we find simple greedy decoding suffices, which is the most robust and well-performed method across benchmarks.
Setup. We compare three scoring methods using point-wise scoring: (1) Single-Basic: Greedy decoding (one run, temperature=0); (2) Single-Avg: Average scores across runs (temperature=1.0). (3) Single-Prob: Probability scores across token "0" to "9". Detail calculation can refer to Appendix A.3.3.
Results. Single-Basic achieves the best alignment performance, shown in Figure 2 (right). While being the simpliest, it outperforms Single-Avg by +1.9 and Single-Prob by +1.4 on average. This may be because greedy decoding's robustness stems from its consistency- avoids noise introduced by stochastic sampling (Single-Avgl or tokenization artifacts (Single-Prob), reflecting the model's deterministic quality assessment.

---

Published as a conference paper at COLM 2025
Tulu (SFT)
MT-Bench Tum2
MT-Bench Tumn
Low-Varimnee
Tulu (SFT)
Mić Variance
MT- -TT-Bench Tuml
Single-Tum
MT-Bench Tuml
High-Varianee
InsTag Quality Filtering
Multi-Tul
ent
WildB ench
248
26R
1F2
50.6
veBench
150
55.6
veBench
2010
360
56.6
AlpacaEval 2
AlpacaEval's
Eurus Suite
Eurus Suite
ArenaHard
ArenaHard
Figure 3: Performance across different benchmarks, comparing different response score variances with top quality tagged by InsTag method (left) and number of context turns (right). We prioritize instructions with low score variance, while multi-turn context offers marginal gains only on MT-Bench Turn 2 but no broad improvement.
4.3 Instruction Selection
Instruction quality is critical for preference learning, yet existing methods prioritize human- curated artifacts (Zhou et al., 2023; Lu et al., 2023), neglecting the preference signal. To better leverage it, inspired by (Li et al., 2024a), which elicit fine-grained distinctions (i.e., small, consistent differences in response quality across LLMs) provide richer signals for alignment. To test this, we propose variance-based instruction selection, prioritizing instructions with low score variance across diverse LLMs. Our experiments demonstrate that:
hypothesize that inference-stable instructions
Takeaway 2: Low-variance instructions outperform high-variancd ones across all benchmarks, with variance-based filtering surpassing prior LLM-based methods; multi-turn context provides only marginal MT-Bench Turn 2 gains, prioritizing instruction quality over conversational complexity.
4.3.1 Variance-Based Instruction Selection: Fine-Grained Distinctions Matter
To prioritize instructions that expose nuanced distinctions between responses, we propose filtering by score variance. For each instruction x;, we sample N responses from diverse LLMs, annotate their scores [ and compute the variance v; Low-variance instructions are prioritized as they indicate fine-grained distinctions critical for alignment.
We sample N responses per instruction, annotate scores via Single-Bac Basic strategy, and calculate variances, detailed distribution can refer to Appendix A.4. We compare three settings: (1) Low: o; < 1.5; (2) Mid: 1.5<v; 3; (3) High: 0;>3.
Setup.
Results. Low-variance instructions setting achieves the best performance compared to higher variance ones, as shown in Figure 3 (left), excelling in AlpacaEval (+3.7) and ArenaHard (+4.6). This indicates that low-variance instructions force models to learn subtle, policy-aligned preferences rather than relying on obvious errors or lacking of specificity (higher variance pairs).
4.3.2 Variance Quality Tags: Preference Signals Outperform Standalone Judgments
While LLM-based method like InsTaag (Lu et al., 2023) filters instructions via LLM-as-a- Quality-Judge, demonstrate that dreieeence-driven variance filtering better aligns with preference learning.
Setup. We compare: (1) Variance-Based:: Low-variance instructions (v; 1.5) from Sec- tion 4.3.1; (2) Ing: Top-quality instructions (score=5) jointly tagged by Llama-3.1 Instruct and Soven25--2B-Innstrue (see Appendix A.1.2) while controlling other settings.
7

---

Published as a conference paper at COLM 2025
WildBench
LiveBeneh
ArenaHard
AlpacaEval
Average
40
35
Seare
30
25
20
Low
Moderate Margin
High
Low-Score Response
Mid-Score Response
High-Score Response
Pure-
Low-
Mid-
Pure- On-Policy
Margin
Margin
Off-Policy
Mix
Figure 4: Performance across different benchmarks, comparing different relative score margins (left), absolute score thresholds (middle), and on on/off-poiccy mixing strategies (right). We prioritize pairs with moderate score margin and higher absolute score, while ensuring hybrid mixing of on/ on/off-policy responses.
Results. Our variance-based instruction selection method achieves +2.2 average improve- ment over instructions tagged as high-quality by InsTag, as shown in Figure 3 (left). This holds even when InsTag uses two annotators for comprehensive evaluation. This indicating that leverage the lateral preference signals outperform standalone judgements.
4.3.3 Multi-Turn us. Single-Turn Context Analysis
Having identified low-variance instructions as critical for alignment performance, we ex- tend our analysis instruction structure- sppectrre-seccficallly.y whether multi-turn conversational contexts improve preference learning. While variance-based filtering prioritizes task speci- ficity, multi-turn instructions may enhance models' ability handle dialog-specific nuances. To test this, we split single-turn and multi-turn contexts instructions, constructing the preference pairs the same way as before.
Results. Multi-turn context instructions yield marginal gains on MT-Bench Turn 2 (+0.7), with no improvement on single-turn benchmarks, as shown in Figure (right). This suggests that multi-turn instructions slightly enhances chat-specific capabilities, while current benchmarks underemphasize multi-turn alignment, implying that while multi- turn instructions show niche utility, their value hinges on future benchmarks prioritizing conversational depth over single-turn tasks.
4.4 Response Pair Construction
Effective response pair construction must balance three competing objectives: (1) Signal Clarity (contrasts should be unambiguous), (2) Response Quality (both responses must be competent to avoid trivial comparisons), and (3) Policy Alignment (mix on /ffff-pplyyy responses to avoid distribution shift). Prior work often optimizes one dimension at the expense of others-we systematicalll address all three through:
Takeaway 3: Optimal pairs combine moderate margins (A = 2 or 3) for clear
contrasts, high absolute scores (s(x, yy) 2 8) for quality, and hybrid mixing (one on-policy one off-policy response) for for effective preference modeling.
balanced approach
4.4.1 Relative Score Margin: Balancing Noise and Signal
While DPO in Equation (3) treats all preference pairs equally, we find moderate score
margin (A = 2 or 3) optimizes alignment performance.
Setup. Using Llama-3.1-70-IInstruee and Single-Basic annotation, construct pairs (ywr vi) with low (A 1), moderate (A = 2 or 3), and high (A 2 4) margins, balancing dataset size across groups. In order to keep enough preference pairs under different settings, we choose split thresholds based on score margin distribution in Appendix A.4.
Results. Moderate margin achieves higher (+1.29/+5.42) average performance than low / margins, as shown in Figure 4 (left), excelling in instruction following (AlpacaEval 2: +3.35/ +7.42) and reasoning (ArenaHard: +0.2/+11.8 compared to high margin setting.
8

---

Published as a conference paper at COLM 2025
This is because moderate margin provides clear preference signals without oversimplifying the learning objective, avoiding noise (low A) or overfitting (high A).
4.4.2 Absolute Score Threshold: Prioritizing High-Quality Responses
Given that the moderate relative score margin matters, here we want to investigate the effect of absolute score thresholds. We find that preference pairs with higher absolute scores (e.g., 9 vs. 7) outperform pairs with identical margins but lower scores (e.g., 3 vs. 1), emphasizing the importance of overall response quality.
Setup. We construct preference pairs (yv, y1) for instruction based on the absolute score of chosen response s (x, Vuu) with controlled relative score margin distribution. We defined three settings: (1) High: sxx, Vio) 2 8; (2) Mid: s(x, = 7; (3) Low: s(x, yw) S We choose the threshold based on our observation of the score distribution, detailed in Appendix A.4.
Results. High-score pairs achieve the most significant performance across all benchmarks, leading a +9.35 compared to low-score pairs overall, as shown in Figure 4 (middle), with the largest gains in instruction following (AlpacaEval 2: +9.22) and reasoning AArenaHarda +17.1). This indicates that while DPO optimizes relative preferences, high-score pairs provide clearer learning signals, as both responses are competent but distinguishable. Low-score pairs risk amplifying noise from inherently poor responses.
4.4.3 OnlOff-Policy Mixing: Diversity for Effective Contrast
There is line of research validating the effectiveness of on-policy samples in preference learning (Dubey et al., 2024; Lambert et al., 2024a). However, how to best mix the on/off off policy samples remain uncertain. Here we investigate the effect of different mix strategies.
and off-polic responses (external models). We compare 4 mixing strategies based on the mix rate of on-policy samples: (1) Pure-Off: Pairs from off-policy only (totally C3 6 pairs); (2) Pairs from 5 responses including 1 on-policy and 4 off-policy (totally C2 = 10 pairs); (3) Mid-Mix: on-policy paired with every off-policy (totally 4 pairs); (4) Pure-On: Pairs from on-policy only (totally G 6 pairs). All strategies sample 4 pairs for each instruction with identical margin distribution.
Setup. For each instruction, we annotate on-policy (Llama-3.1-Tulu-3--33
Low-Mix:
Results. Mid-Mix strategy yields the best result across all benchmarks compared to Pure-Off (+1.38) and Pure-On (+2.56) (Figure 4, right). This indicates that a moderate mix rate of on/ on/off /n/off-policy responses ensures every DPO update contrasts the current policy with external responses and avoids overfitting to static datasets while maintaining policy relevance.
4.5 Combined Impact
To quantify the synergistic impact of our findings, we incrementally integrate optimal components into a baseline DPO pipeline, measuring performance gains stepwise. Starting from Tulu 3 (SFT), we build progressively while maintaining identical hyperparameters and dataset size (14k pairs):
Baseline: DPO with vanilla preference data using Pair-Basic annotation (wo.
Bargin/qualit control).
Annottation:: Change to Single-Basic strategy.
+Response Pairs: Moderate margins and high absolute scores, with Mid-Mix on/ off-f policy pairs.
+ Instructionns: Leverage low-variance filtering for instruction selection.
Cumulative integration yields +5.3 average performance gain steadily, as shown in Figure 1 (right). The change into Single-Basin strategy and higher absolute response scores showing the significant improvements (+1.78, +1.6); detailed results for each benchmark can refer to Appendix C.2. Since our experiment only leverages 14k pairs, we believe the combined impact will continue to scale. The size of preference datasets in previous state-of-the-ars models (Dubey et al., 2024; Lambert et al., 2024a) is far larger than this.
Results.

---

Published as a conference paper at COLM 2025
4.6 Cross- Pross-Verffication
To validate the robustness of our findings, replicate key experiments using the instruc- tions from UltraFeedback dataset and Qwen2.5-72B-Inttree as annotator. Results remain consistent across both configurations (see Appendix C.3), confirming the generalizability of our AIR framework's principles under varying instruction sources and annotator capabili- ties. This consistency underscores that our insights reflect intrinsic dataset design dynamics, not implementation-speiiii artifacts.
5 Conclusion
In this work, we propose AIR, a framework to systematically dissect preference datasets into three core Iomponenss-Anntaaiiii their alignment impact. Through systematic experiments, we establish that streamlined annotation with generative scoring, low-variance instruction selection, and response pairs balancing score margins and absolute quality collectively elevate alignment performance. These principles shift dataset design from ad hoc scaling to component-awart optimization, enabling practitioners build reproducible, high-quality datasets. Future work includes extending AIR to larger-scale datasets and diverse algorithms with dynamic component interactions. By bridging dataset design and alignment outcomes, our framework advances preference learning through interpretable, data-centric innovation.
Instructions, and Response Pairs-and quantify
Acknowledgments
This work was supported by Being Advanced Innovation Center for Future Blockchain and Privacy Computing, Alibaba Group through Alibaba Innovative Research Program and Shanghai Artificial Intelligence Laboratory.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arviv preprint ar xrXiv:2003.08774 2023.
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. arXiv preprint arXiv:2402.10571 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. aiv preprint axXiv:2204.05882, 2022.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(344)3344-45, 1952. ISSN 00063444, 14643510. URL http:///ww www.:ssstotoostaeee.rr. orgg/stabee/2340222
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. arXiv preprint ar Xio:2310.00377, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint ar Xiv:24007.22 2778888 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: simple way to debias automatic evaluators. Xiviv preprint ar axXi::2404.0447,, 2024.
10

---

Published as a conference paper at COLM 2025
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. survey on alu--s-a-judge. arXio preprint ar Xix:2411.115994 2024.
Bingxiang He, Ning Ding, Cheng Qian, Jia Deng, Ganqu Cui, Lifan Yuan, Haiwen Hong, Huan-ang Gao, Longtao Huang, Hui Xue, Huimin Chen, Zhiyuan Liu, and Maosong Sun. The right time matters: Data arrangement affects zero-shot generalization in instruction tuning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Compulational Linguistics: ACL 2025, pp. 222-243 Vienna, Austria, July 2025. Association for Computational Linguistics. TSBN 979-8-89176-256-5. doi: 1118653//1/2225.... findings--acl.13, URL https://aclanthologg org/2025. findings-acl. 13/.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Open- rlhf: An easy-to-use, scalable and high-performance rlhf framework. aiviv preprint ar Xit:2405.11143 2024.
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentan- gling best practices for learning from preference feedback. arXiv preprint ar arXiv:2406.09279, 2024.
Saeed Khaki, Jinjin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. Xiv preprint ar Xix::2402..10338 2024.
Joongwon Kim, Anirudh Goyal, Aston Zhang, Bo Xiong, Rui Hou, Melanie Kambadur, Dhruv Mahajan, Hannaneh Hajishirzi, and Liang Tan. A systematic examination of pref- erence learning through the lens of crrruction-follwiing arXiv preprint ar Xixv:24122.1528,, 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjingg Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. 2024a.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khy- athi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: 2024b.
Evaluating reward models for language modeling. arXiv preprint ar
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and jing Xiao. From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7602-7635, Mexico City, Mexico, June 2024a. Association for Computational Linguistics. doi: (0.18653/vvl htt.:/222.aaacllloooo 421. URL httt::://aclaattholloooooo org/ 2024. www.caaalll.o2202220 421/.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena- hard and benchbuilder pipeline. arXiv preprint ar arx::22406.1195 2024b.
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmark- ing lims with challenging tasks from real users in the wild. ar preprint xrXiv:2406.04770) 2024.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and jingren Zhou. # instag: Instruction tagging for analyzing supervised fine-tuning of large language models. arxiv preprint arXit:2308.07074 2023.
11

---

Published as a conference paper at COLM 2025
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
C Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2777 277734 2022.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 5wen2.5 technical report, 2025. URL
2412. 15115.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728 287441,, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint ar XxXo:1707.06344 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint ar arXiv:2402.03300 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. Xiv preprint arXit:2312.11880 2023.
Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong- Helpsteer2-prefeeence Complementing ratings with preferences. Xrv preprint ar axXio:2410.01257, 2024.
Colin Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Jain,ain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination-free Ilm benchmark. arviv preprint ar XrXiv:2406.19311 2024.
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. f-dpo: Direct preference optimization with dynamic B. Advances in Neural Information Processing Systems, 37:129944-129966, 2024.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned Ilms with nothing. ar arXiv preprint ar Xiv:::2406. 206.04066, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https: / http://axxvv..orro wttp/bs/////00 abs/ 10671.
Michihiro Yasunaga, Leonid Shamis, Chunting Zhou, Andrew Cohen, Jason Weston, Luke Zettlemoyer, and Marjan Ghazvininejad. Alma: Alignment with minimal annotation. ar Xiv preprint xxXt::2112.04300 2024.
12

---

Published as a conference paper at COLM 2025
Ping Yu, Weizhe Yuan, Olga Golovneva, Tianhao Wu, Sainbayar Sukhbaatar, Jason We- ston, and Jing Xu. Rip: Better models by survival of the fittest prompts. ar preprint mmXiv:2501.18578 2025.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing Ilm reasoning generalists with preference trees. ar Xiu preprint ar Xrio:2404.02073 2024.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wild- chat: Im chatypt interaction logs in the wild. ar preprint ar wii::2405.0147777 2024.
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. arviv preprint ar arXii:2330903888 2023a.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595- 46623, 2023b.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan lyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, (6555006-50021, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. Xiv preprint Xit:1909.00853 2019.
A Dataset Details
A.1 Instructions
A.1.1 Instruction Datasets
We utilized two datasets: GPT-4-generated ShareGPT and UltraFeedback? Here, we provide detailed overview of each dataset.
ShareGPT. It comprises a multilingual corpus of multi-turn conversational exchanges curated from high-quality GPT-4 interactions. It contains 103,415 entries across three subsets: 6,206 refined GPT-4 dialogues focusing on knowledge-intenss programming, and multilingual translation), 58,674 entries reformatted from ShareGPT-V3S with informal multilingual dialogues, and 38,535 Chinese-specific exchanges addressngg technical queries and creative tasks. The dataset features conversations in Chinese (both simplified and traditional), English, Japanese, and Korean, with each entry structured as sequential human-AT interaction pairs. In this paper, we use its subset with 58,674 entries reformatted from ShareGPT-V3.
tasks (e-g., reasoning,
UltraFeedback. The UltraFeedback dataset represents large-scale preference collection framework designed for training reward models and critical evaluators. It aggregates 63,967 prompts from six heterogeneous sources including UltraChat, ShareGPT, and domain- specific QA datasets, generating 256k responses through diverse LLMs ranging from com- mercial systems (GPT-44, GPT-3.5 Turbo) to open-source models (LLaMA variants, Falcon, MPT). Each prompt elicits four responses annotated by GPT-4 across four dimensions: instruction adherence, truthfulness, honesty, and helpfulness. The dataset's design em- phasizes diversity through stratified sampling of instructions and heterogeneous model selection, coupled with fine-grained numerical and textual feedback annotations. In this paper, we all its instructions.
ShareGPT4 conversations were filtered by a 4096-token maximum length constraint and concatenated across dialogue turns, yielding 33,717 refined instructions. UltraFeeddbakk underwent a basic 4096-token length filtration to ensure response consistency.
Inttps://luggingface http:///gggggaaeeeooo/////////// co/ddtaaeets/hhbiing2//// sharegpt gpt4
2nttps://huggingffan
http:///wwgggggggee..//./aa///// wttp///aattee//eeeeeeeeeeee traFeedback
13

---

Published as a conference paper at COLM 2025
The Proportion of Instructions with different Scores
0.35
0.30
Percentage
0.25
0.20
0.15
0.10
0.05
0.00
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4,5
5.0
Score
Figure 5: Score distribution of instructions evaluated by Llama-3.1-70B-Instruc and Qwen2.5-7BB-nstuuee (averaged).
A.1.2 InsTag Details
As mentioned in Section 4.3.2, inspired by Instag(Lu et al., 2023), the quality of instructions used for preference learning can be evaluated by LLLM-as-a-Quality-lddee Specifically, we utilize two perr--ero open-source LLMs - Instruct- score the instructions on a 1-5 scale, then calculate the average of the two scores as the final evaluation. The distribution of average scores is shown in Figure 5.
and Qwen2.5-72BB
The prompt for instruction quality evaluation is follows:
Prompt for Evaluating Instruction Quality
You are an expert evaluator tasked with rating the quality of the single or multi-turn dialogue based on its clarity, specificity, and coherence.
The rating scale is as follows:
Very Poor: The dialogue is unclear, vague, or incoherent. It lacks essential information and context.
Poor: The dialogue is somewhat unclear lacks important details. It requires significant clarification.
Average: The dialogue is moderately clear and specific. It may require some additional information for complete understanding.
Good: The dialogue is clear, specific, and mostly well-formed. It provides sufficient context for understanding the user's intent.
Excellent: The dialogue is very clear, specific, and well-articulated. It contains all the necessary information and context for providing a comprehensive response.
## Conversation History (conversation_history
Please directly evaluate the overall quality of the response by levels the following format:
Dialog quality: [Very Poor/Poor/Averge/Good Exxcelleet
A.2 Responses
Our experiments leverage 17 open-source LLMs (Table 2) spanning diverse architectural families:
14

---

Published as a conference paper at COLM 2025
Model Family
Model Name
Avg. Token Length
Avg. Score
Llama
MetaaLlamaa33--B--nsttcce
440.67
6.95
Meta-Llama-33-700--nsstttt
431.78
7.13
Llama-2-13b-chat-hf
405.27
6.59
Llama-3.1-Tul::-88-SFFT
317.62
6.69
Gemma
gemma-2-2b-it
478.53
6.90
gemma-2-9b-it
375.03
7.11
gemma-2-27b-it
1002.39
6.14
Phi-3
Phi-9-mini-128--insttuct
588.54
3.59
Phi-3-smal-112k--isstrtt
366.03
6.80
Phr33mmcdium-228--nisrtutt
492.73
6.11
Qwen
Qwen2-1.5B-istrruct
283.18
5.45
Qwen2-7B-Instruce
375.74
6.85
Qwen2-72B-Instruct
350.10
7.05
Mistral
Nixxtal-&x7B--nstrrtt-0111
327.91
6.84
Mistral-7B--nstruct-00.3
343.15
6.84
Others
DeepSeek-V2-Lite--CNe MiniCPM3-4B
319.52
6.54
260.63
6.37
Table 2: Overview of LLMs in the model pool: model families, average token lengths of responses, and average annotation scores.
Llama Family (Meta & A12): Includes parameter-scaled variants (8B, 70B) with enhanced tokenization and grouped query attention (GQA): 1aae--llaaaa-1------ntttee Meta- -lama-3.-Tull---88----
Llama-3.177B-Insstrue Llama-2-13b-chat-
Gemma Family (Google): Lightweight models (2B-27B) optimized for dialogue and
instruction following: gemma-2-2b-it, gemma-2-9b-it, gemma-2-27b-it
Phi-3 Family (Microsoft): Compact yet powerful models with 128k context windows, employing block expansion and quantization-awar training:
hiii-3-mmeciumm----iinnttt
Qwen2 Family (Alibaba): Scalable models (1.5B-72B) with sliding window attention for efficient long-context processing: Qwen2-1.5B-nnstrucc, Qwen2-7B-Instruct, Qwen2-72B-
Instruct.
Mistral Family (Mistral AT): High-performance architectures, including the mixture-of-
experts (MoE) Mixxtrll&xxB--Insttrrt--11111 and its dense counterpart Mistral-7B-Instruu v0.3.
Specialized Models: DeepSeek-V2-Lit--Chatt Hybrid attention mechanisms for effi- ciency; MiniCPM3-4B: Compact 4B model with multilingual support.
Here a generation prompt example:
Generation Prompt Example
<>>
 Summarize the main ideas of Jeff Walker's Product Launch Formula into bullet points as it pertains to a growth marketing agency implementing these strategies and tactics for their clients... 

Well, I'm quite partial to good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchenl


 Summarize the main ideas of Brendon Burchard's 's Experts Academy into bullet points as it pertains to growth marketing agency implementing these strategies and tactics for their clients... 

Well, I'm quite partial to a good squeeze fresh lemon juice. adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchenl

15

---

Published as a conference paper at COLM 2025

 What are the mental triggers in Jeff Walker's Product Launch Formula and "Launch" book? book? 

A.3 Annotations
A.3.1 Annotator Models
In this study, utilize three annotator models assess and refine the quality of generated responses. These include classifier-based model, (ooor---ewarrd-Gmmm-------- and two generative models, Llama-3.1-70--Intrrucct and Qween225-72B-Instrruc, Here we report their score on RewardBench' in Appendix A.3.1, illustrating their alignment capabilities and reliability in assessing response quality.
Model
Score
Rank
94.3
Llama-3.1-70B-Instruuut Qween225-7B-lInstruct
84.0
40
82.3"
46"
Table RewardBench (Lambert et al., 2024b) scores and leaderboard rankings. 'Self- reported score (not officially listed leaderboard).
Scores reflect model performance at the time when conducting the experiments, with Qwen- 2.5-72B-Instruct's score independently verified through evaluation.
A.3.2 Annotation Strategies: Prompt Design for Response Scoring
We evaluate annotation strategies for prompt design using generative models, con- sidering three key dimensions: response count, guideline complexity, and explanation requirements. By combining different levels within each dimension, we design total of six distinct prompt strategies, ensuring comprehensive assessment of annotation methods.
Response Count. This dimension determines whether responses are evaluated individu- ally or in comparative manner.
Single: The model assigns a score to one response at a time, without explicit comparison to alternatives.
Pairwise Comparison: The model evaluates two responses or more simultaneously, requiring a relative assessment of their quality.
Guideline Complexity. Guidelines provide evaluators with explicit criteria to structure their assessments.
The model assigns scores purely based on its internal judgment without external evaluation criteria.
w/o. Guidelines:
Basic scoring instructions are provided, ensuring minimal level of consistency evaluation.
Coarse Guidelines:
Fine-Grained Guidelines: erence questions, generated based on the user query, to assess response quality in structured manner.
Fine-grained evaluation incorporates task-specific pref-
Inttps://uuginggfaae http://iinnfaae...ooooooor////// oo/Skyyorr/kkyyoorr--eettt-----m 2-27B-v0.2 Pnttps://huggingface comm://aaallllmmm/LLaam.e 1-
9nttps://huggingffan wttpee..////e//eeerrr.a Innsstuu
http://hhggggg.eeeecc///ceeee//a bench
16

---

Published as a conference paper at COLM 2025
Explanation Requirements. This dimension controls whether the model justifies its scor- ing decisions.
w/o. Explanation: The model outputs only numerical score without explaining its reasoning.
w. Explanation: The model first provides a reasoning process before assigning a final score.
By systematically combining these dimensions, we construct six distinct annotation prompts, ranging from minimal (Single-Base Basic: scoring one response without guidelines or explanation) Seaeeed--xxlaiidd--neeee
to highly structured fine-grained guidelines and detailed explanations). Below, we describe each strategy in detail.
comparing two responses using
Single-Basic.
This is the simplest annotation strategy, where the model evaluates each single response without any additional guidelines or explanations.
Prompt for Single-Basic Annotation Strategy
You are an expert evaluator tasked with providing a simple numerical score for the response given the conversation history and user query.
## Conversation History
< <lbegin <bbegin_histor|>>>>>>> history|>>
(dialogue history)
< <|end history/s
## Current User Query
< begin query|> (instruction)
<lend _quer|>>
## Response to Evaluate
reeginnessposeef response]
lnd response|>
Please provide your evaluation by directly scoring the overall quality of the response from 0 to in the following format:
SCORE: [0-9]
Pair-Basic. For pairwise comparisons, this strategy asks the model to assign independent scores to two responses without any additional explanations.
Prompt for Pair-Basic Annotation Strategy
You are an expert evaluator tasked with comparing two responses to the same query.
## Conversation History
<|begin.history/> history|>
(dialogue history)
< <|end history|>
## Current User Query
|begin_qerry/>
Iinstruction]
<lend query|>
17

---

Published as a conference paper at COLM 2025
## Response A
< gregin.resporne a|>
(response }}
<
## Response B
< [eeeinnrespoose bb/>
(response_b]
< </end response b/>
Please provide your evaluation by directly scoring the overall quality of the responses from 0 to 9 in the following format exactly, where is the worst and 9 is the best.
Your response MUST follow this exact format: SCORE A: [0-9]
SCORE B: [0-9]
Pair- Pair-Explained. A pairwise evaluation strategy where the model first highlights the differ- ences between two responses before assigning scores, improving transparency in preference selection.
Prompt for Pair-Explained Annotation Strategy
You are expert evaluator tasked with providing simple numerical score for the response given the conversation history and user query.
## Conversation History
<
tieeginhistooy/:
hiaaogue_historyy
< <|end history|>
## Current User Query
< <begin.query/>
(instruction]
|ndd query|>
## Response A
< begin response a/>
(response_a) < <lend_
## Response B
<
b//>
(response b)
< <|end response b|>
Please provide your evaluation by first pointing out the pros and cons of both responses without polite phrases as short as you can, ensuring that the order of the responses does not affect your judgment. Then rate the overall quality of the responses from 0 to 9, with 0 being worst and 9 being best.
Your response MUST follow this exact format:
EXPLANATION:
[Your detailed comparison of both responses]
SCORE_A: [0-9]
SCORE_B: [0-9]
18

---

Published as a conference paper at COLM 2025
Pair-Guided. This is comparison strategy where predefined scoring guidelines provided to help the model evaluate and score two responses independently without explanation. The Rough scoring guidelines as follows:
8-9: Exceptional response that excels in all aspects.
6-7: Strong response with minor room for improvement.
4-5: Adequate response with some notable gaps.
2-3: Poor response with significant issues.
0-1: Severely inadequate or irrelevant response.
Prompt for Pair-Guided Annotation Strategy
You are an expert evaluator tasked with providing a simple numerical score for the response given the conversation history and user query.
## Conversation History
<
histor//s
[dialogue history)
< </end history|>
## Current User Query
< -berrn
(instruction] < <lend_query/>
## Response A
| response.a/>> (repponse_a] < <|end_
## Response B
< |egin response b/>
(response_b) < <eedoneespo
Scoring Guidelines:
- -8-9: Exceptional response that excels in all aspects
6-7: Strong response with minor room for improvement
4-5: Adequate response with some notable gaps
2-33: Poor response with significant issues
00-1: Severely inadequate irrelevant response
Please provide your evaluation by directly scoring the overall quality of the responses from 0 to 9 in the following format exactly, where is the worst and 9 is the best.
Your response MUST follow this exact format: SCORE_A: (0-99) SCORE_B: [0-9]
This is a combination of Pair-Guided and Pair-Explained, requir- ing the model to follow scoring guidelines and explicitly articulate the reasoning behind its comparison before assigning scores.
Ei--Guiddd-EEplainee
Prompt for Tain-Guided-Explainee Annotation Strategy
You are an expert evaluator tasked with providing a simple numerical score for the response given the conversation history and user query.
19

---

Published as a conference paper at COLM 2025
## Conversation History
< llbeeiin history|>
(dialogue history)
< </end history//>
## Current User Query
< Dvegin_querrl
(instruction]
< <|end query/>
## Response A
response_aa] responespo
## Response B
< begin response b/>
(response_b) < <end_rresponee.bl>
Scoring Guidelines:
- -8-9: Exceptional response that excels in all aspects
6-7: Strong response with minor room for improvement
4-5: Adequate response with some notable gaps
2-33 Poor response with significant issues
0-11: Severely inadequate irrelevant response
Please provide your evaluation by first point out the pros and cons of both responses without polite phrases as short as you can, ensuring that the order of the responses does not affect your judgment. Then rate the overall quality of the responses from to 9, with 0 being worst and being best.
Your response MUST follow this exact format:
EXPLANATION:
[Your detailed evaluation based on the scoring guidelines]
SCORE A: [0-9]
SCORE_P B: [0- 9]
This is the most detailed annotation strategy, in- corporating task-specific preference questions in the pairwise evaluation. The model must address each question for both responses before scoring. The prompt includes:
Bei--Guided-Explaine--Fiii
Prompt for Pair-Cuiddd--xpliinee-----e--- Fine-Grrine Annotation Strategy
You are an expert evaluator tasked with providing a simple numerical score for the response given the conversation history and user query.
## Conversation History
<
<lbegin_history//
(dialogue history)
< </end history
## Current User Query
< <|begin.query query|>
(instruction]
< end >
20

---

Published as a conference paper at COLM 2025
## Response A
< <lbegin.response_a response.a a/>
(response_a)
< <|end response.
## Response B
rpeggnneespoone..be
response b}
<lend response
Please evaluate both responses by first answering the following task-specific preference questions with Yes/Ne /N followed by a brief explanation. Then rate the overall quality of the responses from 0 to 9, with 0 being worst and 9 being best. Avoid polite phrases and be as concise as possible. The order of responses should not affect your judgment.
## Taec-fp Preference Questions (qqestions.prompp)
Your response MUST follow this exact format:
EXPLANATION:
Response A:
(explanation.prompp)
Response B:
(
[Overall comparison exilanation considering the fine-grained evaluation results]
SCORE_A: [0-9]
SCORE_B: [0-9]
Your scoring should be based on how many task-specific questions were answered with "Yes" and the quality of fulfillment for each criterion.
For fine-grained annotation, we first Llama-3.1-8B. Instruct to generate task-specific preference questions based on the user query. These questions are then integrated into the fine-grained annotation prompt to guide the evaluation process. The annotation model responds to each preference question with "Yes" or "No", accompanied by a brief expla- nation, before assigning final quality score. The explanation format follows "[Yes/ [Yees/No- [Brief explanation]" and is repeated for each preference question. The prompt template to generate preference questions is as follows:
Prompt for Generate Fine-Grained Preference Questions
You task analyzer that examines conversations and queries to identify task categories and generate relevant preference questions. Your analysis helps evaluate responses meet user requirements.
RULES:
1. Analyze both the conversation context (if any) and final query to understand the
complete task
2. Category should be brief (2-4 words) but precisely describe the task type
3. Generate 4-7 preference questions that:
-Focus on whether the response directly addresses user's 's specific request
Check if all parts the user's query are fulfilled
Verify if the response provides exactly what was asked for
21

---

Published as a conference paper at COLM 2025
Consider any context from previous conversation turns
4. Questions should help verify if a response completely satisfies what the user asked for
IMPORTANT: You MUST respond ONLY with valid JSON object in exactly this format:
"category": "brief task description",
"preference questions":
specific question 1?",
specific question 2?""
sseecific question 3?",
}}
Do not include any other text or explanation.
Below is an example for generated preference questions:
Example for Generated Preference Questions
User Query: Summarize the main ideas of Jeff Walker's Product Launch Formula into bullet points it pertains growth marketing agency implementing these strategies and tactics for their clients...
category:
product summary
preference questions:
Does the response provide clear summary of the main ideas of Jeff Walker's Product Launch Formula?
Are the bullet points specifically tailored for growth marketing agency implementing these strategies for clients?
Does the response accurately convey the key takeaways from the Product Launch Formula for this context?
Are the bullet points concise and easy to understand?
all the essential aspects of the Product Launch Formula relevant to growth marketing agencies?
Does the response
These fine-grained preference questions serve approximately objective criteria for evalu- ating response quality, ensuring that annotations capture meaningful distinctions in model outputs.
We also analyze the correlation between the proportion of "Yes" answers to preference ques- tions and the overall response score. We conduct experiments two datasets, ShareGPT-o3 and UltraFeedbant using scatter plot distributions to analyze the relationship between pref- erence question agreement and final scores. As shown in Figure there is clear positive correlation between the proportion of "Yes" answers and the assigned score score-respone responses with higher agreement on preference questions consistently receive higher scores. Con- versely, we observe negative correlation with the proportion of "No" answers, indicating that responses failing to meet preference criteria tend to receive lower scores.
A.3.3 Scoring Methods
Here introduce the detailed calculation of three scoring methods:
Single-Basic. this method, a single inference run is performed using greedy decod- ing. The score is assigned based solely on the model's deterministic response. Since randomness is introduced in this approach, ensures high consistency across repeated evaluations.
22

---

Published as a conference paper at COLM 2025
Progertio
quesiions
Pegartton preference quedicess
7000
10000
5090
6-
Proportion of Yes
5990
Proportion of 'N'
6000
0
0
4-
0
0
9000
-000
2-
2000
2000
0-
Final Score
Final Score
Progetion of "Yes' preference (UtraFeedback)
Progartion
preference (UlraFeedback)
7-
7000
12009
5000
Proportion of "Yess
Proportion of No
10009
5000
4000
3000
5000
2000
4000
1
1000
2000
Final Score
Final Score
Figure 6: Correlation between preference question agreement and final score. A clear positive correlation between the proportion of "Yes" answers and negative correlation with the proportion of "No" answers can observed from both ShareGPT and Feeebake datasets.
The final score in this method is computed by averaging the scores from five independent inference runs, each performed with temperature setting of 1.0. The final score is calculated as the average of the individual scores:
Single-Avg.
1 N
Savg
S =
where S; represents the score from the i-th run.
Single-Prob. This method calculates the final score by considering the probability distribu- tion over possible score tokens (from 9) predicted by the model. During inference, the model outputs raw scores (logits) for each possible score. These logits then converted into probabilities using the softmax function. The final score is obtained by weighting each score by its corresponding probability.
We let logit, represent the raw output score for score k. The corresponding probability P₂ is computed using the softmax function:
logit,
P
k= 0,,,,,
55)
eoggi
Then, the final score Sprob is computed by summing the weighted probabilities of each score:
S prob
Σ kP
(6)
23

---

Published as a conference paper at COLM 2025
559.3
50
500
Aveage Token Length
400
-395-1
465.2
353.5
252.1
300
30
200
20
12K
10
0.9155
Score
Score Margin
Figure Left: Average Token Length by Score. Right: Score Margin Distribution.
Model
Batch Size
Max Length
Learning Rate
Beta
Epoch
LR Scheduler
Llama-331Tultc3388--FTT
16
4096
5e-7
0.1
Cosine
Table 4: Hyperparameters for DPO training.
where P3 is the predicted probability for score k, and k is the score value (ranging from 0 to 9). This method captures the model's confidence in each possible score and is particularly useful in cases where the model's 's preferences are uncertain or borderline.
A.4 Final Dataset Statistics
Token Length and Average Scores. Table 2 presents the token length distributions and aver- age annotation scores using Single-Bast Basic strategy for each model in our dataset, calculated using the Llama-3 tokenizer ensure consistency.
Length Distribution Across Scores. We analyze the correlation between response token length and assigned scores (0-9). Figure 7 (Left) shows the average token length for each score, indicating no significant length bias in annotation.
Score Margin Distribution. The distribution of score margins is presented in Figure (Right), where margins of 0 and are the most common, followed by larger margin frequen- cies tapering off.
Annotation Consistency Across Annotation Strategies. The consistency of preference annotations across different strategies visualized in Figure 8. The results demonstrate high consistency between those annotation methods.
Variance distribution for multiple responses per instruction. In Section 4.3.1, we calculate variances of scores across different responses to the same instruction. The distribution of the variances is shown in Figure 9, where we observe a long-tailed distribution of instruction variances. Our experimental results demonstrate that using instructions with lower variance yields better DPO training outcomes.
B Experiments Setup
B.1 Training Details
For preference learning, we present some key hyperparameters related to DPO in Ta- ble 4. Additionally, we utilize the OpF framework (Hu et al., 2024) to conduct full-parameter DPÒ of Llama--.11TTluu------- ments utilize these hyper-parameters consistently.
on 80GB A800s. All of our DPO experi-
24

---

Published as a conference paper at COLM 2025
ShareGPT-V3 Consistency
Consistency
L,000
033322
18333
B0232
0.6833
LO000
63023
9.324
6.274
08333
6.003
-00
3.00
AOpaii
0.000
3.8431
10229
68428
010614
0.0003
1.3900
642.4
010154
3.45
0.9122
18935
882327
1,000
PG
03333
0.3307
0.8220
2.0228
38000
0.000
S8aik
18544
08141
58eac
0.8916
2.309
3.29
roxes
Figure Annotation consistency across annotation strategies. UltraFeedback.
Left: ShareGPT; Right:
Probability Density Function of Instrcution's Variance distribution
0.5
0.4
Density
0.3
0.2
0.1
0.0
10
Variance
Figure 9: Score variance distribution for variance-based method.
B.2 Evaluation Details
This section provides detailed descriptions of the evaluation benchmarks used in our experiments, including the capabilities assessed, evaluation methodologies, and primary metrics.
MTBench (Zheng et al., 2023b). MTBench evaluates -nstruction-folloiing and conversa- tional capabilities through diverse, multi-turn dialogue prompts covering tasks such as math, reasoning, writing, and coding. The benchmark consists of 80 test examples. Evalua- tions are subjective, using GPT-4o as the judge model to score responses from 1 to 10 based on helpfulness, correctness, coherence, and related aspects.
ArenaHard (Li et al., 2024b). Are consists of 500 challenging crowd-sourced conver- sational prompts specifically designed to stress test advanced reasoning, complex instruction follow-up, and nuanced knowledge retrieval capabilities. Evaluations are subjective and per- formed using the LLM-as-a-Judge method, with GPT-4-0314 serving as both the judge and the baseline model. The primary metric is the Elo rating derived from pairwise comparisons.
AlaacaEval 2.0 (Dubois et al., 2024). AlpacaEval 2.0 assesses costruction-following accuracy and general-purpose alignment using 805 uiserggeeerate instructions spanning general
25

---

Published as a conference paper at COLM 2025
knowledge, reasoning, writing, and coding tasks. Evaluations are subjective, using GPT-4 Preview (11/06) as both the baseline and the auto-annotator. The main metric is length- controlled (LC) win rate, which alleviates length biases inherent in GPT-4 evaluations.
WildBench (V2) (Lin et al., 2024). WildBench (V2) benchmarks model performance using 1,024 challenging real-world user queries selected from extensive human-chatbot inter- actions. Evaluations systematically use task-specific checklists and provide structured explanations to justify scores and comparisons. Judges can vary; specifically used GPT- 40 and primarily employed WB-Score, which individually assesses response quality rapidly and cost-effectively.
Eurus Evaluation Suite (Yuan et al., 2024). The Eurus evaluation suite rigorously tests model capabilities across coding, mathematics, logical reasoning, and instruction-followigg tasks using carefully curated, expert-generated prompts. It includes diverse set of benchmarks: HumanEval, LeetCode, and MBPP for coding; GSM-Plus, SVAMP, ASDiv, Theorem and general math evaluations for mathematics; Big Bench Hard (BBH) for reasoning; and IFEval for instruction-folowwii Evaluations primarily utilize objective methods based on exact match accuracy and computational correctness. Detailed descriptions are available at https://github
LiveBench (White et al., 2024). LiveBench is dynamic benchmark designed to mitigate test set contamination by regularly introducing new tasks derived from recently published datasets, arXiv papers, news articles, and IMDb movie synopses. In experiments, used the 2024-11-25 version containing 1,136 questions. Evaluations are objective, based on verifiable ground-trune answers.
C Addition Experimental Results
C.1 Further Experiments On/Off-Policy Mixing
To further investigate on/off-policy responses' effect in preference dataset, based on the Mid-Mix configuration in Section 4.4.3 where preference dataset composes of 50% on-policy and 50% off-policy responses, we designed two experimental groups:
OfChosen-OffReeect where all chosen responses were high-scoring on-policy responses and all rejected responses were low-scoring off-policy responses;
OffChosen-OnReeee the inverse configuration using high-scoring off- responses as chosen and low-scoring on-policy responses rejected.
Setup
ArenaHard
WildBench
LiveBench
Mid-Mix
34.7
27.98
25.5
Offhesen-OffReject
27.6
22.01
20.5
OffChosen-OnReeect
15.9
12.43
19.3
Table Results for OnChosen-OffRRejec, OffChosen-OnRejet and Mid-Mix setups.
The results for these two setups, along with Mid-Mix, are presented in Table 5. Notably, while maintaining identical on-policy off-policy response ratios to the Mid-Mix setup, both setups exhibit substantial performance degradation across benchmarks compared to Mid- Mix. We hypothesize this stems from the homogeneity on-policy data originating from single SFT model. When exclusively employing on-policy data in either chosen or rejected response, LLMs may exploit superficial patterns, thereby learning shortcut heuristics that undermine preference learning efficacy.
Crucially, the OffChosen-OnRReeed setup demonstrates catastrophic aligmment collapse. This phenomenon arises from the DPO loss formulation shown in Equation (3), the alignment process systematically suppresses the likelihood of LLM's own outputs when off- policy responses are designated chosen while on-policy responses serve as rejected targets.
26

---

Published as a conference paper at COLM 2025
Such dynamics violate the fundamental assumption of preference alignment, ultimately destabilizing the learning objective.
C.2 Detailed Results for Combined Experiments
In Figure (right), we demonstrate the performance gains achieved through additive integration of components within our AIR framework. In this section, we quantitatively illustrates the detailed improvements contributed by each integrated element across all benchmarks.
Setup
MT-Bench
WildBench
Furus
LiveBench
ArenaHard
Average
Tula-3-88-SFT (Baseline)
6.38
12.50
55.77
24.3
13.8
34.03
w. Vanilla Pref. Data
25.65
24.1
30.2
40.37
Point-Wise Scoring Pref. Data
6.83
26.47
57.79
24.4
33.8
42.15
Moderate score Margin among responses
7,00
25.80
57.48
25.9
33.8
42.60
Higher Absolute Response Score
29.25
57.96
35.4
On/Onnn// On/Off-Policy Responses Hybrid Mixed
7.04
31.22
58.11
26.5
36.8
44.61
Inst. with Lower Resp. Score Variance
7,23
32.43
58.38
26.4
38.9
45.68
Table 6: Details for AIR Framework's Additive improvement across all benchmarks, resulted by stepwise integrating different principles. Scores are normalized 100-point scale for averaging.
C.3 Cross-verification
In Section 4, we used Llama-3.1-70B-instee for the annotation of preferences and used instructions from ShareGPT to generate responses. Through comprehensive experiments, we determined the optimal design for the AIR framework. In this section, we present an analysis of the robustness of the AIR framework from two critical perspectives: the preference annotation model and instruction sources in preference data.
C.3.1 Robustness Analysis of Annotation Model
To analyze the robustness of annotation model, we use Qwwn-225-72B-nnstruuu (Yang et al., 2024) for preference annotation while maintaining identical experimental settings to those in Section 4.
Results for annotation strategy are shown in Table which demonstrates the Single- Basic configuration consistently outperformed pairwise scoring approaches across all five benchmarks.
We present our experimental results for response pairs in Figure 10, it clearly shows that pairs with moderate score margin, higher absolute score, and mixing on/off an/off-poicyy responses achieves better results, in align with Llama-based annotation experiments.
resulls
The evaluation results for the effect of variances across different response score from the same instruction is shown in Figure 11, we can find that except for Eurus, the low-variance instruction achieves the best performance on all other benchmarks, which consistent with the conclusions of the main experiments.
C.3.2 Robustness Analysis of Instruction Sources
To analysis the robustness of instruction sources, we employ instructions in UltraFeedback (Cui et al., 2023) for response generation while maintaining identical experimental settings to those in Section 4.
We present the results for annotation strategy in Table 8, which demonstrates even with the modification in instruction source, the Single-Basic setup has the best overall performance across all benchmarks.
27

---

Published as a conference paper at COLM 2025
Annotation Strategy
MT-Bench
WildBench
Eurus
LiveBench
ArenaHard
Avg.
Tulu-3-88-SFT (Raseline)
6.38
12.50
55.77
24.3
13.8
34.03
Single-Basic
6.81
27.47
57.85
26.5
37.2
43.42
Pair-Basic
6.73
26.86
57.30
21.8
33.1
41.27
Pair-Guided
6.67
26.92
57.21
21.5
36.6
41.79
Pair-Explained
6.81
26.29
57.40
22.3
35.4
41.90
Mei--Guiddd-Exppliine
6.62
26.21
57.40
20.7
34.7
40.04
Pail-Guided-EXplainne-FFF 6.73
24.10
56.55
23.3
32.8
40.81
Table 7: Impact annotation strategies on DPO performance. Qwen-225-72B-Insttrus is employed for preference annotation while maintaining identical experimental setups to those specified section 4.2.2. FG denotes Fine-Grained. Single-Basic shows superior performance, which aligns with the results obtained from Llama-based annotation.
WildBench
LiveBench
ArenaHard
AlpacaEval 2
Avrage
35
Score
30
25
Low Margin
Moderate Margin
High
Low-Score Response
Mid-Score
High-Score Response
Pure-
Low-
Mid-
Pure-
Margin
Response
Off-Palicy
Mix
Mix
On-Policy
Figure 10: Performance across different benchmarks, comparing different relative score margins (left), absolute score thresholds (middle), and on/ on/off-policy mixing strategies (right). Qwen-2.5-72B-nnttee
employed for preference annotation while maintaining identical experimental setups to those specified in section 4.4. We prioritize pairs with moderate score margin, higher absolute score, and hybrid mixing of on/ an/off-policy responses, which is in align with Llama-based annotation experiments.
Annotation Strategy
MT-Bench
WildBench
Eurus
LiveBench
ArenaHard
Avg.
Tulu-3-8B-SFT (Baseline)
6.38
12.50
55.77
24.3
13.8
34.03
Single-Basic
6.58
27.17
25.4
36.0
42.66
Pair-Basic
6.55
24.59
58.90
23.5
34.2
41.70
Pair-Guided
26.56
58.38
35.0
41.19
Pair-Explained
6.38
26.09
58.70
22.6
32.5
40.74
Hin-Guiddd--Explaine
6.56
27.43
58.30
23.4
34,9
41.93
Mnir-Guided-Exxlaineed
6.46
23.61
58.46
24.5
31.4
40.51
Table 8: Impact of 6 annotation strategies on DPO performance. Instructions from Ultrafeed- back is employed for response generation while maintaining identical experimental setups to those specified in section 4.2.2. All strategies ensure identical instruction sets and pair counts used for DPO training. FG denotes Fine-Grained.
28

---

Published as a conference paper at COLM 2025
Tulu 3 (SFT)
MT-Bench Tum2
Low- Variance
MT-Benchh Turnl
Mid- Mid-Variance
High- Variance
WildBench
7.0
66
17.2
186
66.0
iveBench
21.0
15.0
36.0
20.0
304
354
AlpacaFvaa
25.0
8.5
Eurus Suite
ArenaHard
Figure 11: Performance across different benchmarks, comparing different response score variances. L.ven-2.5-2B-IInttrue is employed for preference annotation while maintaining identical experimental setups those specified in section 4.3.1.
29


Reviews:
[{"review_id": "nj6HuQVQFh", "reviewer": "Reviewer_EMwy", "summary": "This work systematically disentangle the impact of annotations, instructions and response pairs on the efficacy of preference datasets. They conduct controlled experiments to study each dataset component in open-source models. The work yields actionable insights to understand preference dataset design and can be useful for practitioners.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The controlled analysis of each component's impact allows for a systematic study and nuanced understanding of these components. \n2. The work provides various actionable insights to improve performance such as low-variance instructions or generative-RM for scoring. \n3. The experiments have good coverage of various components and design choices. \n4. The takeaways are empirically validated across various benchmarks and cross-validates by replicating with a different instruction source and annotator model.", "reasons_to_reject": "1. The study mainly focuses on the isolated impact of each dataset component and does not study the interactions between these components that may affect LLM performance. \n2. While LLM based preferences are widely accepted, this work did not adequately study the impact of LLM annotations vs Human annotations and caveats with LLM annotations (for e.g. LLMs tend to favour their own generations) and how these impact preference dataset design.", "questions_to_authors": "1. Can you report the error bars across all results to better situate the performance differences with each component (Figure 2, Table 1would be most helpful) ? \n2. In line 204, \"Explanations/coarse guidelines offer negligible gains (≤ 0.28) over simpler baselines\" why is this considered negligible gains when difference between Pair-guided and Pair-explained is 0.45 on average ? Did you implement a single-explained to better understand this effect ? \n3. In line 207,  \"The trend toward intricate annotation workflows may inadvertently introduce noise, advocating for streamlined designs.\" I am not convinced that this is a valid conclusion from Table 1. The difference in performance across various annotation designs are not well-investigated to understand whether this is noise or not. \n4. Did you also study which RM (classifier or generative) had higher agreement with human annotations ?\n5. How would these findings change or not change with the base model or a different architecture ?\n\nThese papers can be cited to better support this work:\n1. Kim, Joongwon, et al. \"A Systematic Examination of Preference Learning through the Lens of Instruction-Following.\" arXiv preprint arXiv:2412.15282 (2024).\n2. Yin, Yueqin, et al. \"Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts.\" arXiv preprint arXiv:2402.10958v1(2024).\n3. Bansal, Hritik, John Dang, and Aditya Grover. \"Peering through preferences: Unraveling feedback acquisition for aligning large language models.\" arXiv preprint arXiv:2308.15812 (2023).\n4. Panickssery, Arjun, Samuel Bowman, and Shi Feng. \"Llm evaluators recognize and favor their own generations.\" Advances in Neural Information Processing Systems 37 (2024): 68772-68802.", "extra_scores": {}}, {"review_id": "bicVKyVzfH", "reviewer": "Reviewer_E4jA", "summary": "This paper introduces the AIR (Annotations, Instructions, and Response Pairs) framework to systematically analyze the independent impacts of three key components—Annotations, Instructions, and Response Pairs—on aligning large language models (LLMs) with human preferences. Through experiments, it identifies three empirically validated principles: simplifying annotations with generative scoring, prioritizing low-variance instruction selection, and optimizing response pairs for quality and diversity. Aggregating all strategies, the authors observed substantial gain over baseline.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors tackle important problem. Constructing preference dataset has been a core component of language model alignment but hasn't been thoroughly investigated. This work investigate various strategies and execute a large number of ablation studies.\n\n2. The resulting dataset outperforms baseline while only maintaining small portion.", "reasons_to_reject": "The current paper presents a series of controlled experiments and their results. To enhance the robustness and credibility of the work, it would be beneficial to include explicit hypotheses for each experiment and additional supporting experiments to validate these hypotheses.\n\nAll experiments in the paper were conducted using a single model, specifically **Llama-3.1-Tulu-3-8B-SFT**. Expanding the scope to include more diverse models would significantly strengthen the findings. Models such as OLMo-2-1124-7B-SFT [1] or openbmb/MiniCPM-S-1B-sft [2] could serve as valuable candidates for additional experimentation.\n\nAddressing these concerns would likely elevate the overall quality and impact of the paper, warranting a higher score, potentially up to 7.\n\n[1] https://huggingface.co/allenai/OLMo-2-1124-7B-SFT\n\n[2] https://huggingface.co/openbmb/MiniCPM-S-1B-sft", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VOQ1IVogPS", "reviewer": "Reviewer_xkuT", "summary": "Preference learning for LLMs requires preference annotations, instructions, and response pairs; these three aspects are typically conflated in current work, which makes it difficult to disentangle their effects. The authors propose AIR, an analysis framework that attempts to isolate, analyze, and optimize for each of the three aspects separately. For Annotations, the authors investigate the effects of annotator model choice, prompt design, and scoring methods (e.g., pairwise vs scoring). For Instructions, they investigate score variance and number of prompt turns. For Response, they investigate how to construct pairs from scored examples, varying the score margins, thresholds, and on/off-policy mixing. Preference data is generated by 17 open-source LLMs; instructions through ShareGPT and UltraFeedback; all experiments are conducted on the Llama-3.1-Tulu-8B-SFT base model with 30k preference pairs and vanilla DPO; and alignment is assessed using 6 benchmark datasets across a range of domains. Based on experiments conducted using this framework, the authors recommend the following for alignment dataset design: (i) use point-wise scoring, (ii) select instructions with low score variance across different LLMs, and (iii) optimize for high quality response pairs with clear preference score margins. Combining all strategies yields substantial performance gains.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This manuscript contains a wealth of insights for preference dataset curation\n- The framework is a useful way of disentangling different aspects of preference learning dataset curation. While not every combination of strategies is explored in this manuscript due to length, the framework will help others conduct such studies in the future.\n- Experiments are thorough and support the findings; confirmatory experiments with another base model would be nice", "reasons_to_reject": "N/A", "questions_to_authors": "- Some of the recommendations borne of this framework are dependent on each other (e.g., it makes sense to adjust score variance in the context of point-wise scoring, but this would not apply in settings whether the annotations are pairwise). It would be useful to provide some recommendations that generalize beyond the wholesale set of recommendations identified in the work, since there may be partial constraints on dataset collection. This could be in the context of future work (the manuscript is dense enough as is) and how the AIR framework allows for discovering other performant mixtures of strategies for data collection.\n- In some cases, the impacts of combining different interacting strategies is not super clear, e.g., in Figure 4, how do relative score margins, score thresholds, and on/off-policy mixing strategies  interact? Are these plots made while holding other strategies constant?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers all agreed that this is a solid study of what makes for a good preference tuning dataset. The takeaways from the careful experiments done in this paper help to crystallize (and/or refute) common wisdom in the field, and they will be very helpful for researchers and developers building preference datasets. I encourage the authors to update their manuscript to account for the reviewer comments, e.g., the addition of the new base model and clarifications on the scope.
Decision: Accept