Data-Centric Human Preference with Rationales for Direct Preference Alignment
Authors: Hoang Anh Just
Ming Jin
Anit Kumar Sahu
Huy Phan
Ruoxi Jia
Venue: COLM 2025 (2025)
Keywords: data-centric AI
rationales
TL;DR: We introduce rationales to boost learning from provided human preference pairs in direct preference training.

Abstract: Aligning language models with human preferences through  reinforcement learning from human feedback is crucial for their safe and effective deployment. The human preference is typically represented through comparison where one response is chosen over another for a given prompt. However, standard preference datasets often lack explicit information on why a particular choice was made, presenting an ambiguity that can hinder efficient learning and robust alignment, especially given the high cost of acquiring extensive human annotations. While many studies focus on algorithmic improvements, this work adopts a data-centric perspective, exploring how to enhance learning from existing preference data. We propose augmenting standard preference pairs with rationales that explain the reasoning behind the human preference. Specifically, we introduce a simple and principled framework that leverages machine-generated rationales to enrich preference data for preference optimization algorithms. Our comprehensive analysis demonstrates that incorporating rationales improves learning efficiency. Extensive experiments reveal some advantages: rationale-augmented learning accelerates convergence and can achieve higher final model performance. Furthermore, this approach is versatile and compatible with various direct preference optimization algorithms. Our findings showcase the potential of thoughtful data design in preference learning, demonstrating that enriching existing datasets with explanatory rationales can help unlock improvements in model alignment and annotation efficiency.

Published as a conference paper at COLM 2025
Data-Centric Human Preference with Rationales for Direct Preference Alignment
Hoang Anh Just Virginia Tech just@vt.edu
Ming Jin
Anit Sahu Oracle anit. sahu@gmail com
Huy Phan
Virginia Tech jinming@vt.en edu
Amazon
huypq@amazon.coo co. uk
Ruoxi Jia Virginia Tech
ruovijia@vt.cm edu
Abstract
Aligning language models with human preferences through reinforcement learning from human feedback is crucial for their safe and effective deploy- ment. The human preference is typically represented through comparison where one response is chosen over another for a given prompt. How- ever, standard preference datasets often lack explicit information on why a particular choice was made, presenting an ambiguity that can hinder efficient learning and robust alignment, especially given the high cost of acquiring extensive human annotations. While many studies focus on algorithmic improvements, this work adopts data-centric perspective, exploring how to enhance learning from existing preference data. We pro- pose augmenting standard preference pairs with rationales that explain the reasoning behind the human preference. Specifically, we introduce a simple and principled framework that leverages machine-generater ra- tionales to enrich preference data for preference optimization algorithms. Our comprehensive analysis demonstrates that incorporating rationales im- proves learning efficiency. Extensive experiments reveal some advantages: rationale-augmented learning accelerates convergence and can achieve higher final model performance. Furthermore, this approach is versatile and compatible with various direct preference optimization algorithms. Our findings showcase the potential of thoughtful data design in prefer- ence learning, demonstrating that enriching existing datasets with explana- tory rationales can help unlock improvements model alignment and annotation efficiency.
1 Introduction
Human preference optimization is critical stage in preparing large language models (LLMs) for real-world deployment. Its primary goal is to align model behavior with human expectations and prevent undesirable outputs (Christiano et al., 2017; Stiennon al., 2020; Bakker et al., 2022). This alignment process typically relies on datasets containing prompts paired with ranked responses, which encapsulate human preferences. Popular methods like Reinforcement Learning from Human Feedback (RLHF)(Ouyang et al., 2022) leverage this data by first training reward model to mimic human judgments and then using reinforcement (earning(Schulman et al., 2017) to optimize the LLM policy against this reward signal. More recently, direct preference optimization (DPO) (Rafailov et al., 2024) and its variants offer an alternative approach, directly optimizing the policy using preference pairs through a loss function derived from an implicit reward model, thus avoiding the complexities of training explicit reward model.
Despite their effectiveness, these methods face significant challenges. Models can overfit to the preference dataset (Azar et al., 2024), suffer performance degradation broader
1

---

Published as a conference paper at COLM 2025
tasks (Pal et al., 2024), learn to exploit the rewards (Amodei et al., 2016), or generate excessively long and unhelpful responses (Park et al., 2024). Furthermore, acquiring high- quality human preference data is notoriously costly and time-consuming Zheng et al. (2023); Tan et al. (2024); Liu et al. (2024); Zhou et al. (2025). Yet, insufficient data can lead to underfitting, failing to achieve robust alignment Jinnai & Honda (2024). Much research has focused on mitigating these issues through algorithmic advancements, such as regularizing the optimization objective I'al et al. (2024); Amini et al. (2024); Park et al. (2024) or developing entirely new learning formulations Ethayarajh et al. (2024); Hong et al. (2024); Yuan et al. (2023); Munos et al. (2023); Swamy al. (2024); Wu al. (2024). While recent techniques such as SimPO (Meng et al., 2024) and SPPO (Wu et al., 2024) have demonstrated remarkable performance gains, often enabling smaller models to outperform larger ones on benchmarks like AlpacaEval Dubois et al. (2024), these improvements largely stem from modifications to the objective function or the generation entirely new preference pairs.
However, human preference data are inherently valuable, representing the ground truth for alignment, and acquiring more is often prohibitively expensive. Generating synthetic data, while scalable, risks introducing biases or deviating from nuanced human intentions. This motivates a shift in focus: instead of solely pursuing algorithmic refinements or creating new datasets, can extract more value from the existing, carefully curated human preference data? Our study adopts this data-centric perspective, specifically exploring how to enhance model alignment using direct preference alignment methods by improving the way models learn from the available human preference data. We pose the central question: How can we help the model better learn from existing human preference data to align more effectively with human expectations?
key limitation of current preference datasets lies in their opacity regarding the reasons behind a preference. Given a prompt and two responses, why is one preferred over the other? While the distinction may be obvious in some cases (e-g,, factual correctness), becomes highly ambiguous when responses are closely matched in quality or exhibit subtle trade-offs. Superficial features like response length offer unreliable cues; longer responses might be favored for comprehensiveness in one context, while shorter ones are preferred for conciseness in another. Furthermore, human preferences can be subjective and stem from diverse, unstated reasoning. Without explicit explanations explaining these underlying criteria, the model faces a significant learning challenge. This ambiguity can lead to data inefficiency, requiring more examples to discern patterns, or worse, cause the model to learn spurious correlations, hindering true alignment and potentially degrading performance.
Current Pair-Wise Preference Dataset
Prompt
Chosen
Rejected
Rationale
tiger large camivorous mammal [...]
you help
response above preferred because
a tiger?
tiger Do you know that? [..]]
Pair-Wise Preference Dataset with Rationales
Figure 1: Comparison between the current pair-wise preference dataset used for preference learning and the enriched dataset with added rationales.
For the reasons outlined above, we propose a natural extension to the standard preference dataset structure by incorporating rationales. We define a rationale as an explanation accompanying each preference pair, explaining why the chosen response was preferred over the rejected one for the given prompt. This approach aims to provide the model with deeper contextual understanding during preference learning. This idea also draws inspiration from social studies (Mitchell et al., 1986; Chi et al., 1994; Crowley & Siegler, 1999, Williams & Lombrozo, 2010), showing that adding explanations to answers improves one's understanding of the problem compared to individuals who do not provide explanations.
2

---

Published as a conference paper at COLM 2025
We hypothesize that by enriching existing human preference datasets with rationales, models undergoing direct preference alignment can develop a more nuanced understanding of the underlying human values. This enhanced comprehension is expected to translate into improved preference modeling, higher performance on alignment benchmarks, and greater annotation efficiency, allowing models to learn human preferences more effectively from the available annotated data before potentially needing to resort to further costly data collection.
Contributions. This paper provides a new data-centric perspective on direct preference learning. We list the summary of contributions:
I We introduce rationales into the human preference learning framework, where rationales explain the reasons behind the preference for a particular response and derive straight- forward formulation for the preference function extend the rationales and show how to adapt our method to current direct preference learning algorithms such as DPO.
I We analytically examine the impact of the rationale on preference training through the lens of information theory and empirically show the impact of preference learning with rationales.
In a broader context, our approach presents a new paradigm for data-centric research in language modeling: rather than focusing pruning samples to distill the most informa- tive pieces from a dataset Albalak et al. (2024), we explore how to enrich each sample's information content and examine its impact. The promising results presented in this paper demonstrate the effectiveness of enhancing individual samples' information content in pref- erence learning and suggest that this approach may hold potential for improving learning in other domains.
2 Related Work (extended version in Appendix A)
Aligning large language models (LLMs) using techniques derived from human preferences is important step for current LLMs (Casper et al., 2023; Christiano et al., 2017; Ziegler et al., 2019; Stiennon et al., 2020). While initial methods often relied on explicit reward modeling followed by reinforcement learning algorithms like Proximal Policy Optimiza- tion (Shhlann et al., 2017; Ouyang et al., 2022), newer approaches such as Direct Preference Optimization (Rafaliaa et al., 2024) bypass explicit reward models. De- spite their success, both PPO-based RLHF and DPO face challenges, such as overfitting to the preference data, underfitting due data scarcity (Azar et al., 2024; Pal et al., 2024), and reward hacking (Amodei et al., 2016). To address these issues, researches introduce length-regularized DPO (R-DPO) (Park et al., 2024), odds ratio preference optimization (ORPO) (Hong et al., 2024), Kahneman-Tversil or ranked responses methods (Yuan et al., 2023). General preference modeling, moving beyond Bradley-Terry
optimization (KTO) (Ethayarajh et al., 2024),
assumptions (Bradley & Terry, 1952; Bertrand et al., 2023), offers alternatives such as Nash preference optimization (Munos et al., 2023; Swamy et al., 2024; Rosset et al., 2024; Wu et al., 2024). However, these methods focus primarily on algorithmic enhancements. A separate line of data-centric work aims to improve alignment by synthe- sizing entirely new preference datasets, often leveraging stronger models or self-critique mechanisms (Yang et al.; Pace et al.; Meng et al., 2024; Wu et al., 2024; Wang et al., 2024). Our work pursues distinct, orthogonal approach. Instead of modifying the optimization algorithm or generating new preference pairs, we focus on enhancing the utility of existing preference datasets for direct optimization methods by introducing rationales that explain why one response was preferred. While related data-centric concepts which incorporate critiques to preference data have shown promise, particularly to refine reward models (Ye et al., 2024; Ankner et al., 2024; Yu et al., 2024), our rationales are designed for direct prefer- ence optimization methods and differ from critiques in design. Particularly, our rationales explain the difference between the responses, the critiques made for each preference individually and independent of other preference. We provide an extended discussion of related work in the Appendix A.
3

---

Published as a conference paper at COLM 2025
3 Method
In this section, we introduce the incorporation of rationales into preference learning and show the derivation of adapting current methods. We present a demonstration of extending the direct preference optimization (DPO) algorithm to incorporate the rationales, while similar extensions can be applied to other variants of DPO. Further, we analyze theoretically the possible impact of rationales through the perspective of information theory.
3.1 Preliminaries
Notations. Let D denote the pair-wise preference dataset of size N, D (x(0, Ye .yi
where x(0) is a context, yo
(i) is the preferred/ / chosen/ winning response the context x(()) over the pntreferred/rejecteerr /losing response y/( Let πi and Tref denote the policy to be preference optimized and the reference policy respectively. In our setting, the policies are the language model to be preference trained and the base or supervised fine-tuned SFT model, respectively. To compute the joint probability of the autoregressive language model 7 generating the response given the prompt x, we compute the product of probabilities
after observing each token: Tty xx) = n, ( yo:).
Reward Modeling with DPO. the RLHF process (Christiano et al., 2017; Ziegler et al., 2019; Stiennon et al., 2020; Bai et al., 2022; Ouyang et al., 2022; Rafailov et al., 2024), the goal is to align the language model towards human preferences. The preferences ranking from the dataset D is assumed to be sampled from the latent reward function r(x, y) and the preference function is assumed to be generated by the Bradley-Terry model (Bradley & Terry, 1952): p" (yw > y)(x) = (x,
r rxxx, y1)), where o is the sigmoid function. The reward function then can be estimated by minimizing the log-likelihood of the fol- lowing objective C(r,
r(x, Then the next step to tune the language model with the reward model as follows by maximizing the re- wards and not diverging from the fixed reference model: maX (Ex-Dyy=xppllx) [r(x,
(x IDD (r(zr(
BDI.. [xq [y)(y[x)] (err]// (yxx)] where B a hyperparameter measuring the divergence between two policies. Alternatively, with a reparametrization of the Bradley- preference model (Rafailov al., 2024), the preference function can be expressed in terms of pol-
icy n²::
p' (yw w|l)
=o =o (f Hoatyyu [x]
(x)]x)
Blog
xet [uile) (yr
Thus, to estimate the policy, Rafailov et al. (2024) proposes to directly minimize the log- likelihood of the following DPO loss: Cimm: Fipppaiii loge (Alog
nylsur))
flog mufyyl
F Lit 37
3.2 Formulation of Preference Learning with Rationales
Although preferences are modeled given the preferred and unpreferred responses, there are nuances in the responses that are obscure for the model to comprehend and catch the differences between them. Therefore, our goal is to help the model learn preferences by providing guidance cues in the preference tuning process, which we call rationales. The rationales explain why given response is preferred over the other response. For that reason, we extend the current preference learning with a data-centric technique to incorporate rationales, and we term this the rationale-enriched preference function, where the updated preference function is formulated as p" (yw > -),r(x) and is the rationale from the updated dataset D'
(1 y/ Vi
(i) , Ji=1 By the chain rule, arrive at:
p" Usa (.rls) (* xxx) "00-""(1x300> m.).
where p* (yw > yı(x) is the pair-wise preference term modeled in Section 3.1, and pr(r|x, vi) is the probability of the rationale r given the context x and the preference Yo > yr. Given the policy π²' we can retrieve the probability of generating the rationale r given the context and the preference yo > Vı, π (r|x, [xx yw > y1). Similarly when retrieving
4

---

Published as a conference paper at COLM 2025
the probability of generating responses Y/ and y/ for the prompt x, which given in the preference dataset D', we can also retrieve the probability of generating rationale / given X,yay, and y/, where (x, Yur 1)) ~ D'. In practice, explain why the response Yv is preferred over the response y1 for the prompt x and retrieve the probability of generating the rationale Thus, p(r|,, yw > yt) Tyg X, > y1)-
ask the policy language model no to
Adaptation to DPO Loss. After deriving the rationale-enriched preference learning func- tion, we extend the DPO method to incorporate rationales. By substituting Equation and p(r[x, V/w > y/) into Equation 2, we can express the rationale-enriched preference function in terms of policy. We can then estimate the policy by optimizing To through maximum likelihood using the following objective over the updated preference dataset (i) (i)
D -xx(1, Vi y1 [(O) i=11 which we term as rationale-DPO (RDPO):
Axpro Find) (1 loger AAlog
Moo [P]
Ma xx)
flog
Ma(3hix)
logg 9)
where y is the added hyperparameter for weighting the impact of rationales the loss. We also provide experiments on adapting ORPO and SimPO with rationales in Section 4.
4 Evaluation
In this section, we evaluate the impact of rationales on direct preference learning. We conduct multiple experiments with two main goals in mind: (1) understand how the added rationales affect the efficacy and efficiency of current preference learning algorithms, and (2) determine the significance of rationale quality for effective learning.
4.1 Experimental Setup
Datasets. For our analysis, we focus on three preference datasets: Orca DPO Pairs (Intel, 2024), which is a pairwise preference dataset version of Orca (Mukherjee et al., 2023), a bina- rized UltraFeedback (Tunstall et al., 2023), which is a pair-wise version of UltraFeedack (Cui et al., 2023), and Anthropic Helpful and Harmless, which is a huamn preference dataset about helpfulness and harmlessness. For each dataset, we take 512 fixed samples as test set for winrate evaluations. We generate rationales and add rationales to the current datasets. We refer readers to Appendix C.9 for details on generating rationales. Given the test sets, we sample the responses from models trained with preference learning methods and compare the performance between the models by measuring the winrates between the corresponding responses.
Models. We investigate preference training on various large language models: Mistral-7B- v0.1, Diiistal--B-Instructtet (Tiang et al., 2023), Beahhr--B-BBttaTunnntat et al., 2023), and Llama3-8B- (A Meta,,, 2024). We use GPT-4o (Achiam et al., 2023) as judge to evaluate the responses generated by the models and retrieve the winrate scores. Please see Appendix C.11 for details. We provide full results with ablation on hyperparameters in Appendix C.2.
Methods. In our experiments, we study the integration of rationales into preference learn- ing frameworks, such as DPO (Rafailov et al., 2024), which requires the SFT model for the reference model, and ORPO (Hong et al., 2024) and SimPO (Meng et al., 2024), which do not. To ensure fair comparison between DPO and RDPO, we fine-tune the base model with supervised fine-tuning (SFT) only using the chosen responses from the preference dataset. We extend the code implementation from human-aware loss functions (HALOs) reposi- tory (Ethayarajh et al., 2023) to adapt to our methodology and borrow the hyperparameters from each of the above methods in our study.
4.1.1 Learning Dynamic with Rationales
Versus SFT. We carefully examine the dynamics of preference learning by studying how adding rationales to the current preference learning algorithms impact performance. We compare the responses generated by the preference-aligned model against the responses generated by the SFT model and measure the winrates scores using the GPT-4o evaluator. To
5

---

Published as a conference paper at COLM 2025
study data annotation efficiency, train the models various training data sizes, ranging from 1,000 to 12,000 data points, for both RDPO and DPO training. We observe on the left side of Figure 2 that both models, DPO and RDPO models, achieve better winning rate over the SFT model (more than 50%) on the Orca dataset with an increasing winning trend when training data increases. Additionally, we observe the DPO model converges to around 60% winning rate at the 9,000 mark against the SFT model, the RDPO model achieves this rate at a smaller training data size with 3,000 training data points. Furthermore, we observe that the RDPO model can reach an even higher winning rate against the SFT than the DPO model, reaching above the 66% winning rate. We see a similar observation with the models trained on the Ulttaaeeddbac dataset on the right side of Figure 2. While RDPO can increase the computation time due to the addition of rationales, the model trained on rationales can converge earlier with fewer data points than DPO. This is especially important as the cost of collecting human preference data is high. Thus, improving annotation efficiency can potentially save further training costs. Additionally, with enough computation, RDPO can reach a better model than DPO does.
Moreover, we observe for the case of the UltraFeedback dataset, with more training data, the performance of the DPO model decreases. This can be attributed to the prob- lem of DPO overfitting and exploiting length in longer responses Azar et al. (2024); Park et al. (2024). Indeed, the UltraFeedback dataset con- tains chosen responses that are longer on average (1,305 character length) than the re- jected responses (1,124 char- acter length). unlike the Orca dataset (784 and 978, respec- tively).
RDPO Winning Rate
DPO Winning Rate
Orca: Winrate VS SFT
ULtrafeedback122: Winrate VS SFT
Mith Varying Training Data Size
Mith Varying Training Data Size
Better Performance
0.875
Minrate
0.
1.000
A. 0.000
8:575
9.575
0.500
3x
Annotation Saving
2016 4920
0980
9900
19690
12900
2000
6800
2000
12000
Training Data Size
Training Data Size
Figure 2: Winrate comparison between the models trained with RDPO and DPO. Left: Winrate against the SFT model trained on the Orca dataset. Right: Winrate against the SFT model trained on the Ultrafeedback dataset. X-axis denotes the training data size used for preference training of DPO and RDPO models.
Versus DPO. While we used SFT as a proxy to compare the per- formance between the DPO- and RDPO-trained models, here we di- rectly compare the responses be-
RDPO Winrate
With Varying Training Data Size
0.675
tween these two models to mea-
0.658
sure the winrate for Orca and UI-
Winrate
0.625
traFeedback datasets. We choose DPO-trained model checkpoint for
0.600
each dataset, where the winrate of the DPO model against the SFT model has converged, which is at 11,000 and 12,000, respectively. In Figure 3, we observe that the model trained with RDPO gener- ates better responses on average than the model with DPO, even when trained with little as 1,000 data points. With increasing train- ing data, RDPO model improves the winrate to reach above 60% in
0.575
0.559
Orca
Ult Ultrabeedback
0.525
2008
4888
6888
8008
18800
12898
Training Data Size
Figure 3: Winrate of RDPO model against the DPO model on respective datasets, Orca on the left and Ul- frabeed on the right. The purple dashed line de- notes the 0.5 mark.
both datasets. RDPO-trained model generates more of the preferred responses than the DPO-trained model does, even with 10x fewer training points.

---

Published as a conference paper at COLM 2025
Adaptation to ORPO. To demonstrate the flexibility of our rationale-enriched preference learning frame- work, extend the ORPÒ preference learning algo- rithm Hong et al. (2024), which omits the SFT step, to include the rationales similar to the RDPO loss, and we call it RORPO. As shown in Table 1, rationales can en- hance the performance of ORPO and achieve a better winrate over the vanilla ORPO trained model. By suc- cessfully adapting rationales to both ORPO and DPO, we emphasize the simplicity of the framework as well as the effectiveness of rationales in preference learning. We further study the adaptation of these methods with ratio- nales and evaluate the preference traineed models on the instruction-following benchmark, AlpacaEval 2.0 (Li et al., 2023b; Dubois et al., 2024).
ORPO
44
56
RORPO
Table 1: Adapting rationales to the ORPO preference learning algorithm on Mistral-7B-v0.22 Instruct (Orca). Comparing the winrate of the ORPO- (Left) against the RORPO-trained model (Right).
Evaluation of the Impact of Rationales with AlpacaEval 2.0. Here, we study the flexibility of our method by extending different preference learning methods, such DPO (Rafailov et al., 2024) and ORPO (Hong al., 2024), with rationales. In this experiment, we evaluate the performance of the trained models on the automatic instruction-followin AlpacaBval 2.0 benchmark (Li et al., 2023b) with GPT-4-turbo as judge and report the raw winrate, the length-controlled (LC) winrate (Dubois et al., 2024), which is robust against the verbosity bias that the raw winrate inherently entails, and the average response length. We train the instruction-tuned models, Intel-DPO-Pairs dataset.
and 1lama-3-8B-Insttee on the
We observe in Figure 2 that DPO trained model improves the win- rates both models compared to the original model. Additionally, RDPO models further increase the win rates. We note that in the case of the Mistral model, the winrate de- crease with ORPO preference train- ing, which might be due to the lack of the reliance on the reference model, the behavior which is also observed in Ethayarajh et al.. Furthermore, af- ter adding rationales, the (LC) win-
Dlistall7B-Inntruutte
Hlama-3.-8B-Instree Winrate
WR
Winrate
Length
LC WR
Length
Original
17.11
14.72
1676
22.92
22.57
1899
DPO
19.52
15.81
1632
26.02
22.52
1759
RDPO
22.42
17.87
1627
27.55
23.69
1750
ORPO
12.84
12.24
1782
23.11
20.27
RORPO
20.45
16.27
1618
26.55
23.45
1758
Table 2: The performance comparison of the original model, ORPÓ trained model, and DPO/ORP with rationales (RDPO/RORPO) trained model on the AlpacaEval 2.0 benchmark.
rate not only increases but also surpasses of the original model. These results show the helpfulness of adding rationales into preference learning. Interestingly, we also note that rationale based models increase the winrates while their average response lengths are decreased compared to average response lengths of the original model, which is not similar observation as seen in some current methods, such as SPPO (Wu et al., 2024) or SimPO (Meng et al., 2024).
SimPO: Full Size Dataset Here, study the effectiveness of training with rationales on the full size dataset, and we adapt SimPO to incorporate rationales, and train the RSimPO model, accord- ingly. In particular, we train the LLaMA-3.1-8B. Instruct model on the UltraFeedback dataset and Anthropic HH dataset. After proper filtering, we are left with around 40K samples for each dataset. To show the effectiveness of RSimPO, we observe in Table 3 that RSimPO achieves higher winrate against the SimPO trained model, achieving over 65% winrate on both datasets agains the SimPO model. Furthermore, to show the efficacy of RSimPO, we train the model on half of the dataset, and we still observe better performance compared to the fully trained SimPO model on 40K for both data sets.
LLaMA-3.1-8B-
UltraFeedack
Anthropic HH
Dataset Size RSimPO Winrate 61
20K
40K
20K
40K
68
58
65
Table 3: The winrate of the RSimPO trained model on 20K and 40K dataset versus the SimPO trained model on the full 40K dataset.
7

---

Published as a conference paper at COLM 2025
4.2 Investigation of the Impact of Rationales
Rationale-Only Optimization. We study an the impact of the introduction of rationale term into the preference learning. To address this, we conducted series of experiments to isolate the impact of each component and evaluate their combined effect. Specifically, we investigated an extreme case where the rationale loss alone drives preference optimization, with the DPO alignment loss set to zero. This approach was based on the hypothesis that rationales inherently encode preferences by combining preference-response pairs, the preferences themselves, and the associated reasoning processes, thereby providing rich and effective training signal. For these experiments, we fine-tuned Mistral-7B-Innstructtvo. on the Orca dataset across three settings: RDPO (combining DPO and rationale loss), DPO (excluding rationale loss), and Rationale-Only (excluding DPO loss). The results, as shown in the Table 4, reveal that rationales alone can substantially improve model performance, achieving a high win rate of over 61% without explicit pairwise preference modeling, This improvement likely stems from the informational richness embedded in rationales, which compensates for the absence of pairwise alignment.
While DPO also demonstrated majority win rate against the SFT baseline, training with both ratio- nale and preference losses (RDPO) consistently achieves the highest win rate (64.5%) across both general and detailed settings. This highlights the benefit of integrating rationales into the preference objective, effectively leveraging the strengths of both losses to produce superior performance.
RDPO
DPO
Rationale-Only
General Detailed
64.5
59.1
61.8
59.1
Table 4: The impact of different compo- nents of RDPO by measuring the win- rate of the target model against the SFT model. The results on Mistral-7B-v0.2- Instruct model using the Orca dataset.
These findings underscore the complementary na- ture of the rationale SFT loss and the pairwise alignment loss. While DPO explicitly optimizes reward margins, the rationale prediction loss pro- vides supplementary supervision, enabling the model to learn the reasoning underlying response preferences. This integration not only strengthens the selection process but also accelerates training convergence. By combining these two approaches, RDPO amplifies their individual strengths, resulting in more efficient and effective preference learning.
Generated By
RDPO vs DPO Winrate
Mistral-7B (eirall-BB-iae Instruct-v0.2
76-23
50-46
Baama--8BIInstruce
73-27
52-45
Phi3-Mini-4K
75-25
Misttrl-7B-insttrutt-0222 Llama3-8B-Instruu
Trained On
Table 5: Studying the impact of different source of rationale generation (Y-axis) for the Orca dataset on the model training with RDPO (X-axis). Winrate of the RDPO model against the DPO model.
Given the prompt:
<prompt>
Which of the following assistant's responses is preferred and strictly follows the prompt question? Why this assistant's ss response:
<chosen>
is preferred over the response provided below: <rejected>
On a high and general level, why the response above is preferred over the response below? Provide general, high-level explanation for your reasoning wi whouu going into the response's ss details.
Prompt 1: Prompt to the LLM to retrieve general rationale.
8

---

Published as a conference paper at COLM 2025
API Rationale Cost
$0.13
$0.19
$0.26
$0.32
S0.39
Annotations Used
1K
1.5K
2K
2.5K
3K
Annotations Saved
3K
6K
6.5K
6.8K
210K
vs SFT Winrate
54%
56%
58%
60%
62%
Table 6: Cost-analysis breakdown of using RDPO with the OpenAl API (gpt-4o-mini model) to generate rationales, highlighting performance improvements relative to the number of annotations used and annotations saved.
Rationale Source. While collecting the human-annotated rationales could be high-quality, in practice, it is costly with time and resources. Therrefore, we resolve to language models to generate rationales. In our experiments, we use the base models to create them. Here, we study the rationales coming from other sources and how they impact the RDPO training. We generate rationales for the Orca dataset on three different models: and Phi3-Mini-4K Abdin et al. (2024). Then, use these rationales to train the first two models. Results from Figure show us consistent winrates against the DPO model with slightly better winrate from the same source as the base model. This shows that the rationales can be transferred to other models for preference training with rationales. Especially, leveraging models with small sizes (3,7.8} billion parameters, we can generate rationales to improve preference learning using Prompt 1. However, we observe modest improvements on the Llama-3.1-8B- 1-S--untttt experiment, which can attributed to two key factors. First, the inherent capability of Llama-3.1 1-8-1-188--nnttu surpasses v0.2, making substantial gains more challenging to achieve on this stronger baseline. Second, the general preference datasets like Orca and Ultrafeedback, which include pre-existing responses, may not be fully optimized for Llama-3.1 --nttttuttt...t
(Lamma--B-IInttree
Quality of Rationales. To investigate the quality of generated rationales, we incorporated the more capable LLM, GPT-4o, to evaluate the generated rationales terms of helpfulness, relatedness, correctness, and coherence. In practice, we applied two primary criteria to ensure the high quality and reliability of the preference data used for training. We discarded data points where the preferred response was rated lower than or equal to the rejected response. This step was crucial for eliminating inherently inconsistent or ambiguous human judgments. We retained only instances where the preferred response received a score of 5 or higher on a 10-point scale. This ensured that the 'preferred' choice genuinely represented a positive example. We further examine the importance of the quality of rationales for preference learning in Appendix C.4, studying how different types of rationales would impact preference learning-
Cost Analysis In this section, we analyze the cost breakdown to assist project owners in evaluating trade-offs. Specifically, we present a cost-benefit analysis of the approach. The table below outlines the cost of using the API to generate rationales for given number of annotations. It also highlights the RDPO win rate compared to the SFT model for each data budget and estimates the number of annotations potentially saved by using RDPO to achieve the same level of performance as DPO.
While wein-w models were used to generate the rationales in study, the table illustrates the associated costs of utilizing an API model, specifically gpt-4o-mini. The pricing for this model is $0.150 per 1M input tokens and $0.600 per 1M output tokens. The results shown in Table 6 are based on the Mistral-7B-v0.--Innttrre dataset. We refer the reader to Appendix C.7 for more details.
model trained on the Orca
A Simplified Theoretical Model for Rationales
To better understand rationale-enhanced preference learning, we employ machinaries from information theory to quantify benefits rationales provide in learning ground truth preferences under simplified assumptions. Formally, given query X, let the preference Z be binary random variable, with Z = indicating that response Y1 is preferred over response Y2, and = 0 indicating the opposite. Let R denote the rationale, S = (X, Y, Y2), and assume that the dataset D {S₁ R,, 7, " n² is sampled i.i.d. from distribution μ.
9

---

Published as a conference paper at COLM 2025
As intuitive first step, we quantify the benefits using the conditional mutual information between the true preferences and rationale-implied preferences given the input- pairs, ie., g[Z; ( (R)[s), where g(R) capture the preference inferred from the rationale R. Intuitively, this mutual information quantity characterizes the added value of rationales in understanding true preferences, measuring how much additional information rationales provide beyond what can be inferred from input-response pairs S. Our analysis demon- strates a closed-form relationship between rationale informativeness and its alignment with true preferences (see Appendix B.1 for detailed derivations of all results).
Next, we quantify the benefits provided by rationales by directly analyzing the generaliza- tion error for preference learning with and without rationales. Compared with analyzing the informativennss of rationales outlined above, this approach offers more direct mea- sure of the impact of rationales on learning outcomes. Note that the generalization error of preference learning still differs from the winrate metrics used in our experiemnts for evaluating the generation of preference-aligned models. However, they are intuitively connected under the DPO loss: the minimized test loss also indicates learning the optimal data generation policy that achieves the highest reward. We defer the detailed statement of the theorem and proof to Appendix B.2. In particular, our theoretical derivation shows that training with rationales can lead to improved lower generalization error when the rationale does not contain irrelevant information other than those predictive of the preference Z, and the learning process only captures the rationale information that useful to predict Z. Despite being derived from simplified model, our theoretical insights align well with experimental observations. For instance, our evaluation in Section C.4 demonstrates that detailed rationales, which may contain extraneous information, achieve lower sample ef- ficiency compared to more general rationales that focus on preference-predictve content; furthermore, when we manually inject noise into rationales to misalign them with true preferences, we observe hampered preference learning, as indicated by lower winrates.
6 Conclusion & Limitations
In this work, we explored and validated a data-centric perspective on direct preference learning for aligning language models. Standard approaches train models on preference pairs (preferred vs. dispreferred responses), but often lack explicit information about why one response is superior, leading to ambiguity that can hinder learning efficiency and require substantial, costly datasets. To address this limitation and extract more value from existing human preference data, we introduced the concept of augmenting these pairs with ratio- nales, which explain the reasoning behind each preference. We presented straightforward framework for integrating readily available, machine-gent seaebine-geerttedd rationales into existing direct preference optimization pipelines. Our extensive empirical evaluations demonstrate approach significantly enhances the learning process: it in-
that this aitionale-augmentee crease the effective use of each annotated data point, improves model convergence to higher performance levels, and helps mitigate undesirable behaviors. Furthermore, theoretical insights grounded in information theory corroborate the observed efficiency gains provided by rationales. Ultimately, our findings underscore the significant potential of thought- fully designing the structure of preference data itself. Enhancing existing clttasets with explanatory rationales offers a powerful and complementary strategy alongside algorithmic advancements for achieving more robust and efficient language model alignment. To date, we have integrated rationales into our training process and successfully trained models with up 8 billion parameters. We encourage further investigation into the impact of rationales on preference learning, particularly exploring larger models and datasets. To facilitate research this area, we make our code and datasets publicly available. With the development of unpaired preference learning algorithms, such as KTO Ethayarajh et al. (2024), it is important to extend the use of rationales to handle unpaired responses in future work, e.g., such as provided in the UltraFeedback dataset Cui et al. (2023) contains rationales for single responses without pairwise comparison. We also would like to study the impact of rationales on trained the reward models, which might have different impact than the critiques from existing literature.
10

---

Published as a conference paper at COLM 2025
Impact Statement
This paper presents a data-centric extension aimed at improving preference learning by enabling users to provide rationales for their choices, particularly in cases where such reasoning is not easily inferred by the model. By granting users greater control over data creation, this approach helps guide the model in effectively capturing user-specific prefer- ences. Through responsible data curation and generation, this method has the potential to significantly influence model behavior in real-time applications.
Acknowledgments
Ruoxi Jia and the ReDS lab acknowledge support through grants from the Amazon- Virggnnaa Tech Initiative for Efficient and Robust Machine Learning, the Cisco Award, the Common- wealth Cyber Initiative Cybersecurity Research Award, the National Science Foundation under grants CNS-2424127, IIS-2312794, IIS-2313130, OAC-223922,, and OpenAl API re- search credits.
References
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally your phone. arXiv preprint ar arXiv:2404.1422 14219, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arviv preprint ar xxXiv:2003.08774, 2023.
Al@Meta. Llama 3 model card. 2024. URL https:://gtthhi cww.meta main/MODEL.CARD. md.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et survey on data selection for language models. arXiv preprint ar arXiv:2402.1e 16827, 2024.
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. Xiv preprint arXiv:2402.10571, 2024.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. ar preprint ar Xio::6666006655 2016.
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan Daniel Chang, and Prithviraj Ammanabrolu. Critique-out-loud reward models. In Pluralistic Alignment Workshop al NeurIPS 2024, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arx preprint arXiv:2204.0586) 2022.
Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell- Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35:38176-383 3888199-8888,,,, 2022.
11

---

Published as a conference paper at COLM 2025
Quentin Bertrand, Wojciech Marian Czarnecki, and Gauthier Gidel. On the limitations of the elo, real-world games are transitive, not additive. In International Conference on Artificial Intelligence and Statistics, pp. 2921- PMLR, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3 4):32443333334 1952.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. ar Xiv preprint arXto:2307.15217 2023.
Xin Chen, Hanxian Huang, Yanjun Gao, Yi Wang, Jishen Zhao, and Ke Ding. Learn- ing to maximize mutual information for chain-of-thought distillation. arXiv preprint or drXi::2403.03348, 2024.
Michelene TH Chi, Nicholas De Leeuw, Mei-Hung Chiu, and Christian Va Eliciting self-explanations improves understanding. Cogmitive science, 18(3):439-477, 1994.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Kevin Crowley and Robert S Siegler. Explanation and generalization in young children's strategy learning. Child development, 70(2):304-316, 1999.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high- quality feedback, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arviv preprint ar 4rXiv:2404.04475, 2024.
Jacob Eisenstein, Daniel Andor, Bernd Bohnet, Michael Collins, and David Mimno. Honest students from untrusted teachers: Learning an interpretable question-answering pipeline from pretrained language model. In Workshop Trustworthy and Socially Responsible Machine Learning, NeurlPS 2022.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In Forty-first International Conference on Machine Learning.
Kawin Ethayarajh, Winnie Xu, Dan Jurafsky, and Douwe Kiela. aware loss functions (halos). Technical report, Contextual
Human- AI, 2023.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01300 2024.
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noach,, Haonan Li, Kyle McDonell Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopí, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo. http:///enndooo/rrrerrrrrdddd wtg/records/////oooooooogoogsssr 1esss/1122666...
Peter Hase and Mohit Bansal. When can models learn from explanations? formal frame- work for understanding the roles of explanation data. arXie preprint arXiv:2102.02201, 2021.
Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 14852-14882, 2023.
12

---

Published as a conference paper at COLM 2025
Jiwoo Hong, Noah Lee, and James Thorne. Reference-free monolithic preference optimiza- tion with odds ratio. ari preprint (xX::22433.06611, 2024.
Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Yhen-Yu Lee, and Tomas Pfister. Distilling step-by-step! out- performing larger language models with less training data and smaller model sizes. In Findings of the Association for Compulational Linguistics: ACL 2023, pp. 8003-8017, 2023.
Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 1051-1068, 2023.
Shawn Im and Yixuan Li. Understanding the learning dynamics of alignment with human feedback. ar Xiv preprint arXiix2203.18//1 2024.
Intel. Intel. 05-18.
2024. Accessed: 2024-
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. ar preprint ar Xrii:22310.06825, 2023.
Yuu Jinnai and Ukyo Honda. Anmotation-efficien preference optimization for language model alignment. arXiv preprint ar Xio::2405.11551,, 2024.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, 2017.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-222 222213,,, 2022.
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. Symbolic chain-of-though distillation: Small models can also "think" step-by-step. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2665-2679, 2023a.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models.
www.tatsu-laa//a//aaaaaoo// lalaa/aaaaaaaaaaaaaaaa eval, 2023b.
Aiwei Liu, Haoping Bai, Zhiyun Lu, Xiang Kong, Xiaoming Wang, Jiulong Shan, Meng Cao, and Lijie Wen. Direct large language model alignment through self-rewarding contrastive prompt distillation. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 9688-9712, 2024.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. arv preprint arXio:2212.08410, 2022,
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint ar arXi:22405.1433.
173,, 2024.
Tom M Mitchell, Richard M Keller, and Smadar T Kedar-Cabelli. Explanation- gener- alization: A unifying view. Machine learning, 1:47-80, 1986.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arx preprint ar Xri::2306.00700) 2023.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. ar preprint (XXi:23312000886 2023.
13

---

Published as a conference paper at COLM 2025
Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. Wt5?! training text-to-text models to explain their predictions. arx preprint WMXiv:2044.14544 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference generation for improved reward modeling. In ICLR 2024 Workshop on Navigating and Addressing Dala Problems for Foundation Models.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positie ppo-posiiive.. arv preprint ar Xii::2402.111 13228, 2024.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. Xre preprint arXiv:2403.19159 2024.
Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C Lipton, Graham Neubig, and William W Cohen. Evaluating explanations: How much do explanations from the teacher aid students? Transactions of the Association for Compulational Linguistics, 10:359-375, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4932-49422 2019.
Andrew Slavin Ross, Michael C Hughes, and Finale Doshi Velez. Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint arXiv:1703.03717, 2017.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint wxXfo:2404.03715 2024.
Daniel Russo and James Zou. Controlling bias in adaptive data analysis using information theory. In Artificial Intelligence and Statistics, pp. 1232-1240. PMLR, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint ar (tt:17077006447, 2017.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXio preprint arXiv:2401.04056 2024.
Zhen Tan, Alimohammad Beigi, Song Wang, Ruocheng Guo, Amrita Bhattacharjee, Bohan Jiang, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation: A survey. arx preprint ar X0:4244402., 2024.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
14

---

Published as a conference paper at COLM 2025
PeiFeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren. Pinto: Faithful language reasoning using prompt-generated rationales. In The Eleventh International Conference on Learning Representations.
Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught evaluators. CoRR, 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ar Xiu preprint ar Xii::2203.11171, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 55:24824-24837 2022.
Sarah Wiegreffe, Ana Marasović, and Noah A Smith. Measuring association between labels and free-text rationales. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10266-10284, 2021.
Joseph| J Williams and Tania Lombrozo. The role of explanation in discovery and generaliza- tion: Evidence from category learning. Cognitive science, 34(5):776- 806, 2010.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. aiviv preprint ar arXic:2405.00675 2024.
Aolin Xu and Maxim Raginsky. Information-thcoreti analysis of generalization capability of learning algorithms. Advances in neural information processing systems, 30, 2017.
Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. Rled: Rein- forcement learning from contrastive distillation for Im alignment. In The Twelfth Interna- fional Conference Learning Representations.
Zihuiwen Ye, Fraser Greenlee-Scott, Max Bartolo, Phil Blunsom, Jon Ander Campos, and Matthias Gallé. Improving reward models with synthetic critiques. arXiv preprint ar Xiv:2405.20850 2024.
Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, al. Self-generated critiques boost reward modeling for language models. ar preprint ar XrXr:22411.16646, 2024.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhí: Rank responses to align language models with human feedback without tears. arXiv preprint ar 0ete22304-05302, 2023.
Omar Zaidan, Jason Eisner, and Christine Piatko. Using "annotator rationales" to improve machine learning for text categorization. In Human language technologies 2007: The confer- ence of the North American chapter of the association for computational linguistics; proceedings of the main conference, pp. 260-267, 2007.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595- 46623, 2023.
Xin Zhou, Yiwen Guo, Ruotian Ma, Tao Gui, Qi Zhang, and Xuanjing Huang. Self- consistency of the internal reward models improves self-rewarding language models. Xiv preprint arXit:2502.08892, 2025.
15

---

Published as a conference paper at COLM 2025
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. Xiv preprint ar Xic:1909.08593, 2019.
16

---

Published as a conference paper at COLM 2025
Appendices
A Extended Related Work
18
Theoretical Derivations
B.1 Mutual information analysis
20
B.2 Theorem B.3: Generalization Bound
21
B,3 Supporting Lemmas
23
B,4 Proof of Theorem B.2 and Derivation of Regimes
25
C Additional Experimental Results
29
C.1 Experimental Details
29
C.2 Ablation Study on Models and Hyperparameters.
29
C33 Ablation on the Number of Epochs
29
C.4 Rationale Quality Analysis
30
C.5 Additional Results
31
C.6 Response Comparison
31
C.7 Log Probabilities.
................................
32
C.8 Cost Analysis
33
C.9 Rationale Generation
................................
33
C.10 Comparison of Rationales
34
C.11 Evaluating Responses with LLM Judge
35
................................
C.12 Evaluating Generated Rationale with LLM as Judge
36
C.13 DPO and RDPO Generated Responses
37
17

---

Published as a conference paper at COLM 2025
A Extended Related Work
RLHF with Reward Modeling. Tuning large language models to align their outputs towards human preferences is crucial for controlling model behavior and maintaining desirable boundaries (Casper et al., 2023). To achieve this, RLHF has been introduced; aligning models through preference training (Christiano et al., 2017; Ziegler et al., 2019; Stiennon et al., 2020). Schulman et al. (2017) describe a method that typically involves two stages. The first stage learns a reward model using a preference dataset often modeled under the Bradley-Terry model Bradley & Terry (1952). The second stage fine-tunes the target policy model to maximize the rewards from the reward model, employing algorithms such as proximal policy optimization (PPO) proposed by Schulman et al. (2017) and adopted in Ouyang et al. (2022). A direct preference optimization (DPO) method that implicitly models the reward function was introduced by Rafailov et al. (2024). However, Azar et al. (2024) observe that RLHF and DPO are prone to overfitting due to the assumptions of the Bradley-Teryy model. Conversely, Pal et al. (2024) explore the possibility of DPO underfitting when dealing with challenging responses that are difficult for the model to distinguish. Additionally, Park et al. (2024) note that DPO can exploit response length to maximize reward, proposing a reggah-regularized DPO (R-DPO) to address this issue. This should not be confused with our rationale-based DPO (RDPO) method. Interestingly, we observe that if rationales mention conciseness as feature, then the length of responses is significantly reduced compared to SFT and DPO responses. The learning dynamics during preference tuning are analyzed in Im & Li (2024), emphasizing the importance of high- quality preference datasets for effective learning. They find that the more distinguishable the response pairs, the easier it is for the model to learn, leading to faster convergence. This has also been observed in Pal et al. (2024). However, designing such datasets is challenging, and it also remains important for models to learn from such nuanced responses, which appear in practice. We try to address the difficulty of model learning from intricate preferences by providing rationales during preference training. To improve efficiency over DPO, which requires an intermediate step to train the reference model, odds ratio preference optimization (ÖRPO) was introduced by Hong et al. (2024) to eliminate this step. Another method by Ethayarajh et al. (2024) adapts the Kahneman-T Tversky human utility model to handle preference datasets with a single response (either chosen or rejected), removing the need for training the model on both responses. Conversely, Yuan et al. (2023) propose preference method that considers multiple ranked responses for prompt and optimizes over them. Our method can complement these methods by adding rationales into training. In this paper, we demonstrate an extension of our framework to ORPO.
General Preference Modeling. Reward modeling, however, can incentivize undesirable behaviors, such as "reward hacking" (Amodei et al., 2016), where agents maximize rewards without achieving the desired objective. Overfitting is another challenge, as exemplified in Azar et al. (2024). While effective for comparing two responses, the Bradley-Tene Terry preference modeling relies on the assumption of transitivity, which may not hold true in practice (Bertrand et al., 2023). To address this, Munos et al. (2023) introduced general preference modeling, which directly learns general preferences by formulating two-player, constant-sum game between policies. The goal is to maximize the probability of generating the preferred response against the opponent. The solution is the Nash equilibrium of this game, where payoffs are derived from the general preference function. Building upon this work, Munos et al. (2023) proposed an algorithm for the regularized general preference model, while Swamy et al. (2024) developed a solution for the unregularized formulation and introduced self-play preference optimization (SPO) as an iterative algorithm to reach the optimal solution. However, SPO suffers from data inefficiency due to its two-timescale update rules. To address this, Rosset al. (2024) introduced an efficient direct Nash optimization (DNO) method that leverages the DPO formulation in practice. P Additionally, Wu et al. (2024) proposed an efficient, scalable, iterative self-play method that generates responses generally preferred over others.
While previous efforts have introduced algorithmic enhancements for preference tuning, they have been limited to the existing framework of preference datasets with prompts and ranked responses. In contrast, our work is first to introduce rationales, a data-centric solution, into preference learning.
18

---

Published as a conference paper at COLM 2025
Learning with Rationales. The supervised learning framework typically involves training a model to learn the ground truth label for a given p without providing explicit explanations for the associations, which can lead to the model learning incorrect cues. To mitigate this issue, rationales have been integrated into the framework, offering explanations for the given associations. These rationales initially were generated by humans (Zaidan et al., 2007; Ross et al., 2017; Hase & Bansal, 2021; Pruthi et al., 2022). However, due to the high cost of human labor and the development of more capable large language models, rationales are now often automatically generated by these models, reducing the need for human involvement (Wei et al., 2022; Kojima et al., 2022). Given rationales, they have been used as guiding aids by incorporating them directly into the prompt during the training phase (Rajani et al., 2019; Zelikman et al., 2022; Huang et al., 2023) or the inference stage (Wei et al., 2022; Kojima et al., 2022; Wang et al., 2022). Besides using them as additional context within the prompt, rationales can also serve as labels to train models to generate such explanations for their predictions (Wiegreffe et al., 2021; Narang et al., 2020; Eisenstein et al.; Wang et al.; Ho et al., 2023; Magister et al., 2022; Li et al., 2023a). In similar manner, rationales have been applied in knowledge distillation, where they are generated by more capable models to supervise weaker models (Hsieh et al., 2023; Chen et al., 2024). In parallel with these advancements, we introduce rationales into the preference learning landscape, where rationales are used to explain the preference of one answer over another. Our findings demonstrate the effectiveness of rationales in preference learning, even when generated by the same model or a smaller-sized model.
Synthetic Preference Data Generation. Synthetic preference data generation plays a pivotal role in preference learning by creating new annotated datasets that capture user choices or preferences Yang et al.; Pace et al.; Meng et al. (2024). These methods focus on producing preference pairs that can serve as training data for models, enabling the exploration of diverse scenarios and reducing reliance on costly manual annotations. Furthermore, Wu et al. (2024); Wang et al. (2024); Meng et al. (2024) try to further synthesize the preference examples by iteratively generating new response pairs in on-policy manner. However, our approach diverges fundamentally from this objective. While synthetic data generation targets the creation of new datasets, our work emphasizes enhancing existing, fixed datasets with provided preferences pairs by incorporating rationales, thereby enriching preference annotations with explanatory depth. This distinction highlights the complementary nature of these methods: data generation addresses the early stage of creating foundational datasets, whereas rationale augmentation enhances the interpretability and utility of existing data. Attempting to compare these approaches directly would obscure their unique contributions. Instead, their synergy lies in how synthetic data generation can produce the raw preference pairs that are later refined through rationale augmentation, advancing the overall preference learning pipeline. Exploring how these two approaches can be combined to create synthetic datasets enriched with rationales is an exciting direction for future work, holding promise to further enhance the capabilities and generalizability of preference learning models.
19

---

Published as a conference paper at COLM 2025
B Theoretical Derivations
We begin with defining several standard quantities to be used throughout this section. Definition B.1. Let X, Y and Z be arbitrary random variables, and let DKL denote the KL divergence. We denote Px as the marginal probability distribution of X, and Pyjx the conditional distribution.
The entropy of is given by: H(X) P(X x) log P(X x).
X is a binary variable with p = P(X = 1) = P(X = 0), then we use H(p) for H(X).
The joint entropy of two random variables, Y),, is the entropy of their joint distribution. The conditional entropy of given Y, (X)), is:
H(X H(X, H(Y).
The mutual information between X and Ys:
I(X;Y) = DRL DxI
The disintegrated mutual information between X and Y given is:
(x;)) DxI DxI 1
The corresponding conditional mutual information is given by:
T(X;YIZ) (x;))
all entropies involved are finite, it can shown that I(X;Y) = H(Y) - X
B.1 Mutual information analysis
Formally, given query X, let the preference binary random variable, with
indicating that response Y1 is preferred over response Y2, and Z = 0 indicating the opposite. Assume that the aationale-impliss preference indicating rationale that supports Y1 being preferred, and R = 0 otherwise. For example, the rationale mentions that Y1 is more concise and informative than Y2, then R 1. However, there can be cases where R + Z, the rationale may not always align perfectly with the actual preference.
a binary random variable, with R
For the analysis, we consider the following model: 1) The rationale depends on the query- response (QR) pair S = (X, Y2) and the preference Z,, and is characterized by parameters and a, where: β = PRR 1 of consistency, and a = P(R = 1| = due to inconsistency. 2) The preference Z is modeled as P(Z 1[S) = f(S) + e, where f(S) captures the preference based on the observable query-response pair S, and the additive noise term human preference that are not captured in S. The term is referred to as "bias" in the main text that accounts for the difference between the true preference and prediction based on the query and responses alone.
P(R 0/7. 0) represents the precision rate 0|2 S) represents the recall error
simple way account for unobserved factors influencing the complex
Theorem B.2. Under the given assumptions, the mutual information I(Z; R(S) is given by:
B(p +e)
H(p + e) (p(p + +e) a(1 -6))) ((p+t) + a(1 p E)
a(p (()+e)
(1
p e))) a(p+t)
+)++ + p+1
20

---

Published as a conference paper at COLM 2025
where p = f(S). The mutual information [[Z) satisfies the following properties in three distinct regimes:
1. Uninformative rationale regime: If 0.5, then [(Z;
Maximally informative rationale regime: If ß and = 0, then [(Z;R[S) = H(p +e).
3. Moderately informative rationale regine: If ß = 0.5 + y and 0.5 Y, where 0 < < 0.5, then [(Z R|)) increases with "y, ranging from when = 0 (uninformative rationale) to H(p 1epp+e +) when Y 0.5 (maximally informative rationale).
The theorem highlights that the potential benefits of including rationales in the training process for preference learning tasks may vary in different regimes.
Highly informative rationale (B 1 and information is solely determined by the entropy of the preference prediction from the query and responses, (()(S) + c). Let us interpret f(S) as the query-response-deennde dependent) confidence generator that only depends on S, and с captures the idiosyncrasies preference. If the true probability P(Z= 1|s) = f(S) + e is less than 0.5, i.e., Z is most likely to be O, a positive e >> means that the true probability P(Z = 1[ is less extreme than the confidence score f(S) as it gets closer to 0.5 with increasing On the contrary, negative e < 0 means that the true probability P(Z 1|5) is more extreme than the QR-dependent confidence f(S) as it gets closer to 0 with increasing /e|.
Regime 1:
O): In this case, the mutual
(QR-
such as unknown confounding factors that influence the probability
From the perspective of the QR-dependent confidence generator f(S) that tries to explain the preference based on QR pairs, positive c > 0 would make it look more confident than it should be, i.e., overconfident, while a negative e < would make it less confident (or more conservative) based on the QR sequence than it should be.
it is likely to be overconfident based on the QR pairs, i.e., <> 0, then the more positive e is, the more risk there is of being overconfident in the QR-dependent predictor f(S). In this case, there is a lot of mutual information in (Z; R)S), so having rationales can "soften" " the potential overconfidence by bringing additional information other than the QR pair, which would otherwise occur in QR pairs alone, as in traditional reward modeling. Similar analysis holds when P(Z = 1|S) = f(S) e is greater than 0.5, i.e., Z is most likely to be 1.
Key message: Rationales are most useful when the reward modeling based on QR alone tends to have bias (i.e., overconfident).
Regime 2: Uninformative rationale (6 = a = 0.5): In this regime, the rationale provides no additional information about the preference, and the mutual information [(2; R)S) is zero.
Regime 3: Moderately informative rationale (high precision & 0.5 and low recall error &= 0.5 % where < y < 0.5): In this regime, can shown based on dereivative increases (more informative rationale), the terms involving 7 in the
analysis that numerators and denominators of the conditional entropies become more prominent. The mutual information will increase with y, the rationale becomes more informative about the preference.
B.2 Theorem B.3: Generalization Bound
Next, we analyze the sample complexity of training language models with and without rationales to predict preferences. We consider two regimes: 1) Training with rationale: Let fra Ara ()) Pe..ID denote the parameters of the language model trained to predict given S and R. 2) Training without rationale: Let 6un = Aun (D.,)) ~ P denote the output parameters trained to predict Z given only S, where Di is a dataset D with rationales removed. Given a loss function é that measures the prediction of preference Z, the (mean) generalization error is (H, is the expected loss the true distribution (true risk) and En[[(()) is the empirical risk.
ED,0-A(()) [()) ED (O)] where Fy [{())]
We introduce the following conditions on the relationship between S, R, and Z, and the learning process: 1) (R()) 1/1 and R) < 1/2, i.e., the rationale R is informative about
21

---

Published as a conference paper at COLM 2025
Z (small 172) without excessive irrelevance (small 1/1) 2) I( SIZ, S 8 for some small positive constant 6, i.e., the learned model Bma does not capture much additional information from S beyond what Z and R already provide. Condition 1 is supported by effective procedure to generate useful rationale R. To justify Condition 2, if the learning algorithm is designed to focus on capturing the information in R, which is highly informative about Z (per Condition for small 1/2), we can show that the model 6ra can accurately predict Z without needing to capture much additional information from S beyond what already present in Z and R. We provide rigorous but partial justification in Appendix B.3.
Theorem B.3 (Generalization bounds). Suppose the loss function l is o-subgaussian under the true data distribution. Under conditions and 2, we have:
20
gen(H,, Ama) S V (I(0na Z) + 5+ -17)),
(3)
202
gen Au)) S
(1(0um 2.) + I(0umiS|Z)) SIZ))-
(4)
The proof relies on the mutual information-based generalization bounds (Russo & Zou, 2016; Xu & Raginsky, 2017) and the decomposition of the mutual information terms for both training regimes using the chain rule (see Appendix B.2). The terms I(fra; Z) and I(Puni Z) can be expected to be similar as long as both regimes achieve good prediction of Z. Under the conditions of the theorem, we can observe that the sample complexity reduction depends on the gap between [fu S)Z) and 5 + 1/1; training with rationales can lead to improved sample efficiency when the rationale does not contain irrelevant information other than those predictive of the preference Z, i.e., 1/1 is small, and the learning process only captures the rationale information that is useful to predict Z. The theoretical insights are supported by our experimental results. For instance, our evaluation in Section C.4 demonstrates that detailed rationale achieves lower sample efficiency compared to more general rationales, which contain less irrelevant information beyond what is predictive of the preference; furthermore, we showed that irrelevant rationales, i.e., large value of //1, indeed hamper learning.
For the proof, recall that given a loss function L, the (mean) generalization error is A)) = - [{)) true distribution (true risk) and ED is the empirical risk. For fair comparison between gen and gen( jt,
ED[ where [p[((() is the expected loss on the
gen(u,,A)r.. Ara,, some technical nuances arise. The key difference from the typical setup in Xu & Raginsky (2017) is that the true data distribution ş includes the distribution for the rationale R, but the training regime gen(j,, this information. However, it may seem unclear initially whether we should include that in the generalization bound, since R is indeed generated based on Z, corresponding to the true Markov chain: S + Z + R.
does not explictly use
To clarify, the Markov chain for the training without rationale is: - + R, with additional arrows Z + Oun and 5 + Oun, but no arrow from R to Pun.
Intuitively, we should account for this difference by arguing that, conditioned on the preference 7, the learned model Oun is conditionally independent of R. However, due to this difference, it seems prudent to reason from first principles.
Let's start by choosing the distributions P and Q for the Donsker-Varadhan variational representation of the KL divergence. We set P = Ps,R. and Q = n² & Pamm where y is the distribution for (S, R, Z). Then, for any measurable function f, we have:
Q(P 2 R, - log E (0, (5,R.229)) (
(5)
Now, choose f(s, R, 0) = (() (0)) for some X E R, where (0) is the em- pirical loss on the dataset D and f. is the expected loss under the true distribution μ. Substituting this into (5), we get:
Q(P[))= 2 (0) - (e))
log (10.R.20)--(e((660 (,00)
120²
> A(B((D(O)] - (())(()))
(6)
2n
22

---

Published as a conference paper at COLM 2025
where the second inequality follows from the fact that (p( 0/ /t/ Vn-sugaaussinn under Q due to the subgaussian assumption on the loss function.
As (6) holds for any À E IR, it must also hold for the A that minimizes the right-hand side. This minimum occurs at A n n(E((e (0)) (()) yielding:
@(PIIQ) 2 202 [D El (())
Taking the square root of both sides, we have:
20
gen Aun) 1 V D(P(IQ).
The key observation here is that Ps conditionally independent of R given and Z in this training regime. Therefore,
(s, since 8un is
(Pl DKI. (Ps, || @ Paun)
DKL (Pomls, ps,z [R @ Poun)
T, um),
where Mi denotes the marginal distribution of μ over S and Z (i.e., excluding R).
Hence, we have:
20
(un) S
Note that we use I(S, un) instead of I(S, R, Z, am,, which is the key difference from the typical setup in Xu & Raginsky (2017), where the learned model is assumed to depend on all the data.
We can then arrive at the result for training without rationale by noting:
((0um:S2)) 1(0umi 2) I(0uni S|Z).
For training with rationale, we have:
I(Ora; S, R, I(frai Z) + I(fra; R + I I(fra:
< [(Orai 2) + H(R(Z) + 1(0ra;S\Z,R)
S [(Ora: Z) + 1/1 +,
where the first inequality due to I(Ora; R(Z) < H(R Z),, and the second inequality follows from conditions and 2. We can now apply [Thm. 1|Xu & Raginsky (2017) to yield the result.
B.3 Supporting Lemmas
Lemma B.4. Let be the estimate of Z based on 0. Let P. = P(Z * -212,0) be the probability of error in predicting Z using Suppose P. 6. Then, have that:
Pe z HZ)) T(R; 8) ( H(e)
log Z
0, 2 Z
Proof:
First, let's define an indicator variable E for the enror event: E
We
1, * Z
have
H(Z = -
H(E)2,0) H(2)E,2,))
-H(P.) P2H =,,2,0 + (1 P.) 0,
H(c) + log ZZ).
23

---

Published as a conference paper at COLM 2025
Hence, have
H(Z H(e) H(Z) 1(Z,0) H(C)
P. Z
(8)
log [Z|
log [z|
This part of the proof follows from Fano's inequality.
Now, since I(Z;0) I( I(Z,, R) have
1( (Z2, I(
(RR;0) + [(Z - [RR
< I(R;8) + +(Z;8|R)
Plugging in (8) obtains the result.
Note: I(Z; 0 R) 0 if 0 is trained only R, i.e. Z and conditionally independent given R.
Lemma B.5. Let 2 be the estimate of Z based on 0. Let Pe = P(Z + 2/2,0) be the probability of error in predicting using Assume that Pe S 0.5. Then, we have that:
P. S H(Z) (R (R)
(9)
log2
Proof: By definition of E from Lemma B.4, we have
H(Z = H(Z]2,0) H(Z,E)2,0)
H(E H(Z|E,,2,0)
2 H(E)2,0)
Pe log((P/ + (1- (11]))lo111 P.))
2 logg
Hence, we have P. < H(Z log2. Since H = H(Z) - I(Z:0), and that
T(Z:0) (7, T(R:0]Z)
I(R;0) [(Z;6]R) I(R;OIZ)
2 ((R;0) H(R/Z) + H(R(0,
2 ((R;6) H(R[Z),
the bound follows.
Partial justification of Condition 2: Consider the Markov chain - - with additional arrows of Z 0 and R where Z is the preference, R is the rationale, and 0 is trained model used to predict Z. From both Lemma B.4 and B.5, we see that as long as captures the information of R, i.e, [RR; 8) large, and does not contain excessive irrelevant information other than Z, i.e., H HR[Z) is small, the prediction error of Z from model can be well-controlled. Specifically, based on the lower and upper bound analysis, we can conclude that the probability of error Pe decreases with increasing I(R;0) or decreasing H(R]Z). This implies that the model does not need to capture additional information from S to achieve high prediction accuracy for Z, i.e., 1(Ora: SI,, R) is small. In other words, the incorporation of in the training of era guides the model to easily predict Z without resorting to finding potentially irrelevant information from S. A full justification of the condition hinges on a detailed analysis the specific algorithm and is beyond the scope of this study.
24

---

Published as a conference paper at COLM 2025
B.4 Proof of Theorem B.2 and Derivation of Regimes
Recall the definition of a, f.) in Sec. B.1. Now, we derive the relationship between the mutual information [Z; R[s) and these parameters:
P(Z = 1| P(R = 1 = 1, 1|S)
P
P(R = 1 = - 1|5) =
(01) P(R = 2,
B(f(5) +c)
B(f(s) +e) + a(1 (
Similarly, we have
a(f(5) +
P(Z =
a(f(S) + +) f(1)-
These equations show that the probability of Z given the query X, the preferred response Y1, the dispreferred response Y2, and the rationale depends on both the informativeness of the rationale, through and B, and the informativeness of the query and responses, through f(S). Using the above equations, we get the conditional entropies as follows:
B(f(S) +
H(Z|S, R = 1) = H
((f(()) +c) + a +c) f()) -())-))
a(f(s) +
H =0) = (a(((S) + + B(1 f(S) E)
and substituting to the mutual information equation, we get:
I(Z;R|S) H(Z]))
Σ P(R = (()== = [0,1]
Then, we compute each probabilities P(R[S) as follows:
P(R 1|5) = Σ P(R 1 = zZS)P z|S)
=E(0.1}
= B(f(S) + + a f(S) -c),
P(R = 0 1 P(R =1{))
1 (B(f(S)+ +e) +a(1 f(S)
and substitute them back into the conditional mutual information term, where we define p f(S):
5(p +e)
[[2;R|S) = H(p -1-----0--0000--00-0------------ B(p + + a -p -
a(p +
- (1- (p(ppp++e) + +a(1-p-e)) a(p+e) B(1 p11-ppp-
To study the influence of the parameters a, f, e on the mutual information, we consider the following edge cases:
Regime 1: Highly informative rationale / x and low noise a R 0 = Rationale sufficient statistics. In this case, the conditional probabilities are simplified to
P(Z 1|s, 1) f(S) +
1,
f(S) +e
0 0.
P(Z = 1 = 0) 1 1- 1- ((S)
25

---

Published as a conference paper at COLM 2025
Thus, the mutual information becomes as follows:
I(Z; H(f(s) P(R (1)
- -P(R = ((S)H(() - (()(() +c).
In this regime, the conditional mutual information is solely determined by the entropy of the preference prediction, (f)(s) +e). We notice that the entropy function is concave and reaches the max value at the 0.5 mark.
Regime 2: Uninformative rationale & ~ 0.5 and high noise a R 0.5.
For this case, the conditional probabilities become:
P(Z 1|5, R 1) f(S) +
+ f(s)) -c= P(Z 1|5, 0)
and the mutual information equals to:
I(Z: H(f(S) +e) H(f(s)+
which shows that the rationales provides 0 information about the preference given the prompt and responses.
Regime 3: Moderately informative rationale 5 = 0.5 + and lower noise a = 0.5 7.
Given the assumption of ß = 0.5 + and a = 0.5 - y, where < y S 0.5 and y denotes the level of informativeness of the rationale, we substitute this into the conditional mutual information term.
We first substitute into the conditional probabilities and get:
+ ())
P(Z = 1 = 1) =
(0.5 + (05
(0.5 (0.5-))((()++)) +e)
P(Z = 1|5, R = 0) =
(0.5 -
f(S) E)
Then, we compute the following probabiliie:::::
P(R 1|5) Σ P(R 1 = 2,5)P 2|5)
zc(0,1)
= 0.5+ 2y(f(S) + -
P(R = 0|S) = 0.5 --5--2y(((5) 2y(f(5)+cc
Now, we can compute the conditional mutual information term:
[[Z;R[S) = H(f(S)++ +e)
(0.++))(($S++
-00.5 + 2y((() + 0.5)) H
(0.--))((S)+ +e)
- (0.5 - (05-22(((S)+++) - 0.5)) H
(10)
We can now analyze the behavior of the mutual information as a function of y::
When the rationale is uninformative
0, then the mutual information becomes 0, 5) = 0, which is consistent with previous cases, in which uninformative rationales provide additional information about the preference Z demonstrated in Figure 4.
26

---

Published as a conference paper at COLM 2025
1(2,R,))
Mutual Information
0.2-
0.2
0.4
Gamma
Y
Figure 4: The plot of Equation 10 showing the relation between mutual information and gamma y for fixed f(S).
H
0.2
Entropy
0.14
0.1
0.2
0.4
0.5
Gamma y
Figure 5: The plot of Equation 10 showing the relation between the first entropy term and gamma y for fixed f(S).
As rationale becomes more informative about the preference by increasing y, observe that mutual information also increases displayed in Figure 4.
Consider the case that the true probability P(Z 1|S) f(S) + 0.5, the preference is most likely to be 1.
Then, now focus on terms in R(Z) that contain yy As y increases, the first entropy weight term (0.5 + 2y(f(S) + - 0.5)) increases and the second entropy weight term (0.5 (f( + c 0.5)) decreases. Entropy terms also involve 7, and observe that as increase, the ratio grows faster than the denominator. Thus, the first entropy term decreases with y, as the entropy of distribution deterministic one is lower.
(055+)((((S++))
approaches 1, since the numerator
27

---

Published as a conference paper at COLM 2025
H
0.4
Entropy
0.2
0.2
0.4
0.8
Gamma Y
Figure 6: The plot of Equation 10 showing the relation between second entropy term and gamma y for fixed f(S).
Conversely, for the second entropy term, with the increase of Y the ratio (05-))
(0.5 -100111111110055-111111111 approaches O, so the second entropy term also decreases with y.
For the first entropy term, with an increase of y, the weight of the first entropy term increases, but the entropy decreases itself (see Figure 5).
For the second entropy term, with increase y, the weight of the second entropy term decreases, and the entropy decreases itself (see Figure 6).
The net effect the mutual information depends on the relative magnitudes of these changes. However, we can argue that the decrease in the entropy terms dominates the change in their weights due to the entropy function changing more rapidly near the extremes (i.e., when the distribution is close to being deterministic) compared to the middle range.
Thus, with increase in y, the overall contribution of the entropy terms to the mutual information decreases, causing an increase in [ZZ RJS), which indicates that as the rationale becomes more informative about the preference, the mutual information increases.
28

---

Published as a conference paper at COLM 2025
C Additional Experimental Results
C.1
Experimental Details
For DPO-based methods, we fine-tune the base model by supervised fine-tuning (SFT) with the chosen responses from the preference dataset for single epoch. For ORPO, which avoids the reference model, we skip this SFT step. For models trained on RDPO and results reported in Section 4, we use y = 2.0, and for RORPO, we use 7 = 10.0. Similar to baseline methods, we train RDPO and RORPO for epoch. We perform ablation studies on the hyperparameter y and the number of epochs the following sections.
For our winrate scores, report the mean winrates after querying the evaluator 3 times. To reduce the evaluator's order bias, we have additionally shuffled the order of responses. We note that the winrate error bars are within 3% for 512 samples.
C.2 Ablation Study on Models and Hyperparameters
Mistral-7B-v0.1
General Detailed
62
61
63
61
62
60
64
66
63
61
60
62
Y
1.0
1.5
2.0
2.5
3.0
10.0
Mistral-7B-v0.2-Innttte
General Detailed
55
62
57
55
57
56
56
56
57
60
57
59
Y
1.0
1.5
2.0
2.5
3.0
10.0
Zephyr-7B-
General Detailed
58
55
55
57
55
57
48
49
50
51
Y
1.0
1.5
2.0
2.5
3.0
10.0
Table 7: The impact of different values of hyperparameter 7 on the winrate of the RDPO model against the DPO model. The results on various models: Mistral-7B-v0.1 (Top), Mistral-7B-Innttructv002 (Middle), and Zephyr-7B-Bets (Bottom).
Here, we investigate the impact of the hyperparameter 7, ranging from 1.0 to 10.0, on the performance of the model trained with RDPO loss. We provide the winrate scores against the DPO model on the Orca dataset. As we see in Table 7, models trained on either general or detailed rationales can still achieve stronger winrate against the DPO model, except for the case of Zephyr-7B- BB-Bela model, which achieves draw with the DPO model. For this model, the better quality rationales are important for effective preference learning, the general rationales can still improve the performance.
C.3 Ablation on the Number of Epochs
General
Detailed
Epoch 76
74
Epoch 2
75
71
Epoch
74
74
Table 8: The analysis of the number of epochs on the performance. Winrate of the RDPO model trained on the general rationales (left) and detailed rationales (right) against the DPO model, respectively.
29

---

Published as a conference paper at COLM 2025
In the main paper, trained the models with the RDPO loss single epoch similar to DPO (Rafailov et al., 2024). Here, we study the impact of training the models with the epochs. As we observe in Table 8, training with more epochs does not improve the winrate of the RDPO model against the DPO model. The reason could be that the model has already learned the preference well after the first epoch, or it could be that the quality of rationales could be further improved to increase the efficiency of the rationale-based preference learning algorithms.
rationales
C.4 Rationale Quality Analysis
In this section, we examine the importance of the quality of rationales for preference learning. We study different types of rationales and possible errors encountered in rationales, and how these affect the preference learning of the model.
RDPO General Rationales
RDPO Detailed Rationales
DPO
Orca: Winrate VS SFT With Different Rationale Types
Orca: Winrate vs DPO
With Different Rationale Types
0.675
3.050
0.650
0.625
0.625
Winrate
Winrate
0.000
0.600
0.575
0.575
>2x Annotation Saving
3.550
8.525
Training Data
Size
Training Data Size
Figure 7: Measuring the impact of types of rationales on the RDPO performance. Left: Comparing the winrates against the SFT model. Right: Comparing the winrates against the DPO model.
Detailed Rationales General Rationales. Explaining why one answer is preferred over the other can be expressed in multiple ways through many perspectives. Here, we study the level of granularity of the rationales, general (which explains the preference at high level without going into details) and detailed (which explains in details and pinpoints specifically to the prompt and the response). We use language models automatically generate the rationales for the Orca dataset according to our intent. For details on generation prompts, we refer to Appendix C.9. We provide samples of these rationales Appendix C.10. After training the models respective rationales, trained models and the DPO one. Figure on the left shows that the model trained on general rationales with the RDPO loss converges earlier to high winning rate against the SFT model than the model trained on the detailed rationales. The reason could be that the general rationales share common features across the samples (e.g., clarity, conciseness, directness), which lets the model learn quickly and transfer these learning cues to other samples more easily, while detailed rationales might require more time fully comprehend them. However, in both cases, the models trained on these rationales reach better winrates than the DPO against the SFT model. In Figure 7 (right), both RDPO models can have better winrate > 57% against the DPO model with as few as 3,000 training samples, while the DPO model trained on 11,000 samples. We provide results on models trained on additional epochs in Appendix C.3.
compare the winrates between RDPO
Low-Quality Rationales. RDPO has shown efficacy with rationales generated by the off-the-shelf models, even when the models have not undergone preference alignment. However, we want to further analyze the impact rationale's quality on RDPO's per- formance. In particular, we examine the rationale quality in terms of its relevance and
30

---

Published as a conference paper at COLM 2025
Permuted Rationales VS Original Rationales
Opposite Rationales VS Original Rationales
99
10
87
Table 9: The analysis of quality of rationales on the RDPO performance. The winrate comparison between the RDPO models trained on rationales with errors and original rationales. Left: Permuted, irrelevant rationales. Right: Opposite, inaccurate rationales.
correctness. One case of low-quality rationale can be a completely irrelevant rationale to the given pair of responses. To simulate irrelevant rationales, we permute the above- mentioned detailed rationales over different samples so that no rationale is relevant to the context. Training the model on these rationales with RDPO and comparing on original rationales, we show in Table 9 that it achieves less than 1% winrate against the RDPO model trained correct and relevant rationales. To study the impact of correctness, we negate the general rationales to have the opposite meaning and observe that the RDPO model trained on original rationales gets almost 90% winrate. As we note, the quality of rationales is important for improving the preference learning performance, While we showed that rationales generated by the off-the-shelf language models can already bring significant improvement to preference learning, we expect that more deliberate control of the rationale quality can further improve preference learning. We leave the in-depth exploration of strategies for generating quntity-controlled rationales to future work.
trained
Quality of Generated Rationales. Here, we provide the quality of generated rationales judged by the GPT-4o in terms of helpfulness, relatedness, correctness, and coherence. As we observe in Table 10, the rationales generated are of good quality. We also shuffled the rationales to random prompts and also changed the meaning of rationales for sanity check of our evaluator and we observe significant drops in the scores. Please see C.12 for prompting details.
Helpfulness
Relatedness
Correctness
Coherence
Generated Rationales
4.99
4.99
5.00
Shuffled Rationales
3.22
1.98
2.20
Changed Meaning Rationales
2.14
2.63
1.58
Table 10: The quality scores for generated rationales from the LLaMA-31-9DD-Intrruud model for UltraFeedback dataset. The scores are in the range from 1-5.
C.5 Additional Results
We provide comprehensive results AlpacaEval 2.0, Arena-Hard 0.1, and MT-Bench for the models corresponding to the Table below:
LC Winrate
Winrate
Length
Arena-Hard 0.1
MT-Bench
Original
17.1
14.7
1676
12.6
7.6
DPO
19.5
15.8
1632
15.0
7.7
Rationale-Only
20.8
16.2
1630
15.2
7.5
RDPO-General
22.4
17.9
1627
16.2
7.6
RDPO-Detailed
23.2
18.6
1636
16.4
7.8
Table 11: Additional Benchmark Results
C.6 Response Comparison
We compare the responses generated by the DPO- and RDPO-trained models. As shown in Table 12, the average output length by the DPO trained model much longer than the RDPO
31

---

Published as a conference paper at COLM 2025
Avg Output Length
'IriviaQA (Exact Match)
DPO
RDPO
DPO
RDPO
Orca
2021
1364
34,9
35.7
UltraFeedback
2066
1299
31.5
33.1
Table 12: Comparison between DPO and RDPO. Left: The average output lengths of the generated responses the prompts from the test sets of respective datasets. Right: The exact match (EM) performance on the TriviaQA dataset of the preference-trained models on respective datasets.
trained model in the case of the Orca dataset, which more than 60% longer on average. Due to longer output, there might be a chance for a higher occurrence of hallucinations. Therefore, we want to study the correctness of the outputs from these models. For this reason, we use the TriviaQA (Joshi et al., 2017) dataset and using LM Evaluation Harness (Gao et al., 2023) to measure the exact match (EM) accuracy and compare between the models. We see in Table 12 that models trained with the DPO loss experience decrease in performance, compared the models with RDPO loss. As reason, we emphasize the importance of measuring the hallucinations in the generations of both models in future studies. We provide a comparison of the responses from the two models Appendix C.13 and time cost analysis in Appendix C.7 to help model owners better understand the trade-offs of our method.
C.7
Log Probabilities.
To further investigate how rationales may boost preference optimization, we take a deeper look into log probabilities of the preference pairs. Specifically, we take a look the difference of log probabilities between the chosen and rejected response on RDPO trained Llama-3.1- 8B-Instruct model with Orca dataset.
Orca: Log Probability Margin Between Preference Pairs (chosen rejected)
480
RDPO
DPO
200
Log Probabilitt
200
-300
-400
0
200
400
600
800
1090
Example
Figure 8: The log- proobabbayyytyy margin between preferred and rejected responses. A compari- son of the log probability margin of responses of a standard DPO-trained model against our Rationale DPO model. Y-axis denotes the margin as margin as logP(preferred response) -reee(reeected response). X-axis represents 1000 individual examples from the training set, which have been sorted in ascending order according to their calculated log-probability margin.
As shown in Figure 8, at the tails of the plots, where the margins are large, DPO yields higher margins than RDPO. This suggests that DPO may be exploiting certain biases or engaging in reward hacking to maximize reward margins. In contrast, RDPO appears to act as
32

---

Published as a conference paper at COLM 2025
regularizer, mitigating such exploitative behavior. Interestingly, at the 0 level-whers the margins are negligible, indicating that preference pair responses are difficult to differentiate, DPO remains relatively flat, whereas RDPO crosses the 0 line at steeper angle. This suggests that RDPO more effective at handling and resolving ambiguous cases compared to DPO. The investigation into the impact of rationales preference learning suggests that they may serve as valuable cues, potentially acting as regularizer to enhance preference learning.
C.8 Cost Analysis
In this section, we analyze the cost breakdown assist project owners evaluating trade- offs. Specifically, we present cost-benefit analysis of the approach. The table below outlines the cost of using the API to generate rationales for a given number of annotations. It also highlights the RDPO win rate compared to the SFT model for each data budget and estimates the number of annotations potentially saved by using RDPO to achieve the same level of performance DPO.
While oeen-weight models were used to generate the rationales in our study, the table illustrates the associated costs of utilizing an API model, specifically gpt-4o-mini. The pricing for this model is $0.150 per 1M input tokens and $0.600 per 1M output tokens. The results shown in Table 6 are based on the Misstrl-7B--vO2--nstttue model trained on the Orca dataset.
We also report the runtime for RDPO and DPO for one epoch on the Mlama-3.1-BBIInstruu model using 12,000 Orca examples, as follows:
RDPO General: 6770 seconds
RDPO Specific: 6950 seconds
DPO: 3583 seconds.
While processing additional tokens nearly doubles the runtime, RDPO compensates for this by requiring fewer annotations while achieving comparable superior performance to DPO. Furthermore, we report the average response lengths for the Orea dataset, highlighting that rationales are approximately 50% shorter in length compared to chosen rejected responses:
Chosen responses: 786
Rejected responses: 981
Rationale responses: 411
C.9 Rationale Generation
Here, we provide the prompts to generate detailed and general rationales as mentioned in Section 4. In Prompt 1, we ask the model to generate general rationales without going into details and provide high-level reasoning. In Prompt 2, we ask the model to provide specific rationale and go much into detail as possible. These our examples of prompts and we call for further studies to improve the prompts for higher quality rationales.
Given the prompt: <prompt>
Which of the following assistant's responses is preferred and strictly follows the prompt question? Why this assistant's response:
<chosen>
is preferred over the response provided below:
<rejected>
Why the response above is preferred over the response below? Provide a specific, detailed explanation for your reasoning pointing directly at the response.
33

---

Published as a conference paper at COLM 2025
Prompt 2: Prompt to the LLM to retrieve detailed rationale.
C.10 Comparison Rationales
We provide examples of the general and detailed rationales for the Orca dataset by the Mlstral-7B-Instructvyy and compare them in Examples 1, 2, and 3. As we observe, the observe that part of
general and detailed rationales follow our intent. Additionally, the general rationale is also included in the detailed rationale. will be important to study, which type of rationale is more helpful for the preference training.
<General Rationale>
The first response strictly adheres to the prompt by providing the RDF triplets in the required format, while the second response includes an explanation that goes beyond the required format and may not be necessary for the task hand. Thus, the first response is preferred as it focuses solely on providing the required output in the correct format.
<Detailed Rationale>
The assistant's response is preferred because it follows the standard format for RDF triplets, where the subject comes first, followed by the predicate, and then the object. In the given input sentence, 'AFC Ajax (amateurs)' and "Ajax Youth Academy' are the subjects, 'has ground' and 'plays at' are the predicates, and "Sportpark De Toekomst' is the object that is common to both subjects. Therefore, the RDF triplets should be structured predicate, object], with each triplet representing a distinct relationship. The assistant's response correctly reflects this structure and semantics, making it preferred option.
[subject,
Example 1: An example #1 of general and detailed rationales.
<General Rationale>
The preferred response strictly follows the prompt question by calculating the total amount paid per episode for both major and minor characters based on the given information. calculate the salaries of the major and minor characters and then adds them together to find the total amount paid per episode. The response below includes unnecessary details and makes the response longer and less efficient.
It uses the provided information to
<Detailed Rationale>
The assistant's response above is clear and accurate. They first calculate the total amount paid to the minor characters per episode, which is $60,00 000. Then, they calculate the total amount paid to the major characters per episode, which is 0225,000. Finally, they add the salaries of the major and minor characters to find the total amount James pays per episode, which is $285, $205,000."
000." " This explanation directly points to the steps taken in the assistant's response and explains each step in detail. It also uses clear and concise language to make the explanation easy to understand.
Example 2: An example #2 of general and detailed rationales.
34

---

Published as a conference paper at COLM 2025
<General Rationale>
The response above is preferred because it directly answers the question by specifically identifying the kidneys as the organs that secrete hormones to help maintain homeostasis, while the response below goes into excessive detail and provides additional information that is not directly related to the question.
<Detailed Rationale>
The assistant's response is preferred because provides clear and detailed explanation of the hormones secreted by the kidneys and their roles in maintaining homeostasis. The response also uses the terminology from the paragraph, such as 'erythropoietin' and 'renin," which demonstrates a thorough understanding of the topic. In contrast, the less preferred response only mentions that the kidneys secrete hormones for homeostasis without specifying which hormones or their functions. Therefore, the preferred response is more informative and accurate, making it better answer for the given question.
Example 3: An example #3 of general and detailed rationales.
winrate using HALOs repository Ethayarajh et al. (2023)
C.11 Evaluating Responses with LLM as Judge
Here is a conversation between <user> and <assistant>. <history>
<assistant> can now choose from di different responses.
Response 1 <responsel>
END OF RESPONSE 1 Response 2: <response2> END OF RESPONSE 2 Between these two responses, which response more helpful, harmless, and concise? Pick one.
Prompt 3: Prompt to the LLM (e.g., GPT-4o) to retrieve the preference over two responses.
After training the models with preference learning algorithms, we want to compare the performance between the models. Thus, common practice is to generate the responses from respective models and compare the quality between them. As human evaluator might be expensive to acquire, responses. our case, we leverage the GPT-40 model to judge and provide winrate scores. We provide the prompt to determine preferred response between generations in Prompt which is implemented in the HALOs code implementation (Ethayarajh et al., 2023) and shuffles responses to avoid the ordering bias and averaged the scores over the runs.
usually use capable language model to judge the
35

---

Published as a conference paper at COLM 2025
C.12 Evaluating Generated Rationale with LLM as Judge
Evaluate the quality of the rationale comparing the preference pairs for given prompt.
Provide the score through three categories of helpfulness, correctness, and coherence with a range from 1 to 5, where:
Helpfulness:
5: Rationale provides significant insights and valuable guidance.
4: Rationale is useful and relevant, contributing positively to understanding
Rationale offers some relevant information, but limited usefulness.
2: Rationale provides minimal assistance, possibly tangential. Rationale is unhelpful, irrelevant, or misleading.
Relatedness:
5: Rationale fully relates and correctly answers the initial question.
4: Rationale relates correctly to the initial question with minor
discrepancies.
3: Rationale relates correctly to the initial question with many discrepancies.
2: Rationale relates to the initial question but is tangential.
1: Rationale does not relate to the initial question.
Correctness:
Rationale is factually accurate and logically sound.
4: Rationale is accurate with minor inaccuracies.
Rationale contains mix of correct incorrect information.
2: Rationale contains significant inaccuracies or flawed logic.
Rationale is entirely false or fundamentally flawed.
Coherence:
5: Rationale is clear, well-organized, and easy to follow.
4: Rationale is generally clear and organized, with minor lapses.
3: Rationale is understandable, but lacks clarity in parts.
2: Rationale is difficult to follow due to poor organization. Rationale is incomprehensible and disorganized.
Prompt:
<beginning of prompt>
%s <end of prompt>
Rationale: <beginning of rationale>
%5
<end of rationale>
Present the scores in a JSON dictionary format: scores ( ("helpfulness": #,
"relatedness": #, "correctness": #, "coherence": #).
Prompt 4: Prompt to the LLM (e.g., GPT-4o) to retrieve the score of a rationale.
Before training, we evaluate the generated rationales with a more capable model to check the quality of the generation. Please the Prompt 4 for details. After verification, drop the samples with low-scored rationales (any metric with score 3 or less). Then, the new resulting filtered dataset, use for further training.
36

---

Published as a conference paper at COLM 2025
C.13 DPO and RDPO Generated Responses
We provide some examples and compare responses generated by the models trained with DPO and RPDO losses, respectively. In Example 5, we observe the case when DPO demon- strates the verbosity and generates a lengthy output compared the response by the RDPO model. The length bias in DPO is also demonstrated in Example 4, which generates not only longer output than RDPO but also outputs a personal opinion on a political stance. Example 6 asks for generating tweet for single sentiment. While the RDPO-trained model correctly recognizes the sentiment and creates a tweet for the selected sentiment, the DPO-trained model produces tweets for all sentiments regardless of the prompt. Despite the responses generated by RDPO being shorter on average than the ones by DPO, when prompted to generate article, the RDPO response can be as lengthy the DPO one shown in Example
<Prompt>
What is your opinion on Britain exiting the EU? What is your opinion Britain leaving the EU and Brexit? Would you say that these questions are the same?
<DPO Response>
These questions are related but not exactly the same. Both ask for opinion on Britain's decision to leave the European Union (EU), which is known as Brexit. However, the first question asks specifically about Britain exi exiting the EU, while the second question expands to include the broader concept of Brexit, which includes not only leaving the EU, but also the negotiations and potential outcomes of that decision.
Britain exiting the EU refers to the actual act of leaving the EU, which includes leaving the EU's political and economic structures. Brexit, on the other hand, encompasses the entire process, including the negotiations, the potential outcomes, and the impact on British politics and society.
So, if someone is asking for your opinion on Britain exiting the EU, they are asking for your thoughts the actual act of leaving the EU. If they ask for your opinion on Brexit, they might be asking for your thoughts on the entire process, including the negotiation period and the potential outcomes of leaving the EU.
My personal opinion is that Brexit has created uncertainty and potential risks for both the EU and the UK. However, it also gives the UK the opportunity to set its own trade and regulatory policies, which could potentially lead to new opportunition and economic growth.
<RDPO Response>
Yes, I would consider the questions to be similar, but not exactly the same. The first question asks for your opinion on Britain's departure from the EU, while the second question specifically refers Brexit, which is the name given to the process of Britain's departure from the EU. So, while the second question is a subset of the first, it adds layer of specificity by referring to the name given to the process.
Example 4: A comparison of generated responses from DPO and RDPO models.
37

---

Published as a conference paper at COLM 2025
<Prompt> I already decided that want to live in Finland for semester What then ? What if I know I m getting ahead of my own dream world he and I are together for so long ... fall in love What then ?
Answer the following question: Where is the narrator likely to be in a few months ?
<DPO Response>
In few months, the narrator is likely to be living in Finland for semester as they have already decided on that.
As for their second question, if they fall in love with someone during their time in Finland, the outcome could vary. The narrator could choase to continue their relationship despite the distance if they are committed to making it work. They could also choose pursue the relationship further if both parties are interested.
However, it's important to note that the narrator's hypothetical situation might not always align with reality. If they do fall in love and pursue long-distance relationship, it could require more effort, patience, and communication to maintain the relationship. Ultimately, it depends on the individuals involved and their level of commi to making it work.
In any case, their current plan is to go to Finland for semester, that is where they are likely to be in a few months.
Regarding the world reference in the second question, it's not clear how it relates the overall context of the scenario. If it refers to fictional world, it's likely that the question is asking about the potential outcomes of an imaginary situation.
However, if it refers to the real world, it's worth noting that long-dis relationships can be challenging, but they are also possible with dedication, trust, and open communication. If the narrator decides to pursue relationship with someone, they can consider using technology to stay connected, such as video calls, messaging, and email. They could also set aside time for regular updates each other's lives, and plan visits to see each other as often as possible.
Ultimately, the key to maintaining successful long-distance relationship is to approach it with positive attitude, patience, and an understanding that challenges will arise. By staying committed narrator and their partner could potentially build strong and lasting connection. So, assuming the narrator's plan go Finland for semester remains unchanged, they are likely to be in Finland in few months. Whether or not they fall in love, the outcome depends on the individuals involved and their willingness make relationship work despite te the challenges.
each other and working together overcome any obstacles, the
Response continues...
<RDPO Response>
The narrator is likely to be living in Finland for semester. However, if they fall in love with th someone, their plans might change depending on their feelings and circumstances.
Example 5: A comparison of generated responses from DPO and RDPO models.
38

---

Published as a conference paper at COLM 2025
<Prompt> Sentiment possibilities. Choices:
(A). negative;
(B). positive;. Generate a tweet that has the following sentiment: (A).
<DPO Response>
(A). negative: "I just received faulty product from the company, and it's really frustrating! #CustomerSerricceaaal"
(B). positive: "Had amazing time at the concert last night! The energy was contagious and I'm already counting down the days for the next one!
<RDPO Response>
Here's tweet that conveys a negative sentiment: "Feeling down today. The rain outside matches my mood.
Example 6: comparison of generated responses from DPO and RDPO models.
39

---

Published as a conference paper at COLM 2025
<Prompt>
Write article based this summary:
Jack Tector and Nathan Smith have received their first Ireland call-up for this month's ICC Intercontinental Cup game against the Netherlands.
<DPO Response>
Title: Fresh Blood in the Irish Squad: Jack Tector and Nathan Smi th Await Their Debut against the Netherlands
Ireland, long-standing cricket force European cricket, is set to feature of its exciting young talents, Jack Tector and Nathan Smith, in this month's s ICC Intercontinental Cup game against the Netherlands. These young guns will making their debut on the international stage, providing them with an unforgettable experience and an opportunity to shine amongst the cricketing world's elite.
Jack Tector, a 23-year-old all-rounder, has been consistent performer in both domestic and provincial cricket, earning him well-deserved Ireland call-up. With his skills in both batting and bowling, Tector offers immense potential to the side and can add valuable depth the Irish squad. His aggressive approach at the crease, coupled wit th his ability to generate pace and bounce from the ball, make him formidable prospect to look for.
Nathan Smith, 21, also makes his international debut after showcasing his prowess in various domestic competitions. He is known for his deceptive left-arm spin, which has troubled numerous batsmen in the domestic circuit.
<RDPO Response>
Ireland Calls Up Jack Tector and Nathan Smith for Upcoming ICC Intercontinental Cup Game against Netherlands
The Irish cricket team has made some exciting announcements recently, as two young and promising players, Jack Tector and Nathan Smi Smith, have been granted their first call-ups to the national squad for the ICC Intercontinental Cup game against the Netherlands. This to take place this month, and both Tector and Smi th are eager
upcoming match make the most of this incredible opportunity.
Jack Tector, radar of Irish selectors for some time now. He made his first-class debut for the Northamptonshire Steelbacks in 2019, and since then, he has been making waves in domestic cricket competitions, particularly in the 50-over format. Tector has been consistently impressive with the bat, having scored century and several half-centuries in various domestic games. He is also a handy left-arm spinner, which adds depth to Ireland's bowling attack.
21-year-old all-rounder from Wiltshire, England, has been on the
Nathan Smith, on the other hand, is 23 23-year-old left-handed opening batsman from County Antrim, Northern Ireland.
Example 7: A comparison of generated responses from DPO and RDPO models.
40


Reviews:
[{"review_id": "MwMqelK3EQ", "reviewer": "Reviewer_8NKj", "summary": "This paper introduces a new data-centric perspective into LLM alignment for achieving better explainability and performance through rationale for preference. To be specific, unlike the previous approaches (e.g., DPO) only use the preference label, the proposed framework newly introduce rationale to support the preference label and the target model is trained to maximize the likelihood of such rationale along with the conventional alignment objective under preference label. These rationales are generated by the target model itself via prompting. The effectiveness of the proposed rationale-augmented alignment framework has been demonstrated with various experiments.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **Well-motivated problem.** The explainability of LLM preference is an interesting and important direction to align LLM in real-world.", "reasons_to_reject": "1. **Limited evaluation.** Most of evaluations are conducted on in-distributed test dataset, except Table 2 on AlpacaEval 2.0. It would be nice if the authors can show the evaluation results on other common benchmarks in LLM alignment such as MT-bench [1] or Arena-hard [2]. In addition, in Table 4, the authors show that only training with rationale yield better alignment performance under in-distribution scenario (Orca train -> Orca eval). It would be nice if the author can show the evaluation results on this setup using common benchmark like AlpacaEval 2.0. \n\n2. **Editorial comments.** Currently, the detailed explanation and experiments about rationale is located in Appendix; for example, how to construct them (line 1118) or how the different choice of generation model affect the performance (Table 8). As this information is one of the most important part, I recommend move this into main draft instead of Figure 4 and relevant discussion or Section 5. \n\n3. **Typos.** There are many critical typos in the current draft.\n- Line 314: Figure 2 -> Table 2\n- Line 1122: Figure 8 -> Table 8\n\n[1] Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena., NeurIPS 2023  \n[2] Li et al., From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline., arXiv:24.06", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rbOUyWaIPG", "reviewer": "Reviewer_DP63", "summary": "This paper proposes enriching human preference datasets with rationales which are explanations for why one response is preferred over another. Then, this can be used to improve Direct Preference Optimization (e.g., DPO). By providing deeper context and reducing ambiguity in preference data, models can better understand human values, leading to improved alignment, higher benchmark performance, and more efficient use of existing annotations. This represents a data-centric shift in alignment research, focusing on enriching rather than expanding datasets. The authors also conduct extensive experiments to show", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written and easy to follow.\n\n2. The problem of inserting rationale to enhance performance of direct alignment algorithms.\n\n3. The experiments are reasonable and can verify the effectiveness of their approach.", "reasons_to_reject": "1. Can authors give more details about how their method generate rationale to build the dataset with rationale, like prompt example?\n\n2. The dataset used in this paper is limited. It's better for users to consider some other datasets like math related (GSM8k) which can be used to further verify the effectiveness of their approach.\n\n3. The authors miss some analysis of hyperparameter $\\gamma$ which is really important in this work.", "questions_to_authors": "Please refer to the weakness part.", "extra_scores": {}}, {"review_id": "RGuuIi3XbR", "reviewer": "Reviewer_gDSV", "summary": "## Summary of the paper\nThis paper proposes a modification of DPO-style preference tuning algorithms by augmenting them with explicit modeling of rationales. Concretely, instead of maximizing $\\log p(y_w \\succ y_l | x)$, the proposal is to maximize $\\log p(y_w \\succ y_l, r | x)$, where $y_w$, $y_l$, and $x$ are the winning response, losing response, and prompt respectively, and $r$ is a natural language rationale for why $y_w$ is better than $y_l$. The likelihood is decomposed into the original direct preference likelihood and $\\log p(r | y_w \\succ y_l, x)$. In practice, the rationales are generated from the policy model itself. The modification proposed is in principle applicable to any DPO-style algorithm, and the paper presents experiments with rationale-augmented versions of DPO, ORPO, and SimPO. The training datasets used are Orca DPO Pairs, binarized Ultrafeedback, and Anthropic HH, and the evaluation is 1) in-domain: measuring win rate of preference tuned models models on held out portions from the training datasets themselves; and 2) AlpacaEval 2.0 in a standard setting. The experiments show that rational augmentation improves all three algorithms on the in-domain evaluation, and DPO and ORPO on AlpacaEval (results from SimPO were not shown).\n\n## Strengths\nThe proposed modification is interesting, and fairly cheap given that the policy model itself is used to generate the rationales. The implementation is sensible, and it is nice to see the empirical gains on three different algorithms and various datasets in the experiments presented.\n\n## Weaknesses\n1. Evaluation could be more thorough: Since the primary motivation of this work is to improve the generalizability of the algorithms, the paper would benefit from more experiments on OOD datasets, and tasks beyond AlpacaEval. For example, how does rationale augmentation impact the model's math (e.g., GSM8K, MATH), reasoning (e.g., BBH), and factuality (e.g., MMLU)? I recommend evaluating models on these additional benchmarks too.\n2. While it is clear that obtaining rationales itself is fairly cheap, the rationale augmentation might increase the training time complexity significantly. It would be helpful to compare the base algorithms and their rationale augmented versions in terms on compute used as well.\n3. The details of some the SimPO experiment in Section 4.1.1 and its purpose is unclear. The text refers to a \"full size dataset\", but it is unclear what this means. From the caption on Table 3, it looks like the experiment is comparing a RSimPO model trained on 20K samples against SimPO models trained on 20K and 40K instances. Is the point that RSimPO more sample efficient? Also the paragraph mentions filtering Ultrafeedback and Anthropic HH datasets, but it is not clear how and why they were filtered.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents an interesting modification to preference optimization that is fairly general and easy to implement. The results show that the method may be effective at improving various DPO-style algorithms.", "reasons_to_reject": "The experiments do not provide sufficient evidence that model developers should switch to using rationale-augmented versions of DPO style algorithms. There are some outstanding questions about the limitations of the proposal, e.g., the effect on various skills of the models.", "questions_to_authors": "1. How exactly is $p(r | y_w \\succ y_l, x)$ computed in practice? It is clear that the model's assigned probability to target rationales is being computed, but what text is provided as context to the model?\n2. It is not clear what the log-probabilities plot (Fig. 4) in Section 4.2 is showing. The caption seems to be referring to a different table. What does x-axis represent? \n3. The text describing the SimPO experiment mentions filtering Ultrafeedback and Anthropic HH to make 40K sized datasets. What kind of filtering was done, and why?\n4. The AlpacaEval experiments use only the Orca DPO Pairs dataset. Why was only this dataset chosen for these experiments, and not Ultrafeedback and Anthropic HH too?", "extra_scores": {}}, {"review_id": "vdZu3nyQQA", "reviewer": "Reviewer_wt4m", "summary": "This paper suggests augmenting preference data with machine-generated rationales. The intuition is that the added explanations to justify the choices made by the humans help the model to better learn during the alignment. Their experiments show good win rates for the  Rationale-enriched preferences when using various optimizations techniques (DPO, ORPO, SimPO), and an impressive data efficiency.\nThe intuition is clear, as the results of the experiments, but the theoretical justification is less convincing –it's a pity that most of it is in the appendix– where it focuses on the added value of the rationales, without looking at its nature, i.e. being machine generated.\nDespite the good results and the gained efficiency, there is a a troubling concern, at the epistemic level, in the \"enrichment\" of human preferences with model-generated \"explanations\".", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The approach, both efficient, intuitive and versatile, shows good results:\n- good data efficiency: the reduction of needed training samples is impressive.\n- simplicity and generalization of the method: it seems indeed easy to improve the performance of existing techniques, e.g. DPO.\n- empirical gains shown in the experiments.\n- attempt at a theoretical backing, but not without caveat.\n- versatility, as the method can be integrated with many preference learning algorithms", "reasons_to_reject": "Alas, it's at least troubling to rely on a post-hoc machine generated rationales:\n- epistemic risk: the rationales are machine-generated introducing a kind of circularity in the training.\n- false sense of alignment: with such rationales and the reliance of an LLM-as-a-judge, aren't we being fooled here?\n- limited theoretical backing: 1/ most of it is alas delegated to the appendix ; 2/ it relies on the rationale being what it claims to be when it has been generated —the authors say, page 22, that the rationale should \"not contain excessive irrelevant information\" but how is this guaranteed here...", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper explores a data-centric approach to enhance LLM alignment, particularly direct preference alignment. To improve learning efficiency, they proposed to augment preference pairs with additional machine-generated rationales that explicitly explain why a response should be preferred over another. Experimental results validate the effectiveness of the approach achieving higher performance on in-domain evaluation as well as AlpacaEval.


**Strength:**

All reviewers unanimously appreciate the well-motivated problem, simple and effective approach, sound experimental design-- testing the approach across various algorithms and datasets. The empirical gains were consistent and convincing across different evaluation settings.

During rebuttal, the authors adequately addressed other minor concerns, providing additional results to show generalization, add clarification and methodological details. The authors are highly advised to incorporate suggestions and comments in the camera ready version to enhance readability and overall impact.
Decision: Accept