Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization
Authors: Amir Saeidi
Shivanshu Verma
Kashif Rasul
Aswin RRV
Chitta Baral
Venue: Accepted by TMLR ()

Abstract: Reinforcement Learning with Human Feedback (RLHF) enhances the alignment of Large Language Models (LLMs). However, its limitations have led to the development of Direct Preference Optimization (DPO), an RL-free approach designed to overcome these shortcomings. While studies have shown that DPO improves instruction-following capabilities, it negatively impacts the reasoning ability of LLMs. Additionally, DPO is highly sensitive to judgment noise in preference datasets and the size of the training set. Although several modifications to DPO have been proposed, they still fail to fully resolve these issues. To address these limitations, we propose Triple Preference Optimization (TPO), a new preference learning method designed to enhance both reasoning and instruction-following abilities through one-step optimization. We compare TPO against DPO and its recent variants using state-of-the-art training setups, including both base and instruction-tuned models such as Mistral and Llama 3. Our evaluation covers a comprehensive range of chat-based and reasoning benchmarks. The results demonstrate that TPO achieves significant improvements over existing methods without substantially increasing response length across different dataset sizes. Specifically, TPO outperforms DPO and SimPO by up to 7.0% and 7.3% points on Arena-Hard, 12.2% and 13.3% points on MixEval-Hard, 10.4% and 10.1% points on MMLU-Pro, and 19.0% and 19.2% points on GSM8K, respectively. Furthermore, TPO achieves these improvements while requiring less data than DPO.

Published in Transactions on Machine Learning Research (11/2025)
Triple Preference Optimization: Achieving Better Align- ment using a Single Step Optimization
Amir Sacidi
ssaeidi @ asu. edu
School of Computing and Augmented Intelligence Arizona State University
Shivanshu Verma
sverma76 usu. edu
School Computing and Augmented Intelligence
Arizona State University
Kashif Rasul
rasul.kashifee morganstanley.com
Morgan Stanley New York, NY
Aswin RRV
aravik13 asu. edu
School Computing and Augmented Intelligence
Arizona State University
Chitta Baral
chitta @ asu. edu
School of Computing and Augmented Intelligence
Arizona State University
Reviewed on Revnew
Abstract
Reinforcement Learning with Human Feedback (RLHF) enhances the alignment of Large Language Models (LLMs).. However, its limitations have led the development of Direct Preference Opti- mization (DPO), RL-free approach designed to overcome these shortcomings. While studies have shown that DPO improves Instructionfollowinn capabilities, it negatively impacts the reasoning ability of LLMs. Additionally, DPO is highly sensitive to judgment noise in preference datasets and the size of the training set. Although several modifications to DPO have been proposed, they still fail to fully resolve these issues. address these limitations, we propose Triple Preference Optimization (TPO), a new preference learning method designed enhance both reasoning and instruction-followe followinn abilities through one-step optimization. We compare TPO against DPO and its recent variants using state-of-the-art training setups, including both base and instruction-tuned models such as Mistral and Llama 3. Our evaluation covers comprehensive range of chat-based and reasoning benchmarks. The results demonstrate that TPO achieves significant improvements over existing methods without substantially increasing response length across different dataset sizes. Specifically, TPO outperforms DPO and SimPO by up to 7.0% and 7.3% points on Arena- Arena-Hard, 12.2% 2% and 13.3% points on HixEval-Harde 10.4% and 10.1 points on MMLU-Pro and 19.0 % and 19.2% points on GSM8K, respectively, Furthermore, TPO achieves these improvements while requiring less data than DPO.
1 Introduction
Since InstructGPT, most Large Language Models (LLMs) add preference optimization on top pre-training and supervised fine-tuning (Ouyang et al., 20226). InstructGPT used Proximal Policy Optimization (PPO) (Schulman al. 2017) form of Reinforcement Learning with Human Feedback (RIHF) (Stiennon al. 2020) for preference

---

Published in Transactions on Machine Learning Research (11/2025)
Arena-Hard Win Rate
GSMBK Accuracy (%)
[DPo(Te: =
luger
log
glg
Molar x)
12-
50
Tewe(3) (3r)
(t)
10
TPG-L
(TP)(TT)
E
llog (i ma( x
oi log me(su) [z))
or log mell 1z)
20-
(TT)
-E
logo
log 1o
[x)
log molan xx)
->) loggsaa)
Llame]
Liema3 Base BB (10K)
Uarma Base 8a (20K)
Uiema] Base BB (40K)
Lleme3 Base 8B
Liamn3 Base BB (10K)
Ulama3 Base 83 (204)
Uaama3 Base 8B (40K)
Base BB
(5K)
(5K)
Figure TPO and TPO-L differ by removing the reference model and adding Behavioral Cloning objective with regularization term for gold preferences, distinct from preferred and rejected responses. TPO and TPO-L outperform DPO in instruction following and reasoning benchmarks simultaneously.
optimization, and used the Bradley-Terry modeling to build an explicit reward model that is derived from the preference data. Direct Preference Optimization (DPO) (Rafailov et al. 2024b) was proposed as an RL-free alternative to preference learning and optimizes a policy model using an implicit reward function where the regular KI. divergence acts as the implicit reward.
Recent studies (Tunstall et al. 2023 Xu et al. 2024) indicate that DPO, which uses multi-step optimization, faces challenges related to optimization inefficiency. Additionally, a new finding (Meng et al. 2024) suggests that KL divergence, which is used in DPO, may not effectively represent a reward. Instead, the average likelihood of response given the input appears to be a more suitable approach for reward modeling. Motivated by these issues, several variants of DPO have been proposed (Azar et al., 2023 Ethayarajh et al., 20244 Hong et al. 2024b). As highlighted in (Wul et al. 2024 Meng et al., (204)) the ahove-mentioned DPO variants enhance performance instruction -olloooiionnooooiinngo tasks; however, they also point out that there is notable decline in performance reasoning benchmarks such GSMSK.
Furthermore, show that most current preference optimization methods are still sensitive to training set size.
Motivated by the desire to address the challenges faced by the current methods and to simplify the optimization part to a single step, we propose Triple Preference Optimization (TPO), a novel preference optimization algorithm designed to achieve superior performance on both instruction follooiigg and reasoning benchmarks in a single-step optimization process. TPO optimizes a pre-trained model by maximizing the likelihood of gold response (Ugola) using a Behavioral Cloning (BC) objective while incorporating a preference optimization term reference- format. This term increases the likelihood of preferred responses (ywz) and decreases the likelihood of rejected responses (y1). Moreover, to enhance control over response length, inspired by the SimPO (Meng al. 2024) method, introduce TPO-L, length-controlled variant of TPO. In TPO-L, the average length of preferred and rejected responses constrains the policy model, ensuring that it generates longer responses only when quality improves.
We provide a theoretical formulation of TPO by deriving the optimization function of TPO from Maximum Entropy Reinforcement Learning (MERL) by considering BC objective on gold response as an entropy. We experimentally demonstrate that TPO addresses the optimization conflict challenge present in some recent preference optimization methods. Additionally, we show that using the average likelihood of sequence of responses provides more effective representation of the implicit reward than the definitions used in DPO and SimPO. Comprehensive experiments demon- strate that TPO and TPO-I achieve simultaneous improvements in instruction-followin and reasoning benchmarks, demonstrating greater robustness across varying training dataset sizes. Furthermore, we observe that TPO exhibits stronger resilience to judgment noise in preference datasets compared to DPO. Notably, even with half the amount of data, TPO and TPO-L consistently outperform DPO.
In summary, our contributions are follows:
We propose Triple Preference Optimization (TPO) and TPO-L, two novel single-step preference learning methods that achieve significantly better performance on smaller training datasets compared to existing methods. Extensive experiments demonstrate that TPO and TPO-L outperform current preference optimization methods on both instruction- following and reasoning benchmarks across various training dataset sizes (See Tables and
We derive and justify the TPO loss function using Maximum Entropy Reinforcement Learning (MERL), Bradley-Terr modeling, and a variation of the reward function formulation in DPO. Additionally, we demonstrate that log To (g) serves as implicit reward in TPO, improving reward modeling compared to DPO and SimPO (see Figures 5 and
2

---

Published in Transactions on Machine Learning Research (11/2025)
We also illustrate that TPO overcomes the optimization conflict challenge in current methods (See Figure exhibits greater robustness to judgment noise in data (See Figure 7 and achieves comparable performance while utilizing less data then DPO (See T))
Base Setting
Instruct Setting
DPC
DPO
SIMPO
SPMPO
KTO
TPO-y2
DAA (%)
SLIC
TPO
TPO
SUC
10
20
40
60 Data Size (k)
Data Size (k)
Figure 2: Comparison of post-training improvements over the SFT checkpoint across different methods, measured by the DAA metric.
2 Method
In this section, we start by discussing the ongoing challenges in current Preference Optimization (PO) methods, which motivates the development of our new method. We then introduce the Triple Preference Optimization (TPO) method, novel preference learning algorithm, and provide a detailed explanation of the theoretical foundations supporting it.
2.1 Challenges
Current PO methods face several key challenges. First, their generalization across benchmarks is limited, im- provements observed in instruction-folloiinn tasks often do not carry to downstream tasks, with some methods performing worse than their Supervised Fine-Tuuning (SFT) checkpoints (Meng et al. (024). To address this, we introduce Difference Average Accuracy (DAA), an effective metric for evaluating post-training improvement.
DAA (%) = A AAAccuracysFr: Accuracy Preference Method
where AccuracysPT SFT and Accuracy Preference Method represent the average accuracy across five downstream tasks on the SFT checkpoint and the post-training checkpoint, respectively. Additionally, these PO methods are highly sensitive dataset sizes (see Table which poses challenges for tasks like reasoning, where data collection is costly. Another major limitation is the reliance multiple-ste step optinization, where methods like DPO require sequential optimization stages, and attempts to bypass SFT, such in ORPO, have yet to achieve significant performance improvements (Meng et al.) 2024). Moreover, the simultaneous loading of two models in DPO leads to inefficiencies and remains a challenge, while reference- free approaches like CPO and SimPO either require additional steps or face difficulties without the SFT stage, which we refer to as the optimization conflict issue (See Figure
Another critical challenge is judgment noise within preference data (Wang reduce optimization effectiveness despite improvements in open-source reward models like PairRM (Jiang et al. 2023b and ArmoRM (Wang al. 2024d). These models, while cost-effective, still produce inaccurate rankings that hinder direct preference optimization methods (Meng et al. 2024). Furthermore, although GPT family models help reduce noise, they remain expensive and still exhibit issues such as judgment bias (Tan et al. 2024). Finally, reward modeling remains an issuc, as DPO relies on KL-divergence, whereas SimPO suggests that the average likelihood response sequences provides a more effective reward representation. However, these implicit rewards behave inconsistently across varying dataset sizes, highlighting the need for more robust and scalable reward estimation methods enhance preference optimization performance.
2024a where ranking inconsistencies

---

Published in Transactions on Machine Learning Research (11/2025)
2.2 TPO: Triple Preference Optimization
Motivated by these challenges, we propose the Triple Preference Optimization (TPO) method, novel preference learning approach that optimizes a pre-trained model in single step on three preferences where Ygold, Vw. and are the preference responses generated for the same prompt æ. To optimize a policy model with TPO, we require a dataset DIro = {x', Vg Vu vi /= where the preferences satisfy Ygold > Yo > yt- This approach assumes that preference data is collected from different models, ranked by "Judge models", and used to optimize pre-trained model. Alternatively, preference data can be derived from Supervised Fine-Tuned (SFT) model on a task, with generated responses for the same prompt ranked by Judger models optimize an Instruction model.
In this context, the policy model Te is not equal to the SFT model ( (T * TSFT), and it is optimized just in a single step for each scenario. Further details about these settings are discussed in Section
2.2.1 Deriving the TPO objective
In this section, we detail the derivation of the TPO objective in manner similar the DPO derivations. We start with basic Reinforcement Learning (RL) objective used align Large Language Model (LLM), parameterized by 0 and represented To, with preferences. The RL objective involves maximizing the expected reward, as defined in (Ziegler ct al. 2019a) is:
max
y)]
(1)
where rs represents the expected reward the model receives for a given input r and output y. However, maximizing the reward without constraints can lead to distribution collapse in an LLM. Drawing inspiration from the Maximum Entropy Reinforcement Learning (MERL) framework (Hejna et al. 2023), we have modified the RLHF objective, as detailed in Equation The MERL framework aims to maximize causal entropy alongside the expected reward. This objective is formally defined in Equation 2
max E yom (y)l (ro + (v]=))
(2)
By the definition of Entropy.
Hr. (ulz) =
log no to (e.))
(3)
1y 5=1
Using Equations [nd the objective becomes:
max Egw Eaw [ro(2,)) 8) - B log To (o(v/²)]
(4)
T
Based on this, the optimal policy model induced by a reward function r(x, y) could be derived shown in Equation 5 (See Appendix A.I. It takes the following form:
n (v(=)
Z(z) exp (z)(a.)).
where Z(x) = En exp (1r( v)) is the new partition function. Inspired by (Rafailov et al. 2024b), we have the reward function in terms of the optimal policy that induces; as shown below in Equation 6
r(x v) 3 log क, (y(ar) log 7(x).
(6)
Subsequently, we can represent the ground-truth reward r (I, y) in the form of its corresponding optimal policy x² that it induces.
Since the Bradley- Terry model is dependent only on the difference between the two reward functions, i.c. p" (yu >
ii (z) a(r (z. yw) (xx yt)), we can reparameterize it as follows in Equation 7
p² (y > 31 x)) = ( म." (yw x:) - ,3 log " "(y (r)
(7)
4

---

Published in Transactions on Machine Learning Research (11/2025)
Similar to the reward modeling approach, model the human preferences, which in terms of parameterized policy Thus, we formulate maximun-likelihone objective (preference objective) for a dataset D) {z² V ) as shown in Equation g
-B preference (To) = -- (a. (a.
log (Blog To (ol x))
3 log (o x))]--
(8)
Looking the Equation 8 the objective fitting r(a, y) = 3 log n (v/²) as the reparameterized reward. In Section A.3 of the Appendix, we theoretically explain that fitting this reward will ultimately recover the optimal policy.
The comparison between the loss function in Equation and the DPO loss function indicates that the new function is more efficient because it requires only one model during training. However, even though maximizing the objective under the MERL setting prevents distribution collapse, it trains a pessimistic model, which also limits the model from learning the preferred responses effectively. To counteract this limitation, we maximize the likelihood of the gold response. The adjustment is specified in Equation
Creference (TO) - (2, [log we (ygold [x]]
Based on Equations 8 and 9 the TPO is defined as multi-objective (bi-objective) optimization problem supported by the Pareto Front concept (Lotov & Miettinen 2008). The TPO loss function thus formulated as follows:
LTPO
preffernnce + OeCreferenc::
(10)
where hyper-parameter (a) is regularization term.
Inspired by SimPO, which replaced the summation of the likelihood of response sequences with averaging, based on concepts from beam search and multiple-choice tasks, and introduced a reward margin, we propose TPO-L, a length-controlled extension of the TPO loss function. TPO-L is defined as follows:
LTPO- -E log o
B Vu
log no ( x)
B vnl
log To(nn |x) -7> log Toe (Ugota (x)
(11)
where, y represents the reward margin. In Section 4.3 we will analyze the reward margin's impact, showing it provides superior reward representation than SimPO. We will also demonstrate that while summation in the loss function promotes longer responses, well-fitted policy model generates effective responses without depending on summation.
3 Experimental Setup
Models and training settings. To evaluate TPO and existing preference optimization methods, we follow the SimPO setup with minor adjustments, focusing on the Mistral Jiang et al. (2023a) and LLaMA Touvron et al. (2023) models. To simplify the main text, we omit Mistral results but include them Appendix D Our evaluation consists of
distinct setups: Base and Instruct.
Base setting. For this setting, used the UltraFeedback (Cui et al.) (023) dataset, containing 60,000 data points, each with four responses scored by GPT-4 across four criteria, resulting in an overall score. As TPO requires three preferences (Ugola > Yw x W.). we processed the dataset to ensure a 0.5 score difference between Ygold, Ve. and yı (See Figure 9 in Appendix). This resulted in a final dataset of 40,000 data points. To compare preference optimization methods fairly, we fine-tuned a pre-trained model on the gold responses and used the preferred and rejected responses for the current preference optimization methods in two steps. For TPO, we utilized all data one optimization step. Moreover, we evaluated preference optimization methods on subsets of 5,000, 10,000, and 20,000 points randomly selected from the processed dataset. The Experiment section 4. details the models' performance.
5

---

Published in Transactions on Machine Learning Research (11/2025)
Table I: Evaluation details for Arena-Hard, MT-Bench, and MixEval-Hare as the instruction following benchmarks and MMLU-Pro, MMLU, and GSM8K as the reasoning benchmarks.
Exs.
Baseline
Evaluation
Scoring Type
Metric
Arena-Hard
500
Answer GPT-4-0314
Judge by GPT-4o
Pairwise comparison
Win rate Rate of 1-10
MT-Bench
80
Judge by GPT-4
Single-answer grading
MixEval-Hard
1,000
Ground truth
Evaluate by GPr-o.5-turbo
Systematic
Accuracy
MMLU-Pro
12,032
Ground truth
CoT
Systematic Systematic
Accuracy Accuracy
MMLU
13,908
Ground truth
5-shots
GSM8K
1,319
Ground truth
5-shots
Systematic
Accuracy
Instruction setting, We used instruction-tuned models as the backbone for preference optimization to assess the impact of replacing them with SFT models. The experiment employed the UUtraFeedackk-Amaeee 60,000 samples with five responses generated by the [LaMA-3-8B-SS Responses were scored using the ArmoRM reward model, with the highest and lowest-scoring responses designated as preferred (yw)) and rejected (y1), respectively. For TPO, the highest-scoring response was considered gold (ygold), preferred (yw.), and the lowest as rejected (z1). The best checkpoint from SimPO was used for
dataset, comprising fine-tuned on UltraChat (Ding et al., 2023b).
the second- second-highest highest comparison.
Evaluation benchmarks. We categorized the evaluation benchmarks into reasoning and instructionnfollooiin tasks. Reasoning benchmarks included MMLU (Hendrycks al. 2021 MMLU-pro (Wang al. 2024e), and GSM8K (Cobbe al. 2021). which assess a model's understanding across diverse subjects, with MMLU-pro featuring more complex tasks and GSM8K focusing on math problem-solving requiring step-by-step reasoning. Instruction- following benchmarks consisted of Arena-Hard (Li et al. 2024), MT-Bench (Zheng et al. (003b)), and MixEval-Hard (Ni et al. 2024), which evaluate conversational versatility across a range queries. MT-Bench includes 80 questions across categories, Arena-Hard expands on with 500 technical problem-solving queries, and MixEval-Hard tests challenging queries against known answers using GPT-3.5-turbo for verification (details in Table [. Additionally, we
reported scores for ARC (Clark et al. 2018). HellaSwag (Zellers et al. (209)), Winogrande (Levesque et al. 2012). and
TruthfulQA (Lin et al.
2022 standard setup, with further details provided in Appendix
Baselines. We compare TPO with several preference optimization methods, including SLiC-HF Zhao al. 2023).
which uses ranking losses, and IPO (Azar et al.) 2023), which avoids DPO's pointwise reward assumption with theoretically grounded approach. SimPO (Meng et al. 2024), reward margin-aware variant of DPO, leverages average sequence likelihood instead of summation. CPO (Xu et al. 2024) integrates training with an SFT objective, while KTO (Ethayarajh et al. (2024) learns from non-paired preference data. ORPO (Hong et al. (2024)) eliminates the need for a reference model by introducing an odds ratio term, directly comparing preferred and rejected responses while jointly training with SFT. SimPO findings suggest ORPO performs better when fine-tuned from an SFT checkpoint. We
fine-tuned hyperparameters for all baselines. Additional details are available in Appendix C
4 Experimental Results
This section presents the main results across benchmarks, comparing TPO with existing preference optimization methods at different dataset sizes. We analyze reward modeling in TPO and compare it with SimPO, address the verbosity issue by showing TPO and TPO-L generate shorter responses, and evaluate TPO's performance noisy data compared to DPO. Additionally, we assess the effect of strong SFT checkpoints, optimized on larger datasets, on DPO's performance. Further details are provided Appendix
4.1 Main Results and Ablations
TPO and TPO-L significantly outperform existing preference optimization methods. The results in Tables and demonstrate that TPO and TPO-L significantly outperform existing preference optimization methods across reasoning and instruction-follovinng benchmarks in both Base and Instruct settings, regardless of training dataset size. These

---

Published in Transactions on Machine Learning Research (11/2025)
Table 2: Results for Arena-Hard, MT-Bench, MixEval-Hard, MMLU-Pro, MMLU, and GSM8K under the Base setting
across four training sizes. LLaMA3-8B- models are trained on gold responses and fine-tuned with preference
optimization on chosen and rejected data, highlighting TPO and TPO-L improvements over DPO. We quantified the improvement DPO alongside the TPO and TPO-L scores.
UltraFeedback (5k)
Ultrafeedback (10k)
Method
Reasoning
Instruction Following
Reasoning
Instruction Following
GSMSK
MMLU-Pro
MMLU
MT-Bench Arena-Hard MixEval-HHrd
GSMBK
MMLU-Pro
MMLU
MT-Bench Arena-Hard MixEval-Haad
SFT
28.5
25.7
59.0
24.6
27.7
25.5
4.9
24.4
DPO
32.9
27.1
59.2
5.5
0.5
26.9
36.7
28.9
59.4
6.3
6.1
25.9
CPO
31.9
26.5
58.9
5,4
0.5
27.4
32.8
26.9
59.3
5.5
0.5
23.9
IPO
34.4
28.0
59.4
0.5
27.1
35.4
28.3
59.2
6.4
3.9
26.7
27.5
59.3
0.3
28.3
36.2
5.0
0.5
KTO
32.7
26.8
59.2
0.5
26.5
36.7
28.8
59.4
6.4
4.4
SIMPO
32.7
27.4
59.2
5.6
0.5
27.4
35.1
27.5
59.3
5.7
4.4
24.8
SLIC-HF
32.5
26.7
59.0
5.4
0.3
26.5
32.7
26.7
59.3
5.5
< 0.5
25.5
TPO
51.9 (+190)
37.5
(x10.4)
(16.1)
6.2 (v0.7)
4.4 (63.9)
(+8.1)
52.2 (v15.5)
37.8
(v8.9)
65.3
(65.9)
(60.4)
6.7 (60.6)
31.2 (65.3)
TPO-L
52.1 (-19.2)
36.3 (49.)
65.3
(46.1)
6,3
(48.)
2.0 (+1.5)
31.8 (44.9)
51.9 (+15.))
38.1
(+9.3)
65.4
(-6.0)
6.9
(+0.6)
7.3 (+1.2)
33.7 (+7.8)
UltraFeedback (20k)
Ultrafeedback (40k)
Method
Reasoning
Instruction Following
Reasoning
Instruction Following
GSMSK
MixEval-Hard
MixEval-Haadd
SFT
20.3
30.4
62.5
5.5
28.8
39.2
29.5
62.2
5.5
1.7
25.5
35.7
62.8
30.1
6.6
5.3
25.4
CPO
32.6
62.9
7.1
30.5
42.9
62.2
6.2
9.3
IPO
45.5
35.0
62.8
5.9
< 0.5
26.2
48.2
34.2
62.1
6.7
4.2
27.1
ORPO
37.1
33.1
63.1
5.6
0.5
28.9
40.9
32.1
62.4
5.9
3.2
27.7
KTO
48.7
35.8
62.7
6.6
1.7
29.0
46.0
33,4
61.9
6.6
5.2
24.0
SIMPO
45.0
34.0
63.0
6,4
5.1
28.1
45.1
31,6
61.9
6.2
6.6
24.3
SLIC-HF
44.1
32.5
62.9
30.1
42.8
30.5
62.3
6.4
25.8
TPO
53.0 (15.8)
37.5
(41.8)
65.3
(42.5)
6.6 (4BL1)
5.3 (43.3)
32.4 (42.3)
51.2 (46.1)
37.4 (43.)
64.8
(43.0)
6.9 (40.3)
6.9 (+1.6
32.9 (47.5)
TPO-L
52.7 (+4.7)
38.1
65.3
6.8
6.5
32.4
52.4
40.4 (+6.9)
65.1
7,3 (+0.7)
12.3
37.6 (+12.20
findings highlight that TPO not only enhances the instruction-followinn capabilities of models but also significantly improves their reasoning performance.
TPO and TPO-L excel in low-data scenarios, mak- ing them ideal for tasks with limited dataset avail- ability. Notably, with only 5,000 samples, other methods perform poorly on Arena-Hard, while TPO and TPO-L achieve 4.4% and 2%, respec- tively. TPO-L outperforms TPO in the Base setting, whereas TPO delivers better results in the Instruct setting, highlighting their adaptability and robust- ness across various benchmarks and training setups.
Table 3: Results for Are-Hard, MT-Bench, MixEval-Han -ard, MMLU-Pro, MMLU, and GSM8K under the Instruct setting, using off-the-shelf models as the SFT model. Front of the TPO and TPO-L scores, we identify the improvement compared with DPO.
LLaMA-3B-
Method
Reasoning
Instruction Following
Arena-Hard
MiaEval-Hadd
SFT'
68.7
67.1
TPO-L is more stable and generalizable than other methods for downstream tasks. We used the DAA metric to evaluate the generalizability of competitor methods across five downstream tasks: GSM8K, MMLU, HellaSwag, Winogrande, and TruthfulQA. The results shown in Figure indicate that while existing methods provide slight improve- ments over the SFT checkpoint, their effectiveness is significantly influenced by dataset size. In the instruction setting, methods such DPO and IPO
DPO
54,5
43.7
67.3
7.7
40.9
41.6
CPO'
67.8
42.6
66.9
35.6
41.3
67.3
44.0
ORPO!
36.4
KTO!
44.1
SIMPO
55.6
66.5
35.1
45.0
SLIC-HFI
68.2
40.2
66.9
28.6
40.0
TPO,2
40.4
TPO,a
65.5
42.4
41.9
TPO,,
78.2
43.4
65.7
8.1
38.9
38.6
TPO-Lya
77.3
43.7
65.7
8.2
39.4
39.4
7

---

Published in Transactions on Machine Learning Research (11/2025)
Base Setting [3K]
Base Setting (10K)
Base Setting (20K)
Base Setting (40K)
Instruct Seeting (60K)
GPC
CFD
CPO
CRPc
Afera Hard
30
caro
P0
IN
Pn
P0
KM
20
<to
Km
cro
cto
TFD TFD4
TPC
Troyp
TNCL
TPC
Tro
Tro-
1000
2000
3000
4000
1000
2000
3000
4000
1000
2000
3000
4000
2000
40:00
600
800
1000
evg. token length
avg. token length
avg. token length
ovg. loken lengthh
avg. token length
Figure 3: Comparison of Arena-Hard scores and average token length of responses across 500 prompts for different methods.
failed to match the performance of the SFT checkpoint, highlighting their limited impact on downstream tasks. Con- versely, TPO and TPO-L achieved substantial improvements over other methods. However, TPO displayed performance instability across various dataset sizes, while TPO-L consistently provided stable enhancements, attaining the highest DAA values in most of the Base settings. This emphasizes TPO-L's superior stability and generalizability compared to other methods.
TPO demonstrates higher effectiveness in a preference dataset. Recent frameworks have improved the efficiency of generating multiple high-quality responses (Kwon et al. 2023). However, the current two-preference methods may lose valuable data when responses have similar quality. TPO addresses this issue by utilizing three preferences, which use more data and improve optimization. Consider dataset D f2r, y1, 12, Va ya, ys where y11 V/2 > Ys > y/ y5- We fine-tuned models under three combinations of preferences: {z,, , 1/2, 2/5 }. where y/ 1/gold: 1/2 You M5 y/ {a, 31, V/. ys where V/3 = Yu: and {ær y1. y4, 353. where = Vn- The results in Table 3 show that TPO in the second combination achieves the best performance, likely because of its resilience to judgment noise in the dataset. This demonstrates TPO's capability to exceed the current two-preference methods.
4.2 Verbosity Problem
Recent studies (Park et al., 2024 Meng et al. 2024) identify verbosity as a key challenge in alignment algorithms, where models produce overly long responses during post-training without quality gains. SimPO addresses this by replacing sequence summation in the DPO loss with average likelihood, reducing verbosity. In this section, we analyze how TPO and TPO-L mitigate this issue and assess their effectiveness.
The sequence summation in TPO does not lead to verbosity. To analyze verbosity, we used the Length Control setup from Arena-Hart As shown in Figure 3 TPO consistently ranks among the top three methods for average token length across all settings. Notably, even in the 5,000 Base setting, TPO generates shorter responses while achieving the best performance. Further analysis of token lengths revealed that gold responses are not necessarily shorter than preferred or rejected ones, suggesting that TPO mitigates verbosity by enhancing the model's understanding rather than just shortening responses. More details are available in Appendix G Overall, TPO with the summation term does not exhibit verbosity issues, challenging the findings reported in the SimPO paper.
TPO-L improves the length control ability of the TPO. Figure shows that TPO-L enhances TPO's 's length control ability across the Base settings, except for the 5,000 setting. comparison of response length and Arena-Hard performance between TPO-L and SimPO reveals that incorporating the average likelihood of the response sequence has stronger effect on the TPO loss function compared to the DPO loss function.
4.3 Analysis of TPO
This section explores optimization conflict in CPO compared to TPO and analyzes reward modeling in TPO and TPO-L. We examine the reward distribution of TPO against SimPO and SFT checkpoints across different training sizes and
We evaluated the latest checkpoints reported SimPO repository across henchmarks and used GSM8K and MMLU values from SimPO
paper.
8

---

Published in Transactions on Machine Learning Research (11/2025)
Base Setting (5k)
Base Setting (10k)
Base Setting (20k)
Base Setting (40k)
150
TPO SIMPO
TPO SIMPO SPT
TPO
TPO SIMPO
Frequency
SIMPO
100
50
2.5
5.0
-2.
0.0
2.5
5.0
-2.5
0.0
2.5
-2.5 0.0
2r m rly,, x) rly.
Ar ry x) x(
Ar nly.. - -rly. x)
Ar - rly
x)
150
TPQ SIMPO
TPO
TPO
SIMPO
SIMPO
SIMPO
Frequency
100
-4
-3
-2
-4
33
-2
-1
-4
-3
-2
-1
-4
-3
-2
-1
avg log moly
avg log maly,
avg log ma(y)/)) /x)
avg log mc(kw/k)
Figure 5: Reward modeling analysis on the UltraFeedback test set. Top: Comparison of reward distributions for SFT, SimPO, and TPO across different data sizes. Bottom: Evaluation of log (y(a) as implicit reward across varying data sizes.
evaluate the impact of reward margin 7 adjustments in TPO-L on reward accuracy and model performance. Further analyses are provided in Appendix B
TPO resolves the optimization conflict issue in CPO.
Experimental Observation
If Vgold = Yw and a = 1, the TPO loss function reduces to the CPO loss function. However, TPO consistently outperforms CPO across benchmarks and settings. We analyze why TPO achieves better performance than CPO. Theoretical analysis of the CPO loss function shows that it incorporates a Behavioral Cloning (BC) objective into preference optimization to maintain the policy model within the distribution of preferred responses. It was hypothesized that adding a BC objective for preferred data would improve the increasing likelihood of preferred responses during optimization. However, as observed in prior work (Rafailov et al. 2024a), and supported by our findings, conflict arises where the likelihood of yv decreases during the optimization process in CPO, contradicting the intended objective. As shown in Figure 4 although the BC objective is designed to increase the likelihood of yıe in preference optimization, the opposite occurs, with yn. 's likelihood diminishing instead. We refer to this phenomenon as optimization conflict.
objective
PO objective
objective
Phor
Hellrood
Sallin A
Steps
Stops
4
Vlar
Steps
Theoretical Expectation
Conflict
Figure 4: Difference between the theoretical expectation and the experimental observation of the Preference Optimiza- tion (PO) objective part the CPO loss function during optimization.
TPO enhances the reward distribution as the training data size increases. To compare the reward distribution of TPO and SimPO, we calculate avg. log To (ye (r)) and Ar = r(ye, x) r(y,, x), where yu and y denote the preferred and rejected responses in the Ultraeedback test set. As shown in Figure TPO demonstrates consistent increase in the average likelihood of yo with larger training data sizes, whereas SimPO shows only slight improvement. Additionally, the comparison of Ar demonstrates that TPO maintains broader reward distribution across all dataset sizes compared to SimPO. Overall, TPO effectively enhances reward distribution as the training set size grows.
Increasing the reward margin in TPO-L enhances the reward distribution. To examine the impact of the reward margin in TPO-L, we analyzed the 40,000 Base setting by calculating reward accuracy and visualizing distributions on the UltraFeedback test set. Figure shows that increasing the reward margin extends the Ar distribution. While the average likelihood of y initially decreases up to margin of 5.4, further increases encourage the model to enhance the average likelihood of yoe-

---

Published in Transactions on Machine Learning Research (11/2025)
912
Brescar
280
Reewaad Mazgiia
Jone Mani Soara
ACHES Erreet Accepey (5)
Actenay (00
Mentral Stark
Roward
Ronasd
Rowadd
Figure 6: Study of the reward margin 7. Top: Reward distribution under different 7 values UltraFeddback test set. Bottom: Reward accuracy on the UltraFeedback test set and accuracy Arena-Hard, MMLU-Pro, GSM8K, and MixEval-Hard under different y values.
Increasing the reward margin diffferntt benchmarks. The results in Figure that increasing the reward margin significantly enhances GSM8K accuracy but leads to substantial performance drop Arena- Hard at higher margins. Additionally, higher reward margin encourages longer responses, indicating that the average likelihood approach alone cannot fully control verbosity. Minimal changes were observed in MMLU-Pro performance across different reward margins, suggesting a limited impact. MixEval-Haid results indicate optimal performance at a reward margin of 5.4, with a sharp decline beyond this point.
4.4 Robustness of TPO and DPO
This section investigates the robustness of TPO and DPO in various scenarios. We analyze the impact judgment noise in preference datasets, consider real-world conditions where out of three responses receive identical scores, and evaluate DPO's performance when initialized from more robust SFT checkpoint.
DPO
TPO
MMLU-Pro (%6)
40
30
33.5
31.3
50
45.1
51.2
42.5
30
20
GSMBK
30
MixEval
20
22.4
Arena Hard
4
10
1.7
10
50
100
50
100
50
100
100
%
Noise%
Noise
Noise%
Figure 7: Robustness comparison of TPO and DPO under different percentages of judgment noise.
TPO displays stronger robustness to judgment noise than DPO. evaluate TPO and DPO under noisy conditions, the 40,000 Base setting, assuming that judgment noise affects the preference dataset while the gold dataset remains clean. The SFT checkpoint for DPO is fine-tuned on the clean gold dataset. Noise is defined as incorrect decisions between preferred and rejected responses, with 50% and 100% noise levels introduced into the training set.
Figure 7 shows that as noise increases, DPO performance collapses, with scores on GSMSK and Arena-Hard dropping to zero under 100% noise. In contrast, TPO maintains acceptable performance despite some decline across benchmarks. This stability is attributed to the log wa (Ugola (r)) term TPO, which leverages the gold response to mitigate the impact of noisy preference data. These findings highlight the importance of the BC objective on the gold response in maintaining model performance under noisy conditions.
10

---

Published in Transactions on Machine Learning Research (11/2025)
TPO performs reliably across different hypotheses. TPO assumes the dataset satisfies Ygold Yw Vt, but constructing such data can be challenging when two preferences have similar scores. To evaluate TPO under these conditions, we created a 10.000-example dataset from UltraFeedback, ensuring (3gota) = Syww), with 0.5-point difference from the rejected response. Results in T show that TPO performs similarly in this setup to its original assumption. This finding suggests that even when V/oodd and you share the same score, using distinet responses for V/gold and Vv yields better performance compared to the case where Ygold = Yuu which is similar to the CPO loss function.
Table 4: Comparison of TPO assuming gold and preferred responses differ but share the same quality score.
Method
Quality Comparison
GSM8k
MMLU-Pro
MMLU
MT-Bench
Arena-Hard (Avg. # TL.
MixEval-Hard
TPO
Ngold > Vie > yt
52.2
37.8
65.3
6.7
6.7 (683)
31.2
TPO
Ugold = Yv
51.5
37.8
65.3
6.4
8.2 (642)
32.0
TPO is more effective than DPO while using less data. Recent studies (Tunstall et al. 2023 Sacidi et al. 2025 emphasize the importance of the SFT checkpoint in DPO, as it can be fine-tuned with additional data to enhance performance. In this study, we evaluated DPO's performance using different, stronger SFT checkpoints fine-tuned on 20,000 and 40,000 data points within the UltraFeedback 10,000 Base setting. Results in Table show that while DPO fine-tuned on stronger SFT datasets improved performance, especially on Arena-Hard, its reasoning benchmark scores declined with the 40,000 dataset compared to the 40,000 Base setting. Moreover, despite using twice as much data TPO and TPO-L, DPO only surpassed them on Arena-Hard, demonstrating that TPO achieves comparable performance with significantly less data, highlighting its efficiency.
Table Comparison of TPO and TPO-L with DPO, which uses more robust SFT checkpoints.
Method
SFT Data
Preference Data
GSM8k
MMLU-Pro
MMLU
MT-Bench
Arena-Hard
MixEval-Hard
DPO
10K (ygola)
10K (Yur y1)
36.7
28.8
59.4
6.3
6.1
25.9
DPO
20K (Unota)
10K (yen
49.8
35.8
63.0
6.6
7.1
28.5
DPO
40K (Wyota)
10K (yor, 91)
46.8
32.3
62.3
6.6
7.8
25.2
TPO
none
10K (ygold: Dwes Mt)
52.2
37.8
65.3
6.7
6.7
31.2
TPO-L
none
10K (Noold: Moos 3R)
51.9
38.0
65.4
6.9
7.3
33.7
5
Related Works
Reinforcement learning from human feedback.
RLHF is used to align large language models with human pref-
erences and values (Christiano et al.) 2017 Ziegler al. 2019b
Ouyang al.
2022a
Bai al.
2022).
A classical
RLHF setting involves three stages: supervised fine-tuning stage
(Zhou et al.
2023a
Taori et al.
202.3
Geng et al.
2023
Conover et al.) 2023 Köpl et al. 2023 Ding et al.
2023a
Wang et al.
2024b
Chen et al.
2024a|
Xia et al.
2024), , reward modeling stage Gao et al. 2023 Luo et al.
2023
Chen et al.
2024b
Lightman et al.
2023
Havrilla During
ct al. 2024b Lambert et al. 2024). and policy optimization stage Schhulmnn et al. 2017
Anthony et al.
2017
the final stage, Proximal Policy Optimization (PPO) (Schulman et al. (017) more commonly used. This approach
involves optimizing for maximum reward by interacting with a reward model trained using the Bradley-Terry objective. The application of the RLHF framework transcends various applications, such as ensuring safety (Dai et al.) 2023)
enhancing helpfulness (Tian et al. 2024 Wang et al., 2024c), mitigating toxicity (Amini et al.) [0244] Korbak et al. 2023 Zheng et al.) 2023a). searching and navigating the web (Nakano et al. 2021). and improving model reasoning abilities (Havrilla et al. 2024a). While Reinforcement Learning with Human Feedback (RLHF) enhances model
performance, encounters challenges such as the collection of preference data (Casper et al. 2023), unstable training
processes, and the generation of biased or overly verbose responses (Dubois et al. 2024 Singhal al. 2023 Wang
et 2023).
Offline vs. Iterative preference optimization. Owing to the challenges and complexity of online preference optimization techniques(Zheng al. 2023c Santacroce et al., 2023), recent works have proposed simple and efficient offline algorithms. One among them is Direct Preference Optimization (DPO) al. 2023). It makes of
11

---

Published in Transactions on Machine Learning Research (11/2025)
implicit reward, which is in terms of the parameterized policy and reference policy. Due to the absence of an explicit reward model, DPO is limited in its ability to sample preferences from the optimal policy. Addressing this, works like (Zhao et al. 2023 Liu et al. 2024b) explore augmenting preference data using a trained or refined SFT policy. Iterative training methods, which continuously update the reference model with the most recent policy model or generate new preference pairs at each iteration, have been explored by works (Dong et al. 2024, Kim al., 2024 Rosset al. 2024 Cen al. 2024). this work, we experiment an offline setting,
Xiong al. 2024 Yuan et al. 2024 Xie
2024
without any iterative training.
Preference optimization objectives. Besides DPO, various preference optimization algorithms have been proposed. A variety of preference optimization objectives have been proposed besides DPO (Dong et al.) 2023 Liu et al. 2024a
Song et al. 2024 Yuan et al. 2023). However, this family of methods either relies on explicit rewards during preference optimization or demands a fine-tuned reference model. In contrast, TPO avoids this by using three preferences. Similar to TPO, simpler objectives that do not rely on a reference model have been proposed (Hong et al.
2024a Xu et al.
2023). In this work, we compare TPO to a series of offline algorithms, including RRHF Yuan et al.
(2023). SL
Zhao al. (2023), DPO Rafailov et al. (2023), IPO Azar al. (2023), CPO Xu et al. (0024) KTO Ethayarajh et al.) (024), ORPO Hong et al. (2024a), and SimPO Meng et al. (2024), and find that TPO can outperform them in both efficiency and performance. Similar to TPO, SimPO does not impose KL regularization, however, the robustness of its effective learning is dependent on various factors like preference datasets with diverse domain coverage. Whereas TPO is not constrained by this.
6 Conclusion
In this study, we introduced the Triple Preference Optimization (TPO) method, a one-step preference optimization approach that enhances both the instuuttionfoolowing and reasoning abilities of the policy model. We developed TPO-L, length-controlled variant, by defining reward margin and replacing summation with the average response sequence. Our results show that TPO and TPO-L outperform existing methods across benchmarks, effectively address the verbosity problem, and are less sensitive to dataset size, particularly in the case of TPO-L. Additionally, TPO demonstrates greater robustness to noisy data and performs better on less data than DPO.
Limitations and Future works
Exploring the Quality Margin Between Responses. The main hypothesis of TPO focuses maintaining margin difference between gold and preferred responses. Although we evaluated TPO in scenarios where the gold and preferred responses share the same quality score, further exploration of the margin between gold and preferred responses, as well as between preferred and rejected responses, is still needed. Our analysis of the UltraFeedback dataset indicates that increasing the margin between gold and preferred responses significantly reduces the number of qualifying samples, leaving only 10,000 that meet the conditions. Therefore, a more in de-tepth analysis of this area is necessary.
Exploration on Reward Margin. Although TPO-L demonstrated impressive performance across various settings, requires considerable effort to determine the optimal value for the reward margin 7. This limitation restricts the applicability of TPO-L and similar methods, such as SimPO, across different tasks. Developing dynamic approaches to adjust the reward margin automatically based on the specific task could be interesting direction for future work.
the TPO loss function incorporates two objective functions. Recent studies (Dai et a 2023; Zhou et a 2023b) have explored multi-objective preference optimization methods, which require preparing multiple preference pairs. Safety is an important domain where multiple objectives, such as safety and helpfulness, are equally critical. Exploring the impact of TPO on such multi-objective tasks could provide valuable insights. A key advantage of TPO compared to other methods is that data collection for TPO is significantly easier than for objti-oee methods like DPO.
Safety and Honesty. As discussed in Section 2
In-Depth Exploration of Reasoning. As shown in Tables 2 and 12 TPO and TPO-L achieved strong results on reasoning benchmarks. However, further demonstrate TPO's 's effectiveness on reasoning tasks, we encourage researchers to focus reasoning-specific datasets such UnSeenTiimeQA Uddin et al. (2025) and explore preference
12

---

Published in Transactions on Machine Learning Research (11/2025)
optimization designed for reasoning tasks. Developing and evaluating reasoning- based preference datasets would provide deeper insights into TPO's capabilities.
Online Learning. Recent study (Park et al. 2024) suggests that online versions of DPO address the overfitring challenges associated with offline versions, often achieving better performance. Exploring this area could be particularly valuable, as superior results on datasets like UltraFeedbakkAAmokk which closely align with the concept of online policy learning, support the feasibility of online learning approaches.
Ethical Considerations
The authors state that in this work, Al assistants, specifically Grammarly and ChatGPT, were utilized correct grammatical errors and restructure sentences.
References
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with offset. arXiv preprint arXiv:2402. 10571, 2024.
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. ArXiv, abs/2310. 12036, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307. /5217, 2023.
Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentivizer preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2405. 19320, 2024.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. AlpaGasus: Training a better Alpaca with fewer data. In ICLR, 2024a. URL
httpss://penres wttpeeiwwwweeeeeeeeeeeeeeeeeeeee foreet/frrmmiieeee dtt.://oommm...dddgggxh
Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: Disentangled reward mitigates hacking in RLHF, arXiv preprint arX1v:240..0311, 2024b.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. ArXiv,
2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 10Xiv:2110.14168. 2021.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned LLM, 2023. URL https: //ww::://www.... databricks waatabrrccss.com/bl/////0.0..
-o oreere/eeeellllaaaaaeeeeettotttr combb//o//////22/////2
13

---

Published in Transactions on Machine Learning Research (11/2025)
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeebback: Boosting language models with high-quality feedback. 2023.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang, Safe RLHF:
Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310. 2023.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. EMNLP, 2023a.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng. Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023b.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, SHUM KaShun, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling online RLHF. arXiv preprint arXiv:2405.07863, 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled AlpacaEval: simple way to debias automatic evaluators. ArXiv, hass/2004.04455. 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model alignment prospect theoretic optimization. ArXiv, abs/2402.01306, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Xinyang Geng, Arny Gudibande, Hao Liu, Eric Wallace, Pieter Abbecl, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April, 1:6, 2023.
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis. Jane Dwivedi- Yu, Maksym Zhuravin- skyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint 2rXiv:2403.04642. 2024a.
Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yuu Maksym Zhuravinskyi, Eric Hambro, and Roberta Railneau. GLoRe: When, where, and how to improve LLM reasoning via global and local refinements. arXiv preprint arXiv:2402.1096. 2024b.
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive prefence learning: Learning from human feedback without rl. arXiv preprint arXiv:2310.1363) 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazcika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. 2021.
Jiwoo Hong, Noah Lee, and James Thorne ORPO: Monolithic preference optimization without reference model. ArXiv, abs/2403.07691, 2024a.
Jiwoo Hong, Noah Lee, and James Thome. Reference-fre monolithic preference optimization with odds ratio. arXiv preprini arXiv:2403.0769), 2024b.
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie- Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7B. ArXiv, abs/2310.06825, 2023a.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Lim-blender: Ensembling large language models with pairwise comparison and generative fusion. In Proceedings of the 6lth Annual Meeting of the Association for Computational Linguistics (ACL. 2023), 2023b.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, and Chanjun Park. sDPO: Don't use your data all at once. ArXiv, abs/2403. 19270, 2024.
Andreas Köpf. Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Minh Nguyen, Oliver Stanley, Richárd Nagyfi, al. Openassistant model alignment. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
large language
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference Machine Learning. pp- 17506-17533. PMLR, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng. Lianmin Zheng, Cody Hao Yu. Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khy- athi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hanna Hajishirzi. RewardBench: Evaluating reward models for language modeling. ArXiv, abs/2403.13787, 2024.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. 2012.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsoureed data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 1139, 2024.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305. 20050, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In ACL, pp. 3214-3252, 2022.
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, et al. LiPO: Listwise preference optimization through learning-tor torrnnnggt--r...... arXiv preprint 0100022002.01878, 2024a.
Tiaqqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations, 2024b. URL
net fort//foum?eo picoorrmmiiiibbl d=xb ]Swwr
Alexander Lotov and Kaisa Miettinen. Visualizing the Pareto Frontier, pp. 213-243. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008. ISBN 978 778-3-540-88908-3, doi: 10.1007/978-3-540 3-0-------------33333-8 URL httpp:://ddo
org/10. 000.0001000777---55550000 889088-88888-111
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.099: 09588,, 2023.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint 1rXiv:2405.14734, 2024.
Kaisa Miettinen. Nonlinear multiobjective optimization, volume 12. Springer Science & Business Media, 1999.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. WebGPT: Browser-assisted question-answeriig with human feedback. arXiv preprint arXiv:2112.093332 2021.
Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng. Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. Mixeval: Deriving wisdom of the crowd from Ilm benchmark mixtures. arXiv preprint arXiv:2406.66500 2024.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022a.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022h.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403. /9159, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NeurlPS, 2023.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to Q": Your language model is secretly a q-function. arXiv preprint arXiv:2404. 12358, 2024a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. ArXiv, abs/2404.03715, 2024.
Amir Saeidi, Shivanshu Verma, Md Nayem Uddin, and Chitta Baral. Insights into alignment: Evaluating dpo and its variants across multiple tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop). pp. 409-421, 2025.
Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, and Yelong Shen. Efficient RLHF: Reducing the memory usage of PPO. arXiv preprini arXiv:2309.00754, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint wrXiv:1707.06447 2017.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in RLHF, arXiv preprint arXiv:2310.03771, 2023.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In AAAI, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and lon Stoica. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784, 2024.
Rohan Taori, Ishaan Gulrajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An custruction-followiin Ilama model, 2023.
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine-tuning language models for factuality. In The Twelfth International Conference on Learning Representationss 2024. URL /htpps://
openreview www.ewwweeeteee het//orummidd-M222... 222fmm
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Lewis Tunstall, Edward Beeching. Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment, arXiv preprint arXiv:2310.16944 2023.
Md Nayem Uddin, Amir Saeidi, Divij Handa, Agastya Seth, Tran Cao Son, Eduardo Blanco, Steven Corman, and Chitta Baral. Unseentimeqa: Time-sensitive guestion-answeriigg beyond Iims' memorization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1873-19111 2025.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint MiXi::2401006080 2024a.
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. OpenChat: Advancing open-soure language models with mixed-quality data. In ICLR, 2024b. URL AOJyEhWYHE
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang. Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of LLMs for diverse user preferences: Directional preference alignment with multi-objective rewards. In ACL, 2024c.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang, Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Findings of EMNLP, 2024d.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah Smith, Iz Beltagy, et al. How far can camels go? exploring the state instruction tuning on open resources. Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024c.
Zhilin Wang, Yi Dong. Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training cop-performing reward models. arXiv preprint arXiv:2406.088 08673, 2024f.
Tianhao Wu, Janice Lan, Weizhe Yuan, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Thinking llms: General instruction following with thought generation. arXiv preprint arXiv:2410. 10630, 2024.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: Selecting influential data for targeted instruction tuning. In ICML, 2024.
Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit aapppooximtaio for sample-efficient rlhf. arXiv preprint arXiv:2405.2 2024.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang. Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint, In Forty-first International Conference on Machine Learning, 2024.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. ArXiv, abs/2401.08417. 2024.
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.1666 2023.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF; Rank responses align language models with human feedback. In NeurlPS, 2023.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self- rewarding language models. arXiv preprint arXiv:2401. 10020, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Marquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 4791 4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1/11866/11/111914777
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. SLiC-HF: Sequence likelihood calibration with human feedback. ArXiv, abs/2305. 2023.
Chujie Zheng, Pei Ke, Zheng Zhang, and Minlie Huang. Click: Controllable text generation with sequence likelihood contrastive learning. Findings of ACL, 2023a.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In NewrIPS Datasets and Benchmarks Track, 2023b.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of RLHF in large language models part I: PPO. arXiv preprint arXiv:2307.04966 2023c.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan lyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. LIMA: Less is more for alignment. NeurlPS, 2023a.
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one- preference-for-all. Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708, 2023b.
Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, abs/1909 2019a. URL
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08599, 2019b.
18

---

Published in Transactions on Machine Learning Research (11/2025)
Appendix
A Derivation
A.1 Deriving the optimal policy under the Preference Objective
In this section, we derive the optimal policy achieved by optimizing the objective in Equation from Section 2.2. 1 given prompt x, the objective can be analogously written follows:
For
max By-r(vl)) [r(x, y) 3 log (y)2)) s.t. s.t. Σ [1y22)
Next, we form a I Lagrangian for the above objective with À being the Lagrangian multiplier.
Σ ([glz)) > Ex(llx))
C Σ x(yla)((z,))
Differentiating & with respect (y]rr) results in,
Oc
= r(x, v) - 3 log [ggg())))+ + A.
To obtain the optimal policy, we can set the above equation to zero and solve for I (yy
r(a,, v) - 3 log क (y)z( +
> O,
log (y)l) 1
(2 8)
3
* (y/x>) = exp (r( exp
--
1).
3
Since E, (3)22)) = 1, the second exponent is a partition function that does normalization as shown below:
Σ exp 1 r(
exp( - 1) 1,
8
1 1) Σ exp 1 exp
Hence, the partition function 7(x) = E, exp (3r(2.1)) and the optimal policy IT, (y(ar) induced by reward function r(a, y) is therefore given by,
1 Z(x) exp ( v)).
1
x (y]x)) =
(1)
Now, we can express the reward function in terms of an optimal policy by performing some algebraic transformations on Equation [ as shown below,
(vlz).2((x) = exp ((),
19

---

Published in Transactions on Machine Learning Research (11/2025)
Taking logarithm and multiplying by 3 both sides,
r(z, y) = 3 log T. (u/=) + 3 log ((x).
(2)
A.2 Deriving the Gradient of the TPO Objective
In this section, we derive the gradient of the TPO objective:
VaLTro
(x B M log To (V(rres12) + log (B (0(1wl>) 8 log (x())))))
(1)
We can rewrite the RHS of the Equation 1 as
VoLTro - WD
V log no (Vrefla))
Vo log ( log no (v) 3 log To(u)=)))
(2)
(i)
(b)
In equation 2 the part (b) can be rewritten with
u = 3 log (ym - B log (
Vo logoo((u) = o(u)
Voo(u).
Va Valogo(u) á (u)
Vo(u).
o(u)
Using the properties of sigmoid function function a' (w) = (1) (u) and and ((-u) = (u)),
vologa(u)= o(u)( - o(u)
Vo(u)
a(u)
Vologo(u) = o(-u)
Va (u) (3 log (u((=) - B log (x(w)))) [o log (()ul=))))) Va log (og<(1)(<))
(3)
Plugging Equation
3 into Equation 2 we get.
VeLrro = Be [a V log क (Uree (z)
+ Bo log Re (e( 3 log Ne (vwl)))
x Vo log π (vwll)) - Vo log (x))
(4)
A.3 Theory Behind TPO
In this section, we provide theoretical foundation for the TPO algorithm, drawing inspiration from Rafailov et al. (2024b). We observe that the preference optimization objective aligns with the principles of Bradley-Terre Terry model, where the reward parameterization is defined as r(x, 3) = 3 log T (y) Consequently, we optimize our parametric model my in a manner similar to reward model optimization, as shown by Ouyang et al. (2022)). We expand on the theory underlying this reparameterization of the reward function, illustrating that does not constrain the range of reward models that can be modeled and ensures accurate retrieval the optimal policy. We initiate this discussion by following the insights presented in DPO about the equivalent class of reward models.
20

---

Published in Transactions on Machine Learning Research (11/2025)
Definition 1. Two reaari r(z, y) and r' (z, y) equivalent iff (fr(2, -r' (x. v) g(x)) for some function
g.
We can state the following two lemmas, as is apparent that there exists an equivalence relation, dividing the set of reward functions into distinct classes.
Lemma Under the Plackett-Luce, and in particular the Bradley-Terry preference framework, two reward functions from the same class induce the same preference distribution. (Rafailov et al. 2024b)
Lemma 2. Two reward functions from the same equivalence class induce the same optimal policy under the constrained RL problem. (Rafailov et al. 2024b)
The proofs are shown in Appendix A.4
Theorem 1. Under mild assumptions, all reward classes consistent with Plackett-Luce models can be represented with the reparumeterization r r(z, v) = -3 [oga())))) for some model π r(y(ar). (Rafailov al. 2024b)
As proposed in DPO, upon imposing certain constraints the under-constrainet Plackett-Luce family of preference models, such that we preserve the class of representable reward model, it possible to explicitly make the optimal policy in Equation 5 from Section I analytically tractable for all prompts The theorem is elaborated in Appendix We further elaborate on our theoretical basis for defining and optimally addressing the TPO objective within a multi-objective optimization framework.
A.5
Definition Let Js denote Sh objective, S denote the feasible policy space, then mulli-objective optimization setting, policy
S is said be Pareto optimal there does not exist another policy S succ that [((T) S fi(x") for all -,..., and fz( < [5(x*) for least one index j.
Looking at the objectives in Equation 8 and Equation 9 from Section 2.2.1 it is obvious that optimizing them together is non-trivial; that is, there does exist policy that is optimal with respect to both objectives. It can be seen that the objectives are conflicting with each other, especially when Ygold ~ Yus as one objective is maximizing the log probability and the other is minimizing the log probability. This means that the objectives are at least partly conflicting. For multi-objective problem, Miettinen (1999) shows that optimizing one objective and converting the other objective/s This shows that optimizing the TPO objective, which is a bi-objective problem, gives optimal policy that is Pareto optimal as defined in A.3
constraint with upper bound, the solution to this constrained problem is Pareto optimal.
A.4 Proof of Lemma
In this section, we will prove the lemmas from Section A.3
Lemma 1 Restated. Under the Plackett-Luce preference framework, and in particular the Bradley-Terry framework, two reward functions from the same equivalence class induce the same preference distribution.
Proof. Let's consider two reward functions, r(x, y) and (xx y). They are said to be equivalent if they can be related by r' (xx y) = r(z, y) + g(x) for some function g. We analyze this in the context of the general Plackett-Luce model, which includes the Bradley-Teri model (special case when rankings generated by given reward function r(x, y) pr- Given any prompt ær responses y1,... yK. and ranking
2). Here, we denote the probability distribution over
21

---

Published in Transactions on Machine Learning Research (11/2025)
T, we can establish the following:
exp(r' Vr(k)))
Pe [T y1..... ,,.,..,,,,.....
yK, x)
exp(r"
k=1
j=k (x)pr((,,)rr)) g(x))
k
k=
K
exp((,
exp(g(x)) Σ/= exp Br()))
exp(r(x1, exp(r(z,V:(x)) (+(k))
k Zjzk exp(r(z. (r()))
= pr VI.....yK...
This completes the proof.
Lemma 2 Restated. Two reward functions from the same equivalence class induce the same optimal policy under the constrained RL problem.
Proof. Let's consider two reward functions, r(v; y) and (1x, y). They are said be equivalent if they can be related by r' (zz y) r(x, y) (x)) for some function g. Let nT, and n,, be the optimal policies induced by their corresponding reward functions. By Equation from Section 2.2. I for all x, y we have.
1 (F)(
Ty- x) = Ey exp (b)((x)))
exp
1
exp { (g(((2)))) +g(z))
Ey exp (((((1x))) + s(x)))
exp (fr(oss)) exp (30(o))
exp (bo(x)) E, exp (2r(z)))
1 Ey exp (tr(a,n)
cxp (
(3 x).
This completes the proof.
A.5 Proof of Theorem
Theorem 1 Restated. For a parameter 8 > O, all reward equivalence classes can be reparameterized as r(z, 8)
3 log N (y]xz) for some model x))(x))
Proof. Consider a reward function r(z, y). which induces optimal model (y(zz) under the MERL framework, which takes the form as shown in 5 from Section 2.2.1 Following, Equation 2n Section of Appendix, we have:
r(z, v) 3 log.o, (ylax) + /3 log (xx)
where Z(x) = E, exp (5, v)) is the partition function of the optimal policy induced by the reward function r(z, v). Let r' (x, v) be a new reward function such that r' (x, v) = r(x. v) Blog Z(x)). It is obvious that the new reward function is within the equivalence class of r, and we have:
r' (x, -(x,v) log 7(x))
22

---

Published in Transactions on Machine Learning Research (11/2025)
From the Equation I we get
r' (x, y) = (l + Bog (()) - 3 log (xx),
r' (z. y) = 3 log ग., (y|z).
This completes the proof.
For parameter 3 > 0, every equivalence class of reward functions has unique reward function r(z, v), which can be reparameterized as r(x, y) = 33 log क (y(e) for some model π (y)2x)
Proposition 1.
Proof - by Contradiction. Let us assume that we have two reward functions from the same class, such that (z. v) = r(z, y) + g(x). Assume that r (z. y) 1og (ylz) for some model π (y/la) and r(x, y) = 3 log (y/a²) for some model TT (t) such that * We then have,
r (z. v) r(x, v) + g(r)
B (og + g(x)
logg((y(>)) + log exp (3((x))
log x (yl exp g0(x))
B log (y)æ))
for all prompts and completions Then, we must have T exp (30(2)) = =x (ylæ:). Since these are probability distributions, summing over on both sides,
Σ [m()at) exp [o(+))]]
(x0())) = Σ (g]zz) exp (3(()))))))
Since 8 > 0. (z))(z) must be for all Therefore, will have r(z,v) (I,z), which contradicts our initial condition of π # x.
Thus, by contradiction, we have shown that every reward class has a unique reward function that can be represented by the reparameterization Theorem
A.6 Insights into the TPO Update
A deeper mechanistic understanding of TPO can be achieved by analyzing the gradient of the LrPO loss function. The expression of this gradient in relation to the parameters 0 is as follows:
V VoLrro
E (n. 11) D a Ve log iT (Ygolal x:)
likelihood Ugsa
+ Bo (9 log no (mllz) log Ie (yw (z)
increase weight when reward estimate wrong
V log T (Yre (z) V log π (B1|2:) 1]
(2)
increase likelihood of
decrease likelihood yr
where r(z, 3) = 3 log ne (y xx) is the reward inherently determined by the policy model тT. Intuitively, the gradient of the TPO loss function works to increase the likelihood of the gold completions ygold- simultaneously enhancing the preference aspect by amplifying the likelihood of preferred completions Voe and reducing the likelihood of the less-preferred completions y,, which weighed by how incorrectly the implicit reward model orders the preferences.
23

---

Published in Transactions on Machine Learning Research (11/2025)
DPO
TPO
66.3
65-
Reward Accuracy (%
63.2
63.4
60
57.4
55.4
50.3
50
20
Data Size (k)
Figure 8: Comparison of TPO and DPO in terms of Reward Accuracy on the UltraFeedback test set across different training set sizes.
B Further Insights on TPO
Likelihood of response sequence in TPO a better implicit reward than KL-divergence in DPO. In Section demonstrated that in TPO, log ma (y(rr) functions an implicit reward. To compare this implicit reward with that of DPO, we calculated it for both methods the UltraFeedback test set. The results in Figure 8 show that TPO outperforms DPO across various data sizes. These findings suggest that the implicit reward in TPO has stronger effect compared to DPO. Additionally, it is evident that the average of log Ia (ylzr) significantly influences the win rate, particularly with larger dataset sizes.
2.2
Q: Given the following list of numbers, find the median: 1, 5, 8, 17
Process UltraFeedback
DultraFeedback
overall score
0.5
Response
Response
Response
Response
Of course! As responsible and ethical assistant, must first point the question contain valid list numbers..
To find the median of the given list of numbers, first, arrange the numbers ascending order: The median of
0.5
The
To find the median, we first need to put the numbers order from smallest largest: Therefore, the median of the list 7.
median the given is
responses
3/2
3/3
3/4
given list is
Gold
>
Chosen
Rejected
Drro
score:
iscore: 6.5
Iscore: 7.5
1score: 8.5,
Supervised FineTuning (SFT)
Direct Preference
Gold
Pre-trained
Optimization (DPO)
AAligued
Drro
chosen
Triple Preference Optimization (TPO).
Comparison of Optimization Process for TPO and DPO
rojected
Pre-trained
Alligned
Figure Overview of the data and optimization processing. Left Top:
Visualization of the data structure in the UltraFeedhack dataset. Right Top: Selection of gold, preferred (chosen), and rejected responses based overall scores generated by GPT-4. Bottom: Optimization differences between TPO and DPO.
24

---

Published in Transactions on Machine Learning Research (11/2025)
C Implementation Details
We used Llama-3-8B for the Base setting, and Llama-3-8B Innttuutttutt for the Instrcut setting. Hyperparameter tuning is essential for optimizing the performance of preference optimization methods, To identify the best hyperparameters, we explored various learning rates [3c-7, 5e-7, 6e-7, lc-6] and batch sizes [32, 64, 128, 256]. Our observations indicate that preference optimization methods perform best with a batch size of 32 for a training size of 5,000, 32 for 10,000, 64 for 20,000, and 128 for 60,000. However, for large datasets like 60,000, TPO performs best with a batch size of 256. Based on these findings, we fixed these batch sizes for all preference optimization experiments.
Table 6: The hyperparameter values in TPO and TPO-L utilized for each training setting.
Training Size
Model
Method
a
3
?
Learning Rate
Batch Size
5K
Llama-3-Base
TPO/TPO-L
1 0.01
-00.5
5c-7
32
5K
Mistral-vO.3-BBl
TPO/TPO-L
1/0.05
0.01/2
-/1.6
5e-7
32
10K
Llama-3-Bass
TPO/TPO-L
0.01
-/3
5e-7
32
10K
Mistral-vO.3-Bar
TPO/TPO-L
1/0.05
0.01/2
-/1.6
5e-7
32
20K
Mlma-3-Bass
TPO/TPO-L
0.01
-/1.5
5c-7
128
20K
Mistral-vO.3-Base
TPO/TPO-L
0.01/2
-/1.6
5e-7
128
40K
Llama-3-Base
TPO/TPO-L
0.01
-110
5e-7
64
40K
Mestral-vO.3-Ba
TPO/TPO-L
1/0.05
0.01/2
-/1.6
5e-7
64
60K
Alama-3-Instrre
TPO/TPO-L
0.05
0.01/10
-/3
le-6
256
60K
1antral-O.22-nntttvve
TPO/TPO-L
0.05
0.01/2.5
-/0.3
le-6
256
Additionally, we the maximum sequence length to 1024 for Base setting and 2048 for Instruct setting and applied cosine learning rate schedule with 10% warm-up phase for the preference optimization dataset. We followed Table 7
in SimPO (Meng et al. 2024) for search on the hyperparameter ranges used for the baseline methods, while Table 6 lists the hyperparameters for TPO and TPO-L under each experimental setting. Moreover, all the training experiments in this paper were conducted on 8xA 100 GPUs.
MMLU-pro (%)
ARC (25) (%)
TruthfuIQA (0) (%6)
HellaSwag (10) (%)
0.0%
0.05
0.05
0.05
Appra
1.0
37.76
37.48
37.50
10
58.45
58.45
1.0
55.09
Alphe
100
81.91
81.77
12.00
374
554
1.8
17.85
32.43
58.96
9/1
Winogrande (5) (96)
MMLU (5) (%)
GSMBK (5) (9)
Average (%6
Apha 1.5 1.3 0.05
77.98
77:11
77.51
0.05
65.38
65.23
65.19
0.05
52.99
51.63
52.24
0.05
61.30
60.98
01.04
-125
77.51
77.35
77.98
1.9
65.35
65.24
65.15
Aphie
1.0
$2.16
$3.07
$2.01
Aptre
1.0
$1.32
61.22
61.12
$120
1515
29
-S15
77.35
77.51
77.35
1.5
65.34
65.38
65.13
1.5
51.25
50.80
52.46
1.5
61.14
61.03
61.14
0.01
6.1
2.0
0.51
0.1 Beta
2.0
0.61
0.1 Beta
2.0
0.01
D11
2.0
Beta
B4ta
Figure 10: Comparison of TPO on different hyperparameters.
Exploration on and 3 in TPO. As shown in Figure 10 we investigated the influence of a and 8 on TPO performance. Our findings indicate that the optimal performance range lies within a E [0.05, 1] and 3 E [0.01, 0.1]. However, for Instruct models, the best performance was achieved at lower value of 0.05. This suggests that the regularization term a plays crucial role in optimizing different types of models. Given that Instruct models undergo extensive supervised fine-tuning on large instruction datasets, they are already well-regularized. Consequently, lower
https://hugging wwwcce..ccmmme www.//////aa////m/ htape-111aaaaaaaaa-aaa--
Instruc
25

---

Published in Transactions on Machine Learning Research (11/2025)
regularization strength is necessary to prevent overfitting and to allow the model to learn complex patterns from the data.
Impact of Batch size per device on Arena-Hard for the instruction-tuned models. We investigated the performance of TPO under various batch sizes and accumulation configurations while keeping the total batch size fixed at 256 across 8 GPUs. For instance, we experimented with batch size per device of 2 and gradient accumulation steps of 16 (Scenario 1), as well as a batch size per device of 4 and gradient accumulation steps of 8 (Scenario 2). Our results, summarized in Table 7 indicate that smaller batch sizes per device can negatively impact a model's instruction-follvwii capabilities, specifically on Arena-hard.
Table 7: Comparison of TPO
different hyperparameters Arena- Hard.
Method
Scenario
Scenario 2
TPO₂²
42.0
40.6 (-1.4)
TPO TPO,4
42.4
40.8 (-1.4)
39.4
38.2 (-1.2)
D Mistral Results
Table 8: Comparison of TPO and DPO on Instruction Benchmarks in the Instruct Setting for Mistral model.
Method
Arena-Hard
MixEval-Hard
DPO
19.9
36.5
TPO 02
26.2 (+6.3)
36.3 (-0.2)
TPO 93
26.3 (+6.4)
39.1 (+2.6)
TPO
27.2 (47.3)
39.4 (+2.9)
To evaluate the generalizability of the TPO method, repeated all experiments
using Mistral models. For the Base setting, we used Mistral-7B-v0.33 and for the Instruct setting, we used Mistral- Instruct-v0.22 In the Base setting, we utilized the UltraFeedback dataset, following the same setup described Section dataset, which was introduced in the SimPO paper. This dataset was created by generating five different responses per prompt using Mistral-7B-S0t and ranking them with PairRM (Jiang et 2023b), a reward model.
For the Instruct setting, we employed the Satira--UttraFeeeaaaa---aai
Table 9: Comparison of TPO and DPO on Instruction Benchmarks across the Base Settings for Mistral model.
UltraFeedback (5k)
Ultrafeedback (10k)
UltraFeedback (20k)
UltraFeedback (40k)
Method
Arena-Hard MixEval-Hard Arena-Hard MixEval-Hard
Arema-Hard MixEval-Hare
Arena-Hard MixEval-Hard
DPO
1.4
29.1
0.6
31.1
1.2
28.3
1.2
29.2
TPO
4.3 (42.9)
30.8 (+1.7)
6.2 (45.6)
34.4 (+3.3)
4.5 (43.3)
31.9 (+3.6)
7.4 (+6.2)
32.3 (43.1)
The results in Table II indicate that TPO and TPO-L in the Base setting significantly outperform other methods on downstream tasks, particularly GSM8K, MMLU, and MMLU-Pro. Additionally, TPO in the Instruct setting is the only method that achieves better performance on GSM8K compared to the SFT checkpoint. Furthermore, in the Instruct setting, TPO outperforms all other methods in terms of the average accuracy across downstream tasks.
Due to the cost associated with dnstruction-follwwiii benchmarks, we limited experiments to comparisons with DPO methods in the Base and Instruct settings. The results in Tables 9 and 8 demonstrate that TPO not only surpasses DPO on downstream tasks but also outperforms DPO on Arena- and MixEyal--Had. These findings highlight the effectiveness of TPO compared to DPO as a primary method across various settings.
We also examined the impact of different chat templates across various benchmarks. The results in Table injddicate that using the chat completion template designed for Mistral-Base the Instruction version of Mistral reduces instruction-folloiingg performance but positively impacts downstream tasks, particularly GSM8K.
26

---

Published in Transactions on Machine Learning Research (11/2025)
Table 10: Comparison of TPO on different chat templates in Instruct setting for Mistral model.
Method
Chat Template
GSM8K
Arena-Hard
MixEval-Hard
gP0,2
Default
40.6
26.2
36.3
TPOg₂
Pre-trained
43.2 (+2.6)
24.9 (-1.3)
38.9 (+2.6)
E Downstream Task Evaluation
To further investigate the impact of preference optimization methods, we evaluated their performance on various downstream tasks. Results for MMLU and GSM8K are included in the main text, while additional results for MMLU, ARC, HellaSwag, TruthfulQA, Winograd, and GSM8K are presented in Table 12] Following established evaluation protocols, we report results for all models. TPO's primary goal is to enhance model's reasoning ability while simultancously improving its instruction-followii impressive results on both GSM8K and Arena- simultaneously.
performance. As noted in the main text, TPO demonstrated
The results in Table 12 indicate that TPO and TPO-L also achieved strong performance on ARC and showed notable improvements in the TruthfulQA benchmark. Another significant observation is their performance on HellaSwag, where TPO and TPO-L consistently outperformed other methods across all settings. The performance of TPO and TPO-L differs depending on the dataset size. While their results on smaller datasets, such as 5,000 and 10,000 samples, are slightly smaller than other methods, they outperform the alternatives on larger datasets, including 20,000 and 40,000 samples, as well as in the Instruct settings. These findings further support the conclusion that TPO and TPO-L have substantial impact on enhancing a model's performance on reasoning benchmarks.
F
GPT-4o Judgment for Arena-Hard
Comparing the Arena-Hard results in the SimPO paper with our study on the same benchmark highlights significant discrepancy due to differences in the Judgment models used. In Aard, the default Judge is GPT-4-turbo (gpt-4- 106-prevveww,, but we replaced with GPT-4o for our evaluations. The primary reason for this change is the improved reliability of GPT-40, which has shown better performance compared to the earlier version across different benchmarks. Additionally, the lower cost of evaluating each model per judgment was another motivation for selecting GPT-4o.
GPT-4o
GPT-4-turbo
20
Judgmmnt Cost Per Model: GPT-4o: 5.54
GPT-4-turbo: 26$
15
Error Rate (%)
13
10
10
6
5
4
0
Volunteer A
Volunteer
Volunteer
Figure 1 11: Comparative analysis of GPT-4o and GPT-4-turbo, focusing on error rates in judgment of responses for Arena-Hard and cost per model.
We loaded the last checkpoint reported the SimPO repository for the current methods and evaluated them across different henchmarks. We also used values for downstream tasks reported in the SimPO paper.
27

---

Published in Transactions on Machine Learning Research (11/2025)
To validate this decision, conducted experiment identify the best model for judgment. Using the same responses generated by the TPO model on the Arena-Hard benchmark, we evaluated them with both GPT-4-turbo and GPT-4o. We then selected 100 samples and hired three volunteer researchers to assess the judgments generated by the two models. The results, presented in Table IT] show that GPT-4o's 's judgments are more closely aligned with human decisions compared to GPT-4-turbo. Based on this finding, we evaluated all models our study using GPT-4o the Judgment model on Arena-Hard.
TPO and TPO-L achieve better performance than others GPT-4o Judgment. The results in Table demonnsttte that in the Instruction setting, TPO outperformed all other optimization methods, ranking first the Arena-Hard Benchmarks and MT-Bench with scores of 42% and 8.22, respectively. Notably, GPT-4o judgments the trena-Hand benchmarks showed stronger alignment with human perspectives, highlighting its reliability.
G Token Length Analysis
observed that TPO, even with summation in the loss function, produces shorter responses compared to DPO and, in some settings, compared to SimPO. One possible explanation could be that the gold response used in TPO is shorter in length compared to the preferred and rejected responses. In this section, we analyze the average token length of gold, as well as preferred and rejected responses across different settings.
In Section 4.2
Clesen
Natataal
UltroFeddback
Liama3 Ftraaae
Mistral.uttaFeedsskkkkkkkkk
400.0
Auarage Tokon Length
300.0
200.0
0.0
X4
Y2
V)
Settings
Figure Comparison of token lengths for gold, chosen, and rejected responses across various datasets.
In the Base setting, models are fine-tuned on UltraFeedback. The token length analysis in Figure 12 reveals that the average token length for chosen and rejected responses shorter than that of the gold responses. This observation indicates that current preference optimization methods, even with shorter responses, tend generate longer outputs compared TPO checkpoints, as shown in Figure
In the Instruct setting, other models are fine-tuned gold and rejected responses, while TPO explores performance using different chosen responses. It is worth noting that
Arrooaaaaa----MRMMMM and Sattral-UUtrFeeedbaa PairRM datasets provide five ranked responses for each prompt. Even in this scenario, TPO generates shorter responses compared to most current preference optimization methods.
In summary, as shown in Figure [2] the average token length of gold responses not shorter than that of chosen or rejected responses. This indicates that the shorter responses generated by TPO, compared to methods like DPO that use summation in their loss function, are due to the influence of the Behavioral Cloning objective on different response types. This differentiation between preferred and rejected responses is additional advantage of TPO.
We loaded checkpoint reported SimPO repository for the methods evaluated them different benchmarks.
28

---

Published in Transactions on Machine Learning Research (11/2025)
Table 11: Downstream task evaluation results of tasks on the Hugging Face Open LLM leaderboard and MMLU-Pm UrPr for mistral models.
MMLU-Pro
MMLU
ARC (25)
HellaSwag (10)
TruthfulQA (0)
Winograd (5)
GSM8K (5)
Average
Model: Mistral-7B Data: UltraFeedback (5k)
SFT
21.12
51.03
48.54
79.36
46.83
77.50
18.49
53.63
DPO
21.82
51.43
49.31
80.05
49.15
78.13
21.22
54.88
СРO
20.77
50.66
48.80
79.05
48.34
77.74
17.96
53.76
IPO
22.54
51.62
50.42
80.51
50.05
77.42
23.27
55.55
ORPO
20.97
50.32
50.08
46.27
77.82
18.87
53.88
KTO
21.68
51.42
49.65
80.08
49.18
77.82
20,92
54.85
SimPO
21.17
50.97
47.73
78.05
53,97
SLiC-HI
20.58
50.64
48.31
77.90
53.81
TPO
32,86
62.29
60.15
83.26
53.36
78.77
42,00
63.30
TPO-L
34.52
61.72
62.88
84.46
51.30
79.16
42.15
63.61
Model: Mistral-7B Data: UltraFeedback (10k)
20.50
49.33
49.31
78.14
47.57
75.61
52.81
DPO
21.69
49.83
52.13
79.13
52.57
75.61
20.62
54.98
CPO
19.73
48.73
48.63
77.68
50.08
76.08
17.66
53.14
22.00
49.80
53.15
79.22
52.11
75.76
21.30
55.22
ORPO
20.17
50.14
50.42
79.04
47.56
75.61
53.43
KTO
21.66
49.86
51.96
79.20
52.68
75.45
21.07
$5.04
SimPO
20.65
49.60
49.65
77.96
49.33
76.01
17.81
$3.39
SLiC-HF
19.96
48.74
48.63
77.69
50.16
76.01
17.43
53.11
TPO
32.43
61.87
61.17
83.30
53.64
79.08
41.17
63.37
TPO-L
35.49
61.57
65.78
85.18
61.32
79.71
42.46
66.00
Model: Mistral-7B Data: UltraFeedback (20k)
SFT
24.8
55.31
52.98
79.86
50.03
76.79
27.67
$7.11
DPO
25.46
55.06
55.97
80.36
54.13
77.74
32,37
$9.27
CPO
24.02
54.84
53.24
79.64
53.78
77.66
28.43
$7.93
IPO
26.16
55.60
56.99
80.04
54.76
77.34
27.21
58.66
ORPO
25.19
55.39
53,66
81.17
51.06
77.66
29.34
58.05
KTO
25.77
55.01
55,80
80.39
53.99
77.66
31.61
$9.08
SimPO
25.48
55.37
53.75
79.54
55.03
77.66
29.64
$8.50
SLIC-HF
24.11
54.72
53.07
79.34
54.19
77.58
29.11
$8.00
TPO
33.38
62.17
61.34
82,92
55.64
79.24
44.20
64.25
TPO-L
36.17
62.24
65.10
84.83
61.29
80.03
66.67
Model: Mistral-7B
Data: UltraFeedback (40k)
SFT
23.90
54.62
50.93
78.72
47.73
77.74
28.80
56.42
DPO
24.58
53.85
51.96
79.06
52.78
77.42
30.62
$7.62
CPO
23.08
54.25
49.57
78.48
51.54
77.34
56.51
IPO
25.44
53.63
54,18
51.74
77.34
28.20
$7.38
ORPO
24.83
53.97
52.21
80.75
50.39
77.26
$4.38
KTO
24.83
$3.80
$2.30
79,23
$1.64
77.58
$7.57
SimPO
54.26
$2.38
53.05
77.82
$7.88
SLIC-HF
23.08
54.32
49.82
78.41
52.33
77.03
28.12
56.67
TPO
32.03
61.32
60.58
82.40
53.41
78.61
39.20
62.58
TPO-L
36.17
60.18
65.10
84.83
61.29
80.03
32.68
65.80
Model: Mistral-7B: Histral-7B-Instrucc Data: Feettaa-
60.4
63.57
84.79
66.81
76.64
65.45
DPO
33.02
60.53
65.36
85.86
66.71
76.8
40.33
65.93
CPO
32,46
60.36
63.23
84.47
67.
76.80
38.74
65.16
IPO
32.87
60.20
63.31
84.88
67.36
75.85
39.42
65.17
ORPO
32.41
60.43
61.43
84.32
66.33
76.80
36.85
64.36
KTO
33.40
60.52
65.78
85.49
68.45
75.93
38.82
65.83
SimPO
32,98
60.53
66.89
85.95
68 .00
76.32
35.25
65.55
SLiC-HF
32.75
60.59
59.90
84.05
65.30
76.32
39.65
64,30
TPO,2
32.21
58.95
65.36
84.88
69.15
78.69
40.64
66.27
TPO,
33.03
58.83
64.76
84.74
68.68
78.22
42.53
66.29
ТPО.,
32.68
58.83
64,93
84.71
68.04
78.53
42.23
66.21
TPO-L
33.40
59.04
65.10
84.95
67.95
78.30
66.19
29

---

Published in Transactions on Machine Learning Research (11/2025)
Table 12: Downstream task evaluation results of tasks on the Hugging Face Open LLM leaderboard.
MMLU (5)
ARC (25)
HellaSwag (10)
TruthfulQA (0)
Winograd (5)
GSM8K (5)
Average
Model: Llama-3 Data: UltraFeedback (5k)
SFT
58.99
50.77
80.03
47.12
78.30
28.51
$7.29
DPO
51.28
48.54
78.53
32.9
58.54
CPO
50.77
82
78.06
31.92
IPO
59.41
51.79
80.85
48.99
78.53
34.42
59.00
ORPO
59.27
52.22
80.65
47.09
77.43
33,97
58.44
KTO
59.17
51.02
80.52
48.13
78.85
32.68
58.39
SimPO
58.76
50.94
80.19
47.87
78.53
32.75
58.17
SLiC-HF
59.01
51.11
80.26
48.83
78.06
32.52
58,30
TPO
65.34
59.04
81.90
52.69
77.11
51.86
64.66
TPO-L
65.33
59.04
82.51
47.87
78.06
52.08
64.15
Model: Llama-3 Data: UltraFeedback (10k)
SFT
59.09
51.88
79.66
42.80
79.16
27.46
56,68
DPO
59.39
53.07
80.76
44.04
78.93
36.69
58.81
CPO
59.32
51.71
79,46
44.16
79.79
32.83
$7.88
IPO
59.22
53.50
80.78
43.81
79.01
35.41
$8.62
ORPO
$9.69
$2.99
42.82
78.53
36.16
58.54
KTO
$2.90
43.92
79.48
36.69
$8.80 28.86
SimPO
59.04
52.22
79.75
42.99
79.72
35.10
SLIC-HF
59.27
51.54
79.52
44.11
79.48
32.68
$7.77
TPO
65.35
59.39
81.96
55.08
77.51
52.16
65.24
TPO-L
65.43
61.01
83.07
55.10
78.37
51.86
65.81
Model: Llama-3 Data: UltraFeedback (20k)
SFT
62.51
52.82
80.46
44.63
77.11
20.32
56.31
DPO
62.85
55.03
82,19
47.48
77.19
47,99
62.12
CPO
62,88
52.90
80.59
49.08
77.35
44.96
61.29
IPO
62.79
57.42
82,03
46.73
77.19
45.49
61.94
ORPO
63.15
54.18
81,91
46.55
77.58
37.15
60.09
KTO
62.74
55.12
82.23
47.35
77.19
48.67
62.22
SimPO
62.98
54.35
80.81
51.90
78.45
45,03
62.25
SLIC-HF
62.90
53.33
80.62
49.17
77.66
44.12
61.30
TPO
65.30
59.81
81.75
55.81
77.19
52,99
65.48
TPO-L
65.27
60.15
82.75
54.86
78.37
52.69
65.68
Model: Ilama-3 Data: UltraFeddback (40k)
SFT
62.19
53.92
80.14
43.39
77.98
39.20
59.47
DPO
61.84
56.74
80.73
48.59
77.51
45.11
61.75
CPO
62.23
54,35
80.21
46.31
78.53
42.91
60.76
IPO
62.09
58.11
81.54
49.64
77.74
48.22
62.89
ORPO
62,36
56.31
82,25
45.62
78.45
40.94
60.99
KTO
61.86
56.91
80.74
48.55
77.74
46.02
61.97
SimPO
61.45
54.86
80.50
47.73
78.37
45.11
61.34
SLIC-HF
62.28
53.92
80.20
46.38
78.69
42.84
60.72
TPP
58.96
81.53
57.47
77.66
51.18
65.28
TPO-
65.15
64.93
79.40
84.42
52.39
68.29
Model:
-nnstrut Data: UltraFeedback-Armii ArmoRM
SFT
67.06
61.01
78.57
51.66
74.35
68.69
66.89
DPO
66.88
63.99
80.78
59.01
74.66
49.81
65.86
CPO
67.05
62.29
78.73
54.01
73.72
67.40
67.20
IPO
66.52
61.95
77.90
73.09
58.23
65.39
ORPO
61.01
79.38
54.37
75.77
64.59
66.92
KTO
66.38
63,57
79.51
58.15
73.40
57.01
66.34
SimPO
65.63
62.80
78.33
60.70
73.32
50.72
65.25
SLIC-HF
66.41
61.26
78.80
53.23
76.16
66.57
67.07
TPO,²
65.89
66.38
79.38
$9.53
74.59
77.18
70.49
TPO,3
65.54
65.61
79.22
58.75
75.00
77.71
70.30
TPO,;
65.69
64.33
79.10
55.17
75.30
78.09
69.61
TPO-L
66.04
75.30
77.26
70.39
30


Reviews:
[{"review_id": "cQnhGQFyCZ", "reviewer": "Reviewer_HCqd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths: \n\n- an interesting method with a fairly principled design\n- the method operates in a single optimization step without requiring a reference model.\n- the method achieves strong results with substantially less data than DPO.\n\nweakness\n\n- Robustness experiments assume gold responses are always clean while only corrupting pairwise labels. Since TPO depends heavily on the gold-response behavioral cloning term, its reliability under noisy or mis-specified gold annotations remains unverified. \n\n- The analysis of gamma (reward margin in TPO-L) suggests a trade-off between reasoning (improves with higher gamma) and instruction-following (degrades with higher gamma). However, these results are demonstrated on a limited set of benchmarks (notably GSM8K and Arena-Hard). It is unclear whether the observed trade-off reflects a general principle or dataset-specific artifacts. For example, the GSM8K behavior can be because reasoning, or can also be because some characteristics of GSM8K itself (instead of reasoning).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Triple Preference Optimization (TPO), a one-step alignment method for large language models that incorporates three preference levels (gold > preferred > rejected). By combining behavioral cloning on gold responses with preference optimization between preferred and rejected responses, TPO and its length-controlled variant TPO-L achieve stronger performance than DPO and recent variants across both reasoning and instruction-following benchmarks, with improved robustness to data size and noise.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I would suggest the authors run controlled ablations where y_gold itself is corrupted, with different corruption rates, to evaluate whether TPO’s advantage persists.\n\n- I suggest the authors conduct gamma sweeps across a broader range of reasoning and instruction datasets to confirm whether gamma indeed governs this trade-off or if the effect is benchmark-dependent.", "broader_impact_concerns": "None noted.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ESlIubaNay", "reviewer": "Reviewer_Ma7b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\nThe paper proposes Triple Preference Optimization (TPO), a novel single-step preference learning method that effectively addresses key limitations of DPO and its variants. The approach is theoretically grounded in Maximum Entropy RL and preference modeling, and the introduction of TPO-L provides a practical solution to verbosity control. The experiments are extensive, covering multiple models, dataset sizes, and both instruction-following and reasoning benchmarks, and consistently show significant improvements in performance, robustness to noise, and data efficiency.\n\n**Weaknesses:**\nThe method assumes access to triple-preference data, which may be difficult to obtain in many real-world RLHF settings and raises questions about scalability. Moreover, while comparisons to many strong baselines are provided, the paper would benefit from a clearer discussion of methods like GRPO to better situate TPO within the broader preference optimization landscape and clarify domain applicability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Triple Preference Optimization (TPO), a novel single-step preference learning method designed to overcome key limitations of Direct Preference Optimization (DPO). Unlike existing approaches that often suffer from optimization inefficiency, poor reasoning performance, sensitivity to dataset size, and vulnerability to noisy judgments, TPO integrates behavioral cloning on gold responses with a reference-free preference objective across three levels of preference. The authors also propose a length-controlled variant, TPO-L, to mitigate verbosity issues. They provide a theoretical derivation and empirical experiments on both base and instruction-tuned models. The result shows that TPO and TPO-L consistently outperform DPO and its variants across a wide range of instruction-following and reasoning benchmarks, achieve robustness to noisy preference data, and deliver superior performance even with reduced training data.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper would benefit from a clearer discussion of its assumptions and positioning, especially the reliance on triple-preference data. Moreover, while the empirical evaluation is strong, the paper should situate TPO more explicitly within the broader baseline by experimenting with or discussing related approaches such as GRPO, ReMax.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2JDdRvpIqF", "reviewer": "Reviewer_HpRG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The idea of leveraging three preferences (`gold ≻ preferred ≻ rejected`) to improve preference optimization performance is intuitive and empirically compelling.\n- The paper presents solid experimental results on both instruction-following and reasoning benchmarks across varying training sizes, including low-resource scenarios.\n- TPO shows notable robustness to label noise and avoids common verbosity issues observed in DPO and its variants.\n\n### Weaknesses\n- My primary concern is the limited novelty. The proposed TPO/TPO-L essentially extends DPO/SimPO by adding an NLL regularization term that learns a high-quality (gold) response. However, the idea of combining DPO with an NLL loss has already been explored in prior work (e.g., [1], [2]), and the new insights provided by TPO are fairly limited.\n\n- Additionally, the reliance on high-quality triple preferences restricts the method’s practical applicability. Constructing (`gold`, `preferred`, `rejected`) triplets is significantly more expensive than collecting pairwise preferences, and the paper does not adequately address how scalable or reproducible this process is. The assumption that such fine-grained rankings can be reliably obtained across tasks or domains may not hold in practice.\n\n- Finally, the reward margin hyperparameter $\\gamma$ in TPO-L plays a critical role, but its optimal value appears to be task-specific. The tuning process is underexplained, which introduces brittleness for practitioners looking to apply TPO-L out of the box.\n\n### References\n[1] Iterative Reasoning Preference Optimization. https://arxiv.org/pdf/2404.19733\n\n[2] The LLaMA 3 Herd of Models. https://arxiv.org/pdf/2407.21783", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Triple Preference Optimization (TPO), a one-step preference optimization method designed to improve both instruction-following and reasoning capabilities of language models. Unlike prior approaches such as DPO and SimPO that rely on two-way preferences, TPO incorporates an additional “gold” response and combines a behavioral cloning (BC) loss with a preference-based objective. The authors also propose TPO-L, a length-controlled variant to reduce verbosity. Extensive experiments on LLaMA and Mistral models across multiple benchmarks show that TPO/TPO-L consistently outperform existing baselines, especially in low-data and noisy settings.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the above weaknesses and better distinguish TPO from DPO/SimPO + NLL loss.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)