TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees
Authors: Weibin Liao
Xu Chu
Yasha Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning from Human Feedback
Language Models
Preferences Learning
Primary area: reinforcement learning
TL;DR: We introduce Tree Preference Optimization (TPO), that does not sample paired preference responses from the preference tree; instead, it directly learns from the entire preference tree during the fine-tuning.

Abstract: In the domain of complex reasoning tasks, such as mathematical reasoning, recent advancements have proposed the use of Direct Preference Optimization (DPO) to suppress output of dispreferred responses, thereby enhancing the long-chain reasoning capabilities of large language models (LLMs). To this end, these studies employed LLMs to generate preference trees via Tree-of-thoughts (ToT) and sample the paired preference responses required by the DPO algorithm. However, the DPO algorithm based on binary preference optimization is unable to learn multiple responses with varying degrees of preference/dispreference that provided by the preference trees, resulting in incomplete preference learning. In this work, we introduce Tree Preference Optimization (TPO), that does not sample paired preference responses from the preference tree; instead, it directly learns from the entire preference tree during the fine-tuning. Specifically, TPO formulates the language model alignment as a Preference List Ranking problem, where the policy can potentially learn more effectively from a ranked preference list of responses given the prompt.  In addition, to further assist LLMs in identifying discriminative steps within long-chain reasoning and increase the relative reward margin in the preference list, TPO utilizes Adaptive Step Reward to adjust the reward values of each step in trajectory for performing fine-grained preference optimization. We carry out extensive experiments on mathematical reasoning tasks to evaluate TPO. The experimental results indicate that TPO consistently outperforms DPO across five public large language models on four datasets.

Published as a conference paper at ICLR 2025
TPO: ALIGNING LARGE LANGUAGE MODELS WITH MULTI-BRANCH & MULTI-STEP PREFERENCE TREES
Weibin Liao Xu Chu Chu Yasha Wang*
School of Computer Science, Peking University
Center on Frontiers of Computing Studies, Peking University
National Research and Engineering Center of Software Engineering, Peking University
X Lisoweibin@stu. pku. edu en, chu.xuêpku.eda edu.
com http:///tttttbccoomttttaaaaaa//s
ABSTRACT
In the domain of complex reasoning tasks, such as mathematical reasoning, recent advancements have proposed the use of Direct Preference Optimization (DPO) to suppress output of dispreferred responses, thereby enhancing the long-chain reasoning capabilities of large language models (LLMs). To this end, these stud- ies employed LLMs to generate preference trees via Tree-of-thoughts (ToT) and sample the paired preference responses required by the DPO algorithm. However, the DPO algorithm based on binary preference optimization was unable to learn multiple responses with varying degrees of preferenceedisprefeeene that provided by the preference trees, resulting in incomplete preference learning. In this work, we introduce Tree Preference Optimization (TPO), which does not sample paired preference responses from the preference tree; instead, it directly learns from the entire preference tree during the fine-tuning. Specifically, TPO formulates the lan- guage model alignment as a Preference List Ranking problem, where the policy can potentially learn more effectively from a ranked preference list of responses given the prompt. In addition, to further assist L.MS in identifying discriminative steps within long-chain reasoning and increase the relative reward margin the preference list, TPO utilizes Adaptive Step Reward to adjust the reward values of each step in the trajectory for performing fine-grained preference optimization. We carry out extensive experiments on mathematical reasoning tasks to evaluate TPO. The experimental results indicate that TPO consistently outperforms DPO across five publicly large language models on four datasets.
INTRODUCTION
Long-chain reasoning task (Wei et al., 2022; Xiong et al., 2024), such as commonsense reasoning and math reasoning, is one of the critical capabilities in large language models (LLMs) (Lai et al., 2024). This task is particularly challenging as it often involves numerous reasoning steps. Any mis- take in these steps can lead an incorrect final answer. Initially, some studies utilized various data augmentation techniques during the supervised fine-tuning (SFT) phase to enhance the reasoning ca- pabilities of LLMs (Shao et al., 2024; Tang al., 2024; Xin et al., 2024). However, phenomenon of pessimism suggests that the positive feedback provided by SFT alone cannot prevent LLMS from generating erroneous reasoning pathways. Hong et al. (2024) indicated that, during the SFT phase, as the probability of preferred outputs increases, the probability of dispreferred outputs also rises. This phenomenon makes the models more prone to errors long-chain reasoning. Consequently, is necessary to develop methods to mitigate the likelihood of dispreferred outputs.
Recently, Direct Preference Optimization (DPO) (Rafailov al., 2023) has been proposed for align- ing LLMs using paired preference data. Compared to the traditional Reinforcement Learning from Human Feedback (RLHF) (Christiano al., 2017) framework, DPO has gained popularity due to its simplicity and reduced memory requirements. Recent studies have utilized DPO to suppress
Corresponding Author.

---

Published as a conference paper at ICLR 2025
LLM generates multi-branch & multi-step preference tree
Direct Preference Optimization (DPO)
(p)
Vw
Y1
maximum likelihood
preference data
language model
(Q & A)
Tree Preference Optimization (TPO)
implicit reward
target rank
alignment
Yz
Y3
sim
Σ
Yn latent space
Σ
language model
preference tree
adaptive step reward
PLR
Figure 1: The framework of TPO regards preference modeling as a more general Preference List Ranking (PLR) problem and employs Adaptive Step Reward for achieving finer-grained preference optimization.
dispreferred responses in LLM outputs (Lai et al., 2024; Xie al., 2024). For this purpose, they have employed LLMs to generate preference trees via Tree-of-thoughts (ToT) (Yao et al., 2023), and based the inherent characteristics of the preference trees, they have collected the paired prefer- ence data required for DPO training. general, existing works employ heuristic methods to sample paired preference data, typically manifesting as the random selection of reasoning trajectories that can/cannot correctly answer the question as preferred/dispreferred responses (Jiao et al., 2024). Al- ternatively, reasoning trajectories may manually selected (particularly evident in the choice of dispreferred responses) to ensure the quality of the data (Lai et al., 2024); however, this approach further introduces costly manual annotation efforts.
Although sampling-hase based strategies have been proven effective, we consider them to be inferior solution that constrained by the fact that DPO supports only binary preference data input. We still seek preference learning algorithm tailored specifically for preference trees. This is demonstrated by the following points:
1. Preference trees typically yield unbalanced preference responses, with large number of dispre- ferred responses being randomly filtered out and not incorporated into the DPO, resulting a low data utilization efficiency for the model. Additionally, due to the inherent nature of the tree structure, these dispreferred responses encompass varying degrees reward. For instance, although neither response /1 nor V/2 leads to the correct outcome, 1/2 may contain more correct reasoning steps than y1. resulting in an inequality among dispreferred responses. We contend that DPO based binary rewards is unable to explore the critical information within failure trajectories, we propose to introduce preferences with varying reward values to facilitate more robust preference optimization.
The responses in preference tree may share a portion of sub-trajectories, which leads to lower reward margin between preferences, especially when large number of shared sub- trajectories are present. This issue has not been considered in the existing DPO algorithm. We contend that the lower reward margin may prevent the model from discerning the differences be- tween preference pairs. Consequently, we need to adaptively adjust the step rewards to enable fine-grained optimization.
Motivated by the aforementioned points, this work, we introduce Tree Preference Optimization (TPO), which does not sample paired preference responses from the preference tree; instead, it directly learns from the entire preference tree. Specifically, TPO decouples the preference tree into multi-branch & multi-step responses and performs preference optimization. To align LLMs from
2

---

Published as a conference paper at ICLR 2025
multi-branch preferences, TPO formulates LM alignment problem and establishes a connection between LM alignment and Learn-to-Rand (Liu et al., 2009; Li et al., 2024b), enabling the LLMs learn alignment more effectively from preference lists. To align LLMs from multi-step preferences, TPO proposes the Adaptive Step Reward that adjusts the reward value of each step based the correlation scores between pairs of steps. This mechanism aids LLMs in conducting preference optimization from more discriminative steps. Although TPO a preference learning algorithm that introduces the reward value, unlike traditional reinforcement learning algorithms, the reward used in TPO is not obtained through a learning- basedd algorithm, implying lower learning variance.
general Preference List Ranking
Our contributions are summarized as follows:
1. In the context of long-chain reasoning tasks, we consider issues related current DPO algo- rithms, specifically its low data utilization efficiency when aligning large language models on preference trees, as well as the inability of DPO based on binary rewards to explore critical in- formation in failure trajectories. To tackle these challenges, we propose TPC, the first preference optimization algorithm designed specifically for tree-structured preference data.
2. We propose the Preference List Ranking optimization objective that connects LM alignment and Learn-to-Rante establishing a framework that enables preference modeling from responses with varying reward values. addition, we propose the Adaptive Step Reward mechanism, which addresses the issue of reduced reward margin between responses generated from preference trees due to shared sub-trajectories.
3. We conduct extensive experiments to validate the effectiveness of TPO and discuss its general- ization to out-of-distribution datasets.
PRELIMINARIES
2.1
DIRECT PREFERENCE OPTIMIZATION
Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017) is an effective method for enhancing the robustness, authenticity, and safety of LLMs (Ouyang et al., 2022), which directly optimizes L.MSs according to human preferences by maximizing the reward value of the model's s responses. The reward function defined based on the Bradley-Terry (BT) model (Bradley & Terry, 1952) of preferences. Specifically, for preferred response Yw and dispreferred response V under the same prompt x and data distribution D, the BT model stipulates that the human preference distribution p" can be expressed as:
PD (yw 31 |x) (r" (x, (z,yw) - r" (z, 31))
where PTD (y/ > 31) denotes the probability that Yoe is preferred against Y1, ((x) 1+expi=x) denotes the sigmoid function, and y* denotes some latent reward model, which we do not have access to. The aligmment of language models is commonly regarded optimization problem with a Kullback-Leibe (KL) constraint on reward values, formalized as follows:
mar [r- (z, y)] s.t.E.-DDKk KL [To Tre x) S
where my denotes the aligned policy model, Kref denotes the reference policy model. To prevent reward hacking and ensure that Ta does not deviate too much from the Tref (Amodei et al., 2016), a regularization term is typically added to the objective function (Stiennon al., 2020), the problem is transformed into:
marEa--Dy--rtll [x" (1.,0) v)] 3DxL [TT( x)el (v x)]
(3)
where the hyperparameter controls the KL divergence between सo and Tref- In general, RLHF encompasses two training phases, including reward model training, and policy model training. How- ever, the ultimate performance of RLHF is highly sensitive various hyperparameters across these
3

---

Published as a conference paper at ICLR 2025
two phases, requiring careful tuning. To circumvent this complex training process, Rafailov et al. (2023) introduced Direct Preference Optimization (DPO), which directly utilizes paired preference data to optimize the policy model, bypassing the reward modeling stage by directly substituting this closed-form solution Eq. 1. Specifically, given an input prompt x: and a pair of preference data (yur yt) the goal of DPO is maximize the probability of the preferred response yhe and minimize the probability of the dispreferred response yielding the following DPO objective:
To (:)
Tre (yt x)) )
To (1) x:)
(TP Fref) -B log
(llo
B log
(4)
Tref (Yw x))
where the Blog me(u)))
(e)(y)))) can be regarded as an "implicit reward" (Rafailov et al., 2023), and the objective of DPO is to align the "implicit reward" directly with human preference data.
2.2 TREE-STRUCTURE REASONING POLICY FOR LLM
Multi-step Reasoning Following the standard reasoning setup of LLMMs, given policy и instan- tiated by L.M and an input prompt can step-by-step generate trajectory of reasoning steps y (. 8k) x(- ([-(a: by autoregressively predicting the next token. The standard reasoning setup assumes that y encompasses the complete list of reasoning steps, with each step Sk comprising multiple tokens. The step-by-step long-chain reasoning process is most famously used in Chain-of- Thought (CoT) (Wei et al., 2022).
Multi-branch Reasoning Self-Consistnncy (Wang et al., 2023) was the first to introduce multi- branch reasoning. Given an input prompt x, the policy и generates N trajectories of reasoning steps (01 ... MNN x--1z), where y; = (s (si.... .xx). Ultimately, Self-Consistency selects the most probable final answer by marginalizing over the reasoning trajectories.
Tree: Multi-branch & Multi-step Reasoning Recent works have further extended CoT and Self- Consistency to tree-like structure, referred to as the Tree-of-Thoughe (ToT) (Yao et al., 2023). Specifically, ToT no longer confines its application the initial prompt but extends engaging in branching reasoning at any intermediate state subsequent to given steps. Given the state [x, $1, 1 of an LLM the reasoming trajectory, ToT employs a Thought Generator G(n,S,
to propose N next planning steps [s] ................................ 8k ). Compared to CoT, ToT possesses broader space for cognitive exploration and can circumvent the generation of repetitive responses within the same context.
METHODOLOGY
We propose Tree Preference Optimization (TPO), preference learning algorithm tailored for pref- erence trees generated by L.LMs via Tree-of-Thoughts. TPO learns preference list with varying re- ward values using a Preference List Ranking objective, and utilizes Adaptive Step Reward achieve fine-grained alignment of step rewards.
3.1 ALIGNING LLMS FROM MULTI- PREFERENCES
)== with M samples: given prompt r, there is a response list y = (yi, yN) of size N, and each response y is associated with reward value The responses y are generated by the policy n, while the reward values derived from human raters or an inaccessible reward model. Typically, ( E [0, 1] with higher reward values indicating better responses.
Problem Definition TPO defines the dataset D (x((), y((), v(0))
M
Connection Between LM Alignment and Learn-to- Rank TPO follows the definition of the clas- sie Learning to- co-Rank (LTR) (Liu et al., 2009; Li et al., 2023; Liao et al., 2024c) problem: the op- timization objective is to learn ranking model that outputs the relevance scores for all documents given a query. In the context of LM alignment, TPO treats prompt as the query and responses y as documents. Inspired by Rafailov et al. (2023), TPO further regards the normalized "implicit reward"
4

---

Published as a conference paper at ICLR 2025
(denoted as (r1,-.. ,ry) (8 log merlvi(2) ".. : log [tr E [0, 1| as evaluation
To [v(])
xelux (x)
of the model's relevance scores for a and y. Overall, TPO establishes the following connection between LM alignment and Learn-to-Rank.
LM Alignment <- Learn-toRann
prompt, = query
(5)
responses, = documents
implicit rewards, r = scores
Preference List Ranking The LTR algorithm defines the ranking loss function based rewards v of responses y and predicted scores r, to train the model n
LLTR = B(x ((V,)) []].
(6)
where / is the loss function. TPO proposes the Preference Ranking Loss LPLR instantiate Specifically, TPO utilizes the relative reward margin between each pair of preferences in the pref- erence list to train the policy æ. To further consider the absolute position of preferences within the list, inspired by Burges et al. (2006), TPO introduces the Lambda Weight (Burges et al., 2006) to optimize LRRL, in order to perceive the impact brought about by the change in the positions two preferences. Ultimately, PRRL is mathematically represented as follows:
LPRL -BZy.VWD
Ar.
Σ 11
log (r: -r3)
Ar. = 12 - -2% log(1 + +
(8)
log(1 + +(())
where r(i) is the ranking position of yi in the ranking permutation induced by For more detailed information on Lambda Weight, please refer to Burges et al. (2006). is worth noting that when the length of the preference list N and the Lambda Weight not introduced, LPRL degenerates into the naive DPO loss.
3.2 ALIGNING LLMS FROM MULTI-STEP PREFERENCES
Problem Definition TPO follows the definition of multi-step reasoning as described in Sec. 2.2, introducing y/ = (51, **, ,8) consisting of K steps. Due to the characteristics inherent in tree- structured reasoning, for any two reasoning trajectories y; (s)...., skx) and yj = (s)..... s (To simplify, TPO assumes that yi and yj possess steps equal length.) there exist sub-trajectories which are content sharing action sharing.
Steps of content sharing: yi and have traversed the same sub-trajectory (81, "
,88-1) and
branched off at state Sg. Due to the (s) + +)) x-122), which resulting in S / S
Steps of action sharing: Expanding on content sharing, even though the (s * st). the high degree of semantic similarity the execution of identical actions results in S S
Adaptive Step Reward In the naive DPO algorithm, the "implicit reward" margin is step- independent, that is, for responses yi and yy. the "implicit reward" margin mathematically defined as follows:
Te(ys x) R.M 3 log To (y: x:) Tree (yj
No (yz
x)) x))
T x) Teaf (s) x) Trat (8
log (5(
x)) x))
3 log
3 log
(9)
To mitigate the reduced reward margin resulting from shared steps, TPO introduces the Adaptive Step Reward mechanism to discriminatively assign rewards for each step. Specifically, TPO employs adaptive weight w to adjust reward margin between step pairs, and instantiates w as cosine similarity in the semantic space. The adaptive R.M can be mathematically expressed as follows:
5

---

Published as a conference paper at ICLR 2025
emb(s)) emb(s))
(8log To( [x) ( x)
To( (2) Tree ( x))
R.M Σ ((1
B log
(10)
||mm(( ( [Il]emb(sg) [|l
where emb(-)) is the operation for semantic vectors generation. It is worth noting that when sf. and s/ manifest as steps of content sharing, the current step pairs yields provided R.M = 0 due to the no (x)
log [+(s]|z) 3 log Tia (s] Ultimately, the overall algorithm of TPO is detailed in Appendix Alg. 1.
O.
Teat (e]
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Network Architecture Our experiments were based on various base models, including Qwen2 models (Bai et al., 2023) of various sizes (Qwen2-1.5B-Instree and Owen2-7B-Instrudee Meta- Mama-3-8B-Instruue (Touvron et al., 2023), -iistal-7B-Insttuut---.. (Jiang et al., 2023) and the DeepSeekMath-7B-nnstee
(Shao et al., 2024) that has been specifically fine-tuned for mathemat- ical tasks. We also introduced DeepSeekMath-75 7B-RL (Shao et al., 2024), which underwent rein- forcement learning by Shao et al. (2024), the baseline model.
Training Datasets Typically, when faced with complex mathematical problems, LLMs struggle arrive the correct final answer even when employing ToT methods. To ensure that the pref- erence tree can generate trajectories capable of reasoning to the correct answer, we have expanded upon the existing dataset. Lai et al. (2024) proposed a dataset that provides 10,795 paired preference data, completely composed of mathematical problems, with complete correct and incorrect reason- ing trajectories provided for each problem. As shown in Fig. 2(a), starting from any intermediate (Bai et al., 2023) for further
step in the correct reasoning trajectory, we utilized Owen2-7B-Instreee step-by-step reasoning resulting in trajectories with varying degrees of preference. Ultimately, we collected 10 trajectories for each problem, including least correct trajectory and one incorrect trajectory from original data (Lai et al., 2024), with the remaining eight were generated by Qwen2- 7B Instreut. We utilized ChatGPT to assign scores ranging from -100 to 100 for each trajectory in order to obtain the rewards for these trajectories. To avoid incorrect judgments by ChatGPT, we provided the correct trajectory as reference and employed ReACT shown in Fig. 2(b). The prompts for data generation and ChatGPT scoring can be found in the Appendix. B. Fig. 2(c) il- lustrates the distribution of reward values for this dataset. The statistical findings reveal that we ultimately gathered preference data corresponding to reward value distribution of 53.74 + 69.27. noteworthy that Bai et al. (2023) shows that the dataset was derived from the MetaMath (Yu et al., 2023), MMIQC (Liu & Yao, 2024), and AQuA (Ling et al., 2017) datasets. We have ensured that these datasets do not overlap with the subsequent evaluation data.
Evaluation Datasets We introduced three types of tasks, Math (in-distribution)) Coding and Reasoning (outtof-distributtion) to assess the effectiveness of TPO. For the Math tasks, we con- sidered the following datasets: MATH (Hendrycks et al., 2021), SVAMP (Patel et al., 2021), AS- Div (Miao et al., 2021) and GSM-Plus (Li et al., 2024a). For the Coding tasks, we considered the HumanEval (Chen al., 2021) and MBPP (Austin et al., 2021) datasets. For Reasoning task, we considered the BBH (Suzgun et al., 2023) and MMLU (Hendrycks et al.) datasets. It is worth not- ing that BBH tasks require multi-step reasoning, whereas MMLU does not. The prompts used for evaluating these datasets can be found in the Appendix. B. We evaluated these datasets with pass@) accuracy.
Implement Details We performed the TPO and DPO on the models mentioned above. We used the Py Torch library to implement all the algorithms based on the open-source HuggingFace trans- formers (Wolf, 2019) and Transformer Reinforcement Learning (TRL) (von Werra et al., 2020). The experiments were conducted on 8 NVIDIA-RTX3000 trained the model for epoch, using batch size of 1 for each GPU. The learning rate was set to
GPUs. For each experimental setup,

---

Published as a conference paper at ICLR 2025
104
Question: 5x5
Correct Reasoning Trajectory:
SreatoningS
Current Reasoning Trajectory:
SreasoningS
ranking based on score
80
20
60
80
80 100
Reasoning:
100
Score:
ChatGPT
(a) Preference Tree Generation
(b) Reward Evaluation
(c) Distribution of Reward Values
Figure 2: (a) illustrates the data generation pipeline we used, where we start from the intermediate steps of the original correct reasoning trajectories and generate new reasoning trajectories step by step. The steps represent preferred reasoning steps, the steps denote dispreferred reasoning steps, and the steps indicate reasoning steps with unknown preference. (b) shows how we introduced
ChatGPT to score each reasoning trajectory, with scores ranging from 100, 100]. We provided ChatGPT with correct reasoning trajectories as a reference and employed ReACT to improve score credibility. (c) presents the distribution of reasoning trajectories across various score intervals.
5e-7. The hyperparameter used in Eq. 4 for DPO was set to 0.5. We utilized the AdamW optimizer and cosine learning rate scheduler, with warm-up ratio set to 0.1.
4.2 EXPERIMENT RESULTS
Results on Math Task We conducted evaluations on four mathematics reasoning datasets to verify the performance of TPO in-distribution datasets. We employed the CoT (Wei et al., 2022) strategy for reasoning without using any demonstrations. Results are shown in Table. 1. We summarize the key takeaways as follows:
TPC comprehensively outperformed the SFT and DPO algorithm across all datasets, across various LLM size settings (Qwen2-1. 1-221.1B-Innsture and Weven--BB-nssrruee and whether the L.MMs were fine- tuned in the domain of mathematics (Qwen2-7B-Insee Insstruct and SeeeSeeeMatth7B-Insteee lar experimental results were also observed on Meta-Llama-3-SB-Innsee v0.3 (see Appendix Table. 4). In many cases, TPO also surpassed existing specialized reinforce- ment learning methods (DeepSeekMath- 7B-RL-BBRRL---- vs. DeepSeekMath- Dar-aaathh-B---sttuttttPPO with only a slight disadvantage on the ASDiv dataset. We further discovered that TPO can help LLMs ex- ceed baseline models that are 5x larger. For instance, on SVAMP dataset, the performance of However, after fine-
Simi- and Mistral--B-Inntrrutt
Qwen2-1 1--2-1.5B-Insii Ins-tutt was significantly inferior to that of -wwen-7BB-Insrreee tuning with TPC, Qwen2-1.5B -...5----InttuutttPPP outperformed Qwen2-7B 7B--truutttt by 1.7% in accu- racy. DeepSeekMath-77 IIInntueet, fine-tuned with TPO, has significantly surpassed GPT-3.5 Turbo.
Results on Coding and Reasoning Task and two reasoning datasets verify the performance of TPO on -uitrf-distribution datasets. Results are shown in Table. 2. We summarize the key takeaways as follows:
We further conducted evaluations on two coding datasets
For coding tasks, in the vast majority of cases, TPO aided in improving the performance of LLMs on out-of-distribution datasets, surpassing the DPO algorithm. However, on the HumanEval dataset, and Qwen2-7BBInstrruc exhibited decline in performance after undergoing TPO, a phenomenon similarly observed with the DPO approach. Notably, DeepSeckMath-78 Instruct experienced an improvement in performance on the HumanEval dataset after the preference alignment, regardless of whether the DPO or TPO algorithm was used. We speculate that the cause of this phenomenon that the Qwen2 series models have already been fine-tuned on the HumanEval dataset, leading to "catastrophic forgetting" (Xuhong et al., 2018; Liao et al., 2022: 2024b) in the Qwen2 models after the preference aligmment, where they forgot the coding knowledge originally leamed the HumanEval dataset. In contrast, MatpSeekMath-17 Math-7BBIInstruc is model specifically designed for mathematical reasoning and did not acquire coding knowledge during its previous training phases. For reasoning tasks, we observe that TPO once again surpasses existing algorithms on the BBH dataset. However, the MMLU dataset, the performance across all algorithms is similar. We

---

Published as a conference paper at ICLR 2025
Experimental results Math task (In-Distribution) using the SFT, IPOO and TPO algo- rithm. The best results for each large language model setting are indicated in bold. The best results across all settings are highlighted with background, while the second-best results indicated with background. We report results using pass accuracy.
Table 1:
LLMs
open
general
MATH
SVAMP
ASDiv GSM-Plus
Avg.
Qwen2- --5tttttttttt
1.5B
19.52
23.90
35.76
20.05
24.81
Qwen2-1 (wen2-1.5B-hie 1.5B--InsttucttSFFF
1.5B
20.80
28.77
38.32
21.87
27.44
Qwen2-1.55
1.5B
20.98
29.30
40.13
21.52
27.98
Qwen2-1.55 Been2-1,5B-Instructtee ---tuntttcccOOOOO
1.5B
22.88
35.60
46.28
24.12
32.22
Owen2-7B-Instie
7B
53.92
33.90
48.38
44.72
45.23
TB-InstructtSFF
7B
54.92
46.40
53.28
45.40
50.00
Qwen2-7B 7B-nntttuutttDPOO
7B
54.26
44.69
54.32
50.28
50.89
Qwen2- TBInstrrcct+TC
7B
55.46
48.20
59.22
54.82
54.43
DeepSeckMat-7B-Innsne
7B
X
43.92
84.10
90.10
59.78
69.48
DespSeckMatt-7BB-I.
50.70
86.70
90.94
64.07
73.10
7B
45.18
84.30
90.45
61.09
70.26
48.66
85,98
90.16
71.92
DeepSekkaah--7B-nnnnueee
7B
51,30
86,80
90.61
64.73
73.36
GPT-3.5 Turbo
37.80
83.00
90.60
61.20
68.15
GPT-4
x
69.70
92.60
85.60
85.68
Table 2: Experimental results on Coding and Reasoning task (Out-of-Distribution) using the SFT,
DPO and TPO algorithm. The best results for each large language model setting are indicated in bold. We report results using pass@) accuracy.
Coding
Reasoning
LLMs
size
open
general
MBPP
HvmanEval
BBH
MMLU
Owen2-1.5B-Inme
1.5B
45.11
46.34
32.76
55.97
Owen2-1.55Blnnsruct5533
1.SB
45.63
46.20
33.10
55.90
1.SB
45.11
44.51
34.72
56.03
1.5B
/
46.62
43.90
37.49
55.99
Qwen2-77B-Inttrut
7B
58.90
75.00
62.43
70.78
Qwen2-7B--ie
Owenn--7B-Instruct-S777
7B
62.91
77.44
64.75
70.74
Owen2-78-Insttueett
7B
59.11
68.32
65.58
70.58
Owen2-77--nstmucc+PPP
7B
61.65
65.85
69.62
70.63
DecpSSekkMarh--BBBnnnnne
78
60.90
56.10
61.85
54.44
DeepSeek-Mahh-78-nnnnnc--s
7B
59.65
56.10
62.79
54.56
DeepSeck.Math785
7B
65.91
56.10
62.74
54.98
DeepSeekMath-7B.
7B
63.26
57.25
62.68
54.22
TrepSeekMath-77--Insttttee Instruct++PP
7B
66,42
59.15
62.99
54.99
GPT-3.5 Turbo GPT-4
82.50
76.80
70.10
70.00
83,50
85.40
86.70
86.40
attribute this to TPO acquiring long-chain reasoning abilities during fine-tuning on mathematical tasks, which generalize effectively to other domains such coding and reasoning. However, since MMLU tasks do not require long-chain reasoning, the TPO algorithm does not achieve performance improvements this dataset. Notably, TPC does not lead performance degradation of LLMs on MMLU tasks, indicating that commonsense knowledge is preserved during the TPO fine-tuning process.
8

---

Published as a conference paper at ICLR 2025
Table 3: Ablation Studies of TPO Math tasks. The green font indicates the performance loss incurred after the removal of the respective module. Results show that the absence of any module leads to a degradation in performance.
Methods
MATH
SVAMP
ASDiv
GSM-Plus
Avg.
Qwen2-7B- BBIInttuutt
TPO
55,46
48.20
59.22
54.82
54.43
wlo Adaptive Step Reward w/o Preference List Ranking
-5.08
47.84.0036)
58. (--.5)
54.36(0044)
54.00
54.36
(5.67(2253)
56.84.2.88
51.10
51.99(2244)
DecpScckMatth-7Bnnsstie
TPO
51.30
86.80
90.61
64.73
73.36
wlo Adaptive Step Reward
50,92 .0.38)
86.68...11
90.53 0.088
64.32
73.11(-0.55)
wlo Preference List Ranking
49.11(-2.19)
86.29(-0.5)
90.20i-0.41
63.02-1771
72.16-1120)
Ablation Studies We verified the effectiveness of each module by removing some modules from TPO and evaluated the modified models using the Qwen2-7B-Instee Instruct and DeepSeekMath-777 Instruct across four mathematical reasoning datasets. The experimental results were presented in Table. The results indicated that the absence of both the Adaptive Step Reward and the Prefer- ence List Ranking modules leads to a degradation in performance of TPO, with the Preference List Ranking module's s removal resulting in an average performance of 2.44%) and 1.40% These re- sults suggest that the Preference List Ranking module aids LLMs in learning from wider variety of preference lists with different reward values, thereby facilitating more robust preference alignment. Regarding the Adaptive Step Reward, provide further discussion through t-test and case studies in the Appendix. C.4 and Appendix. C.5.
4.3 ANALYSIS OF DISPREFERRED RESPONSES
Comparison of DPO with Varying Reward Values We performed DPO using preference pairs with different reward values and evaluated the results using Swwen-7B-Inssrruc on the ASDiv and GSM-Plus datasets. Specifically, we employed correct reasoning trajectories as preferred responses and sampled dispreferred responses with different reward distributions sampled from incorrect trajectories. The mean reward values with corresponding standard deviations were 56. 75.4+ 46.4+19.5) Our experimental results are presented in Fig. 3(a). The results indicate that dispreferred responses with different reward values have varying degrees of impact on the model's performance. Fig. 3(a) shows that when dispreferred responses with lower mean rewards (strong dispreference) or higher mean rewards (weak dispreference) are used, the performance of DPO is inferior. However, the best performance of DPO is observed when dispre- ferred responses with moderate mean reward are used. We argue that some dispreferred responses with lower rewards are less valuable for learning due their significant discrepancy from preferred responses. Conversely, dispreferred responses with higher rewards pose challenges for the DPO algorithm learn because of their smaller difference from preferred responses. Therefore, it is necessary to select dispreferred responses with moderate rewards to facilitate more effective DPO learning. Nonetheless, TPO still outperforms all DPO baselines, suggesting that introducing more dispreferred responses and aligning language models from preference lists with different reward values concurrently contributes to stronger preference learning.
Analysis on Size of Preference List To better understand the effect of Preference List Ranking, we conduct analysis on multiple choices list sizes of TPO, and evaluate TPO on the ASDiv and As illustrated in Fig. 3(b), the size of the Pref-
GSM-Plus datasets using Qwen2-7B-Instree erence List Ranking increases, the performance of TPO shows a steady growth, which is observed across both datasets. We argue that it is beneficial model preferences using preference lists with varying reward values, and further increasing the size of Preference List Ranking can enhance the performance of preference learning.

---

Published as a conference paper at ICLR 2025
OPO)7.4:677
55
TPO
59
ASDiv
453575.4:35.5)
199186-4:19.5)
60
Pass@1 Accuracy
Pass@1 Accuracy
55
53
45
40
ASDN
GSM Plus
2
Datasets
List Size
Figure 3: (a) illustrates comparison between TPO and DPO using various reward value distribu- tions for dispreferred responses the ASDiv and GSM-Plus datasets. The numbers in the legend following each group of DPO algorithms represent the mean and standard deviation of the reward values for dispreferred responses. The results indicate that TPC consistently outperforms DPO. shows performance of TPO with different list sizes on the ASDiv and GSM-Plus datasets. TPO ben- efits more and monotonically as the list size increases.
LIMITATIONS AND FUTURE WORKS
Despite the promising results obtained in our work, it is important to acknowledge the limitations. The first limitation that TPO may introduce stronger form of "catastrophic forgetting". The results in Sec. 4.2 indicate that while TPO exhibits excellent performance on in-distribution datasets, may suffer from performance degradation on out-of-distribution datasets. We provide a more in- depth discussion Sec, 4.2 and attribute this issue to "catastrophic forgetting" (Xuhong al., 2018; Liao et al., 2022; 2024b). Existing strategies to mitigate "catastrophic forgetting" include memory replay (Miao et al., 2024; Babakniya al., 2024), regularization constraints (I.iao al., 2022; 2024b), and meta-learning (Gupta et al., 2020; Son et al., 2024), among others. Incorporating these techniques into the TPO training procedure could potentially improve the generalization of TPO on out-of-distribution datasets.
The second limitation due to imbalanced distribution of the preference tree reward values, as shown in Fig. 2(c). We analyze the reasons for this as follows: (1) Autoregressive LLMs, including ChatGPT, tend to assign either high or low values (Xiong et al., 2023). Although we employ the Re- reasonable evaluation, this issue remains to be addressed. (2) Our data generation pipeline, as depicted in Fig. 2(a), adopts a strategy of generating additional responses starting from correct trajectories. This strategy ensures that the preference tree contains at least one preferred response and that the reasoning generated from intermediate nodes includes
ACT strategy to prompt ChatGPT to provide
correct reasoning paths to diversify the reward values. However, once the preceding trajectory in the generated path already includes the key steps to solve the problem, the subsequent steps become easily inferable, leading to a higher distribution tree. In future work, we aim to introduce more effective ToT strategies, such as MCTS (Xie et al., 2024), to ensure the generation of higher-quality data. Additionally, we will employ techniques such as prompt optimization (Shin et al., 2020), multi-model collaborative scoring (Talebirad & Nadiri, 2023), and self-consistency (Wang et al., 2023) to enhance the reliability of the scoring procedure.
reward values in the preference
CONCLUSIONS
In this work, we propose TPO, preference learning algorithm designed specifically for preference trees input. TPO enhances DPO by addressing two critical issues: (1) DPO only supports binary preference input and cannot model preferences from preference lists with varying reward values. (2) DPO exhibits lower reward margin when dealing with reasoning involving long chains of trajec- tories with shared sub-trajectories. We evaluate the effectiveness of TPO on extensive experiments, and the experimental results indicate that TPO consistently outperforms DPO in-distribution data and shows promise for its generalization to ditt----istriittionn data.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (Grant No. U23A20468). (The authors would like express their sincere gratitude to all individuals and insti- tutions who contributed to this research. Their valuable support and constructive suggestions greatly facilitated the completion of this study.)
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprini arXiv: 160.:0660006655 2016.
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, al. Program synthesis with large language 2021.
models. arXiv preprint arXiv:2108.077
Sara Babakniya, Zalan Fabian, Chaoyang He, Mahdi Soltanolkotabi, and Salman Avestimehr. A data-free approach to mitigate catastrophic forgetting in federated class incremental learning for vision tasks. Advances in Neural Information Processing Systems, 36, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hul- lender. Learning to rank using gradient descent. In Proceedings of the 22nd international confer- ence on Machine learning, pp. 89-96, 2005.
Christopher Burges, Robert Ragno, and Quoc Le. Learning to rank with nonsmooth cost functions. Advances in neural information processing systems, 19, 2006.
Zhe Cao, Tao Qin, Tie-Yas Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach listwise approach. In Proceedings of the 24th international conference on Machine learning, pp. 129-136, 2007.
Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, and Jun Zhu, Noise contrastive align- ment of language models with explicit rewards. arXiv preprint arXiv:2402.05369, 2024a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Yuxin Chen, Junfei Tan, An Zhang, Zhengyi Yang, Leheng Sheng, Enzhi Zhang, Xiang Wang, and Tat-Seng Chua. On sofimax direct preference optimization for recommendation. arXiv preprint arXiv:2406.09215, 2024b.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
Gunshi Gupta, Karmesh Yadav, and Liam Paull. Look-ahead meta learning for continual learning. Advances in Neural Information Processing Systems, 13:11558-11 11598, 2020.
Dan Hendryeks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Dan Hendrycks, Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103.03874 2021.
11

---

Published as a conference paper at ICLR 2025
Jiwoo Hong, Noah Lee, and James Thome Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2(4):5, 2024.
Rolf Jagerman, Zhen Qin, Xuanhui Wang, Michael Bendersky, and Mare Najork. On optimizing top-k metrics for neural ranking models. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.
2022.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Fangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F Chen, and Shafiq Joty. Learning planning- based reasoning by trajectories collection and process reward synthesizing. arXiv preprint arXiv:2402.00658, 2024.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of Ilms. arXiv preprint wrxiv2240.118629 2024.
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of Iims as mathematical problem solvers. arXiv preprint arXiv:2402. 19255, 2024a.
Yuchen Li, Haoyi Xiong. Linghe Kong. Rui Zhang, Fanqin Xu, Guihai Chen, and Minglu Li. Mhrr: Moocs recommender service with meta hierarchical reinforced ranking. IEEE Transactions Services Computing, 2023.
Yuchen Li, Haoyi Xiong, Linghe Kong, Jiang Bian, Shuaiqiang Wang, Guihai Chen, and Dawei Yin. Gs2p: a generative pre-trained learning to rank model with prarraaamtttrzitionn for web-scale search. Machine Learning, pp. 1-19, 2024b.
Jiayi Liao, Xiangnan He, Ruobing Xie, Jiancan Wu, Yancheng Yuan, Xingwu Sun, Zhanhui Kang, and Xiang Wang. Rosepo: Aligning lim-based recommenders with human values. arXiv preprint arXiv:2410.12519 2024a.
Weibin Liao, Haoyi Xiong, Qingzhong Wang. Yan Mo. Xuhong Li, Yi Liu, Zeyu Chen, Siyu Huang, and Dejing Dou. Muscle: Multi-task self-supervised continual learning to pre-train deep models for x-ray images of multiple body parts. In International Conference Medical Image Comput- ing and Computer-Assisted Intervention, pp. 151-161. Springer, 2022.
Weibin Liao, Qingzhong Wang, Xuhong Li, Yi Liu, Zeyu Chen, Siyu Huang, Dejing Dou, Yanwu Xu, and Haoyi Xiong. Mtpret: Improving x-ray image analytics with multi-task pre-training. IEEE Transactions on Artificial Intelligence, 2024b.
Weibin Liao, Yifan Zhu, Yanyan Li, Qi Zhang, Zhonghong Ou, and Xuesong Li. Revgnn: Negative sampling enhanced contrastive graph learning for academic reviewer recommendation. ACM Transactions on Information Systems, 43(1):1-26, 2024c.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gener- ation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lonng Papers), pp. 158-167, 2017.
Haoxiong Liu and Andrew Chi-Chih Yao. Augmenting math word problems via iterative question composing. arXiv preprint 2xXi::2401.09003 2024.
Tie-Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends@ in Informa- fion Retrieval, 3(3):225-331, 2009.
Hao Miao, Yan Zhao, Chenjuan Guo, Bin Yang. Kai Zheng, Feiteng Huang, Jiandong Xie, and Christian Jensen. A unified replay-based continuous learning framework for apatio-tempora prediction on streaming data. arXiv preprint 2404 14999, 2024.
12

---

Published as a conference paper at ICLR 2025
Shen- Yun Miao, Thao-Chure Liang, and Keh- Su. A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprint wwXi::2106.1572,, 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402. 13228, 2024.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2023.
Antoine Scheid. Etienne Boursier, Alain Durmus, Michael Jordan, Pierre Ménard, Eric Moulines, and Michal Valko. Optimal design for reward modeling in rlhf. arXiv preprint arXiv:2410.17055, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03:: 242.03330, 2024.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2020.
Jaehyeon Son, Soochan Lee, and Gunhee Kim. When meta-learning meets online and continual learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3022 2020.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, et al. Challenging big- beg-bench tasks and whether chain-of thouhhtttt can solve them. In Findings of the Association for Computational Lin- guistics: ACL 2023, pp. 13003-13051, 2023.
Yashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Harnessing the power of intelligent Ilm agents. arXiv preprint arXiv:2306.00 033114 2023.
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. In Forty-first International Conference Machine Learning, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini 13771,, 2023.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https:/ / / thub. www.coo////ooooo hoom//hugiaa ni eace/rrrl 2020.
Xuanhui Wang, Cheng Li, Nadav Golbandi, Michael Bendersky, and Mare Najork. The lambdaloss framework for ranking metric optimization. In Proceedings of the 27th ACM international con- ference information and knowledge management, pp. 1313-1322, 2018
13

---

Published as a conference paper at ICLR 2025
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le. Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 15:24824-24833 2022.
Wolf. Huggingface's transformers: State-of-the natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. beta-dpo: Direct preference optimization with dynamic beta. arXiv preprint arXiv:2407.08639 2024.
Fen Xia, Tie-Yan Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning, pp. 1192-1 1199, 2008.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min- Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shich. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.004:3 2024.
Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Deepseek-prover: Advancing theorem proving in Ilms through large-scale synthetic data. arXiv preprint arXiv:2405.14333. 2024.
Haoyi Xiong, Zhiyuan Wang, Xuhong Li. Jiang Bian, Zeke Xie. Shahid Mumtaz, and Laura E. Barnes. Converging paradigms: The synergy of symbolic and connectionist ai in Ilm- llm-empowered autonomous agents, 2024. URL https: arxiv. arg.abs/ 2407 08516.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li. Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306. 13063, 2023.
LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International Conference on Machine Learning, pp. 2825-2832 PMLR, 2018.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances Neural Information Processing Systems, 36, 2023.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A ALGORITHM DESCRIPTION OF TPO
Algorithm 1: TPO Training Algorithm
Input Dataset D (xx"), (0) v A=1, Policy Model
Output: Aligned Policy Model me
Initialize No and fref with ; for (z, y, E D do
// Preference List Ranking
for Ms y 8 > Vi
// Calculate the Lanbda weight
Ai.j 8;
// Adaptive Step Reward
for st E y (1) E y
19
// Adjust reward margin based on semantic similarity between
paired steps
R.M., Eq. 10;
1/ Calculate the preference list ranking loss CPRL + Eq. 7 =MMa))r---
Update Policy Model: Te VLRRI:
Return Tg.
B PROMPTS USED IN THIS WORK
Prompt used for data generation. We employ the following prompts to synthesize the relevant data for preference trees. To ensure that the generated trajectories contain some correct reasoning steps, we provide the initial few reasoning steps the prompts and allow LLMs to generate the subsequent reasoning steps.
Prompt used for data generation.
[System]
You are a helpful assistant.
[Instructions]
### Given the question, please provide the steps solve it.
### Question: {question)
### Your answer should strictly follow the following format.
Step 1:
Step 2:
Step
### Please reason step by step, and put your final answer within boxed Your An- swer).
Step 1: {step_1}
Step 2: (step 2}
Step 3:
Prompt used for generating reasoning trajectory scores using ChatGPT. We utilize the fol- lowing prompts to instruct ChatGPT to score the reasoning trajectories within preference trees. To
15

---

Published as a conference paper at ICLR 2025
ensure the reliability of the scores, provide ChatGPT with genuine reasoning trajectories reference and employ the ReACT to facilitate ChatGPT in generating the scoring rationale.
Prompt used for generating reasoning trajectory scores using ChatGPT.
[System]
You are a helpful assistant.
[Instructions]
### Given the question, standard answer, and current answer, give a score for the current answer.
### Question: {question)
### Standard Answer: {standard answer]
###
Current Answer: (current answer)
### You only need to give the score, and you also need to provide a detailed com- parison with the standard answer to give the reason for your score.
### Provide a reward score between -100 and 100 for the answer quality, using very strict standards. Do not give a full score above 95. Make sure the reward score is integer.
### If the final answer of the current answer is incorrect, please give lower score. ### Your answer should strictly follow the following json format. Please note that only the following JSON is provided and no additional response content is required.
"reasoning":
"score": 3983
### Your Answer:
Prompt used for solving Math problems. In our assessment of TPO performance, we employ the following prompts to address relevant Math tasks, including MATH (Hendrycks et al., 2021), SVAMP (Patel et al., 2021), ASDiv (Miao et al., 2021) and GSM-Plus (Li et al., 2024a) datasets.
Prompt used for solving Math problems.
[Instructions]
Solve the following math problem step-by-step. Simplify your answer as much as possible. Present your final answer as within boxed Your Answer)
{question}
Prompt used for solving Coding problems on HumanEval dataset. In our assessment of TPC performance, employ the following prompts address relevant Coding task Hu- manEval (Chen et al., 2021) dataset.
Prompt used for solving Coding problems on HumanEval dataset.
[Instructions]
Write Python code to solve the task. Write Python function solve the following problem: Present code in "'python" "python (question]
16

---

Published as a conference paper at ICLR 2025
Prompt used for solving Coding problems on MBPP dataset. In assessment of TPO perfor- mance, we employ the following prompts to address relevant Coding task on MBPP (Austin et al., 2021) dataset.
Prompt used solving Coding problems on MBPP dataset.
Instructions]
Write Python code solve the task.
Write a Python function to solve the following problem: Present code in "'python"
pyython
question)
[test
Prompt used for solving Reasoning problems on BBH dataset. our assessment TPO per- formance, we employ the following prompts to address relevant Reasoning task BBH (Suzgun et al., 2023) dataset.
Prompt used for solving Reasoning problems on BBH dataset.
[Instructions]
Answer the following question.
{question]
Let's think step by step.
Prompt used for solving Reasoning problems on MMLU dataset. In our assessment of TPC performance, we employ the following prompts to address relevant Reasoning task on MMLU (Hendrycks et al.) dataset.
Prompt used for solving Reasoning problems on MMLU dataset.
Instrructios]
Please answer the following multiple-choice questions. (question]
Answer:
C ADDITIONAL EXPERIMENTAL RESULTS
C. ADDITIONAL EXPERIMENTAL RESULTS MATH TASKS.
Table. 4 presents the performance of the TPC algorithm mathematical tasks using Mama-L_lama-3 8B-Instruct and Mistral-78 Instrruct--O... as backbone LLMs. The experimental results demonstrate that TPO consistently outperforms the existing baseline algorithms across different backbone LLMs.
C.2 COMPARISON WITH ADVANCED DPOS
Considering that TPO introduces potential data augmentation, we further introduce two new DPO baselines that extract complete pairwise data from the preference lists.
given data [z, V1, 3/..... ym] with the preference ranking V1 > y/2 >,..., > yn.
DPO*: DPO* extracts paired preferences [ar, (31.32), (V1, Va). .... (v1 Vn.)
DPO+: + extracts paired preferences (ar, (v1, 82), (u2, ya), ... (Un 1, yn)]
17

---

Published as a conference paper at ICLR 2025
Table 4: Experimental results Math task (In-Distribution) using Meta-Llama-3-88 Laslllaa------nntttuu Instructttt and Datrral-B--nstuuct--O0 as backbone LLMs. The best results for each large language model setting indicated in bold. We report results using pass@ accuracy.
LLMs
size
open
general
MATH
SVAMP
ASDiv
GSM- Plu
Avg.
Veta-Llama-23 Instruct
8B
28.50
74.30
75.57
48.77
56.79
Meta-Llamaa-3-88 3-
28.08
73.30
74.92
47.50
55.95
Meta- Sta--Llama-3-88 DB-sstttcc+D
8B
29. 29.12
73.10
75.73
47.66
56.40
Meta-LLama-33 Jaama-3-------- InstructtPPP
8B
29.58
77.70
80.91
53.25
60.36
Mistral 7B Instruct-OO.3
7B
14.30
53.60
60.36
30.97
39.81
Mistral 7B Bo---nntttttt.---444
7B
14.62
56.00
62.46
33.60
41.67
Mistral-7B-Instnun -et--mtttttttO34-DPP
7B
15.98
70.44
71.85
38.62
49.22
Mistral-7B-Inn -Bt.----nnntturco00.....0
17.56
72.90
75.24
51.82
Table 5: Experimental results on Math task (n--Disrribution) using the DPO, DPO", DPO and TPO algorithms. The best results for each large language model setting indicated in bold. We report results using pass@ accuracy.
Methods
MATH
SVAMP
ASDiv
GSM-Plus
Avg.
Qwen2-7B 7B-Inssruct+DDOO
54.26
44.69
54.32
50.28
50.89
2on2-7BB-istruuc+-DPO
55.02
46.85
57.96
53.26
53.27
Dren2-7BB-Inttrucr-DDoo
54.78
46.13
55.30
50.31
51.63
Qwen2-77-lnntruutt+PP
55.46
48.20
59.22
54.82
54.43
DeepSeekMaah-77--nntne----
48.66
85.98
90.16
62.86
71.92
DeepSsekMaath--BB-nnsnu----
50.33
86.20
90.61
63.57
72.68
49.75
85.78
89.78
63.11
72.11
DeepSeekMah-77-Instteee
51.30
86.80
90.61
64.73
73.36
We conducted experiments on Qwen2-7B-Instee and DeepSeckMathh7BBmng -B-nstuut and the exper- imental results are shown in the Table. 5. The experimental results indicate that TPO outperforms all the baseline models. We argue that all variants of DPO focus solely on the relative likelihood between chosen and rejected preferences. However, this approach causes the likelihood of the cho- sen preferences to decrease during the optimization process (Chen al., 2024a; Pal et al., 2024). In contrast, TPO introduces lambda weights into the ranking algorithm to provide absolute positional information for preferences within the list, mitigating data likelihood decline issues.
C.3 ANALYSIS OF VARIOUS RANKING LOSSES
We further introduced various ranking losses, including Pointwise SoftMax (Cao et al., 2007), Pair- wise Logistic (Burges al., 2005), and Listwise MLE (Xia al., 2008), to further confirm the validity of choosing Lambdal Loss for TPO. The experimental results are shown in Table. 6:
We have the following observations: Pointwise SoftMax demonstrates the poorest performance, in- dicating that learning only the reward values of preferences is insufficient. The relative relationships between preferences are particularly important. Pairwise Logistic and Listwise MLE perform worse than LambdaLoss. This is because both approaches only consider the relative relationships between preferences while neglecting their absolute positions in the ranked list. This limitation has been shown in existing Learn-to-Be Rank literature (Wang et al., 2018; Jagerman et al., 2022) to be detri- mental to ranking optimization. also underscores the motivation for introducing lambda weights in this work.
18

---

Published as a conference paper at ICLR 2025
Table 6: Performance of TPO using various ranking losses Math tasks. The best results for each large language model setting are indicated in bold. We report results using pass @ accuracy.
Ranking Loss
MATH SVAMP
ASDiv
GSM-Plus
Avg.
Qwen2-7B-Instrucc
Pointwise SoftMax
52.13
43.22
52.16
48.77
49.07
Pairwise Logistic
55.02
46.85
57.96
53.26
53.27
Listwise MLE
54.10
45.89
54.16
52.33
51.62
LambdaLoss
55.46
48.20
59.22
54.82
54.43
DeepSeekMath Instructt
Pointwise SoftMax
48.59
84.26
87.89
60.96
70.43
Pairwise Logistic
50.33
86.20
90.51
63.57
72.65
Listwise MLE
50.10
85.74
89.22
62.41
71.87
Lambdal.oss
51.30
86.80
90.61
64.73
73.36
Table 7: The t-test experimental results for analyzing the effectiveness of Adaptive Step Reward are reported, including the mean value and standard error of pass @ accuracy. denotes p 0.01, and i denotes < 0.05.
Methods
MATH
SVAMP
ASDiv
GSM-Plus
Owen2-7B-Insteee
TPO w/o Adaptive Step Reward p-value
55.54=0.06#
48.00:10.061
59.2540.11
54.86110.04
15.18::0.07
47.80=10.07
58.60::0.15
54.33
0.0004
0.0361
0.0013
1.0970e-9
TPO
5..09
86.85+10.071
90.83+0.11
64.67-10.04
w/o Adaptive Step Reward
50.78+10.08
86.55-10.10
90.46++0.08
64.34-10.04
p-value
9.9973e-5
0,0167
0.0129
2.8655e-6
C.4 VALIDATION OF ADAPTIVE STEP REWARD EFFECTIVENESS
To assess the statistical significance of our proposed Adaptive Step Reward, we performed 20 inde- pendent experiments for both TPC and TPO w/o Adaptive Step Reward under identical experimental conditions. The performance metrics from these experiments were then analyzed using an indepen- dent I-test to compare the means and standard error the two groups. The experimental results are shown in the Table. 7.
The null hypothesis assumed no significant difference in performance between TPO and TPO w/o Adaptive Step Reward. A p-value less than 0.01/0.05 was considered statistically significant, in- dicating that the observed differences were unlikely to occur by chance. The results showed that TPC consistently outperformed the TPO w/o Adaptive Step Reward, with statistically significant improvements in pass@ @ accuracy, validating the effectiveness of Adaptive Step Reward.
C.5
CASE STUDIES
To better understand the motivation behind the Adaptive Step Reward, we provide two cases for further analysis below.
19

---

Published as a conference paper at ICLR 2025
Casei. The critical errors in the steps are highlighted red font.
**Prompt:** #
Let A = 2 2+ i, B = - 2
5 3i, and = -3- 62.
Calculate (A + - +++++=55+++
Let's think step by step.
Step 1:
First, need to find the sum of all the complex numbers in the problem.
We are given A = 2 + B = 3 2i, O -, P i Q 5 + 37, S = 5 -3, and
T
Gi.
We need to find (A + 0 + Step 2:
** Response; **
Let's add all the complex numbers together. First, add and B:
A + B (2 + (3 21) = 2 2i
Then, subtract O:
5 5-i- (-4 1 9
Next, add
- i (-i) 9- - 2i
Then, calculate Q S:
(5 (5) (5 3i) 25 15i 15i 16
Next, add Q * S:
2i
25- 2i
Finally, add T:
25 2i + (-3 61) 25 2i
22
So. (A + B 0 P Q + 22 8i.
** Responses **
First, let's add all the given complex numbers:
A+ B (2 i) + (3 21) = i
-0 -0=-(-4)=44 =4
p=
5+5 = (5) (5 31) = 25 15i
15i -
9 = =166,
-=-3- Gi.
Now, let's add these results together: 5-i+4 i
6i 22 8i.
** Responses **
Let's add all the complex numbers together. First, add and B:
A+ B (2
+ (3 21) 2 2i =5
Then, subtract 0:
5 5-i- (-4) = 5 9-1
Next, add P:
9 - i (-i) 9 - 2i
Then, add Q:
9- 2 (5 3i)
2 14+
Next, multiply S:
(14+ (5 3i) -00 70 421 3= 67
37i
Finally, add T:
67 37i + (-3 61 67 371
64 43i
So. (A+ B 04
P Q
S) T =
64
43i.
20

---

Published as a conference paper at ICLR 2025
Case2. The critical errors in the steps are highlighted red font.
**Prompt:**
If Jeremy played a three-letter word on a triple word score earn thirty points, and before the word score was tripled, the first and third letters were each worth one point, what was the value of the middle letter before the word score was tripled? Let's think step by step.
Step 1: The word is three-letter word, and the first and third letters each worth one point, so the total value of the first and third letters before the word score was tripled is 1+1 2 points.
Step 2:
** Response""
The word score was tripled to earn points. So, the original word score before tripling was 30/3 10 points.
** Response2 **
The total score of the word before applying the triple word multiplier was calculated as 30/3 10, representing the base score.
** Responsez **
The word score was tripled to earn 30 points. So, the value of the middle letter before the word score was tripled 30 28 points.
noteworthy that, Case Responses: and Responses2. despite having significantly differ- ent expressions, both follow the correct steps, whereas Responsez, though similar to Responser in expression, fails to prioritize multiplication and thus yields an incorrect result. the ground truth ranking, the preference ranking is Response; > Responsez during LLMs' preference learning, since implicit reward are computed token by token, the sig- nificant overlap of tokens between Responsey and Response; leads to incorrect ranking of Response; Response3 > Responsez. A similar phenomenon can also be observed in Caséz.
Responsez. However,
Based on this observation, propose the Adaptive Step Reward mechanism leveraging step-level semantic similarity to adjust the reward margin between preferences. We emphasize the importance of semantics in preference ranking and believe this is particularly critical in tasks like mathematical reasoning, which emphasize semantic similarity rather than token overlap.
We visualized the reward margins during the training process for the two cases in Fig. 4. The experimental results indicate that when the Adaptive Step Reward in TPO is disabled, LLMs exhibit smaller reward margin between Responses; and Responses;; which share high degree of token overlap. Conversely, when Adaptive Step Reward is enabled, LLMs demonstrate smaller reward margin between Responses and Responses2. which share similar semantics. This confirms the effectiveness of the Adaptive Step Reward, as it encourages preference optimization to focus more on semantic information rather than token overlap or sequential order.
C.6 ANALYSIS OF 3 IN TPO
We further conducted additional experiments investigate the impact of on the performance of DPO and TPO. We used Qwen2-7B-Instnne as backbone LLM. The experimental results are shown in Table. The results indicate that reducing 3 DPO from 0.5 0. or even 0.01 improves the performance of LLMs. However, it is noteworthy that TPO's performance also improves and consistently outperforms DPO.
In DPO, 3 is a parameter that controls the deviation from the base reference policy. Therefore, when a smaller is used, means that the constraints between the policy and the reference policy are relaxed, making it easier for the policy to adapt the training task, leading improved performance. This holds true for TPO as well. However, exist work (Wu et al., 2024) show that smaller values of B are always better, excessive deviation from the reference policy can lead more forgetting.
21

---

Published as a conference paper at ICLR 2025
TPO w/o Adaplive Step Reward
TPO
3,0
Reward Marqin
2.5
2.0
2.0
Cases
1.5
1.5
1.0
Responsey Response, Responsey Resporsee
Response, & Response, Responses Response)
0.0
50
100
150
200
250
50
100
150
200
250
Number Training Steps
Number of Training Steps
Reward Margin
J
Reward ra Margin
Casity
Responsey Response;
Response; & Response;
Responsey Response,
Response, Response,
0-
50
100
150
200
250
100
150
200
250
Number Training Steps
Number of Training Steps
Figure 4: Study of Reward Margins.
Table 8: Experimental results of DPO and TPO across different B settings on Math task. The better results for each large language model setting are indicated bold. We report results using pass accuracy.
Methods
MATH
SVAMP
ASDiv
GSM-Plus
Avg.
bela=0.5
DPO
54.26
44.69
54.32
50.28
50.89
TPO
55.46
48.20
59.22
54.82
54.43
beta=0.1
DPO
54.94
45.10
55.02
50.89
51.49
TPO
56.16
48.50
59.54
55.36
54.89
beta=0.01
DPO
55.30
45.30
55.34
51.19
51.78
TPO
56.48
48.50
60.03
55.54
55.14
which may cause a loss in performance other tasks. Thus, we view the choice of 3 as trade-off between optimizing LLM performance on specific tasks and ensuring generalization across tasks.
D DISCUSSION ON RELATED LITERATURE
There are several contemporaneous literature (Chen et al., 2024b; Liao et al., 2024a; Scheid al., 2024) with TPO that also analyze issues related to preference ranking in lists. We present the op- timization objectives of TPO and related literature (Chen al., 2024b; Liao al., 2024a; Scheid et al., 2024) in Table. 9.
S-DPO (Chen et al., 2024b) employs SoftMax to maximize the reward margin between positive preference and all negative preferences. However, S-DPO does not account for the contrasts between negative preferences, nor does consider the absolute positional information of preferences within the list. Although Rose- DPO (Liao et al., 2024a) does not consider the ranking of the preference list, similar to TPO, both Rose-DPO and TPO use additional weights to adjust the reward margin. The difference lies in the adjustment mechanism: while Rose-DPO is based uncertainty, TPO adjusts
22

---

Published as a conference paper at ICLR 2025
Table 9: Optimization Objectives for Preference Alignment. Where eg used in Rose-DPO represents the uncertainty assessment of preferences.
Methods
Objective
S-DPO (Chen et al., 2024b)
log
( log Eyeey, exp B log ma(ywla)
9 log Tent (sh (2)
Nowe (arc (2)
Rose DPO (Liao et al., 2024a)
(1 - cs) (3log (Mi
3 log
Turlyw (xr)
-c (3log
matyr (ar]
Blgg (2( 12)
(the (t)
exp(8 log motoein)
ODPO (Scheid al., 2024)
log I
Ej exp( log Fourloy1n)
Xs. E Diie log a (3 log ro(ull))
3 log mo(yy Ferllyy
TPO
the margin according to the semantic similarity between preference pairs. ODPO (Scheid et al., 2024) utilizes Maximum Likelihood Estimation (MLE) to optimize list-wise preferences. However, existing Learn-to-Rank literature (Wang et al., 2018; Jagerman et al., 2022) suggests that list-MLE not an ideal ranking optimization objective, as it enforces strict list ordering without considering the actual label values.
23


Reviews:
[{"review_id": "c4fhNpfuuO", "reviewer": "Reviewer_2RYw", "summary": "This work proposed TPO which enables direct preference alignment for long-chain reasoning tasks.  By sampling with tree-of-thoughts, lists of generated CoTs would be paired with preference signals, which are used for LLM alignment. Limited by the original DPO, which only enables pairwise preference optimization, TPO proposed list ranking formulation to extend DPO framework. Empirical studies on several mathematical reasoning tasks show better performance.", "questions": "1. Could the authors explain the link between DPO on chain-of-thought reasoning, which from my understanding is the major contribution of this paper, and the proposed Preference List Ranking optimization. Specifically, I was wondering if extracting pairwise samples from the ranking list could simply solve the challenge?\n\n2. Could the authors explain their means of acquisition of such ranking list preference signals, which could be relatively more expensive and harder to find? Any intensive human annotation process involved in this paper?\n\n3. Could the authors try to discuss the potential alternative ranking list preference alignment methods I provided in Weaknesses?\n\n4. Could the authors provide some insights or experimental results, if applying their TPO on knowledge-intensive CoTs? Since in such tasks knowledge fidelity (factuality) could be entangled with logical reasoning steps (tree-of-thought sampling), the alignment process is not as straightforward as mathematical or coding problems. In such cases, would TPO still work under some guarantees with some lower-bounded confidence?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well motivated and well presented.\n\n2. The connection between LLM alignment and learn-to-rank problems is well explained.\n\n3. The proposed implicit reward model formulation in Eq(10) is novel.", "weaknesses": "1. Regarding the preference list ranking, there might have been some studies on its direct optimization approaches [1-3]. The authors could consider discussing their unique challenges or difference to those works. Ideally, the authors might consider adapting their ideas as baselines for a more comprehensive comparison.\n\n2. The backbone general LLMs are restricted to only one type, Qwen. The authors might considering experimenting on more types of LLMs, including Llama-3, Phi-2, Mistral, etc.\n\n3. The authors could consider including supervised fine-tuning and/or instruction fine-tuning baselines, which could help the readers better understand the potential performance upper bound on the specific downstream tasks.\n\n4. It would be critical for the authors to also demonstrate the generalizability of the LLMs after TPO on general benchmark or metrics, which might prevent potential chances of model overfitting on the downstream tasks.\n\n5. Although the proposed DPO is proposed for general chain-of-thought reasoning, only maths and coding tasks are evaluated. More commensense, multi-hop question-answering, and especially knowledge-intensive tasks could be evaluated.\n\n[1] Chen, Yuxin, et al. \"On Softmax Direct Preference Optimization for Recommendation.\" arXiv preprint arXiv:2406.09215 (2024).\n\n[2] Liao, Jiayi, et al. \"RosePO: Aligning LLM-based Recommenders with Human Values.\" arXiv preprint arXiv:2410.12519 (2024).\n\n[3] Scheid, Antoine, et al. \"Optimal Design for Reward Modeling in RLHF.\" arXiv preprint arXiv:2410.17055 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t4xoF9E0K4", "reviewer": "Reviewer_UFQG", "summary": "The paper presents Tree Preference Optimization (TPO), an enhancement to Direct Preference Optimization (DPO) for LLM alignment. Unlike DPO’s binary preference structure, TPO uses ranked preference lists and an Adaptive Step Reward to adjust the reward margin, allowing the model to learn from multi-branch and multi-step reasoning paths and improve accuracy in complex tasks.", "questions": "See Weakness", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is well-written and easy to follow.\n2.\tThe paper proposes using the classical Lambda weight from Learning-to-Rank (LTR) to construct the TPO loss, which enhances the accuracy of list ranking. This is an interesting design.\n3.\tThe design of a loss function for multi-branch and multi-step scenarios is a highly significant research direction.The paper makes several attempts in this direction.", "weaknesses": "1. The most critical issue is that I believe obtaining noise-free, list-wise preference data is very costly. A more essential challenge is how to reliably construct such data. The construction method in the paper likely introduces substantial noise, and I am skeptical about the positive impact of aligning the llm to a noisy list ranking.\n2. The Adaptive Step Reward mechanism proposed is overly intuitive, lacking more rigorous experimental analysis and theoretical support.\n3. The experimental design in the paper is somewhat unfair, as DPO only uses pair-wise data, whereas TPO leverages additional augmented data. I believe the list data should also be converted into pair-wise form, allowing for a fair comparison. This would enable analysis of whether list-wise data is indeed more effective than pair-wise when the total information content is kept constant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wEcEohcTzb", "reviewer": "Reviewer_Svzf", "summary": "This paper presents Tree Preference Optimization (TPO), a novel framework that decomposes Tree of Thoughts into multi-branch and multi-step response patterns for fine-grained preference optimization. The framework reconceptualizes language model alignment as a generalized Preference List Ranking problem, which facilitates more effective alignment learning from preference lists. TPO introduces the Adaptive Step Reward mechanism, which adjusts step-wise rewards based on inter-step correlations to focus on discriminative steps, thereby enhancing the optimization efficiency.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and presents comprehensive empirical results with publicly available implementation code, demonstrating the effectiveness of the proposed approach.\n\n- This paper presents a novel approach by decomposing Tree of Thoughts into a hierarchical structure of multi-branch, multi-step responses and leveraging Learning-to-Rank algorithms for preference optimization.", "weaknesses": "- The rationale for selecting LambdaLoss over other LTR loss functions in Section 3.1 requires further clarification. Maybe you can justify this choice by highlighting the specific advantages of LambdaLoss in your application.\n\n\n- In Equation 10, I notice that RM=0 in Line 298, while my understanding suggests that cosine similarity should be 1 when the content is shared. Could you please clarify this point?  Maybe you can provide more explanation of your *Adaptive Step Reward*.\n\n- The ablation studies for Adaptive Step Reward appear limited. Based on Table 3, its impact seems modest. Further investigation into the effectiveness and mechanism of Adaptive Step Reward would strengthen the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)