ICLR 2025

Published as a conference paper at ICLR 2025
TPO: ALIGNING LARGE LANGUAGE MODELS WITH
MULTI-BRANCH & MULTI-STEP PREFERENCE TREES
Weibin Liao♠, Xu Chu♠♢♡∗, Yasha Wang♠♡
♠School of Computer Science, Peking University
♢Center on Frontiers of Computing Studies, Peking University
♡National Research and Engineering Center of Software Engineering, Peking University
liaoweibin@stu.pku.edu.cn, chu xu@pku.edu.cn
https://github.com/MrBlankness/TPO.git
ABSTRACT
In the domain of complex reasoning tasks, such as mathematical reasoning, recent
advancements have proposed the use of Direct Preference Optimization (DPO)
to suppress output of dispreferred responses, thereby enhancing the long-chain
reasoning capabilities of large language models (LLMs). To this end, these stud-
ies employed LLMs to generate preference trees via Tree-of-thoughts (ToT) and
sample the paired preference responses required by the DPO algorithm. However,
the DPO algorithm based on binary preference optimization was unable to learn
multiple responses with varying degrees of preference/dispreference that provided
by the preference trees, resulting in incomplete preference learning. In this work,
we introduce Tree Preference Optimization (TPO), which does not sample paired
preference responses from the preference tree; instead, it directly learns from the
entire preference tree during the fine-tuning. Specifically, TPO formulates the lan-
guage model alignment as a Preference List Ranking problem, where the policy
can potentially learn more effectively from a ranked preference list of responses
given the prompt. In addition, to further assist LLMs in identifying discriminative
steps within long-chain reasoning and increase the relative reward margin in the
preference list, TPO utilizes Adaptive Step Reward to adjust the reward values of
each step in the trajectory for performing fine-grained preference optimization.
We carry out extensive experiments on mathematical reasoning tasks to evaluate
TPO. The experim...