Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs
Authors: Xuan Zhang
Chao Du
Tianyu Pang
Qian Liu
Wei Gao
Min Lin
Venue: NeurIPS 2024 poster (2024)
Keywords: large language model
reasoning
tree of thought
Primary area: natural_language_processing
TL;DR: A novel method called Chain of Preference Optimization (CPO), which leverages the supervision generated by the self-reasoning process (i.e., Tree-of-Thought) to enhance the reasoning ability of LLMs.

Abstract: The recent development of chain-of-thought (CoT) decoding has enabled large language models (LLMs) to generate explicit logical reasoning paths for complex problem-solving. However, research indicates that these paths are not always deliberate and optimal. The tree-of-thought (ToT) method employs tree-searching to extensively explore the reasoning space and find better reasoning paths that CoT decoding might overlook. This deliberation, however, comes at the cost of significantly increased inference complexity. In this work, we demonstrate that fine-tuning LLMs leveraging the search tree constructed by ToT allows CoT to achieve similar or better performance, thereby avoiding the substantial inference burden. This is achieved through \emph{Chain of Preference Optimization} (CPO), where LLMs are fine-tuned to align each step of the CoT reasoning paths with those of ToT using the inherent preference information in the tree-search process. Extensive experimental results show that CPO significantly improves LLM performance in solving a variety of complex problems, including question answering, fact verification, and arithmetic reasoning, demonstrating its effectiveness. 
Our code is available at [https://github.com/sail-sg/CPO](https://github.com/sail-sg/CPO).

Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs
Xuan Zhang 12, Chao Dut Tianyu Pang', Qian Liu', Wei Gao?, Min Lin
SSa AI Lab, Singapore
School of Computing and Information Systems, Singapore Management University xuanzhang- 2020@phdcs. smu edu. ag; weigaoßsmu.. edu. ag; {duchao, liuqian, tianyupang, linmin]@sea. com
Abstract
The recent development of chain-of-thour of-thought (CoT) decoding has enabled large language models (LLMs) to generate explicit logical reasoning paths for complex problem-solving. However, research indicates that these paths are not always deliberate and optimal. The tree-of-thought (ToT) method employs tree-searching to extensively explore the reasoning space and find better reasoning paths that CoT decoding might overlook. This deliberation, however, comes at the cost significantly increased inference complexity. In this work, we demonstrate that fine-tuning LLMs leveraging the search tree constructed by ToT allows CoT to achieve similar or better performance, thereby avoiding the substantial inference burden. This is achieved through Chain of Preference Optimization (CPO), where LLMs are fine-tuned to align each step of the CoT reasoning paths with those of ToT using the inherent preference information in the tree-search process. Extensive experimental results show that CPO significantly improves L.LM performance in solving a variety of complex problems, including question answering, fact verification, and arithmetic reasoning, demonstrating its effectiveness. Our code is available at http:iiuuub.ommiliili/CCCCC
Introduction
Recent advances in large language models (LLMs) have shown that constructing reasoning chains
is eritical to improving their problem-solving capabilities [1, 2, 3, 4, 5, 6, 7]. A representative method is chain-of-thought (CoT) [i], which prompts LLMs to generate intermediate reasoning steps, i.e., thoughts, thereby constructing explicit reasoning paths (as depicted in Figure 1(a)). While straightforward and intuitive, recent research observes that Co'I' can often overlook optimal reasoning paths and exhibit unconscious style of answering due its single-path focus [8, 9]. To foster more deliberate and conscious reasoning style, Yao et al. [8] propose tree-of-thought (ToT), which generates multiple branching thoughts at each step of the reasoning process and conducts self- evaluation for pruning and planning to search for reasoning paths (as shown in Figure 1(b)). However, despite improving reasoning quality, ToT significantly increases computational complexity, which limits its practical application. This raises the question: Can the strategic depth of ToT be integrated into CoT to enhance its effectiveness while maintaining efficiency?
Existing research has initially provided a positive answer to the above question [10, 11, 12]. A natural strategy is treat the reasoning path discovered by ToT for each instance target for supervision, and then fine-tune L.MS to improve their CoTT reasoning abilities [11, 12]. Several methods have been proposed to improve this approach, including using advanced tree-search techniques like Monte
Work done during Xuan Zhang's associate membership at Sea Al Lab. *Correspondence to Chao Du.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Chain of Thought (Col)
(b) Tree Thought (ToT)
Chain Preference Optimization (CPO)
Input
Input
Input
DPO
Preference data
Output
Ourput
Output
Output
Selected thoughts.
Training.
Collecting preference data (of each step) through ToT then tuning LLM via DPO.
inference: same as CoT.
Pruned thoughts.
Inference.
Comparison of CoT,, ToT,, and CPO methods, where each node illustrates a step in the reasoning process, forming coherent language sequences aimed at solving a problem. The highlighted path indicates the chosen reasoning trajectory. In the CoT method, the LLM generates only one new node at each step, and all generated nodes are used to build the final reasoning path. For ToT, the LLM produces k new nodes at each step, but only the top n-best nodes are kept, with the rest being pruned. CPO, nodes marked with trophy represent preferred thoughts, while those marked with numbers are nodes that can be utilized to create preference data. This method uses the search tree structure from ToT to develop paired preference data, subsequently training LLMs to align with these preferences through DPO.
Figure 1:
Carlo tree-search (MCTS) and employing external reward models [12, 10] for pruning and planning gather better reasoming paths as supervision. The effectiveness of these approaches therefore largely dependent on the quality of the best-discovered reasoning path.
In this paper, we identify limitation in these approaches: they overlook the non-optimal reasoning thoughts generated during the tree-search process, which naturally provides additional preference information. Specifically, ToT inherently generates multiple alternative thoughts at each reasoning step, and pruning is performed according to their evaluated qualities. This tree-search process constitutes a preference discovered reasoning path are preferred over those that do not. Moreover, this could shed even more insights than the final best-discovered reasoning path, non-optimal reasoning paths (and thus preferences) exist at each step in the tree-search.
all intermediate thought
appearing in the best-
Inspired by recently developed reinforcement learning from human feedback (RLHF) techniques like direct preference optimization (DPO) [13], we propose Chain-of-Prefereeee construct paired preference thoughts
Optimization (CPO) to
fully exploit the inherent preference information. Specifically, at each reasoning step according to the search tree of ToT and then train LLMs to align with these preferences using the DPO algorithm (as illustrated in Figure I(c)). The paired preference thoughts constructed based on the above intuition: at each reasoning step, we categorize thoughts as preferred dispreferred based on their inclusion in the final paths chosen by ToT. With such preference data, CPO enables L.Mss to generate the path preferred by ToT using CoT decoding at inference time.
We conduct extensive experiments to evaluate the effectiveness of CPO. Experiments seven datasets using LLaMA [14] and Mistral [15] as base models demonstrate that CPO is highly effective in teaching LLMs the preferred thoughts of ToT at each reasoning step, leading improvement of up to 4.3% compared to the base models. Additionally, the experiments reveal that superior performance the ToT method, which average
average accuracy
CPO can achieve comparable requires more than 50 times longer for inference.
Related Work
Reasoning with L.MMs. LLMS have been shown to perform better when prompted engage in multi-step reasoning [1, 2, 3]. Many studies have focused on improving the generated reasoning paths

---

by post-editing [16] or accessing external knowledge [3, 17]. A distinct approach, more relevant to our interests, transforms the linear reasoning structure into a non-linear format such a tree or graph [18, 19, 20, 8, 9, 21], which combines thought evaluation with search algorithms like depth-first search (DFS) [22]. Different from our proposed CPO, these methods require searching during inference, which significantly increases latency.
LLM seff-improving. Reinforcement learning (RI.) has increasingly been applied to LLMs by treating them as RL agents for alignment with human feedhack [23, 24, 25, 26]. Recent advances demonstrate the potential of using LLMs for self--enerating data to augment fine-tuning processes [27, 28, 29, 30, 31, 32]. For instance, reinforced self-training methods [33, 34, 35, 36, 37] introduce mechanisms to curate new high-quality examples and iteratively enrich the training dataset for enhancing model performance. Nevertheless, these methods typically rely on either an external reward model [35, 34] or labeled data [33]. In contrast, approaches like self-rewarding [29, 38] utilize LLMs themselves to evaluate the generated content, aligning more closely with our method. However, these strategies still require initial seed data [29, 38], necessitating human annotation. Our work differs from previous methods as it does not rely on any ground-truth data, allowing LLMS to self-learn from their own feedback. Additionally, our approach constructs feedback in chain fashion, focusing on reasoning steps, an aspect overlooked by prior works.
Monte Carlo tree-search for LLMs. Monte Carlo tree-search (MCTS) is robust algorithm for navigating complex decision-making environments, commonly employed in strategic board games such as AlphaGo [39, 40, 41, 42, 43]. MCTS methodically constructs search tree, balancing exploration and exploitation, simulates various outcomes, and updates utility estimates based on these simulations. Recent studies have shown that MCTS can enhance the decoding process in LLMs [11, 44, 45, 21, 12]. However, the primary challenge with MCTS is the high latency during inference, particularly in difficult reasoning tasks [46, 47]. While some approaches have attempted to optimize LLMs by leveraging reasoning paths identified through MCTS [11, 12], these methods still rely on labeled data and require separate policy and value models to explore and evaluate potential moves at the tree's leaves. In contrast, our CPO approach eliminates the need for human annotations and simplifies the tuning of LLMs without the necessity for additional models.
3 Background
In this section, we formalize our notation and provide a brief overview of key prior knowledge for our method. We denote language sequences by lowercase letters, e.g., y. z, to represent sequence of tokens. The output distribution of LLM parameterized by is denoted by Ty.
3.1 Chain-of-Though Prompting
Chain-of-thougle (CoT) [1] is a method that prompts LLMs to generate chain of reasoning steps before the final answer, as shown in Figure 1. It introduces series of intermediate thoughts, denoted as 21, 2n. that link an input to an output where n is the total number of reasoning steps. For instance, if is a combination of demonstration examples and the input question and y is the final answer, each intermediate thought forms a coherent language sequence representing part of the overall reasoning path toward the final answer. The demonstration examples consist of set of CoT demonstrations, which serve exemplars in the prompting process. The intermediate reasoning thoughts are sequentially sampled from the distribution 2, ~ To( [a., 21, .. 2111) and the output then derived from y ~ TT( (x. 21, 2n))
3.2 Tree-of- Tree-of-Thoughh Prompting
Tree-of-though (ToT) [8] enables LLMs to explore multiple reasoning paths before answering a given question, as illustrated in Figure 1. This approach models the LLM reasoning task as search over a tree, where each node represents a thought step in the reasoning path. ToT comprises two main components, both implemented through prompting LLMs: 1) the thought generator and 2) the state evaluator. The thought generator constructs several new thoughts for the next step based on the current state. Subsequently, the state evaluator generates scores for each new thought and selects the
3

---

Input
Collection
Input
Thought generation
Thoughts:
Preferred
thought:
Is thought helpful onswer the question?
Dispreferred
self- evaluation
thought:
Scores:
Preference
thoughts
BFS + Pruning
X
collected:
Next step generation
Output
Thoughts:
D
Figure 2: The framework of our CPO method. The left part illustrates the process of generating. evaluating, and pruning thoughts, while the right part demonstrates the collection of preference thoughts. The shaded path represents the final selected reasoning path. Thoughts marked with trophy indicate preferred data, while sibling nodes without trophy are marked as dispreferred.
n-best thoughts for further search. The final result is determined by the search algorithm (e.g., BFS or DFS) applied over the selected thoughts until the reasoning process reaches a conclusion.
3.3 Direct Preference Optimization
Direct preference optimization (DPO) is method for directly optimizing an LLM to align with preference data [13], c.g., human feedback [13, 48, 49]. More specifically, RLHF traditionally frames the application of human feedback to enhance the performance of LLM within the context of an RL problem. However, DPO reformulates the reward modeling and RL fine-tuning phases in RLHF to single optimization problem. The objective function of DPO aims to maximize the ratio of probabilities for the preferred responses and optimize the LLM to imitate human preferences.
Given the generations (11, 92) x (ū]x²) conditioned on input these pairs are evaluated and ranked according to specific criteria. Preference data then constructed from these ranked pairs, denoted by Yar > it xr where Dwr and in denote the preferred (winning) and dispreferred (losing) completions between V1 and V2. respectively. The DPO objective formulated follows:
LDPO Tref) llo (Blogg (Blog
To (+)))
To (in
B log
Trer (n (r)
Tee (81 (x)
where the logistic function, the hyperparameter 3 regulates the penalty imposed for the deviations from the base reference model Tee--
Our Method: Chain of Preference Optimization
Unlike previous methods that train LLMs to learn the complete reasoning paths [10, 50, 11, 12], our approach leverages the preferences over thoughts generated each reasoning step, which are often discarded prior works. Our key insight that non-optimal thoughts generated during the tree-search process in ToT provide valuable preference information that enhance LLM's reasoning ability. A major advantage of our method is that it utilizes this supervision only during training, thereby avoiding high inference latency. Our approach consists of two components: synthesizing the chain of preference thoughts (i.c., the preference thoughts in a chain fashion) and training with the CPO objective.
4.1 Synthesizing the Chain Preference Thoughts
Our procedure for synthesizing and collecting preference thought pairs closely follows the inference process of ToT [8]. An overview of our method is shown in Figure 2. Specifically, the detailed process is divided into three parts: 1) thought generation, which generates multiple thoughts for each

---

reasoning step; 2) state evaluation, which evaluates each thought; and 3) search and collection, which finalizes the preference thoughts.
Thought generation. Given a state s;-1 = [a,, 21. ................................ 2 representing partial solution with the input r and the sequence of thoughts (21.-. 21-1] so far, we sample & thoughts for the next reasoning step:
~ (o((((s-1)) = To(2; 21, " ; for j = =1, ,,..,,
Conditioned on the initial input x, which contains the demonstration examples and the question to be answered, and the previous thoughts 21; #2,... 1, the L.MM generates multiple thoughts for the next reasoning step. Specifically, it follows the format of demonstrations, starting with the prefix "Step i, and samples & thoughts {2 jj=1- We control the model to pause the end of 2, by setting the generation of the string "Step + 1," as the stop criteria. ? As a result, we obtain new states s [r, Z1, -1-1, ( for j 1,
State evaluation. Given different states { we utilize the LLM to reason about the states and evaluate their progress toward solving the problem, climinating the need for an external reward model or human annotations. To evaluate state 5/ the input to the LLM includes specific demonstration examples for the evaluation process, the input question I, and all the thoughts in the state (i.e.,
[21, 24-1, 2t (.). The LLM follows the format of demonstrations to generate a verbal justification first, followed by classification result from two classes: likely and impossible. The classification results are then used to assign a score, with likely = 10 and impossible =
The prompt template used in our evaluation consists of two parts: (1) the general guidelines, and (2) task-specific demonstration examples. To minimize the effects of randomness and bias, we shuffle the order of demonstration examples [51] and repeatedly sample the generated justification and evaluation results. We then calculate the average score for the state s("), The general guideline prompt for the evaluation is as follows: Evaluate whether the thought helps in partially or directly answering the original question hiikely/imossebbee)
Search and collection. We use BFS with pruning as the search algorithm select the reasoning paths. After evaluation, we retain the n-best thoughts with the highest evaluation scores and proceed to the next step of generation. When the LLM generates a thought containing "so the final answer is:", the search algorithm concludes and returns the selected paths.
As shown in the right part of Figure 2, after finalizing the reasoning paths, the thoughts within the selected paths are marked as preferred (i.e., winning) thoughts. For each preferred thought at the i-th step 2!" we construct corresponding dispreferred (i.e., losing) thoughts. First, identify the parent state s which includes all the previous thoughts leading to z Each child thought of s that is not included in the selected path is chosen as a dispreferred thought 2f compared to 2," This process results in the preference pair (2,", 2/) for the state s We highlight that the constructed dataset D includes preference data at every step of the reasoning chain. This per-step paired preference supervision is usually overlooked in previous methods [11, 12].
4.2 Training with the CPO Objective
Once we have obtained the chain of preference thoughts D, we can proceed with optimization. For the i-th step, given the previous reasoning thoughts s'-1. 1: the probabilities of generating z/" and af are denoted as To (2)") [x, s and <0(2/10, p2. s )) respectively. To optimize the LLM on this pair of preference thoughts,
directly substitute it into Equation 1:
Li(mo) log a Blog T0(2)" sx 3 llg
To( [r, 85
Teel 2 [r, s(1))
Trerz//2z s
"The "stop criteria" is used control when generation should stop, which is implemented via a function input in Hugging Face's Transformers Library.
5

---

Thus, the objective function for CPO can be formulated as follows:
CCPO(Te: Tet) (B sf 1ND (Ca(Ta)
(4)
5
Experiments
In this section, we empirically validate that CPO improves the reasoning ability of the base model, and uncover several insightful findings.
5.1 Settings
Datasets and evaluation metrics. We focus our research on three types of reasoning tasks: Question Answering (QA), Fact Verification, and Arithmetic Reasoning. For QA, we conduct experiments on three widely used datasets: Bamboogle [17]. WikiMulciHopQA [52], and HotpotQA [53]. For fact verification, we use three datasets: Fever [54], Feverous [55], and Vitamine [56]. For arithmetic reasoning, we test on the SVAMP dataset [57]. Our choice of tasks was driven by the performance of the models using the ToT method, which showed improvements in QA, fact verification, and arithmetic reasoning. We use 4-shot prompting for each dataset, with CoT demonstrations manually constructed by previous works [1, 17, 2]. Detailed experimental configurations can be found in Appendix A. For evaluation metrics, we report the accuracy and the average latency of generating the answer per instance. More metrics can be found in Appendix B.
Baselines. To validate the effectiveness of our proposed CPO, we consider the following baselines: 1) CoT [1], which prompts the LLM to generate a series of reasoning steps before producing the final answer. In our experiments, we use CoT with greedy decoding assess the model's reasoning capabilities without any tuning. 2) ToT [8], which requires the LLM to explore multiple reasoning paths via tree search before generating the final answer. We use ToT to select reasoning paths and construct datasets to improve LLM's reasoning ability in the following TS-SFT baseline and our CPO method. 3) TS-SFT [11], which finds reasoning paths through tree search (i.e., ToT in our implementation) and then uses these paths during the supervised fine-tuning (SFT) process (referred to as SFT in Section 5.3 and 6).
Implementation Details Our experiments are based widely used LLMs, specifically LLaMA2- 7B/13B [14] and Mistral-7B [15]. For efficient fine-tuning, we use Low-Rank Adaptation (LoRA) adapters [58]. In all experiments,
set the regularization controller 3 to 0.1, generate 10 new thoughts for each state, and retain the top 5 thoughts after pruning at each step of reasoning. The temperature is
0.9 for SVAMP and 0.4 for the other datasets. The learning rates for DPO and SFT are 5e-6 and 1e-5, respectively. We use batch size (with accumulation) of 32 and optimize the LLM with AdamW [59]. For LoRA, the rank is set 8, and a is set to 16. All experiments are conducted on NVIDIA A100 GPUs. The latency reported in Table 1 is based on single NVIDIA A100 40GB. Both training and inference are performed using the Accelerate [60] backend. We train the LLMs for 4 epochs with early stopping based on the performance validation set. To mitigate the influence of randomness, all experiments are repeated three times with different random seeds, and the average results are reported.
randomly sampled
5.2 Overall Results on Reasoning
Table
summarizes the performance across various reasoning tasks. We have the following findings:
CPO improves LLM's reasoning ability. As shown in Table 1, CPO enhances the reasoning ability of the base LLM,, achieving an average improvement of 4.3% and a maximum improvement of 9. 9.7% across all tasks and L.LMs compared the CoT approach. This indicates that CPO effectively im- proves the LLM's reasoning capabilities. Notably, CPO achieves these improvements without requir- ing additional numan-annotater data, which is particularly beneficial in resource-constrained settings.
CPO has a lower latency than ToT but comparable performance. Although ToT consistently im- proves performance over CoT, incurs high latency due to the need to generate and evaluate multiple thoughts at each reasoning step during inference. This process produces numerous tokens, resulting

---

Table 1: Experimental results for ToT, CoT, TS-SFT, and our proposed CPO across complex tasks including question answering, fact verification, and arithmetic reasoning are presented. means significantly better than the best baseline (TS-SFT) with p < 0.01. Bold denotes the best method, and the second best the top method is ToT.
ToT
CoT [1]
TS-SFT [11]
CPO (ours)
Acc. (%)f
Latency (s/ins.)
Acc. (%)t
Latency (s/ins.)
Acc.
Latency (sfins.)
Acc. (%)f
Latency (s/ins.)
(%)
LLaMA2-7B
Bam.
33.6
1168.4
29.6
37.2
30.4
36.5
32.0*
38.2
Question
2Wiki.
28.6
847.6
26.3
35.7
27.6
35.5
29.7
35.7
Answering
Hot.
23.0
1100.7
21.0
45.5
22.7
44.8
24.0*
41.1
FVR.
47.3
2087.1
45.8
33.8
47.5
34.0
53.2
36.8
Fact
FVRS.
47.5
2539.5
44.3
40.6
46.0
40.4
49.0"
41.2
Verification
Vita.
50.7
2639.3
47.3
35.9
51.0
40.1
52.7*
40.1
Arithmetric
SVA.
42.7
1861.1
37.7
33.3
43.1
30.2
46.0"
32.1
Average Performance
39.1
1749.1
36.0
37.4
38.3
37.4
40.9
37.9
LLaMA2-13B
Bam.
53.8
1318.3
48.0
50.5
50.8
49.6
52.0
50.3
Question
2Wiki.
36.3
1097.1
28.3
67.0
29.0
66.5
30.3*
60.4
Answering
Hot.
32.0
1271.0
29.0
65.2
30.3
65.5
30.3
63.8
FVR.
48.8
3139.8
48.2
45.4
48.8
44.0
49.2
43.9
Fact
FVRS.
51.3
3433.2
50.0
61.1
48.8
58.0
50.7*
68.2
Verification
Vita.
52.5
2933.6
46.3
48.3
49.7
51.5
54.0"
58.6
Arithmetric
SVA.
45.7
2115.3
40.3
46.2
44.6
46.4
50.0*
48.1
Average Performance
45.8
2186.9
41.4
54.8
43.1
54.5
45.2
56.2
Mistral-7B
Bam.
46.4
4399.6
41.6
46.2
41.6
45.0
45.6*
47.0
Question
2Wiki.
28.4
2356.9
26.7
45.1
31.0
44.2
31.7
46.3
Answering
Hot.
30.0
4698.0
28.0
58.4
28.6
56.2
29.4
56.9
FVR.
61.4
3291.3
57.9
41.8
60.2
41.7
59.9
40.6
Fact
FVRS.
50.5
5537.8
48.0
51.0
49.7
49.5
54.0*
53.0
Verification
Vita.
52.2
4698.2
47.7
44.8
50.3
45.9
53.7*
46.6
Arithmetric SVA.
66.0
4623.7
65.3
41.4
59.0
41.3
69.3*
44.9
Average Performance
47.8
4229.4
45.0
47.0
45.8
46.3
49.1*
47.9
in significant computational and memory overhead [61]. In contrast, CPO shifts this computational burden the training phase, maintaining the low latency of CoT (i.e., 57.5x faster than ToT on average) during inference while providing comparable or superior performance. This demonstrates that our CPO can deliver enhanced reasoning capabilities without compromising efficiency.
CPO surpasses TS-SFT on average. Despite both CPO and TS-SFT using ToT to generate training data (where implementation of ToT remains consistent), CPO exhibits average improvement of 2.7% and reaches a maximum increase of 10.3%. A key factor behind this performance is the CPO's ability to fully utilize ToT's reasoning process. Specifically, CPO effectively leverages both selected and unselected thoughts at each reasoning step, whereas TS-SFT only uses information from the selected paths, offering CPO with clear advantage. detailed discussion of the effectiveness of CPO is presented in Section 5.3.
5.3 Component-wise Evaluations

---

35
35
Accuracy (%)
Bave Model
CPO of Lower CPO
Accuricy (%)
Base Model
CPO Lowest
SFT
CPO
31.5
31.5
28
28
(a) Effect of dispreferred thoughts selection. Base Model: The base LLM. CPO w/ Lowest: CPO using
(b) Effect of per-step preference supervision. Base Model: The LLM without any fine- tneetuning. SFT: SFT of the base LLM (i.e., our TS-SFT baseline). FPO:
the lowest-scoring thoughts as dispreferred. CPO w/ Lower: CPO using all lower-scoring thoughts as dispreferred. CPO w/ All: CPO using all thoughts not the selected paths as dispreferred.
DPO of the base LLM on paired full paths. CPO: CPO
of the base L.M on paired per-step thoughts.
Accuracy (%)
34.0
31.7
Accurccy (%)
33.0
32.0
31.2
29.5
31.0 304
30.4
Base Model
29.0
40
80
120
160
200
0.25
0.5
0.75
Number of instances utilized
Percentage of dispreforred thoughts utilized
(c) Effect of the number of instances in generating paired thoughts. The number indicates the number of instances (e.g.. questions in the QA task) used to generate paired thoughts for our CPO.
(d) Effect of dispreferred thoughts in optimization. The percentage indicates the proportion of data used for CPO, which uses dispreferred thoughts.
Figure 3: Component evaluations and analysis on the Bamboogle dataset using the LLaMA2-7B as the base model.
Effect of selection methods of dispreferred thoughts. We analyze the impact of different methods for selecting dispreferred thoughts on model performance. As shown in Figure 4, experiment with three strategies based on evalua- tion scores for each thought: 1) CPO w/ Lowest: Only the lowest-scoring thoughts in each reason- ing step are dispreferred thoughts. CPO w/ Lower: Thoughts with evaluation scores lower than the selected paths dispreferred thoughts. 3) CPO w/ All: All thoughts not in the selected paths are considered dispreferred thoughts. We ensured an equal number of training samples for each strategy. Note that the evaluation score at each intermediate reasoning step (apart from the final one) determines whether create the next reasoning step but not which thoughts are preferred. For example, as shown in the figure, even though the score of 32 is higher than 23, the thought with a score of 23 is preferred since it is part of the selected path.
Iapur
Evaluation Score:
Lower:
CPO m/
14
Ourput
Figure 4: Different strategies for selecting dispre- ferred thoughts and their impact on model perfor- mance. At each reasoning step, three strategies are used select dispreferred thoughts based on their reasoning scores: 1) CPO w/ Lowest: Selects only the thought with the lowest score. 2) CPO w/ Lower: Selects all thoughts with scores lower than the preferred thought. 3) CPO w/ All: Selects all thoughts dispreferred as long as they are not the preferred thought.
The results in Figure 3(a) show that the perfor- mance differences among these strategies are minimal. This suggests that the distinction between preferred and dispreferred thoughts is better determined in the selected reasoning path rather than intermediate evaluation scores. To obtain greater number of preferred thoughts for each instance to create paired preference thoughts, we chose the CPO w/ All strategy.
Effect of numbers of training data. To assess the impact of the number of training data used in optimization, we conduct an ablation analysis by varying the number of instances (e.g.. questions in the QA task) used to generate paired preference thoughts, ranging from 0 to 200. As illustrated in

---

Table 2: Effect of different kinds of training data on the Bamboogle dataset using the LLaMA2-7B as the base model.
Data
Description
SFT
CPO
Single- Task Uniform QA Mixed-Type
Training only on specific task (Bamboogle) data. Training on 3 datasets of the same type (QA) as the test task.
30.4
32.0
31.2
35.2
Training on all 7 different types of data.
29.6
35.2
Figure 3(c), observe that with an increase in the number of instances, the model's performance initially declines and then rises. Specifically, when trained with data generated from less than S0 instances, the model's performance is even worse than without any training, likely due to overfit- ting [62], which leads to performance degradation. However, as the number increases to 120, the model's 's performance consistently improves. Optimizing with paired thoughts from 120 instances, the model's performance surpasses that of the base model. When the number exceeds 120, the model's performance converges, indicating balance of data for training.
Sensitivity of data mixture. We explore the performance of the CPO method across diverse data settings to assess its adaptability and learning efficiency from various data types. As shown in Table 2, we specifically examine three different data configurations: 1) single task data, 2) uniform QA data, and 3) mixed-type data. Our findings indicate that CPO demonstrates performance improvements of 3.2% in both settings and 3, suggesting its robust ability to harness diverse data sources to enhance learning outcomes. In contrast, the SFT method exhibits comparable performance across these settings, indicating a different sensitivity to data diversity. It is worth noting that, to ensure fairness, although find that mixed data leads to better performance, the experiments in Table I conducted using individual datasets for training, consistent with the baselines.
6 Analysis
Do we need dispreferred information? We explore the impact of dispreferred thoughts on model performance by gradually incorporating these thoughts into the training data. Initially, we introduce dispreferred thoughts for their corresponding preferred counterparts and apply CPO to this segment of the data. For preferred thoughts without dispreferred counterparts, we implement SFT on these data. Consequently, the percentage of dispreferred thoughts incorporated can also be viewed as the proportion of data processed using CPO. We adjust the inclusion percentage of dispreferred thoughts from 0% to 100%. An inclusion 0% indicates that we utilize SFT solely on the preferred thoughts, i.e. the baseline TS-SFT. Conversely, an inclusion of 100% signifies our CPO, where the entire dataset includes paired preferred and dispreferred thoughts.
Why is chain level optimization important? As shown in Figure 3(d), we find that increasing the percentage of dispreferred data inclusion consistently improves model performance. This suggests that dispreferred thoughts are beneficial during the optimization process, highlighting the importance of leveraging both preferred and dispreferred thoughts for enhancing the model's reasoning capabilities.
Irput
CPO
Unlike our CPO. an alternative approach to construct preference data using complete reason- ing paths, i.e., using the selected full reasoning paths as preferred and other paths as dispreferred data, as shown Figure 5. This method essen- tially applies DPO at the full-path level, referred to here as Full-path Preference Optimization (FPO). However, FPO encounters a significant issue where the gradients of the longest com- mon prefix (LCP) tokens in paired data cancel
Ouiput
Figure Illustrations of two different ways to con- struct paired preference data: 1) CPO: Paired pref- erence data are constructed at each thought step. 2) FPO: Paired preference data are constructed only at the full path level.

---

out, which we call the LCP gradient cancellation issue. For example, for the preferred path F.
[5, =.4 and, 9, +, =, and the dispreferred path ğr [5, +, 4, =, 9, and, 9, +.2 =; 15).
the gradient will only be computed for the last token where the two sequences diverge.
To mathematically illustrate how LCP gradient cancellation happens in FPO, consider yor [Pim: Wn+1] and ÿt [pis:: fn41], where is the longest common prefix sequence between Du. and it. The gradient of FPO is given by:
V (To C(O) Vo (log To (Bho log T (()) =C(0)- Ve
log Ta (Pim (z) + log To To(w (a,t1la,PMi
To ne (pimlæ)
log Toe To (h4+1|es Pt:)))
where C(O) is a scalar function that does not affect the direction of the gradient and can be absorbed into the learning rate.
We can clearly see that the gradient terms of the common prefix tokens (highlighted with boxes) cancel each other out. This issue also exists in DPO training [63], but FPO suffers more frequently and severely due to the longer ICPP between paired data constructed by tree search. As an empirical evidence, we observe the LCP length accounts for 28% of the total length in the Bamboogle dataset. CPO, on the other hand, constructs preference data at every step in the reasoning chain, allowing optimization of the LLM on steps the reasoning path. This means the common prefix can be optimized at its own step, ensuring that the gradient still exists for the common prefix.
We also compare FPO CPO empirically in Figure 3(b), which further substantiates this observation. Switching to FPO led to a relative performance decrease of 4.6%, even worse than the baseline SFT that does not utilize any information from dispreferred data. This underscores the importance of per-step preference thoughts for CPO.
7 Conclusion
In this work, we introduce novel method called Chain of Preference Optimization (CPO), which leverages the supervision generated by the self-reasoning process (i.e., tree-of-thoughts) to enhance the reasoning ability of LLMs. Experiments on three different LLMs across seven different datasets demonstrate that CPO can consistently improve the performance of the base model by 4.3% on average without sacrificing inference speed. Furthermore, our method also substantially outperforms strong baseline TS-SFT and even achieves comparable performance to the ToT method, which requires approximately 57.5 times more inference time.
In future work, plan to integrate CPO with other reasoning algorithms, such Graph-of- Thoughts [9] and AlphaZero-like tree search [1]]. Furthermore, we intend to explore the potential of using weaker LLM evaluate a stronger one within the CPO framework, facilitating weak-to-strong alignment [64].
Acknowledgement
We thank Cunxiao Du for the insightful derivation of the Longest Common Prefix (LCP) gradient cancellation and for the valuable contributions the discussions throughout this work. We also appreciate Fangkai Jiao's helpful feedback. Additionally, we thank the anonymous reviewers for their constructive comments and suggestions that helped improve this paper.
References
[1] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Shain-of-though prompting elicits reasoning in large language models. NeurlPS, 35:24824-24833 2022.
[2] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2022.
10

---

[3] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Le. et al. Least-to-most prompting enables complex reasoning in large language models. In ICLR, 2022.
[4] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In ICLR, 2023.
[5] Ting Zhang, Ivana Clairine Irsan, Ferdian Thung, and David Lo. Cupid: Leveraging chatgpt for more accurate duplicate bug report detection. arXiv preprint arXiv:2308, 10022, 2023.
[6] Xinyang Hu, Fengzhuo Zhang, Siyu Chen, and Zhuoran Yang. Unveiling the statistical founda- tions of mfain-of-thoughe prompting methods. arXiv preprint arXiv:2408. 14511, 2024.
[7] Fengzhuo Zhang, Boyi Liu, Kaixin Wang, Vincent Tan, Zhuoran Yang. and Zhaoran Wang. Relational reasoning via set transformers: Provable efficiency and applications to marl. Advances in Neural Information Processing Systems, 0835585-358335 2022.
[8] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
[9] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
[10] Fangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F Chen, and Shafiq Joty. Learning blanning-based reasoning by trajectories collection and process reward synthesizing. arXiv preprint arXiv:2402.00658. 2024.
[11] Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero- like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309. 17179, 2023.
[12] Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. To- ward self-improvement of I1ms via imagination, searching, and criticizing. arXiv preprint arXiv:2404. 12253, 2024.
[13] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[14] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[15] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[16] Ruochen Zhao, Xingxuan Li, Shatiq Joty, Chengwei Qin, and Lidong Bing. Verify-and-edit: A knowledge-enhanced chain-of-thought framework. In ACL, pages 5823-5840, Toronto, Canada, July 2023.
[17] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03335, 2022.
[18] Fangkai Jiao, Yangyang Guo, Xuemeng Song, and Liqiang Nie. MERIt: Meta-path guided contrastive learning for logical reasoning. In Findings of ACI. ACL, 2022.
[19] Yao Yao, Zuchao Li, and Hai Zhao. Beyond chain-of-thought, effective graph-of-thought reasoning in large language models. arXiv preprint arXiv:2305.16582, 2023.
11

---

[20] Jieyi Long. Large language model guided tree-of ofttthhgtt.. arXiv preprint arXiv:2305.08291, 2023.
[21] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhit- ing Hu. Reasoning with language model planning with world model. arXiv preprint ar0iv:2305. /4992, 2023.
[22] Thomas H Cormen, Charles Leiserson, Ronald Rivest, and Clifford Stein. Introduction to algorithms. MIT press, 2022.
[23] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurlPs, 11-27730-277444 2022.
[24] Jing Xu, Megan Ung, Mojtaha Komeili, Kushal Arora, Y-Lan Boureau, and Jason Weston. Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback. arXiv preprint arXiv:2208.0322) 2022.
[25] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586 2022.
[26] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01693 2023.
[27] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. 2022.
[28] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020 2024.
[29] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models strong language models. arXiv preprint arXiv:2401.0133 2024.
[30] Fangkai Jiao, Zhiyang Teng, Bosheng Ding, Zhengyuan Liu, Nancy F. Chen, and Shafiq R. Joty. Exploring self-supervised logic-enhanced training for large language models. In NAACL. Association for Computational Linguistics, 2024.
[31] Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. arXiv preprint arXiv:2401.0m 08967, 2024.
[32] Ansong Ni. Jeevana Priya Inala, Chenglong Wang, Oleksandr Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao. Learning math reasoning from self-sampled correct and partially-correct solutions. arXiv preprint 1rXi::2205.143188 2022.
[33] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahem, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08990 2023.
[34] Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, et al. Rest meets react: Self-improvement for multi-step reasoning Ilm agent. arXiv preprint arXiv:23 / 10003, 2023.
[35] Zonghan Yang, Peng Li, Ming Yan, Zhang, Fei Huang, and Yang Liu. React meets actre: When language agents enjoy training data autonomy, 2024.
[36] Chaojie Wang, Yanchen Deng, Zhiyi Lv, Shuicheng Yan, and Bo. Q+: Improving multi-step reasoning for Ilms with deliberative planning. arXiv preprint 241::2406.142888 2024.
12

---

[37] Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang. Qian Liu, Arunesh Sinha, Pradeep Varakan- tham, and Min Lin. Bootstrapping language models with dpo implicit rewards. arXiv preprint urXiv:2406.09766) 2024.
[38] Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipali, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. Llm2llm: Boosting Ilms with novel iterative data enhancement. arXiv preprint arXiv:2403. 15542220.. 2024.
[39] Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter Cowling, Philipp Rohffshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in games, 4(1):1-43, 2012.
[40] Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning, pages 282-293. Springer, 2006.
[41] Mark HM Winands, Yngvi Björnsson. and Takesh Saito. Monte-carlo tree search solver. In Computers and Games: 6th Internutional Conference, CG 2008, Beijing, China, September 29- 29-October 1, 2008, Proceedings 6, pages 25-36. Springer, 2008.
[42] Guillaume Chaslot, Sander Bakkes, Istvan Szita, and Pieter Spronck. Monte-carlo tree search: A new framework for game ai. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 2008
[43] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-3333 2017.
[44] Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Making ppo even better: Value-guided monte-carlo tree search decoding, arXiv preprint arXiv: 15028, 2023.
[45] Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint arXiv:2311.0425. 2023.
[46] Zichen Liu, Siyi Li, Wee Sun Lee, Shuicheng Yan, and Zhongwen Xu. Efficient offline policy optimization with a learned model. In International Conference on Learning Representations, 2023. URL https: / /arxxvixvvvva/////// wwtg////////////// aat///aab/22.////////////aa////a 05980.
[47] Jessica B Hamrick, Abram L Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Witherspoon, Thomas Anthony, Lars Buesing, Petar Veličković, and Théophane Weber. On the role of planning in model-based deep reinforcement learning. arXiv preprint arXiv:2011.044 2020.
[48] Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang. Haifeng Zhang, and Jun Wang. Token- level direct preference optimization. arXiv preprint arXiv:2404.1 1999,, 2024.
[49] Zixuan Liu, Xiaolin Sun, and Zizhan Zheng. Enhancing Ilm safety via constrained direct preference optimization. arXiv preprint arXiv:2403.02475 2024.
[50] Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce 1lms stee without human annotations. CoRR, abs/2312.08935, 2023.
[51] Xavier Amatriain. Prompt design and engineering: Introduction and advanced methods. arXiv preprint 1.Xi::2401.1442. 2024.
[52] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060. 2020.
[53] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600. 2018.
13

---

[54] James Thome Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 809-819, 2018.
[55] Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. Feverous: Fact extraction and verifica- tion over unstructured and structured information. arXiv preprint arXiv:2106.057000 2021.
[56] Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin c! robust fact verification with contrastive evidence. arXiv preprint urXiv:2103.0854) 2021.
[57] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191 2021.
[58] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09653 2021.
[59] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[60] Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Mare Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https:// http//gittttb/tt com/huggingf ace/ accelerate, 2022.
[61] Jinwei Yao, Kaiqi Chen, Kexun Zhang, Jiaxuan You, Binhang Yuan, 7eke Wang, and Tao Lin. Deft: Flash tree-attention with io-awareness for efficient tree-search-based I1m inference. arXiv preprint 40X1v:2404.00222,, 2024.
[62] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[63] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.1322 13228, 2024.
[64] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong gener- alization: Eliciting strong capabilities with weak supervision. arXiv preprint 122.092312.099390. 2023.
[65] Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non- autoregressive neural machine translation. arXiv preprint 1xXiv:/711.0220 2017.
[66] Cunxiao Du, Zhaopeng Tu, and Jing Jiang. Order-agnostic cross entropy for non-autoregressive machine translation. In International Conference on Machine Learning, pages 2849-2859 PMLR, 2021.
[67] Cunxiao Du Zhaopeng Tu, Longyue Wang, and Jing Jiang. agram-OAXE Phrase-based order-agnostic eross entropy for non-autoregressive machine translation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5035-5045, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.
[68] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286. PMLR, 2023.
[69] Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, et al. Glide with a cape: A low-hassle method to accelerate speculative decoding. In Forty-first International Conference Machine Learning, 2024.
14

---

[70] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024.
[71] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapky: Lim knows what you are looking for before generation. arXiv preprint arXiv:2404.1444 2024.
[72] Cunxiao Du, Hao Zhou, Zhaopeng Tu, and Jing Jiang. Revisiting the markov property for machine translation. In Findings of the Association for Compulational Linguistics: EACL 2024, pages 582-588, 2024.
[73] Cunxiao Du Zhaozheng Chen, Fuli Feng. Lei Zhu, Tian Gan, and Liqiang Nie. Explicit interaction model towards text classification. In Proceedings of the AAAl conference on artificial intelligence, volume 33, pages 6359-6366, 2019.
[74] Xuan Zhang and Wei Gao. Towards Ilm-based fact verification news claims with a hierarchi- cal step-by-step prompting method. arXiv preprimt arXiv:2310.00305, 2023.
[75] Xuan Zhang and Wei Gao. Reinforcement retrieval leveraging fine-grained feedback for fact checking news claims with black-box Ilm. arXiv preprint arXiv:2404. 2024.
[76] Wenxiang Jiao, Wenxuan Wang, J-t Huang, Xing Wang, and Zhaopeng Tu. Is chatgpt good translator? a preliminary study. arxiv. arXiv preprint arXiv:2301.08744 2023.
[77] Ting Zhang, Ivana Clairine Irsan, Ferdian Thung, and David Lo. Revisiting sentiment analysis for software engineering the era of large language models. arXiv preprint 1rXiv:2310.11113 2023.
[78] Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv e-prints, pages arXiv-2404, 2024.
15

---

Societal Impacts and Limitations
Since our CPO does not require any human annotation, it can be directly used. For example, to protect the safety of large models, one can simply provide a constitution, and then fine-tune the LLM to make it more compliant. This also introduces another issue: our method can be adjusted for malicious applications. Our limitation that we still need to generate data through ToT, which is time-consuming process. We aim to accelerate the complex reasoning processes during training data generation by incorporating methods like non aoto-utrrerrssivee generation [65, 66, 67], speculative decoding [68, 69], and KV cache prunning [70, 71, 72]. Additionally, we have only tested this on text language models and have not tried it on vision-language models. Furthermore, the application scope of our method remains restricted to small set of downstream tasks. Expanding its application to diverse tasks, such as text classification [73]. news verification [74, 75], machine translation [76], and software engineering [77], remains an area for future research. Moreover, ethical considerations
must be taken into account, as the potential for misuse could lead to unintended consequences.
A Detailed Experiment Configurations
To maintain reasonable budget, especially given the high computational demand of ToT, we limit each dataset to a maximum of 300 test samples through random sampling. For datasets that contain less than 300 test samples, we instead use all available samples. For training, we randomly select less than 300 instances from each dataset to construct the preference data pairs, without using the ground-truth labels. This is because we observe that more number of training data does not lead to performance improvement as shown Section 5.3. In addition, in our experiments, approximately 200 samples (e.g.. questions in the QA task) on average could generate about 6,531 preference pairs, suggesting that our CPO requires only a small number of samples by design. Constructing preference data is a time-intensive process. The choice of 300 samples represents practical trade-off between efficiency and effectiveness, allowing us to manage resources effectively while still achieving noticeable improvements.
B
Additional Experiments
CPO benefits from iterative learning. Inspired by the iterative improvements achieved in previous research [28, 78], in this section, we explore whether CPO can be further improved by iterative learning. Specifically, we try two distinct iterative training strategies: I. SFT CPO: in iter=0, Start with a base LLM that has not been fine-tuned at all; in iter= SFT the base LLM on the reasoning path selected by ToT (base model); in subsequent iterations (iteration >1), Continue fine-tune the model using the CPO method, based on the chain of preference thoughts constructed by the model in the previous iterations. and 2. CPO only: in iter=0, same as iter=0 in SFT+CPO; in subsequent iterations (iteration >0): Only use the CPO method for training in all iterations, similar to the approach in SFT+CPO after the first iteration.
As shown in Table 3, We find that if use CoT for inference, as the number of iterations increases, the performance of the model gradually improves. In the CPO only setting, the performance improves by 4% after two iterations. However, an intriguing phenomenon is noted: if we use the ToT method for inference on our fine-tuned models, the performance does not consistently rise and sometimes even declines. For instance, in the SFT+CPO setting, after the first round of SFT, the performance with ToT decreased by 2.7%. We hypothesize this may be related to decrease in the diversity of the model's 'ss outputs after fine-tuning, which reduces the search space for ToT, making it challenging to find better reasoning paths. When the performance of CoT and ToT becomes similar, further fine-tuning of the LLM leads to convergence in the SFT+CPO setting and even decline in the CPO only setting.
Comparasion with ReST and self- baseline. The settings of ReST [33] and self- rewarding [28] are different from ours, as discussed in Section These methods rely on either external reward models (ReST) or labeled data (self-rewarding), which makes them not directly comparable to our approach. To ensure a fair comparison with our CPO method, we prompted the LLM itself in the same way as our CPO to serve as the reward model for ReST and as the labeled data
16

---

SFT+CPO
CPO only
Inference Method
iter=0 (Base)
iter=l (SFT)
iter=2
iter=3
iter=0
iter=l (CPO)
iter=2
iter=3
(CPO)
(CPO)
(Base)
(CPO)
(CPO)
CoT
29.6
30.4
30.4
31.2
29.6
32.0
33.6
31.2
ToT
33.6
30.4
31.2
33.6
34.4
33.6
Table 3: Effects of iterative learning on the Bamboogle dataset using the LLaMA2-7B the base model.
annotator for self-rewarding, respectively. As shown in Table 4, the results indicate that, on average, CPO method surpasses both ReST and self- rewardigg under this fair comparison setting.
Model
Bam.
2wiki
Hotpotqa
Fever
Feverous
Vitaminc
Svamp
Average
LLaMA2-7b
Rest
30.4
24.0
22.3
45.5
43.9
51.7
42.3
37.2
SelfR
31.2
25.3
21.0
48.8
44.7
51.3
43.0
37.5
CPO
32.0
29.7
24.0
53.2
49.0
52.7
46.0
40.9
LLaMA2-13b
Rest
48.0
28.3
28.7
46.8
48.0
50.2
44.3
42.0
SelfR
48.0
29.0
30.0
47.8
48.5
51.0
45.3
42.8
CPO
52.0
30.3
30.3
49.2
50.7
54.0
50.0
45.2
Mistral-7h
Rest
43.2
26.7
27.4
59.5
48.3
49.7
63.3
45.4
SelfR
44.0
28.0
28.1
58.2
48.0
50.0
65,0
45.9
CPO
45.6
31.7
29.4
59.9
54.0
53.7
69.3
49.1
Table 4: Performance comparison across different datasets.
More metrics on QA dataset. We include F1 scores for the three QA tasks in Table 5. The results show that the FI performance aligns well with the corresponding accuracy for each task.
Ablation studies across datasets and models. Figure 6, 7, and provide ablations and analysis across various models and datasets. The observed trends remain generally consistent across these different settings.
Bamn (13b)
(13b)
Antentyy (h)
$5.0
$5.11
55.0
17
17.5
43
400
Number pared instanoe ctitied
Nursher unioad
Number unitaed
Figure Effect of the number of instances generating paired thoughts.
Illustrative examples of the reasoning paths preferred by CPO. Table presents examples demonstrating that the paths favored by CPO align more closely with those selected by ToT than by COT, indicating higher reasoning quality in CPO's chosen paths.
17

---

LLaMA2-7b
LLaMA2-13b
Mistral-7b
ToT
CoT
SFT
CPO
ToT
CoT
SFT
CPO
ToT
CoT
SFT
CPO
Bam.
29.5
28.1
29.3
31.8
43.0
40.9
41.4
43.0
42.2
41.9
47.5
2wiki.
26.5
23.6
25.0
26.0
30.7
26.8
27.2
28.3
25.5
24.3
27.3
27.0
hot.
23.4
20.5
22.9
24.7
33.3
31.3
31.3
32.0
29.9
25,9
26.8
28.0
Table 5: Fl scores on QA datasets.
Fever (7b)
Bun, (13b)
Fever (136)
PM)
560-15
PA11
44
880
65.0
47.0
8.5
0.25
0.75
0.25
0.35
Pasonage dfdaaddnna stlioed
diegesfoeed langles
Figure 7: Effect dispreferred thoughts in optimization.
Accuney (%)
Bose Model CPO wi Lowes
w/ Lower CPO w/
Accuracy (%)
Model
52.5
30
Bam. (13b)
Fever (7b)
Fever (13b)
(13b)
Fever (7b)
(13b)
(a) Effect of dispreferred thoughts selection.
(b) Effect of per-step preference supervision.
Figure 8: Component-wise Evaluations and Analysis.
Question 1: When did the author of Annabel Lee enlist in the army?
CoT: Step I, when did Edgar Allan Poe enlist in the army? Edgar Allan Poe enlisted in the army May 27, 1827. Step 2, when did Edgar Allan Poe die? Edgar Allan Poe died on October 7, 1849. Step 3, so the final answer is: October 1849.
ToT: step 1, when did edgar allan poe enlist in the army? edgar allan poe enlisted in 1827.step the army on august 16, 1827 step 2. what year did edgar allan poe enlist the army?
edgar allan poe enlisted in the army 1827 1827.step 3, so the final answer is: 1827.
CPO: step I, when did Edgar Allan poe die? Edgar Allan Poe died October 7, 1849. Step 2, when did Edgar Allan Poe enlist in the army? Edgar Allan Poe enlisted in the army on May 27, 1827. Step 3, so the final answer is: May 27, 1827.
Question 2: What is the nickname of the easternmost U.S. state? CoT: Step 1, who is the scientist at MIT that won the Queen Elizabeth Prize for Engineering in 2013? The scientist at MIT that won the Queen Elizabeth Prize for Engineering in 2013 Daniela Rus's father is Vladimir Rus. Step 3, so the final answer is: Vladimir Rus.
Daniela Rus. Step who is the father of Daniela Rus?
ToT: step what the name the easternmost u.s. state? maine is the easternmost u.s. state.step 2, what the nickname of maine? maine's 's nickname is 'the pine tree state.
CPO: Step 1, what is the name of the casternmost U.S. state? The easternmost U.S. state is Maine. Step 2, what is the nickname of Maine? The nickname of Maine is The Pine Tree State. Step 3, so the final answer is: The Pine Tree State.
Table 6: Illustrative examples of the reasoning paths preferred by CPO. Similar paths between ToT and CPO are in yellow.
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction of the paper clearly outline the main contributions and the scope of the research.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper adequately discusses the limitations of the work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
Theorems, formulas, and Lemmas in the paper are numbered and properly
referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides a clear and comprehensive explanation of the proposed method. Key hapee--arameters critical for reproducing the results are meticulously detailed.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code is provided in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
wttps.cc/ppblccc//eccccccpcppppp gewic//gueee..ee CoodeSummii agionPoiicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experiment setting and details presented in Section 5.1 and A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results are accompanied by significant tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
21

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the computer resources can be found in Section 5.1 and A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / http///eerr....eee ctt.ai/eieeccsuilellliellee
Answer: [Yes]
Justification:
The research conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper adequately discusses the impacts of the work in Section 7.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets used in this paper are properly credited, and the license and terms are respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "sRd9xgmEsr", "reviewer": "Reviewer_VaVd", "summary": "This paper presents Chain-of-preference optimization (CPO) a self-supervised learning extension of Tree of Thought (ToT). Rather than use ToT during test time, which takes exponentially longer than end-to-end sampling, this paper proposes to use ToT at training time to annotate data for DPO fine-tuning and then use the DPO-tuned model end-to-end at test time. Notably, the DPO annotations are performed by the untuned model itself being prompted to label inferences as either useful or not. This differs from other approaches that are trained on only full successful reasoning paths.\n\nFor certain QA and reasoning tasks, the paper presents experimental evidence that CPO yields stronger models than either (1) fine-tuning the same LM on positive examples (TS-SFT) as done in previous papers or (2) running regular chain-of-thought prompting.", "questions": "* What metrics did you use for the QA datasets? many of them use more than exact match, e.g. HotpotQA uses F1.\n* The discussion around ablations in 5.3 is unclear. Are the trends consistent across different models and datasets? \n* It is odd to have only considered up to 200 instances for constructing preference pairs. Why is this experiment limited to such a small maximum? The difference between e.g. 160 and 200 seems much less important than the difference between 200 and 1000 or 1000 and 5000. \n* How many total preference pairs do you end up training on? It would be helpful to include this number in the main body of the paper.\n* How stable is this approach to different prompts for state evaluation? Did you experiment with other prompts?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The approach is an intuitive and straightforward improvement to multi-hop reasoning that that is much faster than tree-based inference procedures.\n2. The approach is appealing as it doesn't rely on correct labels nor external LLMs other than the LLM being fine-tuned on its own preference data.\n3. The methodology is well explained and easily reproducible.", "weaknesses": "1. While the authors run evaluation over several datasets, they only consider 300 questions per dataset which is quite small. A stronger evaluation would use the whole test sets.\n2. Details about the baseline implementation are unclear. The authors refer to both \"TS-SFT\" and \"TS-LLM\" (l.206) which in the original paper refer to different things-- SFT refers to the model fine-tuned on training examples with reasoning traces pulled from both a gold-annotated dataset and/or reasoning traces from the model that led to correct answers. However, TS-LLM in the original paper refers to the result of an iterative refinement process.\n3. The choice of datasets is somewhat odd. E.g. HotpotQA is meant to be evaluated against support documents, which might explain the very low scores. They also compare against approaches (ToT, TS-SFT) that were not evaluated on any of the QA or Fact Verification tasks under consideration.  This paper did _not_ consider the Game of 24 or GSM datasets, which both ToT and TS-SFT did evaluate. \n\nMissing references:\n1. Khalifa et al 2023: [GRACE: Discriminator-Guided Chain-of-Thought Reasoning](https://aclanthology.org/2023.findings-emnlp.1022.pdf)\n2. Li et al 2023: [Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutions](https://openreview.net/pdf?id=4D4TSJE6-K)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and questions. Below, we respond to the comments in ***Weaknesses (W)*** and ***Questions (Q)***.\n\n---\n\n***W1: Evaluation using entire test sets***\n\nThank you for your suggestion. We selected evaluation sets of 300 questions per dataset to manage the high computational demands of evaluating ToT (more than 230,000 A100 GPU hours across seven full test sets and three base models). Sampling a relatively small subset is consistent with common practices in the field; for instance, ToT by Yao et al. [46], Self ask by Press et al. [29], Verify-and-edit by Zhao et al. [50], and Contrastive CoT by Chia et al. [A] also utilize small evaluation sets or sampled subsets for similar reasons. We will release the subset we used to make our results reproducible.\n\nWe also acknowledge that using small evaluation sets could introduce variance in the results. Following your suggestion, we have included evaluations on the entire test sets in $\\\\textrm{\\\\color{blue}Table A}$ of the Rebuttal PDF. Currently, we are only able to provide results for CPO and two baselines (i.e., CoT and TS-SFT), as the computational cost for evaluating ToT is substantial.\n\nOur findings is that the relative improvement of CPO over the baselines is consistent with our previous results. We will incorporate these expanded evaluations in our revised paper.\n\n---\n***W2: Details about the baseline implementation & terminology***\n\nWe apologize for the confusion. Throughout our paper, TS-SFT refers to the model fine-tuned on training examples with reasoning paths discovered by ToT, as described in L176-178. Note that these reasoning paths do not necessarily lead to correct answers, as we do not assume access to ground truth in our setting. The term TS-LLM in L206 was a typo and will be corrected to TS-SFT.\n\nWe will clarify these in the revision and consistently use the term TS-SFT to avoid misunderstandings. Thank you for your attention to detail.\n\n---\n***W3: Choice of datasets and comparison approaches***\n\nWe understand the concerns about our selections and the comparability of our results with established benchmarks.\n\n1. HotpotQA is typically evaluated against wikipedia pages. We selected this dataset because LLMs are generally pre-trained on Wikipedia [B]. This aligns with prior work that tests LLMs on HotpotQA by relying solely on the information encoded in the model's parameters like [C].\n2. Our choice of tasks was driven by the performance of the models using the ToT method, which showed improvements on QA and Fact Verification. \n3. Based on your suggestion, we have added performance for the LLaMa2-7B base model on the full GSM8k test set in $\\\\textrm{\\\\color{blue}Table B}$ of the Rebuttal PDF. As shown in the Table, on LLaMA2-7b-base, CPO enhances performance by 1.5% on GSM8K, compared to CoT. Regarding the Game24, we find ToT can only achieve less than 3% with LLaMA3-8b-base (also observed by [D]). Our method, which solely relies on the inherent capabilities of the LLM for self-improvement without assuming access to human annonated data like [14], faces significant challenges in improving upon these figures. In addition, while ToT was tested on Game24 using GPT-4 in the original paper, GPT-4 does not currently offer an interface that supports DPO finetuning, precluding our ability to apply our methods in that context.\n\n---\n***Q1: Metrics for QA datasets***\n\nWe primarily reported accuracy as our metric (line 171) following [29]. Additionally, we added F1 scores for the three QA tasks in $\\\\textrm{\\\\color{blue}Table D}$ of the Rebuttal PDF. As shown in the table, the performance in terms of F1 scores is consistent with its corresponding accuracy. \n\n---\n***Q2: Ablations in 5.3: Are the trends consistent across different models and datasets?***\n\nFollowing your suggestion, we have included ablations and analysis across different models and datasets in $\\\\textrm{\\\\color{blue}Figure A, B, C(a) and C(b)}$ and  $\\\\textrm{\\\\color{blue}Table C}$ of the Rebuttal PDF. We find the trends are generally consistent across different models and datasets.\n\n---\n***Q3: Limitation of 200 instances for constructing preference pairs***\n\nWe construct preference pairs with up to 200 instances for the following reasons:\n\n1. In our experiments, approximately 200 samples (e.g., questions in the QA task) on average could generate about 6,531 preference pairs, suggesting that our CPO requires only a small number of samples by design. \n2. Constructing preference data is a time-intensive process. The choice of 200 samples represents a practical trade-off between efficiency and effectiveness, allowing us to manage resources effectively while still achieving noticeable improvements.\n\nFollowing your suggestions, we added an analysis of performance trends from 0 to 3,000 samples in $\\\\textrm{\\\\color{blue}Figure C(c)}$ in the Rebuttal PDF. We observed that while performance increases with more samples, the preference data constructed from 200 samples already provides a significant boost in model performance.\n\n---\n***Q4: Total number of preference pairs used in training***\n\nOn average, we trained on 6,531 preference pairs across all three LLMs and seven datasets. We appreciate your suggestion and will ensure to include this specific number in the main body of the paper for clarity and completeness.\n\n---\n***Q5: Stability of approach to different prompts for state evaluation***\n\nIn our prior experiments, we tried different prompting templates for evaluation. We slightly changed the prompts by 1) adding more suggestive information, such as \"a good annonator will classify it as\"; and 2) replacing prompts with synonymous sentences. We find that the reward score returned by the LLM is robust to such kinds of changes. Balancing the performance and the length of different templates, we chose the current prompting template. We will include the discussion in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t2xugVaQTP", "reviewer": "Reviewer_Nj7W", "summary": "The paper presents a novel method to enhance the performance of LLMs in complex problem-solving tasks using Chain of Preference Optimization (CPO). The authors propose fine-tuning LLMs using the search tree constructed by tree-of-thought (ToT), allowing CoT to achieve similar or better results without the heavy tree-searching during inference. The CPO method aligns each step of the CoT reasoning paths with those of ToT, leveraging the inherent preference information in the tree-search process. The authors provide experimental evidence of the effectiveness of CPO, showing significant improvements in LLM performance in tasks such as question answering, fact verification, and arithmetic reasoning.", "questions": "LN35, what is the advantage of CPO comared to existing MCTS methods? And why?\n\nLN117, have you considered online RL that regenerates the chain-of-preferences using the updated LLM?\n\nTable1, I'm wondering how does the method perform on common reasoning tasks like GSM8K and ProofWriter?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel method, Chain of Preference Optimization (CPO), that significantly enhances the CoT reasoning ability of LLMs without increasing the inference load. It offers a convincing comparison with the ToT method, demonstrating that CPO substantially reduces inference time, as evidenced by the decrease in latency from 1749s/ins to 38s/ins on Llama2-7B. Furthermore, the paper shows that CPO not only matches but surpasses the ToT baseline in terms of accuracy across multiple datasets. This indicates that CPO effectively fine-tunes LLMs to generate more optimal reasoning paths, thereby improving their performance in complex problem-solving tasks. The research is significant as it provides a practical solution to the challenge of balancing inference complexity and reasoning quality in LLMs.", "weaknesses": "The paper presents a promising approach to fine-tuning LLMs using CPO. However, there is a significant gap in the evaluation of the potential impacts of this fine-tuning process on other abilities of the pretrained LLMs. This omission could leave readers questioning the broader implications and versatility of the proposed method. Future work should consider a comprehensive evaluation of the fine-tuning process to fully understand its effects on the LLMs' overall performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and questions. Below, we respond to the comments in ***Weaknesses (W)*** and ***Questions (Q)***.\n\n---\n\n***W1: Evaluation of the potential impacts of fine-tuning process on LLM's other abilities***\n\nIn response to your suggestion, we conducted additional experiments using the LLaMA2-7b-base model to assess how fine-tuning on one specific task influences performance on other tasks. The results are presented in the table below:\n|Training|Test|||||||\n|  ----  | ----  | ----  | ----  | ----| ----  | ----| ---- |\n||Bam.|2wiki.|hot.|Fever|Feverous|Vitaminc|SVAMP|\n|-|29.6|26.3|21.0|45.8|44.3|47.3|37.7|\n|Bamboogle|**32.0**|23.3|19.7|45.5|44.3|47.1|41.7|\n|Fever|28.8|25.4|22.3|**53.2**|47.0|49.3|40.3|\n\nIn the table, '-' represents the base model without any fine-tuning. **Bold** indicates testing on the same task as the model was fine-tuned on. \n\nThe results reveal that while fine-tuning on specific tasks often improves performance on the targeted task, it can lead to decreased performance on other tasks. To address this, we are considering strategies such as using a more diverse mixture of data for fine-tuning and incorporating additional regularization techniques. Interestingly, the improved reasoning ability may generalize to other domains (e.g., the improved performance of SVAMP dataset after fine-tuning).\n\nWe will include discussions in the revised version. Thanks for your suggestions.\n\n---\n\n***Q1: LN35, what is the advantage of CPO comared to existing MCTS methods?***\n\nOur CPO utilizes computationally expensive ToT to construct preference data for optimization during the training phase, and can efficiently generate answers through greedy decoding during testing. \nToT is a method that augment the reasoning capabilities of LLMs by using tree search algorithms (i.e., BFS and DFS) to guide multi-step reasoning. While existing methods also explore using MCTS (Monte Carlo Tree Search) to guide LLM reasoning, our choice of ToT offers several advantages:\n\n1. Self-Evaluation: ToT leverages the LLM itself to evaluate its generated reasoning paths, eliminating the need for labeled data to train an additional value function or reward model, which MCTS requires.\n2. Performance: According to our preliminary exploration, MCTS guided LLM decoding does not consistently outperform ToT, which is also observed by [14].\n3. Efficiency: MCTS is generally more resource-intensive than the search algorithm used in ToT (i.e., BFS and DFS), especially in terms of computational resources and time, because it involves a large number of random simulations. Moreover, ToT, with its pruning mechanisms, offers a more efficient tree search process by reducing unnecessary explorations.\n\n---\n\n***Q2: LN117, considering online RL that regenerates the data using the updated LLM.***\n\nWe chose not to employ online RL due to the significant computational overhead it incurs, as each update step involves sampling from the current policy model.\n\nInstead, we opted for an iterative approach as a compromise between online and offline settings. This method reduces computational demands while still allowing for periodic updates from the policy model. Details on our iterative setting can be found in Appendix C of our paper.\n\n---\n\n***Q3: Table1, CPO performance on GSM8K and ProofWriter.***\n\nFollowing your suggestions, we have included our results on GSM8K and ProofWriter in $\\\\textrm{\\\\color{blue}Table B}$ of the Rebuttal PDF. For the ProofWriter dataset, we sampled 300 instances for testing, as justified in our response to ***W1*** of **Reviewer VaVd**. For the GSM8k dataset, we evaluated the models using the full test set. As shown in the Table, CPO enhances performance by 1.5% on GSM8K and 1.1% on ProofWriter compared to CoT, demonstrating its effectiveness on these common reasoning tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7bg2oisnae", "reviewer": "Reviewer_WuBE", "summary": "This paper presents a method called Chain of Preference Optimization (CPO) that fine-tunes large language models (LLMs) using the tree-of-thought (ToT) method to improve the performance of chain-of-thought (CoT) decoding. CPO aligns each step of CoT reasoning paths with those of ToT by leveraging preference information in the tree-search process, resulting in similar or better performance than CoT alone while avoiding the significant inference burden. Experimental results demonstrate its effectiveness.", "questions": "1. Why use DPO instead of other preference optimization algorithms?\n2. Explain why the experiment lacks different self-imporvement baseline algorithms.\n3. There may be a large number of negative samples in the preference dataset constructed by ToT making the dataset unbalanced.  May I ask whether this will have a negative impact on the training stage?", "limitations": "The limitations of the study and the possible negative social impact have been well documented by the authors.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Innovation: The method of constructing preference dataset with ToT has a certain novelty.\n\nWriting quality:  Your paper is well-written, with clear and precise language and a smooth flow of ideas.  The structure is reasonable,  and the logic is sound, making it very enjoyable to read.\n\nExperimental analysis: This paper has made a full experimental demonstration.", "weaknesses": "1.  The experiment lacked a stronger baseline comparison.  In my opinion, the author's method belongs to the self-imporvement methods, so it should be compared with such typical methods, such as the ReST and self-rewarding mentioned by the author in related work.\n\n2.  The ToT approach to dataset construction appears to be computationally expensive.\n\n3.  the author does not seem to provide a task-specific prompt templates (although the general guideline prompt is mentioned in line 141).  This is not necessary, but I think the quality of building a preference dataset can be seriously affected by the prompt template of evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and questions. Below, we respond to the comments in ***Weaknesses (W)*** and ***Questions (Q)***.\n\n---\n\n***W1: Comparision with ReST and self-rewarding baseline***\n\nWe would like to clarify that the settings of ReST and self-rewarding are different from ours, as discussed in L66-L78 of Section 2. These methods rely on either external reward models (ReST) or labeled data (self-rewarding), which makes them not directly comparable to our approach.\n\nHowever, following your suggestion, we have added a comparison with these two baselines in the table below. To ensure a fair comparison with our CPO method, we prompted the LLM itself in the same way as our CPO to serve as the reward model for ReST and as the labeled data annotator for self-rewarding, respectively. The results indicate that, on average, our CPO method surpasses both ReST and self-rewarding under this fair comparison setting.\n\nWe will include the two baselines in the revision.\n\n\n|||Bam.|\t2wiki.|\tHotpotqa|\tFever|\tFeverous|\tVitaminc|\tSvamp|Average|\n|  ----  | ----  | ----  | ----  | ----  | ----  | ----  | ----  | ----  |----  |\n|LLaMA2-7b|\tRest|\t30.4|24.0|22.3|45.5|43.9|51.7|42.3|37.2|\n||\tSelfR|\t31.2|\t25.3|\t21.0|\t48.8|\t44.7|\t51.3|43.0|37.5|\t\n||\tCPO|\t**32.0**|\t**29.7**|\t**24.0**|\t**53.2**|\t**49.0**|\t**52.7**|**46.0**|**40.9**|\t\n|LLaMA2-13b|\tRest|48.0|28.3|28.7|46.8|48.0|50.2|44.3|42.0|\t\n||\tSelfR|48.0|29.0|30.0|47.8|48.5|51.0|45.3|42.8|\n||\tCPO|**52.0**|**30.3**|**30.3**|**49.2**|**50.7**|**54.0**|**50.0**|**45.2**|\t\t\n|Mistral-7b|\tRest|43.2|26.7|27.4|59.5|48.3|49.7|63.3|45.4|\n||\tSelfR|44.0|28.0|28.1|58.2|48.0|50.0|65.0|45.9|\n||\tCPO|**45.6**|**31.7**|**29.4**|**59.9**|**54.0**|**53.7**|**69.3**|**49.1**|\t\n\n\n---\n\n***W2: Dataset construction using ToT is computationally expensive.***\n\nThe computational expense introduced by ToT is limited to the training stage. A key motivation of our approach is to transfer the computational burden from the inference to the training phase. This allows CPO to directly produce answers through greedy decoding during inference, significantly enhancing efficiency.\n\nMoreover, by using only 200 samples to generate preference pairs, we were able to achieve performance improvements efficiently (Please also refer to our response to ***Q3*** of **Reviewer VaVd**).\n\n---\n***W3: Task-specific evaluation prompt templates***\n\nWe apologize for any confusion. To clarify, the prompt template used in our evaluation consists of two parts: (1) the general guidelines, and (2) task-specific demonstration examples (L135-L137).\n\nFor example, here is one of the demonstration examples we used for QA datasets:\n\nQuestion: When did the last king from Britain's House of Hanover die?\nThought: Step 1, when did the last king from Britain's House of Hanover born? \nEvaluation Process:\nThe thought process focuses on the birth date of the last king from Britain's House of Hanover. However, knowing the birth date does not directly help in determining the date of death, which is the actual question. The lifespan of an individual can vary widely and cannot be accurately inferred from their birth date alone. Therefore, this thought process is unlikely to lead to the correct answer without additional information.\nSo the evaluation result is: this thought is impossible to help pariticially or directly answer the question.\nEvaluation Results:\nImpossible\n\nWe will clarify theis and provide task-specific demonstration examples in the revision.\n\n\n---\n\n***Q1: Choice of DPO instead of other preference optimization algorithms.***\n\nOur contribution primarily focuses on the construction of preference data (Please also refer to our response to ***W1*** of **Reviewer ADqw**). We chose to experiment with DPO because it is one of the most typical and widely used preference optimization algorithms. While we acknowledge that DPO is not the sole option available, advanced algorithms like SimPO [F] could potentially yield superior results, which we will leave as future work.\n\n[F] Meng et al. SimPO: Simple Preference Optimization with a Reference-Free Reward, arXiv:2405.14734.\n\n---\n\n***Q2: Lacking self-imporvement baselines.***\n\nPlease see our response to ***W1***.\n\n---\n\n***Q3: Unbalanced preference dataset constructed by ToT.***\n\nIn the constructed preference data, each positive sample is structurally paired with a corresponding negative sample, maintaining a one-to-one mapping at the level of individual pairs, thus achieving balance structurally. However, since ToT selects more paths than it rejects, the overall dataset exhibits an imbalanced distribution with more positive than negative samples. \nDespite this numerical discrepancy, our method oversamples positive instances by design, which could mitigate the potential negative impacts of an unbalanced dataset on the CPO model training. This strategy is also adopted by [G]. Figure 3(a) of our paper further substantiates this hypothesis by exploring various methods for selecting negative samples, which leads to differerent quantities of negative examples. The results in the figure suggests that the number of negative samples has minimal impact on the training outcomes. \n\n[G] Pattnaik et al. Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences, arXiv:2403.07230.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xQmPhIA3yl", "reviewer": "Reviewer_ADqw", "summary": "This paper introduces Chain of Preference Optimization (CPO), a method to enhance mathematical reasoning in large language models by feeding step-level pairs rather than response-level ones into DPO objectives. CPO leverages non-optimal reasoning thoughts from tree-search processes to construct paired preference data at each reasoning step, where both responses share the same content until a certain step and then start varying. Experiments across seven datasets show CPO significantly improves performance compared to Chain-of-Thought approaches, achieving comparable results to Tree-of-Thought methods while being substantially faster during inference. The authors provide a comprehensive analysis of CPO's components and demonstrate its effectiveness in utilizing both preferred and dispreferred thoughts in the reasoning process.", "questions": "See weakness.", "limitations": "See above", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The experiments demonstrate that CPO is effective across different tasks including question answering, fact verification, and arithmetic. \n- This paper conducts an extensive analysis of several factors that can influence the model performance. The insights on scaling effects, data mixture, and dispreferred information can be useful.\n- The paper is well-written and easy to follow.", "weaknesses": "- The major concern is that despite being effective, the method is basically a new use of the DPO algorithm with different inputs (converting response-level pairs to chain-level) and thus it's fundamentally still DPO.\n- In section 6, the authors have explained why chain-level optimization is important. However, according to the explanation, token-level DPO is more natural in locating errors and avoiding LCP gradient cancellation. Then according to your theory, why not directly use token-level optimization, which is exactly what PPO does, but an intermediate chain-level? I suppose there exists an efficiency-effectiveness trade-off in these settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and questions. Below, we respond to the comments in ***Weaknesses (W)*** and ***Questions (Q)***.\n\n---\n\n***W1: CPO is fundamentally still DPO.***\n\n\nThank you for raising this point. We would like to clarify that while our approach adopts the DPO algorithm to fine-tune language models, our core contribution lies in different aspects compared to DPO.\n\nDPO is an algorithm designed to align models (typically LLMs) with a target reward directly through pair preference data, avoiding the need for a surrogate reward model, and is widely used in RLHF for LLMs. The algorithm itself does not specify how to obtain the preference data, which can be sourced from human annotators [26,41,5,40], external reward models [43,1], or the model being fine-tuned itself [14].\n\nOur work falls into a specific methodology for constructing preference data from the model itself. We find that time-consuming inference schemes such as ToT naturally produce thought-level preference data that are helpful for learning more time-efficient inference methods like CoT. This construction, although cost-effective (as it requires no external reward models or human annotation), has not been explored in previous research. Please also refer to our response to ***W1*** of **Reviewer WuBE** for a discussion and comparison with closely related works.\n\nMoreover, the DPO algorithm in our approach is not the only choice. Advanced preference optimization algorithms (such as SimPO [F]) could potentially yield better results, which we will leave as future work. Please also refer to our response to ***Q1*** of **Reviewer WuBE**.\n\n[F] Meng et al. SimPO: Simple Preference Optimization with a Reference-Free Reward, arXiv:2405.14734.\n\n---\n\n***W2: Why not directly use token-level optimization.***\n\nWe opted not to use token-level optimization primarily due to the following reasons:\n\n1. Challenges in Reward Acquisition: CPO relies on constructing preference data from the model’s self-evaluation capabilities. Our preliminary experiments suggested significant difficulties in generating reliable token-level rewards using prompt-based LLM methods, which is crucial for token-level preference data construction.\n2. High Computational Cost: The computational demands and associated costs of token-level ToT are substantially greater than those for thought-level ToT.\n\nConsidering these factors, chain-level optimization presents a balanced approach, mitigating both the computational overhead and the LCP gradient cancellation problem.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Y346q1CXW", "reviewer": "Reviewer_iCyz", "summary": "The paper introduces Chain-of-Preference Optimization (CPO) to improve reasoning in LLMs. CPO utilizes preference data generated during ToT to fine-tune models. This approach improves reasoning without increasing inference complexity.", "questions": "NA", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of CPO is interesting.\n2. CPO achieves enhanced performance without increasing inference complexity.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. While CPO is effective across several tasks, its performance on more complex reasoning benchmarks, such as the MATH benchmark, remains unexplored.\n2. Fine-tuning with reasoning paths generated by ToT may reduce the diversity of generated paths. A systematic investigation into how the generated reasoning paths change before and after fine-tuning is needed.\n3. The effectiveness of CPO is heavily dependent on the quality of reasoning paths generated by ToT. If ToT produces suboptimal paths, the preference data might not be as beneficial.\n4. The paper does not adequately address the applicability of CPO to more advanced models, such as LLaMA3.\n5. While CPO enhances performance, the paper lacks a deep analysis of the interpretability of the optimized reasoning paths. Understanding why certain paths are preferred could provide additional insights and improve trust in the model’s decisions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and questions. Below, we respond to the comments in ***Weaknesses (W)*** and ***Questions (Q)***.\n\n---\n\n***W1: CPO performance on MATH benchmark.***\n\nFollowing your feedback, we included the performance of both CoT and CPO using the LLaMa3-8b-base model in $\\\\textrm{\\\\color{blue}Table B}$ of the Rebuttal PDF. For the MATH benchmark, we tested on a sample of 300 instances, as discussed in our response to  ***W1*** of **Reviewer VaVd**. As indicated in the table, CPO enhances performance by 1.1% on the MATH benchmark relative to CoT, demonstrating its effectiveness on more complex reasoning tasks.\n\n---\n\n***W2: CPO's influnce on the diversity of generated paths.***\n\nIn response to your insightful suggestion, we systematically evaluated the diversity of generated reasoning paths before and after fine-tuning, as measured by distinct-N [H]. The results are summarized in the table below:\n\n||LLaMA2-7b||||\tLLaMA2-13b||||\tMistral-7b||||\n|  ----  | ----  | ----  | ----  | ----| ----  | ----| ---- | ----| ----  | ----| ----  | ----| \n ||distinc-1|distinc-2|distinc-3|distinc-4|distinc-1|distinc-2|distinc-3|distinc-4|distinc-1|distinc-2|distinc-3|distinc-4|\n|Before|0.12|0.408|0.568|0.632|0.035|0.167|0.288|0.366|0.460|0.199|0.329|0.407\n|After|0.039|0.148|0.229|0.287|0.032|0.144|0.272|0.35|0.037|0.134|0.215|0.271|\n\nThe results indicate a decrease in distinct-N values after fine-tuning with CPO, suggesting a reduction in diversity. This phenomenon may stem from our preference optimization algorithm (i.e., DPO), which optimizes the reverse KL divergence and inherently exhibits mode-seeking behavior, thus reducing diversity in generation [I]. We will incorparate this evaluation and analysis in the revision. \n\n[H] Li et al. A diversity-promoting objective function for neural conversation models, NAACL 2016\n\n[I] Wang et al. Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints, arXiv:2309.16240\n\n\n---\n\n***W3: CPO's effectiveness is heavily dependent on ToT's performance.***\n\nWe acknowledge the concerns raised regarding the dependence of CPO on the quality of reasoning paths generated by ToT. While it is true that suboptimal paths from ToT could affect the overall performance, our results indicate that CPO has the capability to occasionally surpass the performance upper bounds of ToT. This is evidenced by the results presented in Table 1 of our paper.\n\nAdditionally, CPO's design is not inherently restricted to using ToT-generated paths. Our framework is adaptable and can integrate with other tree search methods that might yield higher quality reasoning paths, which we leave as our future work.\n\n---\n\n***W4: CPO's performance on LLaMA3.***\n\nThe release of LLaMA3 on April 18, 2024, just 28 days prior to our submission deadline, presented significant challenges in conducting comprehensive experiments across all seven tasks with this new model.\n\nIn response to your feedback, we have included additional results for the LLaMA3-8b-base model’s performance using both CoT and CPO on the ProofWriter and MATH benchmarks in Table B of the Rebuttal PDF. These results demonstrate that CPO enhances performance by 1.1% on both datasets when compared to CoT, affirming its applicability even on more advanced models.\n\n---\n\n***W5: The interpretability of the optimized reasoning paths.***\n\nWe appreciate your emphasis on the importance of interpretability in the reasoning paths optimized by CPO. To address this, we have included an analysis using two specific examples in  $\\\\textrm{\\\\color{blue}Table E}$ of the Rebuttal PDF. These examples illustrate that the paths preferred by CPO closely resemble those selected by ToT rather than CoT, suggesting a relatively higher quality of reasoning in the paths chosen by CPO. A thorough analysis of these patterns and their implications will be provided in the revision to enhance the interpretability and trustworthiness of the model's decision-making process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive feedback, and we have responded to each reviewer individually. We have also uploaded a Rebuttal PDF that includes:

- $\\textrm{\\color{blue}Figure A}$: Effect of the number of instances in generating paired thoughts;
- $\\textrm{\\color{blue}Figure B}$: Effect of dispreferred thoughts in optimization;
- $\\textrm{\\color{blue}Figure C}$: Component-wise Evaluations and Analysis;
- $\\textrm{\\color{blue}Table A}$: Evaluation using entire test sets;
- $\\textrm{\\color{blue}Table B}$: Experiment results: GSM8K on LLaMA2-7B; others on LLaMA3-8B;
- $\\textrm{\\color{blue}Table C}$: Sensitivity of data mixture;
- $\\textrm{\\color{blue}Table D}$: F1 scores on QA datasets;
- $\\textrm{\\color{blue}Table E}$: Illustrative examples of the reasoning paths preferred by CPO.

Meta-review: This study focuses on solving complex reasoning problems using large language models. It introduces the Chain of Preference Optimization (CPO) paradigm to enhance the traditional Chain-of-Thought (CoT) method. The CPO approach uses the Direct Preference Optimization algorithm, a reinforcement learning method, to establish a preference order among paths in the Tree-of-Thought (ToT) and select the most preferred one for query answering. Comparative experiments show the effectiveness of CPO.

All the reviewers’ ratings for this paper are very close, ranging from “borderline accept” to “weak accept”. After the rebuttal, reviewers agree that the chain of preference optimization approach significantly improves the chain-of-thought without increasing the inference load. The experimental results clearly highlight this ability, and even indicate that CPO can sometimes outperform the unpruned ToT baseline. Furthermore, the paper is well-organized with clear notation and a smooth flow of concepts.

Although the paper's strengths outweigh its weaknesses, I highly recommend addressing all reviewers’ concerns and using the conclusions of the fruitful discussion during the rebuttal. Notably, the differences between CPO and DPO, together with the use of CPO instead of MCTS, should be emphasized in the paper. Additionally, the new experimental results and reasoning tasks should be incorporated in the paper or the appendix.
Decision: Accept (poster)