Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs

Authors: xuan  zhang
chao  du
tianyu  pang
qian  liu
wei  gao
min  lin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 18:11:20

Abstract: The recent development of chain-of-thought (CoT) decoding has enabled large language models (LLMs) to generate explicit logical reasoning paths for complex problem-solving. However, research indicates that these paths are not always deliberate and optimal. The tree-of-thought (ToT) method employs tree-searching to extensively explore the reasoning space and find better reasoning paths that CoT decoding might overlook. This deliberation, however, comes at the cost of significantly increased inference complexity. In this work, we demonstrate that fine-tuning LLMs leveraging the search tree constructed by ToT allows CoT to achieve similar or better performance, thereby avoiding the substantial inference burden. This is achieved through \emph{Chain of Preference Optimization} (CPO), where LLMs are fine-tuned to align each step of the CoT reasoning paths with those of ToT using the inherent preference information in the tree-search process. Extensive experimental results show that CPO significantly improves LLM performance in solving a variety of complex problems, including question answering, fact verification, and arithmetic reasoning, demonstrating its effectiveness. 
Our code is available at [https://github.com/sail-sg/CPO](https://github.com/sail-sg/CPO).

Main paper: 
 introduction: 
 Recent advances in large language models (LLMs) have shown that constructing reasoning chains is critical to improving their problem-solving capabilities [1, 2, 3, 4, 5, 6, 7] . A representative method is chain-of-thought (CoT) [1] , which prompts LLMs to generate intermediate reasoning steps, i.e., thoughts, thereby constructing explicit reasoning paths (as depicted in Figure 1(a) ). While straightforward and intuitive, recent research observes that CoT can often overlook optimal reasoning paths and exhibit an unconscious style of answering due to its single-path focus [8, 9] . To foster a more deliberate and conscious reasoning style, Yao et al. [8] propose tree-of-thought (ToT), which generates multiple branching thoughts at each step of the reasoning process and conducts selfevaluation for pruning and planning to search for reasoning paths (as shown in Figure 1(b) ). However, despite improving reasoning quality, ToT significantly increases computational complexity, which limits its practical application. This raises the question: Can the strategic depth of ToT be integrated into CoT to enhance its effectiveness while maintaining efficiency?Existing research has initially provided a positive answer to the above question [10, 11, 12] . A natural strategy is to treat the reasoning path discovered by ToT for each instance as a target for supervision, and then fine-tune LLMs to improve their CoT reasoning abilities [11, 12] . Several methods have been proposed to improve this approach, including using advanced tree-search techniques like Monte where each node illustrates a step in the reasoning process, forming coherent language sequences aimed at solving a problem. The highlighted path indicates the chosen reasoning trajectory. In the CoT method, the LLM generates only one new node at each step, and all generated nodes are used to build the final reasoning path. For ToT, the LLM produces k new nodes at each step, but only the top n-best nodes are kept, with the rest being pruned. In CPO, nodes marked with a trophy represent preferred thoughts, while those marked with numbers are nodes that can be utilized to create preference data. This method uses the search tree structure from ToT to develop paired preference data, subsequently training LLMs to align with these preferences through DPO.Carlo tree-search (MCTS) and employing external reward models [12, 10] for pruning and planning to gather better reasoning paths as supervision. The effectiveness of these approaches is therefore largely dependent on the quality of the best-discovered reasoning path.In this paper, we identify a limitation in these approaches: they overlook the non-optimal reasoning thoughts generated during the tree-search process, which naturally provides additional preference information. Specifically, ToT inherently generates multiple alternative thoughts at each reasoning step, and pruning is performed according to their evaluated qualities. This tree-search process constitutes a preference over all intermediate thought candidates-thoughts appearing in the bestdiscovered reasoning path are preferred over those that do not. Moreover, this could shed even more insights than the final best-discovered reasoning path, as non-optimal reasoning paths (and thus preferences) exist at each step in the tree-search.Inspired by recently developed reinforcement learning from human feedback (RLHF) techniques like direct preference optimization (DPO) [13] , we propose Chain-of-Preference Optimization (CPO) to fully exploit the inherent preference information. Specifically, we construct paired preference thoughts at each reasoning step according to the search tree of ToT and then train LLMs to align with these preferences using the DPO algorithm (as illustrated in Figure 1(c) ). The paired preference thoughts are constructed based on the above intuition: at each reasoning step, we categorize thoughts as preferred or dispreferred based on their inclusion in the final paths chosen by ToT. With such preference data, CPO enables LLMs to generate the path preferred by ToT using CoT decoding at inference time.We conduct extensive experiments to evaluate the effectiveness of CPO. Experiments on seven datasets using LLaMA [14] and Mistral [15] as base models demonstrate that CPO is highly effective in teaching LLMs the preferred thoughts of ToT at each reasoning step, leading to an average accuracy improvement of up to 4.3% compared to the base models. Additionally, the experiments reveal that CPO can achieve comparable or even superior performance to the ToT method, which on average requires more than 50 times longer for inference. 
related work: 
 Reasoning with LLMs. LLMs have been shown to perform better when prompted to engage in multi-step reasoning [1, 2, 3] . Many studies have focused on improving the generated reasoning paths by post-editing [16] or accessing external knowledge [3, 17] . A distinct approach, more relevant to our interests, transforms the linear reasoning structure into a non-linear format such as a tree or graph [18, 19, 20, 8, 9, 21] , which combines thought evaluation with search algorithms like depth-first search (DFS) [22] . Different from our proposed CPO, these methods require searching during inference, which significantly increases latency.LLM self-improving. Reinforcement learning (RL) has increasingly been applied to LLMs by treating them as RL agents for alignment with human feedback [23, 24, 25, 26] . Recent advances demonstrate the potential of using LLMs for self-generating data to augment fine-tuning processes [27, 28, 29, 30, 31, 32] . For instance, reinforced self-training methods [33, 34, 35, 36, 37] introduce mechanisms to curate new high-quality examples and iteratively enrich the training dataset for enhancing model performance. Nevertheless, these methods typically rely on either an external reward model [35, 34] or labeled data [33] . In contrast, approaches like self-rewarding [29, 38] utilize LLMs themselves to evaluate the generated content, aligning more closely with our method. However, these strategies still require initial seed data [29, 38] , necessitating human annotation. Our work differs from previous methods as it does not rely on any ground-truth data, allowing LLMs to self-learn from their own feedback. Additionally, our approach constructs feedback in a chain fashion, focusing on reasoning steps, an aspect overlooked by prior works.Monte Carlo tree-search for LLMs. Monte Carlo tree-search (MCTS) is a robust algorithm for navigating complex decision-making environments, commonly employed in strategic board games such as AlphaGo [39, 40, 41, 42, 43] . MCTS methodically constructs a search tree, balancing exploration and exploitation, simulates various outcomes, and updates utility estimates based on these simulations. Recent studies have shown that MCTS can enhance the decoding process in LLMs [11, 44, 45, 21, 12] . However, the primary challenge with MCTS is the high latency during inference, particularly in difficult reasoning tasks [46, 47] . While some approaches have attempted to optimize LLMs by leveraging reasoning paths identified through MCTS [11, 12] , these methods still rely on labeled data and require separate policy and value models to explore and evaluate potential moves at the tree's leaves. In contrast, our CPO approach eliminates the need for human annotations and simplifies the tuning of LLMs without the necessity for additional models. 
background: 
 In this section, we formalize our notation and provide a brief overview of key prior knowledge for our method. We denote language sequences by lowercase letters, e.g., x, y, z, to represent a sequence of tokens. The output distribution of an LLM parameterized by θ is denoted by π θ .Chain-of-thought (CoT) [1] is a method that prompts LLMs to generate a chain of reasoning steps before the final answer, as shown in Figure 1 . It introduces a series of intermediate thoughts, denoted as z 1 , • • • , z n , that link an input x to an output y, where n is the total number of reasoning steps. For instance, if x is a combination of demonstration examples and the input question and y is the final answer, each intermediate thought z i forms a coherent language sequence representing a part of the overall reasoning path toward the final answer. The demonstration examples consist of a set of CoT demonstrations, which serve as exemplars in the prompting process. The intermediate reasoning thoughts are sequentially sampled from the distributionz i ∼ π θ (•|x, z 1 , • • • , z i-1 ) and the output is then derived from y ∼ π θ (•|x, z 1 , • • • , z n ).Tree-of-thought (ToT) [8] enables LLMs to explore multiple reasoning paths before answering a given question, as illustrated in Figure 1 . This approach models the LLM reasoning task as a search over a tree, where each node represents a thought step in the reasoning path. ToT comprises two main components, both implemented through prompting LLMs: 1) the thought generator and 2) the state evaluator. The thought generator constructs several new thoughts for the next step based on the current state. Subsequently, the state evaluator generates scores for each new thought and selects the n-best thoughts for further search. The final result is determined by the search algorithm (e.g., BFS or DFS) applied over the selected thoughts until the reasoning process reaches a conclusion.Direct preference optimization (DPO) is a method for directly optimizing an LLM to align with preference data [13] , e.g., human feedback [13, 48, 49] . More specifically, RLHF traditionally frames the application of human feedback to enhance the performance of an LLM within the context of an RL problem. However, DPO reformulates the reward modeling and RL fine-tuning phases in RLHF to a single optimization problem. The objective function of DPO aims to maximize the ratio of probabilities for the preferred responses and optimize the LLM to imitate human preferences.Given the generations (ŷ 1 , ŷ2 ) ∼ π(ŷ|x) conditioned on input x, these pairs are evaluated and ranked according to specific criteria. Preference data is then constructed from these ranked pairs, denoted by ŷw ≻ ŷl |x, where ŷw and ŷl denote the preferred (winning) and dispreferred (losing) completions between ŷ1 and ŷ2 , respectively. The DPO objective is formulated as follows:EQUATION)where σ is the logistic function, the hyperparameter β regulates the penalty imposed for the deviations from the base reference model π ref .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The code is provided in the supplemental material. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: The information on the computer resources can be found in Section 5.1 and A.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
our method: chain of preference optimization: 
 Unlike previous methods that train LLMs to learn the complete reasoning paths [10, 50, 11, 12] , our approach leverages the preferences over thoughts generated at each reasoning step, which are often discarded in prior works. Our key insight is that non-optimal thoughts generated during the tree-search process in ToT provide valuable preference information that can enhance LLM's reasoning ability. A major advantage of our method is that it utilizes this supervision only during training, thereby avoiding high inference latency. Our approach consists of two components: synthesizing the chain of preference thoughts (i.e., the preference thoughts in a chain fashion) and training with the CPO objective.Our procedure for synthesizing and collecting preference thought pairs closely follows the inference process of ToT [8] . An overview of our method is shown in Figure 2 . Specifically, the detailed process is divided into three parts: 1) thought generation, which generates multiple thoughts for each reasoning step; 2) state evaluation, which evaluates each thought; and 3) search and collection, which finalizes the preference thoughts.Thought generation. Given a state si-1 = [x, z 1 , • • • , z i-1] representing a partial solution with the input x and the sequence of thoughts [z 1 , • • • , z i-1 ] so far, we sample k thoughts for the next reasoning step:EQUATIONConditioned on the initial input x, which contains the demonstration examples and the question to be answered, and the previous thoughts z 1 , z 2 , • • • , z i-1 , the LLM generates multiple thoughts for the next reasoning step. Specifically, it follows the format of demonstrations, starting with the prefix "Step i," and samples k thoughts {z(j) i } k j=1. We control the model to pause at the end of z (j) i by setting the generation of the string "Step i + 1," as the stop criteria. 2 As a result, we obtain k new states s(j) i = [x, z 1 , • • • , z i-1 , z (j) i ] for j = 1, • • • , k. State evaluation. Given different states {s (j) i } k j=1, we utilize the LLM to reason about the states and evaluate their progress toward solving the problem, eliminating the need for an external reward model or human annotations. To evaluate state s (j) i , the input to the LLM includes specific demonstration examples for the evaluation process, the input question x, and all the thoughts in the state (i.e.,[z 1 , • • • , z i-1 , z (j) i ]). The LLM follows the format of demonstrations to generate a verbal justification first, followed by a classification result from two classes: likely and impossible. The classification results are then used to assign a score, with likely = 10 and impossible = 1.The prompt template used in our evaluation consists of two parts: (1) the general guidelines, and (2) task-specific demonstration examples. To minimize the effects of randomness and bias, we shuffle the order of demonstration examples [51] and repeatedly sample the generated justification and evaluation results. We then calculate the average score for the state s (j) i . The general guideline prompt for the evaluation is as follows: Evaluate whether the thought helps in partially or directly answering the original question (likely/impossible).Search and collection. We use BFS with pruning as the search algorithm to select the reasoning paths. After evaluation, we retain the n-best thoughts with the highest evaluation scores and proceed to the next step of generation. When the LLM generates a thought containing "so the final answer is:", the search algorithm concludes and returns the selected paths.As shown in the right part of Figure 2 , after finalizing the reasoning paths, the thoughts within the selected paths are marked as preferred (i.e., winning) thoughts. For each preferred thought at the i-th step z w i , we construct corresponding dispreferred (i.e., losing) thoughts. First, we identify the parent state s w i-1 , which includes all the previous thoughts leading to z w i . Each child thought of s w i-1 that is not included in the selected path is chosen as a dispreferred thought z l i compared to z w i . This process results in the preference pair (z w i , z l i ) for the state s w i-1 . We highlight that the constructed dataset D includes preference data at every step of the reasoning chain. This per-step paired preference supervision is usually overlooked in previous methods [11, 12] .Once we have obtained the chain of preference thoughts D, we can proceed with optimization. For the i-th step, given the previous reasoning thoughts s w i-1 , the probabilities of generating z w i and z l i are denoted as π θ (z w i |x, s w i-1 ) and π θ (z l i |x, s w i-1 ), respectively. To optimize the LLM on this pair of preference thoughts, we can directly substitute it into Equation 1:L i (π θ ; π ref ) = -log σ β log π θ (z w i |x, s w i-1 ) π ref (z w i |x, s w i-1 ) -β log π θ (z l i |x, s w i-1 ) π ref (z l i |x, s w i-1 ).Thus, the objective function for CPO can be formulated as follows:EQUATION 
experiments: 
 In this section, we empirically validate that CPO improves the reasoning ability of the base model, and uncover several insightful findings.Datasets and evaluation metrics. We focus our research on three types of reasoning tasks: Question Answering (QA), Fact Verification, and Arithmetic Reasoning. For QA, we conduct experiments on three widely used datasets: Bamboogle [17] , WikiMultiHopQA [52] , and HotpotQA [53] . For fact verification, we use three datasets: Fever [54] , Feverous [55] , and Vitaminc [56] . For arithmetic reasoning, we test on the SVAMP dataset [57] . Our choice of tasks was driven by the performance of the models using the ToT method, which showed improvements in QA, fact verification, and arithmetic reasoning. We use 4-shot prompting for each dataset, with CoT demonstrations manually constructed by previous works [1, 17, 2] . Detailed experimental configurations can be found in Appendix A. For evaluation metrics, we report the accuracy and the average latency of generating the answer per instance. More metrics can be found in Appendix B.Baselines. To validate the effectiveness of our proposed CPO, we consider the following baselines: 1) CoT [1] , which prompts the LLM to generate a series of reasoning steps before producing the final answer. In our experiments, we use CoT with greedy decoding to assess the model's reasoning capabilities without any tuning. 2) ToT [8] , which requires the LLM to explore multiple reasoning paths via tree search before generating the final answer. We use ToT to select reasoning paths and construct datasets to improve LLM's reasoning ability in the following TS-SFT baseline and our CPO method. 3) TS-SFT [11] , which finds reasoning paths through tree search (i.e., ToT in our implementation) and then uses these paths during the supervised fine-tuning (SFT) process (referred to as SFT in Section 5.3 and 6).Implementation Details Our experiments are based on widely used LLMs, specifically LLaMA2-7B/13B [14] and Mistral-7B [15] . For efficient fine-tuning, we use Low-Rank Adaptation (LoRA) adapters [58] . In all experiments, we set the regularization controller β to 0.1, generate 10 new thoughts for each state, and retain the top 5 thoughts after pruning at each step of reasoning. The temperature is set to 0.9 for SVAMP and 0.4 for the other datasets. The learning rates for DPO and SFT are 5e-6 and 1e-5, respectively. We use a batch size (with accumulation) of 32 and optimize the LLM with AdamW [59] . For LoRA, the rank is set to 8, and α is set to 16. All experiments are conducted on NVIDIA A100 GPUs. The latency reported in Table 1 is based on a single NVIDIA A100 40GB. Both training and inference are performed using the Accelerate [60] backend. We train the LLMs for 4 epochs with early stopping based on the performance on a randomly sampled validation set. To mitigate the influence of randomness, all experiments are repeated three times with different random seeds, and the average results are reported.Table 1 summarizes the performance across various reasoning tasks. We have the following findings:CPO improves LLM's reasoning ability. As shown in Table 1 , CPO enhances the reasoning ability of the base LLM, achieving an average improvement of 4.3% and a maximum improvement of 9.7% across all tasks and LLMs compared to the CoT approach. This indicates that CPO effectively improves the LLM's reasoning capabilities. Notably, CPO achieves these improvements without requiring additional human-annotated data, which is particularly beneficial in resource-constrained settings.CPO has a lower latency than ToT but comparable performance. Although ToT consistently improves performance over CoT, it incurs high latency due to the need to generate and evaluate multiple thoughts at each reasoning step during inference. This process produces numerous tokens, resulting in significant computational and memory overhead [61] . In contrast, CPO shifts this computational burden to the training phase, maintaining the low latency of CoT (i.e., 57.5× faster than ToT on average) during inference while providing comparable or superior performance. This demonstrates that our CPO can deliver enhanced reasoning capabilities without compromising efficiency.CPO surpasses TS-SFT on average. Despite both CPO and TS-SFT using ToT to generate training data (where our implementation of ToT remains consistent), CPO exhibits an average improvement of 2.7% and reaches a maximum increase of 10.3%. A key factor behind this performance is the CPO's ability to fully utilize ToT's reasoning process. Specifically, CPO effectively leverages both selected and unselected thoughts at each reasoning step, whereas TS-SFT only uses information from the selected paths, offering CPO with a clear advantage. A detailed discussion of the effectiveness of CPO is presented in Section 5.3.Accuracy (%) The results in Figure 3 (a) show that the performance differences among these strategies are minimal. This suggests that the distinction between preferred and dispreferred thoughts is better determined in the selected reasoning path rather than intermediate evaluation scores. To obtain a greater number of preferred thoughts for each instance to create paired preference thoughts, we chose the CPO w/ All strategy.Effect of numbers of training data. To assess the impact of the number of training data used in optimization, we conduct an ablation analysis by varying the number of instances (e.g., questions in the QA task) used to generate paired preference thoughts, ranging from 0 to 200. As illustrated in Figure 3 (c), we observe that with an increase in the number of instances, the model's performance initially declines and then rises. Specifically, when trained with data generated from less than 80 instances, the model's performance is even worse than without any training, likely due to overfitting [62] , which leads to performance degradation. However, as the number increases to 120, the model's performance consistently improves. Optimizing with paired thoughts from 120 instances, the model's performance surpasses that of the base model. When the number exceeds 120, the model's performance converges, indicating a balance of data for training.Sensitivity of data mixture. We explore the performance of the CPO method across diverse data settings to assess its adaptability and learning efficiency from various data types. As shown in Table 2 , we specifically examine three different data configurations: 1) single task data, 2) uniform QA data, and 3) mixed-type data. Our findings indicate that CPO demonstrates performance improvements of 3.2% in both settings 2 and 3, suggesting its robust ability to harness diverse data sources to enhance learning outcomes. In contrast, the SFT method exhibits comparable performance across these settings, indicating a different sensitivity to data diversity. It is worth noting that, to ensure fairness, although we find that mixed data leads to better performance, the experiments in Table 1 are conducted using individual datasets for training, consistent with the baselines.6 Analysis Do we need dispreferred information? We explore the impact of dispreferred thoughts on model performance by gradually incorporating these thoughts into the training data. Initially, we introduce dispreferred thoughts for their corresponding preferred counterparts and apply CPO to this segment of the data. For preferred thoughts without dispreferred counterparts, we implement SFT on these data. Consequently, the percentage of dispreferred thoughts incorporated can also be viewed as the proportion of data processed using CPO. We adjust the inclusion percentage of dispreferred thoughts from 0% to 100%. An inclusion of 0% indicates that we utilize SFT solely on the preferred thoughts, i.e., the baseline TS-SFT. Conversely, an inclusion of 100% signifies our CPO, where the entire dataset includes paired preferred and dispreferred thoughts. ① ① ① Input Output ② ② ② ③ ③ ③ 🏆 🏆 🏆 CPO ≻ ① 🏆 ① ≻ ① 🏆 ① ≻ ② 🏆 ② ≻ ② 🏆 ② FPO ≻ ≻ ① 🏆 ② 🏆 ③ 🏆 ① 🏆 ② 🏆 ③ ① 🏆 ② 🏆 ③ ① 🏆 ② 🏆 ③ 🏆 ≻ ③ 🏆 ③ ≻ ③ 🏆 ③As shown in Figure 3 (d), we find that increasing the percentage of dispreferred data inclusion consistently improves model performance. This suggests that dispreferred thoughts are beneficial during the optimization process, highlighting the importance of leveraging both preferred and dispreferred thoughts for enhancing the model's reasoning capabilities.Unlike our CPO, an alternative approach is to construct preference data using complete reasoning paths, i.e., using the selected full reasoning paths as preferred and other paths as dispreferred data, as shown in Figure 5 . This method essentially applies DPO at the full-path level, referred to here as Full-path Preference Optimization (FPO). However, FPO encounters a significant issue where the gradients of the longest common prefix (LCP) tokens in paired data cancel out, which we call the LCP gradient cancellation issue. For example, for the preferred path ŷw = [5, +, 4, =, 9, and, 9, +, 2, =, 11] and the dispreferred path ŷl = [5, +, 4, =, 9, and, 9, +, 2, =, 15], the gradient will only be computed for the last token where the two sequences diverge.To mathematically illustrate how LCP gradient cancellation happens in FPO, consider ŷw = [p 1:n , w n+1 ] and ŷl = [p 1:n , l n+1 ], where p is the longest common prefix sequence between ŷw and ŷl . The gradient of FPO is given by:∇ θ L FPO (π θ ; π ref ) = C(θ) • ∇ θ (log π θ (ŷ w |x) -log π θ (ŷ l |x)) =C(θ) • ∇ θ log π θ (p 1:n |x) + log π θ (w n+1 |x, p 1:n ) -log π θ (p 1:n |x) -log π θ (l n+1 |x, p 1:n ) ,where C(θ) is a scalar function that does not affect the direction of the gradient and can be absorbed into the learning rate.We can clearly see that the gradient terms of the common prefix tokens (highlighted with boxes) cancel each other out. This issue also exists in DPO training [63] , but FPO suffers more frequently and severely due to the longer LCP between paired data constructed by tree search. As an empirical evidence, we observe the LCP length accounts for 28% of the total length in the Bamboogle dataset. CPO, on the other hand, constructs preference data at every step in the reasoning chain, allowing optimization of the LLM on all steps in the reasoning path. This means the common prefix can be optimized at its own step, ensuring that the gradient still exists for the common prefix.We also compare FPO to CPO empirically in Figure 3 (b), which further substantiates this observation. Switching to FPO led to a relative performance decrease of 4.6%, even worse than the baseline SFT that does not utilize any information from dispreferred data. This underscores the importance of per-step preference thoughts for CPO. 
conclusion: 
 In this work, we introduce a novel method called Chain of Preference Optimization (CPO), which leverages the supervision generated by the self-reasoning process (i.e., tree-of-thoughts) to enhance the reasoning ability of LLMs. Experiments on three different LLMs across seven different datasets demonstrate that CPO can consistently improve the performance of the base model by 4.3% on average without sacrificing inference speed. Furthermore, our method also substantially outperforms the strong baseline TS-SFT and even achieves comparable performance to the ToT method, which requires approximately 57.5 times more inference time.In future work, we plan to integrate CPO with other reasoning algorithms, such as Graph-of-Thoughts [9] and AlphaZero-like tree search [11] . Furthermore, we intend to explore the potential of using a weaker LLM to evaluate a stronger one within the CPO framework, facilitating weak-to-strong alignment [64] . 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper adequately discusses the impacts of the work in Section 7.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The assets used in this paper are properly credited, and the license and terms are respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Since our CPO does not require any human annotation, it can be directly used. For example, to protect the safety of large models, one can simply provide a constitution, and then fine-tune the LLM to make it more compliant. This also introduces another issue: our method can be adjusted for malicious applications. Our limitation is that we still need to generate data through ToT, which is a time-consuming process. We aim to accelerate the complex reasoning processes during training data generation by incorporating methods like non-autoregressive generation [65, 66, 67] , speculative decoding [68, 69] , and KV cache prunning [70, 71, 72] . Additionally, we have only tested this on text language models and have not tried it on vision-language models. Furthermore, the application scope of our method remains restricted to a small set of downstream tasks. Expanding its application to diverse tasks, such as text classification [73] , news verification [74, 75] , machine translation [76] , and software engineering [77] , remains an area for future research. Moreover, ethical considerations must be taken into account, as the potential for misuse could lead to unintended consequences.To maintain a reasonable budget, especially given the high computational demand of ToT, we limit each dataset to a maximum of 300 test samples through random sampling. For datasets that contain less than 300 test samples, we instead use all available samples. For training, we randomly select less than 300 instances from each dataset to construct the preference data pairs, without using the ground-truth labels. This is because we observe that more number of training data does not lead to performance improvement as shown in Section 5.3. In addition, in our experiments, approximately 200 samples (e.g., questions in the QA task) on average could generate about 6,531 preference pairs, suggesting that our CPO requires only a small number of samples by design. Constructing preference data is a time-intensive process. The choice of 300 samples represents a practical trade-off between efficiency and effectiveness, allowing us to manage resources effectively while still achieving noticeable improvements.CPO benefits from iterative learning. Inspired by the iterative improvements achieved in previous research [28, 78] , in this section, we explore whether CPO can be further improved by iterative learning. Specifically, we try two distinct iterative training strategies: 1. SFT+CPO: in iter=0, Start with a base LLM that has not been fine-tuned at all; in iter=1, SFT the base LLM on the reasoning path selected by ToT (base model); in subsequent iterations (iteration >1), Continue to fine-tune the model using the CPO method, based on the chain of preference thoughts constructed by the model in the previous iterations. and 2. CPO only: in iter=0, same as iter=0 in SFT+CPO; in subsequent iterations (iteration >0): Only use the CPO method for training in all iterations, similar to the approach in SFT+CPO after the first iteration.As shown in Table 3 , We find that if use CoT for inference, as the number of iterations increases, the performance of the model gradually improves. In the CPO only setting, the performance improves by 4% after two iterations. However, an intriguing phenomenon is noted: if we use the ToT method for inference on our fine-tuned models, the performance does not consistently rise and sometimes even declines. For instance, in the SFT+CPO setting, after the first round of SFT, the performance with ToT decreased by 2.7%. We hypothesize this may be related to a decrease in the diversity of the model's outputs after fine-tuning, which reduces the search space for ToT, making it challenging to find better reasoning paths. When the performance of CoT and ToT becomes similar, further fine-tuning of the LLM leads to convergence in the SFT+CPO setting and even a decline in the CPO only setting.Comparasion with ReST and self-rewarding baseline. The settings of ReST [33] and selfrewarding [28] are different from ours, as discussed in Section 2. These methods rely on either external reward models (ReST) or labeled data (self-rewarding), which makes them not directly comparable to our approach. To ensure a fair comparison with our CPO method, we prompted the LLM itself in the same way as our CPO to serve as the reward model for ReST and as the labeled data Inference Method 3 : Effects of iterative learning on the Bamboogle dataset using the LLaMA2-7B as the base model.EQUATIONannotator for self-rewarding, respectively. As shown in Table 4 , the results indicate that, on average, our CPO method surpasses both ReST and self-rewarding under this fair comparison setting.Model Bam. 2wiki Hotpotqa Fever Feverous Vitaminc Svamp Average More metrics on QA dataset. We include F1 scores for the three QA tasks in Table 5 . The results show that the F1 performance aligns well with the corresponding accuracy for each task.Ablation studies across datasets and models. Illustrative examples of the reasoning paths preferred by CPO. Table 6 presents examples demonstrating that the paths favored by CPO align more closely with those selected by ToT than by CoT, indicating a higher reasoning quality in CPO's chosen paths.Mistral-7bToT CoT SFT CPO ToT CoT SFT CPO ToT CoT SFT CPO Bam.29. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: The research conform the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).The "stop criteria" is used to control when generation should stop, which is implemented via a function input in Hugging Face's Transformers Library. 


Decision: This study focuses on solving complex reasoning problems using large language models. It introduces the Chain of Preference Optimization (CPO) paradigm to enhance the traditional Chain-of-Thought (CoT) method. The CPO approach uses the Direct Preference Optimization algorithm, a reinforcement learning method, to establish a preference order among paths in the Tree-of-Thought (ToT) and select the most preferred one for query answering. Comparative experiments show the effectiveness of CPO.

All the reviewers’ ratings for this paper are very close, ranging from “borderline accept” to “weak accept”. After the rebuttal, reviewers agree that the chain of preference optimization approach significantly improves the chain-of-thought without increasing the inference load. The experimental results clearly highlight this ability, and even indicate that CPO can sometimes outperform the unpruned ToT baseline. Furthermore, the paper is well-organized with clear notation and a smooth flow of concepts.

Although the paper's strengths outweigh its weaknesses, I highly recommend addressing all reviewers’ concerns and using the conclusions of the fruitful discussion during the rebuttal. Notably, the differences between CPO and DPO, together with the use of CPO instead of MCTS, should be emphasized in the paper. Additionally, the new experimental results and reasoning tasks should be incorporated in the paper or the appendix.

Reviews: review: paper_summary: paper_summary: This paper presents Chain-of-preference optimization (CPO) a self-supervised learning extension of Tree of Thought (ToT). Rather than use ToT during test time, which takes exponentially longer than end-to-end sampling, this paper proposes to use ToT at training time to annotate data for DPO fine-tuning and then use the DPO-tuned model end-to-end at test time. Notably, the DPO annotations are performed by the untuned model itself being prompted to label inferences as either useful or not. This differs from other approaches that are trained on only full successful reasoning paths.

For certain QA and reasoning tasks, the paper presents experimental evidence that CPO yields stronger models than either (1) fine-tuning the same LM on positive examples (TS-SFT) as done in previous papers or (2) running regular chain-of-thought prompting.
strength_weakness: strength_weakness: 1. While the authors run evaluation over several datasets, they only consider 300 questions per dataset which is quite small. A stronger evaluation would use the whole test sets.
2. Details about the baseline implementation are unclear. The authors refer to both "TS-SFT" and "TS-LLM" (l.206) which in the original paper refer to different things-- SFT refers to the model fine-tuned on training examples with reasoning traces pulled from both a gold-annotated dataset and/or reasoning traces from the model that led to correct answers. However, TS-LLM in the original paper refers to the result of an iterative refinement process.
3. The choice of datasets is somewhat odd. E.g. HotpotQA is meant to be evaluated against support documents, which might explain the very low scores. They also compare against approaches (ToT, TS-SFT) that were not evaluated on any of the QA or Fact Verification tasks under consideration.  This paper did _not_ consider the Game of 24 or GSM datasets, which both ToT and TS-SFT did evaluate. 

Missing references:
1. Khalifa et al 2023: [GRACE: Discriminator-Guided Chain-of-Thought Reasoning](https://aclanthology.org/2023.findings-emnlp.1022.pdf)
2. Li et al 2023: [Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutions](https://openreview.net/pdf?id=4D4TSJE6-K)
questions: questions: * What metrics did you use for the QA datasets? many of them use more than exact match, e.g. HotpotQA uses F1.
* The discussion around ablations in 5.3 is unclear. Are the trends consistent across different models and datasets? 
* It is odd to have only considered up to 200 instances for constructing preference pairs. Why is this experiment limited to such a small maximum? The difference between e.g. 160 and 200 seems much less important than the difference between 200 and 1000 or 1000 and 5000. 
* How many total preference pairs do you end up training on? It would be helpful to include this number in the main body of the paper.
* How stable is this approach to different prompts for state evaluation? Did you experiment with other prompts?
limitations: limitations: yes
paper_summary: paper_summary: The paper presents a novel method to enhance the performance of LLMs in complex problem-solving tasks using Chain of Preference Optimization (CPO). The authors propose fine-tuning LLMs using the search tree constructed by tree-of-thought (ToT), allowing CoT to achieve similar or better results without the heavy tree-searching during inference. The CPO method aligns each step of the CoT reasoning paths with those of ToT, leveraging the inherent preference information in the tree-search process. The authors provide experimental evidence of the effectiveness of CPO, showing significant improvements in LLM performance in tasks such as question answering, fact verification, and arithmetic reasoning.
strength_weakness: strength_weakness: The paper presents a promising approach to fine-tuning LLMs using CPO. However, there is a significant gap in the evaluation of the potential impacts of this fine-tuning process on other abilities of the pretrained LLMs. This omission could leave readers questioning the broader implications and versatility of the proposed method. Future work should consider a comprehensive evaluation of the fine-tuning process to fully understand its effects on the LLMs' overall performance.
questions: questions: LN35, what is the advantage of CPO comared to existing MCTS methods? And why?

LN117, have you considered online RL that regenerates the chain-of-preferences using the updated LLM?

Table1, I'm wondering how does the method perform on common reasoning tasks like GSM8K and ProofWriter?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper presents a method called Chain of Preference Optimization (CPO) that fine-tunes large language models (LLMs) using the tree-of-thought (ToT) method to improve the performance of chain-of-thought (CoT) decoding. CPO aligns each step of CoT reasoning paths with those of ToT by leveraging preference information in the tree-search process, resulting in similar or better performance than CoT alone while avoiding the significant inference burden. Experimental results demonstrate its effectiveness.
strength_weakness: strength_weakness: 1.  The experiment lacked a stronger baseline comparison.  In my opinion, the author's method belongs to the self-imporvement methods, so it should be compared with such typical methods, such as the ReST and self-rewarding mentioned by the author in related work.

2.  The ToT approach to dataset construction appears to be computationally expensive.

3.  the author does not seem to provide a task-specific prompt templates (although the general guideline prompt is mentioned in line 141).  This is not necessary, but I think the quality of building a preference dataset can be seriously affected by the prompt template of evaluation.
questions: questions: 1. Why use DPO instead of other preference optimization algorithms?
2. Explain why the experiment lacks different self-imporvement baseline algorithms.
3. There may be a large number of negative samples in the preference dataset constructed by ToT making the dataset unbalanced.  May I ask whether this will have a negative impact on the training stage?
limitations: limitations: The limitations of the study and the possible negative social impact have been well documented by the authors.
paper_summary: paper_summary: This paper introduces Chain of Preference Optimization (CPO), a method to enhance mathematical reasoning in large language models by feeding step-level pairs rather than response-level ones into DPO objectives. CPO leverages non-optimal reasoning thoughts from tree-search processes to construct paired preference data at each reasoning step, where both responses share the same content until a certain step and then start varying. Experiments across seven datasets show CPO significantly improves performance compared to Chain-of-Thought approaches, achieving comparable results to Tree-of-Thought methods while being substantially faster during inference. The authors provide a comprehensive analysis of CPO's components and demonstrate its effectiveness in utilizing both preferred and dispreferred thoughts in the reasoning process.
strength_weakness: strength_weakness: - The major concern is that despite being effective, the method is basically a new use of the DPO algorithm with different inputs (converting response-level pairs to chain-level) and thus it's fundamentally still DPO.
- In section 6, the authors have explained why chain-level optimization is important. However, according to the explanation, token-level DPO is more natural in locating errors and avoiding LCP gradient cancellation. Then according to your theory, why not directly use token-level optimization, which is exactly what PPO does, but an intermediate chain-level? I suppose there exists an efficiency-effectiveness trade-off in these settings.
questions: questions: See weakness.
limitations: limitations: See above
paper_summary: paper_summary: The paper introduces Chain-of-Preference Optimization (CPO) to improve reasoning in LLMs. CPO utilizes preference data generated during ToT to fine-tune models. This approach improves reasoning without increasing inference complexity.
strength_weakness: strength_weakness: 1. While CPO is effective across several tasks, its performance on more complex reasoning benchmarks, such as the MATH benchmark, remains unexplored.
2. Fine-tuning with reasoning paths generated by ToT may reduce the diversity of generated paths. A systematic investigation into how the generated reasoning paths change before and after fine-tuning is needed.
3. The effectiveness of CPO is heavily dependent on the quality of reasoning paths generated by ToT. If ToT produces suboptimal paths, the preference data might not be as beneficial.
4. The paper does not adequately address the applicability of CPO to more advanced models, such as LLaMA3.
5. While CPO enhances performance, the paper lacks a deep analysis of the interpretability of the optimized reasoning paths. Understanding why certain paths are preferred could provide additional insights and improve trust in the model’s decisions.
questions: questions: NA
limitations: limitations: Yes

Comments: title: Thank you for your support and raising the score
Thanks for the expanded results
Thank you for your feedback
Thanks for the responses
comment: We appreciate your detailed comments and suggestions. In the revision, we will include the new results and the discussion on the potential impacts of fine-tuning process. Thank you!
Thanks the author for the update on the potential impacts of fine-tuning process, which confirms my concerns. I'd like to keep my current score.
We greatly appreciate your prompt feedback and for raising the score. We will incorporate these additional experiments into the final revision and ensure clarity throughout the paper. Thank you again for your valuable insights and support!
Thank you for including these additional results. They have significantly improved this portion of the submission, so I will raise my score to a 6.
Thank you for your detailed feedback. We appreciate your thorough review and would like to address each of your comments below.

---

>Were the results on the other 4 datasets not positive findings?

The full test sets for Fever, Feverous, and Vitaminc are indeed quite large, with approximately 10,000 instances each for Fever and Feverous, and over 50,000 for Vitaminc. Due to time constraints during the rebuttal period, we initially prioritized reporting performance within the QA domain. However, we have since conducted additional experiments, and we now present the results on all seven datasets:

Model | | Bam | 2wiki | hotpotqa | fever | feverous |	vitaminc | svamp | Avg.|
|  ----  | ----  | ----  | ----  | ----  | ----  | ----  | ----  |----  |----  |
|LLaMa2-7b	|CoT	|29.6|	23.8|	20.7|	38.6|	51.2|	50.5|	37.7|36.0|
|	|TS-SFT|	30.4|	24.1|	22.7|	40.3|	53.0|	53.0	|43.1|38.1|
||CPO|	**32.0**|	**26.1**|	**24.0**|	**45.1**|	**56.2**|**55.2**|	**46.0**|**40.7**|	
|LLaMa2-13b|	CoT|	48.0|	28.4|	27.0|	47.4|	49.9|	50.8|	40.3|41.7|
|	|TS-SFT|	50.8|	29.0|	28.1|	48.0|	48.8|	53.8|	44.6|43.3|
|	|CPO|	**52.0**|	**30.3**|	**28.5**|	**49.7**|	**50.7**|	**59.6**|	**50.0**|**45.8**|
Mistral-7b|	CoT|	41.6|27.5|24.8|53.6|54.0|53.8	|65.3|45.8|
||TS-SFT|	41.6|30.3|25.1	|**56.0**|55.0|57.1|					59.0|46.3|
|	|CPO|	**45.6**|**31.3**|**25.9**|55.7|**60.1**|**60.0**|						**69.3**|**49.7**|

Our findings indicate that the relative improvement of CPO over the baselines is consistent with our previously reported results. We will clarify it and incorporate these expanded evaluations into the revised paper.

---

> Why is Table B missing the other baselines?
 
Due to time constraints during the rebuttal period, we prioritized demonstrating CPO's potential to improve performance on new tasks, including GSM8k, rather than emphasizing its superiority over other methods. However, as per your suggestion, we have now included the experimental results for ToT and TS-SFT on GSM8k. Since some settings need to be clarified before presenting the results, please refer to our response to the following comments for detailed results and conclusions. 

---

>Moreover, others have found (e.g. in the Gemma report that Llama-2-7B matches your 14.6% performance on GSM using just few-shot prompting. This suggests that CPO is not providing any benefits.

The performance of 8-shot prompting on GSM8k using LLaMA-2-7B, as reported in the Gemma report, is comparable to our CPO using 4-shot prompting (see lines 169-170). We extended our evaluation to 8-shot prompting here, consistent with the Gemma report settings:

||ToT|CoT|TS-SFT|CPO|
|  ----  |  ----  | ----  |  ----  |----  |
|GSM8k|16.2|14.2|14.8|15.3|

We found that our COT approach achieves similar performance to that reported in the Gemma report, with CPO further improving the results to 15.3%. 

---

> Are the TS-SFT and ToT worse than CPO (raising questions about statistical significance), or are they stronger than CPO?

As shown in the table provided in our previous response, CPO outperforms TS-SFT, as confirmed by a bootstrap significance test, which yielded a p-value of 0.0011 (p < 0.05), indicating a statistically significant difference. However, ToT is superior to CPO. This result is expected, as ToT serves as the teacher method for our CPO method, guiding its improvements.

---

>Which datasets are the Figure C plots computed over?

For $\\textrm{\\color{blue}Figure C (a)}$ and $\\textrm{\\color{blue}(b)}$, as labeled on the x-axis, the results are reported for the Bamboogle and Fever datasets, respectively. For $\\textrm{\\color{blue}Figure C (c)}$, we present the results on the Bamboogle dataset. We will ensure clarity when we include these figures in the final revision. The reason for reporting results on only one dataset is the computational cost associated with using ToT for dataset construction (e.g., over 1740 A100 GPU hours for LLaMA 2-7B on the Fever dataset). However, we are continuing to conduct this ablation and will include the results in the revised version.

---

We hope these efforts address your concerns. If you have any further feedback, we will do our best to respond.
Thank you to the authors for their careful rebuttal. I still have concerns about the evaluation given the authors have been selective about which results were shown in the rebuttal PDF.

>  we have included evaluations on the entire test sets in Table A

This is greatly appreciated, though this only includes 3 of the 7 considered datasets in the paper. Were the results on the other 4 datasets not positive findings? Running on those datasets is clearly within scope because you included SVAMP and Fever in Table C.

> added performance for the LLaMa2-7B base model on the full GSM8k test set in Table B

Thank you as well for this result. Table B is a good start but these numbers do not constitute a full results that could be added to the paper. Why is it missing the other baselines? I appreciate the improvements by about a point over CoT. However, TS-SFT (and ToT) consistently improve upon CoT. For the other datasets you were outperforming CoT by 3-7 points pretty consistently, but for GSM it is only 1 point-- are the TS-SFT and ToT baselines somewhere inside this 1 point spectrum (raising questions about statistical significance), or are they stronger than CPO?

Moreover, others have found (e.g. in the [Gemma report](https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf) that Llama-2-7B matches your 14.6% performance on GSM using just few-shot prompting. This suggests that CPO is not providing any benefits.

> [Ablations] 

Thank you for these results, they have somewhat addressed my concern. Which datasets are the Figure C plots computed over?
We greatly appreciate your valuable feedback and the score improvement. We will further polish the paper and incorporate the rebuttal discussions into the final revision. Thank you!
Thanks to the author for solving my confusion. The experiments added by the author have well solved my worries, so I raised my score from 4 to 6.
**References**

[A] Chia et al. Contrastive Chain-of-Thought Prompting, arXiv:2407.03600

[B] Shi et al. Detecting Pretraining Data from Large Language Models, ICLR 2024.

[C] Wang et al. Causal-driven Large Language Models with Faithful Reasoning for Knowledge Question Answering, MM2024

[D] Yang et al. Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models, arXiv:2406.04271

[E] Zhou et al. LIMA: Less Is More for Alignment, arXiv:2305.11206