Advancing LLM Reasoning Generalists with Preference Trees

Authors: lifan  yuan
ganqu  cui
hanbin  wang
ning  ding
xingyao  wang
jia  deng
boji  shan
huimin  chen
ruobing  xie
yankai  lin
zhenghao  liu
bowen  zhou
hao  peng
zhiyuan  liu
maosong  sun
modelbest  inc
Venue: ICML_cc_2024_Workshop_AI4MATH
Published: 2024-05-20 19:00:56

Abstract: We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning. Finetuned from Mistral-7B and CodeLlama-70B, Eurus models achieve state-of-the-art results among open-source models on a diverse set of benchmarks covering mathematics, code generation, and logical reasoning problems. Notably, Eurus-70B beats GPT-3.5 Turbo in reasoning through a comprehensive benchmarking across 12 tests covering five tasks, and achieves a 33.3% pass@1 accuracy on LeetCode and 32.6% on TheoremQA, two challenging benchmarks, substantially outperforming existing open-source models by margins more than 13.3%. The strong performance of EURUS can be primarily attributed to ULTRAINTERACT, our newly-curated large-scale, high-quality alignment dataset specifically designed for complex reasoning tasks. ULTRAINTERACT can be used in both supervised fine-tuning and preference learning. For each instruction, it includes a preference tree consisting of (1) reasoning chains with diverse planning strategies in a unified format, (2) multi-turn interaction trajectories with the environment and the critique, and (3) pairwise data to facilitate preference learning. ULTRAINTERACT allows us to conduct an in-depth exploration of preference learning for reasoning tasks. Our investigation reveals that some well-established preference learning algorithms may be less suitable for reasoning tasks compared to their effectiveness in general conversations. Inspired by this, we derive a novel reward modeling objective which, together with ULTRAINTERACT, leads to a strong reward model. All artifacts produced during this research will be made public.

Main paper: 
 introduction: 
 Current alignment techniques have significantly advanced the development of open-source large language models (LLMs) that effectively meet user expectations and align with human values (Touvron et al., 2023; Tunstall et al., 2023) . On complex reasoning, success has been achieved by specializing models for specific capabilities, such as coding (Wei et al., 2023; Guo et al., 2024a; Zheng et al., 2024) and solving math problems (Fu et al., 2023; Yue et al., 2023; Luo et al., 2023a; Toshniwal et al., 2024) . However, these models still fall short, by large margins, of the most advanced proprietary models in their all-around capabilities to tackle a diverse range of challenging problems. We conjecture that this performance gap can be primarily attributed to (1) the lack of high-quality alignment data and (2) the underexploration of preference learning techniques for improving models' complex reasoning capabilities. In this paper, we take strides towards bridging this gap by addressing both factors and developing EURUS.EURUS consists of a suite of LLMs finetuned from Mistral-7B (Jiang et al., 2023a) and CodeLLaMA-70B (Roziere et al., 2023) . Across a diverse set of complex reasoning benchmarks that are mostly out-of-distribution (OOD), EU-RUS achieves state-of-the-art overall performance among all open-source models. In particular, EURUS excels in solving challenging problems that often require sophisticated planning, reasoning, tool integration, and the ability to interact with and learn from the environment and users.As shown in Figure 1 , on university-level STEM questions TheoremQA (Chen et al., 2023) and competition-level coding problems LeetCode Contest (Guo et al., 2024a) , EU-RUS-70B significantly outperforms all open-source models, achieving comparable performance to GPT-3.5 Turbo.EURUS models are trained on ULTRAINTERACT, our newly-curated, large-scale, and high-quality alignment data specifically designed to improve LLMs' reasoning capabilities. ULTRAINTERACT consists of a diverse set of instructions spanning math, coding, and logical reasoning problems from 12 established datasets. For each instruction, ULTRAINTERACT collects a preference tree that includes:(1) Diverse planning strategies in a unified pattern, such as sequential processing (Wei et al., 2022) and tool creation (Qian et al., 2023) , followed by executing step-by-step actions formatted in either text or code, to provide divserse reasoning trajectories.(2) Multi-turn interaction trajectories with the environment and the critique, to improve models' capabilities to learn from feedback and correct previous errors (Wang et al., 2023b) .(3) Paired correct and incorrect actions organized in tree structures, to facilitate preference learning. In total, ULTRAINTERACT contains 86K instructions and 220K action pairs, where each pair consists of an instruction, a correct response, and an incorrect one. Conceptually, ULTRAINTERACT's data resemble imbalanced binary trees as shown in Figure 2 .ULTRAINTERACT can be used in both supervised finetuning and preference learning. Our experiments show that, using ULTRAINTERACT along with established datasets in instruction fine-tuning already achieves strong performance. ULTRAINTERACT further facilitates preference learning for reasoning tasks, improving the performance even further with KTO (Ethayarajh et al., 2024) and NCA (Chen et al., 2024a) . Surprisingly, applied to an instruction finetuned EURUS model, DPO (Rafailov et al., 2023) hurts the performance.Through careful analysis, we provide evidence that the performance in reasoning correlates with the value of rewards of chosen data-a higher final reward often indicates a better reasoning capability. Besides, our investigation suggests that DPO may be less suitable for reasoning tasks than KTO and NCA. Inspired by this fresh finding, we devise a new objective for reward modeling to augment the Bradley-Terry (Wang et al., 2024) and Code-Feedback (Zheng et al., 2024) ; Middle: HH-RLHF (Bai et al., 2022) ; Right: ULTRAINTERACT. Each instruction in ULTRAIN-TERACT is constructed as a preference tree.objective (Bradley & Terry, 1952) , explicitly encouraging training to increase the absolute rewards of chosen solution and decrease those of rejected data. Furthermore, UL-TRAINTERACT leads to our reward model EURUS-RM-7B, which achieves a better correlation with human annotators than all existing models on AutoJ (Li et al., 2023a) and MT-Bench (Zheng et al., 2023) , including GPT-4 (OpenAI, 2023) . EURUS-RM-7B demonstrates especially strong preference modeling performance on reasoning tasks. 
ultrainteract: tree-structured alignment data for reasoning: 
 Solving complex problems often requires the model's capability in planning and reasoning, integrating with tools, and interacting with and learning from both the environment and the users. This is reflected in ULTRAINTERACT's design choices: (1) Its instructions are diverse, challenging, and of a large scale ( §2.1); (2) It provides multi-turn trajectories that solve the input instruction through multiple turns of interaction with and learning from the environment and critique. At each turn, it breaks down the problem into smaller ones ( §2.2). ( 3) ULTRAINTERACT includes pairwise data to facilitate preference learning ( §2.3).Conceptually, ULTRAINTERACT collects a preference tree for each instruction, with the instruction being the root and each action a node (Fig. 2 ). A trajectory is a root-to-leaf path consisting of a sequence of actions. In each preference tree, all nodes of correct actions and all trajectories ending with correct actions can be used for SFT. Paired correct and incorrect nodes or trajectories can be used for preference learning.We target three representative reasoning tasks: math problem-solving, code generation, and logical reasoning.The complexity, quality, and diversity of the alignment data are crucial to the model's performance (Liu et al., 2023) . Fol- lowing Wang et al. (2023b) , we select challenging problems that GPT-3.5-Turbo fails to solve. We intentionally restrict the selection of the datasets to those with ground-truth solutions, aiming to ensure high-quality oversight signals rather than relying on LLM-as-a-judge annotation (Weyssow et al., 2024) . Besides, the gold solutions also serve as references for the critique model to generate feedback. To promote ULTRAINTERACT's diversity, we pick datasets of different categories. For each dataset, we include distinct reasoning patterns based on question categories or formulations necessary to solve the problems. Table 6 summarizes the datasets selected by ULTRAINTERACT. Except for MATH, none of the training datasets is used in our evaluation. In what follows, we connect the actor model with a Python interpreter as the "environment". Unless otherwise specified, we use GPT-3.5 Turbo as the actor model.Following Wang et al. (2024) , the actor model first decomposes the input problem into several sub-problems and then solves each by generating Python code pieces as actions and using the environment to execute them.To promote solution diversity, the actor model randomly samples one reasoning schema in the form of either CoT (Wei et al., 2022) or modularization programming (Qian et al., 2023; Yuan et al., 2023) . The actor then generates actions in text or code to solve each sub-problem, with each step being marked by explicit notations.Multi-turn interactions with the environment are often necessary to solve challenging problems (Wang et al., 2023b) .To improve such capabilities of the models, ULTRAINTER-ACT collects trajectories in which the actor model interacts with the environment and a critique model (a proxy for user) and refines its action based on their feedback.The environment receives an action from the actor model along with the interaction history, and then the code interpreter returns two kinds of "Observation": (1) Python execution results, either program outputs or error traceback messages;(2) binary feedback, indicating whether the solution is correct or not. Then, the observations along with the history will be passed to a critique model, which locates the errors and provides suggestions for improvements. To avoid potential bias introduced by self-correction (Wang et al., 2023b; Xu et al., 2024) , we adopt a stronger model, GPT-4, as the critique and ensure critique quality by providing GPT-4 with ground truth answers as references.This procedure resembles Wang et al. (2024) . However, we adopt more diverse reasoning patterns to teach LLMs to learn rationales rather than simply memorizing answers (Mitra et al., 2023) , and learn to create and use tools (Qian et al., 2023; Yuan et al., 2023; Qin et al., 2023) . Besides, we believe that it is important for LLMs to learn from the feedback provided by the critique rather than solely from observations of the environment.Unlike open-ended conversations, where human preference is ambiguous and challenging to specify, many reasoning tasks have clear and objective preferences for correct actions. The preference annotation is threfore an evaluation of the correctness of the solutions conditioning ground truth ones, which come with the datasets in ULTRAINTERACT. This eliminates the need for human or LLM-based preference annotation and ensures high data quality. To facilitate preference learning, ULTRAINTERACT pairs correct and incorrect actions.Sampling Paired Correct and Incorrect Actions at Each Turn. For each instruction in ULTRAINTERACT, we sample, from the actor model, a pair of correct and incorrect actions following §2.2. We follow (Cui et al., 2023) to sample the pair from different actor models to ensure response diversity. To prevent models from exploiting shortcuts based on surface features, we exclude instances that fail to pass the Python syntax check.Certain challenging problems in ULTRAINTERACT pose difficulties in obtaining correct actions, even using strong actors such as GPT-4, with nearly zero pass@100 accuracies. To improve the pass rates of the actor models while keeping the expense under control, we sequentially take the following steps.(1) Directly sampling 20 actions and randomly keeping a correct one, if any. (2) If no correct action is obtained, we repeat the above process up to three times, progressively switching from more cost-effective models to the strong yet expensive GPT-4 Turbo.(3) For the remaining difficult problems where no correct action is acquired after the previous two steps, we provide the actor with ground-truth rationales and answers, and then apply various techniques to elicit correct actions. The specific information provided and the techniques applied vary depending on the tasks (Appendix A.2).Tree-structured Action Pairs Across Multiple Turns.After each turn, the correct action concludes its trajectory. We expand the incorrect action into the next turn, and have the actor interact with the environment and the critique to refine its solution ( §2.2). We then repeat the procedures introduced earlier in this section to collect an additional action pair. By expanding the incorrect action, ULTRAINTERACT can provide data to help models learn from feedback, and collect multiple action pairs for preference learning across multiple turns.Conceptually, for every instruction, ULTRAINTERACT constructs a binary preference tree with each action being a node (Figure 2 ). We cap the tree at a maximum of five turns.Additional Instruction-action Pairs for Challenging Problems. We believe the challenging instructions that make it to step (3) above can provide valuable training signals. Therefore, for a subset of these problems with multiple ground truth solutions, we further sample additional correct actions to cover all ground truths. Accordingly, we further sample incorrect actions to pair with these additional correct actions, so that they can be used in both supervised fine-tuning and preference learning.With the tree-structured data, ULTRAINTERACT enables comparisons at every turn, in contrast to comparing only at the last turn (Bai et al., 2022) , and thus can improve the models' interaction ability. Closing this section, Table 1 summarizes some statistics of ULTRAINTERACT, and more details are in Appendix A.4. 
eurus: state-of-the-art open llms in reasoning: 
 ULTRAINTERACT helps us develop EURUS, a suite of LLMs and a reward model (RM).Supervised Fine-Tuning. EURUS-7B-SFT is fine-tuned from Mistral-7B (Jiang et al., 2023a) and EURUS-70B-SFT from CodeLLaMA-70B (Roziere et al., 2023) . First, we perform SFT using all correct actions (287K) in ULTRAIN-TERACT. We find it yields better performance to discard interaction history and train only on correct leaf nodes in each tree. To improve general instruction-following ability, we include into our SFT data mixture UltraChat (Ding et al., 2023) , ShareGPT1 , and OpenOrca (Lian et al., Magicoder-S-DS-6.7B (Wei et al., 2023) , OpenCodeInterpreter (OpenCI for short, DS-6.7B/CL-70B) (Zheng et al., 2024) , DeepSeek-Coder-33B-Instruct (Guo et al., 2024a) , and CodeLLaMA-70B-Instruct (Roziere et al., 2023) .MAmmoTH-7B-Mistral (Yue et al., 2023) , WizardMath-7B-v1.1 (Luo et al., 2023a) , OpenMath (Mistral-7B/CodeLLaMA-70B) (Toshniwal et al., 2024) . 2023). We finetune base models for 1 epoch with a 2e-5 learning rate and 0.1 warmup ratio using a cosine scheduler.For EURUS-7B, we mix 32K UltraChat, 30K ShareGPT, and 50K OpenOrca. For For EURUS-70B, we mix 63K UltraChat, 30K ShareGPT, and 70K OpenOrca.Perference Learning. Based on EURUS-SFT models, we explore three preference learning algorithms, DPO (Rafailov et al., 2023) , KTO (Ethayarajh et al., 2024) , and NCA (Chen et al., 2024a) . Differently from SFT, here we include all multi-turn trajectory pairs in our ULTRAINTERACT (220K) and include all UltraFeedback (Cui et al., 2023) pairs (340K). For hyperparameters, all β is set to 0.1, and λ + /λ - in KTO is set to 1.33 as recommended. We finetune models for 1 epoch with a 5e-7 learning rate and 0.1 warmup ratio using a cosine scheduler.Reward Modeling. Similarly to the preference learning, we use all 220K multi-turn trajectory pairs from ULTRAIN-TERACT; it is further augmented with the 240K single-turn action pairs from ULTRAINTERACT. More details are in the Appendix B. We include all 340K pairs from UltraFeedback and one pair for each instruction from UltraSafety (Guo et al., 2024b) , totaling 3K. EURUS-RM-7B is initialized from EURUS-7B-SFT with a new linear layer. We train RM for 1 epoch with lr=1e-5 learning rate. We also use a cosine scheduler with a warmup ratio of 0.1.Our findings in §6 indicate that the absolute values of rewards make a big difference in the models' reasoning performance. We therefore augment the established Bradley-Terry (BT) objective L BT with an additional term L DR to directly increase the reward of the chosen actions for instances from ULTRAINTERACT, and decrease those of the rejected ones:L ULTRAINTERACT = -log (σ (r θ (x, y c ) -r θ (x, y r )))L BT optimize relative rewards -log (σ (r θ (x, y c )) -log (σ (-r θ (x, y r )))L R: : increase r θ (x,yc) and decrease r θ (x,yr)For instances from other datasets, we train with L BT . θ denotes the reward model's parameters, r θ (•) and r θ (x, y r ) the rewards on the chosen and rejected actions respectively. Our ablation study demonstrates the importance of both L BT and L DR . 
evaluation of eurus-7b and eurus-70b: 
 Evaluation Setup. We consider both single-turn and multiturn reasoning. For single-turn evaluation, we consider HumanEval (Chen et al., 2021) , MBPP (Austin et al., 2021) , and LeetCode (Guo et al., 2024a) for coding, GSM-Plus (Li et al., 2024) , MATH, TheoremQA (Chen et al., 2023) , SVAMP (Patel et al., 2021) , and ASDiv (Miao et al., 2020) for math, and BBH-Hard (Suzgun et al., 2022) for reasoning.We evaluate with pass@1 accuracy. We also use IFEval (Zhou et al., 2023) to assess the instruction-following ability and report the prompt-level loose score. For multi-turn evaluation, we adopt MINT (Wang et al., 2023b) and only consider the coding and math problems. We report the success rate at Turn 5. Please find further details on evaluation setups and evaluations beyond reasoning in Appendix C.As shown in Table 2 , we compare our EURUS with generalpurpose models, and those specialized in coding and math of various sizes. We also summarize the results of GPT-3.5 Turbo and GPT-4 reported in previous works.Results are shown in Table 3 . We summarize the takeaways as follows:EURUS, both the 7B and 70B variants, achieve the best overall performance among open-source models of similar sizes. EURUS even outperform specialized models in corresponding domains in many cases.Notably, EURUS-7B outperforms baselines that are 5× larger and EURUS-70B achieves better performance than GPT-3.5 Turbo. EURUS's instruction-following performance is among the best general-purpose models, substantially better than specialized ones.Preference learning with ULTRAINTERACT can further improve the performance, especially in math and the multi-turn ability. KTO and NCA consistently improve the models' performance in all five math benchmarks and multturn evaluations, while their effects vary in others. Since SFT models only use the single-turn data from ULTRAIN-TERACT while preference learning uses the multi-turn ones, the improvements in interaction ability should also be attributed to ULTRAINTERACT rather than the algorithms alone. Surprisingly, we observe that DPO hurts model performance on most benchmarks. DPO training of our 70B model fails since the rewards go down to -∞. We analyze this phenomenon in §6.1. 
evaluation of eurus-rm-7b: 
 Evaluation Setup. We evaluate EURUS-RM-7B on three RM benchmarks, RewardBench (Lambert et al., 2024) , AutoJ (Li et al., 2023a) , and MT-Bench (Zheng et al., 2023) .Aiming for a more realistic OOD evalation, we exclude the "prior sets" split from RewardBench, since many baselines train on the datasets that this split contains. We compare with PairRM (Jiang et al., 2023b) , Starling-RM-7B/34B (Zhu et al., 2023), UltraRM-13B (Cui et al., 2023) , GPT-3.5 Turbo, and GPT-4. To further explore EURUS-RM-7B's potential in improving models' performance through reranking, we use it to rerank Mistral-7B-Instruct-v0.2's responses on HumanEval, MBPP, GSM8K, and MATH. We report the results of random sampling, self-consistency, and Starling-RM-34B as baselines.Table 4 summarizes reward modeling performance, and Figure 4 plots some reranking results with others in Appendix D.1.EURUS-RM-7B stands out as the best 7B RM overall, and achieves similar or better performance than much larger baselines. Particularly, it outperforms GPT-4 in certain tasks. EURUS-RM-7B achieves a better correlation with human experts than all existing models on AutoJ and MT-Bench, and it achieves comparable performance to the 5× larger Starling-RM-34B on RewardBench. On Re-wardBench, EURUS-RM-7B outperforms all baselines on the "Chat-Hard" split while achieving very competitive performance on the "Reasoning" split. Across the AutoJ splits, EURUS-RM-7B outperforms nearly all existing models, with the only exception being GPT-4's results on Coding.Our training objective is beneficial in improving RM performance on hard problems and reasoning. Table 4 shows that optimizing L DR improves RM's reasoning ability, but BT modeling is still beneficial in equipping RM with abilities in general chatting as suggested in the "Chat-Hard" column, though its effect on reasoning may vary.ULTRAINTERACT is compatible with other datasets like UltraFeedback and UltraSafety, and mixing these datasets can balance different RM abilities. Improving RM's capabilities in reasoning with ULTRAINTERACT does not sacrifice others, which indicates that ULTRAINTERACT can be a great ingredient for the training data mixture of 9 .reward models.EURUS-RM-7B improves LLMs' reasoning performance by a large margin through reranking. EURUS-RM-7B consistently improves pass@1 accuracy across all tasks and performs better than 5× larger baseline Starling-RM-34B. Also, EURUS-RM-7B's reranking performance scales well with #responses per instruction, except a slight decrease in HumanEval when increasing response number form 8 to 16. In contrast, Starling-RM-34B suffers from severe performance drop on HumanEval and it consistently hurts model accuracy on MATH.6. Analysis 
explicit reward as a proxy? hypothesis for preference learning in reasoning: 
 We investigate the reason why DPO behaves differently than KTO and NCA. We start by empirically inspecting the rewards throughout the preference learning process, as shown in Figure 5 . Rewards for chosen rejected data both keep decreasing through DPO, though the rewards for chosen data is still higher hence the loss decreases. In KTO and NCA, the rewards of chosen data keep increasing with those of rejected data decreasing.Therefore, we hypothesize it is the distinction in the trend of rewards that leads to the performance gap between DPO and the other two algorithms. This distinction can be attributed to that DPO, derived from the Bradley-Terry model, only optimizes the relative differences between chosen and rejected data overlooking the absolute values of the rewards. This is a non-issue in alignment with general human values where preference is "relative" and there can be many valid answers to the same input. However, in reasoning tasks, the space of correct answers is much smaller than that of incorrect ones. Further, we notice that the rewards of chosen data in the last training step follow the ranking order of KTO > NCA > DPO, positively correlate with their performance trends. Therefore, we believe that increasing the rewards of the chosen data is especially beneficial in preference learning for reasoning tasks.We study the impact of ULTRAINTERACT and other opensource alignment data on EURUS-7B-SFT's performance. We consider three settings: (1) With original ground-truth answers, which replaces the generated actions with groundtruth rationales and answers from the original datasets. If no rationales are available, we use those from ULTRAINTER-ACT.(2) Open-source data only. . Reward patterns of EURUS-7B preference learning with DPO, KTO, and NCA. For all algorithms, the rewards of rejected data keep decreasing and the margins between chosen and rejected data keep increasing. However, the rewards of chosen data decrease below zero in DPO while keeping increasing and staying positive in KTO and NCA. The absolute values of the reward in the last step (in red) of the three algorithms positively correlate with their performance in Table 3 . only. We evaluate with the same setting as §4.In Table 5 , EURUS outperforms the "Grouth-truth" model on all tasks, confirming the advantage of ULTRAINTERACT's designs of divide-and-conquer and code-as-action patterns, in line with conclusions of concurrent work (Chen et al., 2024b; Wang et al., 2024) . Training only on open-source data without ULTRAINTERACT greatly hurts the reasoning performance, confirming the effectiveness of ULTRAIN-TERACT. Meanwhile, training only on ULTRAINTERACT suffers a performance drop except for BBH, especially in instruction following. We attribute the performance drop to a worse instruction-following ability. This suggests the necessity of mixing ULTRAINTERACT with other alignment data for better all-around supervised fine-tuning. 
related work: 
 Open LLMs in Reasoning. Open-source LLMs have shown remarkable progress in building specialists that excel in mathematics reasoning (Luo et al., 2023a; Yue et al., 2023; Toshniwal et al., 2024) or coding abilities (Roziere et al., 2023; Wei et al., 2023; Guo et al., 2024a; Zheng et al., 2024) . On the contrary, mastering general reasoning capabilities still challenges open models, while the most advanced ones (DeepSeek-AI, 2024; Bai et al., 2023; Touvron et al., 2023; Jiang et al., 2024) are well behind proprietary models. More, these cutting-edge open general-purpose models maintain their alignment recipes confidential, which further hinders the replication and development of open-source reasoning models.Preference Learning for Reasoning. Aligning language models from human or AI preferences has emerged as a prevalent approach in the open-source community (Tunstall et al., 2023; Bai et al., 2023) with the proposal of DPO (Rafailov et al., 2023) and high-quality preference datasets (Cui et al., 2023; Zhu et al., 2023) . Different from open-domain chatbots, preference learning is largely underexplored in complex reasoning. Recent research showed performance degradation when applying DPO on reasoning tasks, but some newly proposed algorithms demonstrated a positive effect (Ethayarajh et al., 2024; Chen et al., 2024a; Mitra et al., 2024; Shao et al., 2024) . However, a deep understanding of preference learning, specifically its efficacy on complex reasoning, is not yet established. 
conclusion: 
 We strive to narrow the huge gap between open-source models and proprietary models from the perspective of alignment.  
: 
 The first AI for MATH Workshop at the 41st International Conference on Machine Learning, Vienna, Austria. Copyright 2024 by the author(s).spired by this, we derive a novel reward modeling objective which, together with ULTRAINTERACT, leads to a strong reward model. Models and data will be made public. Our EURUS-7B is comparable with baselines that are 10x larger and EURUS-70B is the only one on par with GPT-3.5 Turbo. 
 
 Appendix: A.1. Dataset Details Math. We adopt GSM8K (Cobbe et al., 2021) , MATH (Hendrycks et al., 2021b) , MathQA (Amini et al., 2019) , and NumGLUE (Mishra et al., 2022) for mathematic reasoning, and include TabMWP (Lu et al., 2023) for tabular processing. We retain the instructions for all datasets except MathQA, NumGLUE, and TabMWP. MathQA divides problems into different categories according to the topics and annotates the formula that indicates the pattern needed to solve each problem. We apply stratified sampling to sample at most five problems for each pattern and prioritize the problems that come from the long-tail category. Numglue contains eight different reasoning tasks and we discard Task 5 (Reading Comprehension + Explicit Numerical Reasoning), Task 6 (Reading Comprehension + Implicit Numerical Reasoning), and Task 7 (Quantitative NLI) due to the simplicity (Mishra et al., 2022) . For TabMWP, we only keep the questions with difficulty levels 4 and 5 since the rest are too easy for current state-of-the-art models.Code. We focus on programming with Python for the simplicity of integration of the interpreter. We use CodeContest (Li et al., 2022) and TACO (Li et al., 2023b) , two competition-level coding datasets collected from various online platforms. We filter out the overlapped questions. Note that part of the questions in TACO only contain groundtruth solutions and do not contain test cases for evaluation, hence we apply GPT-4 to generate 12 test case inputs (4 basic inputs, 4 edge cases, and 4 large numbers) for each question and then execute the ground-truth solution snippets to produce outputs. Given that the two datasets mainly focus on competition problems that may deviate from real-world daily uses, we exclusively adopt Magicoder-Evol-Instruct (Luo et al., 2023b; Wei et al., 2023) , the only dataset in our selection that does not contain test cases or ground-truth solutions. We employ GPT-4 Turbo to judge the correctness of generated code during interaction, and therefore we do not use this dataset for preference learning since we cannot rigorously construct pairs of correct and incorrect actions limited by the evaluation reliability. We also include WikiTableQuestions (Pasupat & Liang, 2015) for table processing with code.Logical Reasoning. we use the multi-hop reasoning datasets HotpotQA (Yang et al., 2018) and StrategyQA (Geva et al., 2021) , and the logical reasoning dataset ReClor (Yu et al., 2020) . We follow the setting of Wang et al. (2023b) and convert HotpotQA to a generation task, removing the contexts and requiring LLMs to search relevant information using Wikipedia API. Math GSM8K (Cobbe et al., 2021) , MATH (Hendrycks et al., 2021b) , MathQA (Amini et al., 2019) , NumGlue (Mishra et al., 2022) , TabMWP (Lu et al., 2023) Coding CodeContest (Li et al., 2022) , TACO (Li et al., 2023b) , WikiTableQuestions (Pasupat & Liang, 2015) , Magicoder-Evol-Instruct (Luo et al., 2023b; Wei et al., 2023) Logic ReClor (Yu et al., 2020) , HotpotQA (Yang et al., 2018) , StrategyQA (Geva et al., 2021) A.2. Details on Preference Tree Construction Models Adopted for Incorrect Action Sampling. We randomly sample one model from Mistral-7B-Instruct-v0.2, DeepSeek-Coder-33B-Instruct, Mixtral-8x7B-Instruct, and DeepSeek-LLM-67B-Chat to generate one incorrect action to pair with each correct one.Correct Action Generation Based on Ground Truth Annotations.We adopt GPT-3.5 Turbo as the generator to generate correct actions based on ground truth considering the instructionfollowing ability. We provide different access to the ground truth information for different tasks, specifically: (1) For coding, where test cases are black boxes to reference solutions, we provide full access to the solution codes. The actor model will add step marks and corresponding explanations to the ground-truth code to make it easier to understand, or further refine the code for optimization.(2) For tool-free math problems, to avoid the actor model directly copying the answers to pass the correctness checking, we mask the answer numbers in the rationale before providing it to LLMs. This approach can better ensure response quality since it encourages LLMs to generate responses with complete reasoning chains with each step clearly marked.(3) For program-enhanced math reasoning, we first translate the textual rationale into code. Then, we either directly provide it to the actor model to generate plans, or ask the actor model to convert the code into modularization programming and then make plans to create tools to solve problems.We conduct careful decontamination. Firstly, for LeetCode, we apply the Exact Substring Matching Algorithm2 to compare with each instruction in the ULTRAINTERACT and find no overlaps. For others, we perform 8-gram exact matching to compare ULTRAINTERACT instructions with test sets of the same task. We remove those instructions that overlap 8 grams with any test sample.In total, ULTRAINTERACT has 86K instructions and 220K action pairs. The Total # Pairs does not equal Total # Turns in ULTRAINTERACT, since we fail to generate sufficient correct actions for every incorrect action in multi-turn trajectories mainly due to a lack of sufficient ground truth annotations. The total # pairs may not equal # correct answers, either, because it is also difficult and unnecessary to sample incorrect actions for the correct ones for some simple instructions. We present the specific information for each dataset. In particular, we list information on human annotation in each dataset, which plays an important role in correct action generation ( §2.3 and Appendix A.2). All three steps of correct action sampling methods mentioned in §2.3 can be applied to datasets that have rationales, while for datasets only containing answers, only the first two steps are applicable. We do not apply any of the three-step methods to generate correct answers for Magicoder, the only dataset without any human annotation, to construct preference pairs.Regarding pair augmentation, we scale up the pairs by matching every correct action for each instruction with one incorrect action of other turns. This leads to NxN pairs of single-turn actions for a trajectory of depth N. We remove the action pairs consisting of nodes at the same turn, as they are already part of the multi-turn trajectory pairs we included. Next, to avoid overfitting on the training set, we only select instructions with NxN ≤ 10, and for these instructions, we randomly sample at most 9 pairs with each action occurring no more than 3 times. This leads to an augmentation of 240k single-turn action pairs.Detailed Setup in §4. For math, we test both textual reasoning and program-enhanced settings and report the best performance of the two. All evaluations are conducted in 0-shot CoT with two exceptions: BBH uses 3 shots and IFEval does not use CoT. For MINT, we select MATH, TheoremQA, and MMLU-math from "reasoning" as a new https://huggingface.co/datasets/ openchat/openchat_sharegpt4_datasethttps://github.com/bigcode-project/ bigcode-dataset/tree/main/decontaminationhttps://huggingface.co/spaces/lmsys/ chatbot-arena-leaderboard 


Decision: Good paper. It proposes an instruction finetuning and preference learning dataset for reasoning tasks. The dataset contains multi-turn interactions with program execution feedback and critique feedback, which is novel to see. The paper also has a rich empirical section with many evaluations and comparisons to competitor models. Reviewers QhqV and veXk have made some solid points on the weaknesses of this paper, but overall, the contribution of this paper is significant and suitable to present in the workshop.

Reviews: review: paper_summary: paper_summary: This paper introduces EURUS, a suite of LLMs optimized for reasoning, and proposes their training set, ULTRAINTERACT, for both SFT and Reward Modeling. ULTRAINTERACT is characterized by preference trees, diverse planning strategies, multi-turn interaction trajectories with the environment, and critique. By comparing and analyzing popular alignment approaches such as DPO, KTO, and NCA, the authors conclude with key findings in the alignment of reasoning tasks.
strength_weakness: strength_weakness: 1. It would be wonderful to see the scaling curves or experiments from EURUS-RM-7B to EURUS-RM-70B on ULTRAINTERACT, using the assistance from the policy model for selection.
2. According to the Figure 5, I find this figure very interesting, and you effectively demonstrate why DPO negatively impacts performance. I would like to ask if there is any way to improve DPO to avoid this issue, or which part of DPO causes this phenomenon? Could online DPO resolve this problem and enhance reasoning ability?
questions: questions: 1. How should the linear layer of a reward model be structured? Should it use the activation of the last token as input to the linear layer, or should it average the activations of all tokens?
paper_summary: paper_summary: The paper presents an instruction finetuning and preference learning dataset for reasoning tasks like mathematics and coding. The dataset has been generated with GPT-3.5 and GPT-4. It contains positive and negative samples that employ tool use in the form of a Python interpreter and has 18 % multi-turn interactions with program execution feedback and critique feedback. Multi-turn samples come with correct and incorrect actions at each turn. By finetuning publicly available LLMs on this dataset, the authors show strong performance with respect to similarly sized models on reasoning benchmarks. Different preference learning methods are compared (DPO, KTO, NCA). An alternative reward modeling loss is suggested and a reward model trained on this dataset achieves strong performance on reward modeling benchmarks, especially for reasoning tasks.
strength_weakness: strength_weakness: - Important ablations are missing that impact the message of the study.
  - The paper suggests that the dataset is better than comparable datasets because of its use of multi-turn evaluations and the availability of positive and negative actions for every turn. However, there is no comparison to a dataset generated with the same amount of computation / cost that does not use this “tree structure”. It could also be due to its use of (a recent version of) GPT-4 for difficult prompts and critique steps instead of older models, the choice of prompt sources, the inclusion of execution feedback etc. Apart from “use this dataset”, there is limited information to be gained for practitioners intending to produce their own finetuning datasets for reasoning.
  - In a similar vein, there are not nearly enough experiments to support the claims on the general usefulness of different finetuning methods (DPO, KTO, NCA) for preference learning. Each method could necessitate different optimal hyperparameters (learning rate, batch size, $\beta$, number of epochs), and before a general claim comparing the methods can be made, these choices should be ablated carefully. (And this is feasible: finetuning Mistral-7B is cheaper than generating from GPT-4; definite knowledge on methods can be more important than releasing a 70B model that is soon outdated anyway.) Training 70B models is known to be somewhat unstable, and a single failed DPO run is not enough to prove that the method generally fails.
- The analysis on DPO’s failure is incomplete: according to the DPO gradient (“What does the DPO update do?” in the original paper), the likelihood of correct answers is indeed increased while the likelihood of wrong answers is decreased. If therefore the authors observe (implicit) rewards of correct solutions to degrade, this means that the weight update cannot sufficiently distinguish between the behaviors that led to the good and bad outcomes, respectively, and fails at generalizing. Such observations are possible if there are imbalances or correlations in the data, possibly related to the way preference samples are extracted from multi-turn trajectories.
- The proposed alternative reward modeling objective is neither derived/explained nor ablated against different ways of incorporating “absolute terms” into the loss. The proposal amounts to augmenting each preference sample $(c, r)$ (correct, rejected) to three samples $(c, r), (c, 0), (0, r)$, where $0$ stands for a virtual sample with zero reward. Does this correspond to a baselined version of the Bradley-Terry model? Are there alternative ways of incorporating absolute terms into the reward modeling loss?
- The experimental sections should discuss advantages and disadvantages of the method more carefully. For instance, all preference learning methods appear to (at best) maintain / (at worst) severely harm the coding abilities of the 7B models.
- Evaluations could benefit from more samples to decrease variance, all numbers appear to be derived from a single sample per prompt.
questions: questions: - Line 269: “Warmup ratio of 0.1” means 0.1 epochs?
- Which evaluation parameters were used? Greedy decoding?
- Which total batch sizes were used for the finetunings? (This is an important hyperparameter in finetunings on small dataset
paper_summary: paper_summary: This paper introduces a dataset called ULTRAINTERACT for alignment with respect to reasoning tasks. Compared to prior such datasets, it includes more fine-grained information in the form of unbalanced binary trees. They finetune a 7B and 70B model from Mistral, CodeLlama respectively on their new dataset showing improved performance for reward modelling, further bolstered by a new reward objective they introduce.
strength_weakness: strength_weakness: 1. Table 3 indicates that the Eurus models are not uniformly better than previous models, and there are some tasks it performs exceptionally better than, but suffers in others. The variance is most clear in the 7B comparison, likely less in the 70B comparison because there are not many quality models of that size which are open-sourced. However, I think the main claim in the paper is the performance on reward modelling.
questions: questions: 1. Could you compare with Llama-3 8B/70B as well to get the comparison with most up-to-date SOTA models?
2. You mentioned that you choose tasks which are difficult for GPT-3.5. I am curious why not choose tasks which are instead difficult for GPT-4? Is this because the critique model is GPT-4?
3. Could you report the overall expenses of interacting with the OpenAI API for producing ULTRAINTERACT? I think this information can be beneficial for further community efforts in producing such datasets.
4. What are some reasons that you expect make Eurus better on TheoremQA compared to other models, but not for other datasets?

Suggestions:
1. There is a typo in appendix section A.3: "Decomtamination" -> "Decontamination"
2. On line ~150, include the period . inside the quotations “environment”. -> "environment."