Advancing LLM Reasoning Generalists with Preference Trees
Authors: Lifan Yuan
Ganqu Cui
Hanbin Wang
Ning Ding
Xingyao Wang
Boji Shan
Zeyuan Liu
Jia Deng
Huimin Chen
Ruobing Xie
Yankai Lin
Zhenghao Liu
Bowen Zhou
Hao Peng
Zhiyuan Liu
Maosong Sun
Venue: ICLR 2025 Poster (2025)
Keywords: Reasoning
Alignment
Data
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We present Eurus, state-of-the-art open LLM reasoning generalists and its recipe.

Abstract: We introduce EURUS, a suite of large language models (LLMs) optimized for reasoning. Finetuned from Mistral-7B, Llama-3-8B, and Mixtral-8x22B, EURUS models achieve state-of-the-art results among open-source models on a diverse set of benchmarks covering mathematics, code generation, and logical reasoning problems. Notably, EURUX-8X22B outperforms GPT-3.5 Turbo in reasoning through a comprehensive benchmarking across 12 test sets covering five tasks. The strong performance of EURUS can be primarily attributed to ULTRAINTERACT, our newly-curated large-scale, high-quality training data dataset specifically designed for complex reasoning tasks. ULTRAINTERACT can be used in both supervised fine-tuning, preference learning, and reward modeling. It pairs each instruction with a preference tree consisting of (1) reasoning chains with diverse planning strategies in a unified format, (2) multi-turn interaction trajectories with the environment and the critique, and (3) pairwise positive and negative responses to facilitate preference learning. ULTRAINTERACT allows us to conduct an in-depth exploration of preference learning for reasoning tasks. Our investigation reveals that some well-established preference learning algorithms may be less suitable for reasoning tasks compared to their effectiveness in general conversations. The hypothesis is that in reasoning tasks, the space of correct answers is much smaller than that of incorrect ones, so it is necessary to explicitly increase the reward of chosen data. Therefore, in addition to increasing the reward margin as many preference learning algorithms do, the absolute values of positive responses’ rewards should be positive and may serve as a proxy for performance. Inspired by this, we derive a novel reward modeling objective and empirically that it leads to a stable reward modeling curve and better performance. Together with ULTRAINTERACT, we obtain a strong reward model.

Published as a conference paper at ICLR 2025
ADVANCING LLM REASONING GENERALISTS WITH PREFERENCE TREES
Lifan Yuan 2 Ganqu Hanbin Hanbin , Ning Ding's Xingyao Wang', Boji Shan', Zeyuan Liu', 2 Jia Deng", Huimin Chen', Ruobing Xie", Yankai Lin Zhenghao Liu", Bowen Zhou', Hao Peng", Zhiyuan Liu', Maosong Sun'
"Tsinghua University University of Illinois Urbana-Champaige *Peking University *Northeastern University 5 ModelBest.Inc 6 Renmin University of China Tencent li 1ifan40illinot s. edu wanghanbin95estuu pku. edu.
cgq22 ils. ts nnghua edu.cn
ABSTRACT
We introduce EURUS, suite of large language models (LLMs) optimized for reasoning. Finetumed from Mistral-7B. -lama-3-8B, and Mixtral-8x22B. EURUS models achieve state-of-the-ard results among open-source models diverse set of benchmarks covering mathematics, code generation, and logical reasoning prob- lems. Notably, EURUX-8X22B outperforms GPT-3.5 Turbo reasoning through a comprehensive benchmarking across 12 test sets covering five tasks. The strong performance of EURUS can be primarily attributed ULTRAINTERACE newly-curated large-scale, high-quality training data dataset specifically designed for complex reasoning tasks. ULTR INTERACT can be used in both supervised fine-tuning, preference learning, and reward modeling. It pairs each instruction with a preference tree consisting of (1) reasoning chains with diverse planning strategies in a unified format, (2) multi-turn interaction trajectories with the environment and the critique, and (3) pairwise positive and negative responses to facilitate prefer- ence learning. UL.TRAINTERAN
our
allows to conduct in-depth exploration of preference learning for reasoning tasks. Our investigation reveals that some well- established preference learning algorithms may be less suitable for reasoning tasks compared to their effectiveness in general conversations. The hypothesis is that in reasoning tasks, the space of correct answers much smaller than that of incorrect ones, so it is necessary to explicitly increase the reward of chosen data. Therefore, in addition to increasing the reward margin as many preference learning algorithms do, the absolute values of positive responses' rewards should be positive and may serve as a proxy for performance. Inspired by this, we derive a novel reward modeling objective and empirically that leads to stable reward modeling curve and better performance. Together with ULTRAINTERACT, we obtain a strong reward model.
INTRODUCTION
Current alignment techniques have significantly advanced the development of open-source large lan- guage models (LLMs) that effectively meet user expectations and align with human values (Touvron
et al. 2023 Tunstall et al., 2023). On complex reasoning, success has been achieved by specializing
models for specific capabilities, such coding (Wei et al. 2023 Guo al.) 2024a Zheng et al. 2024)
and solving math problems (Fu et al. 2023 Yue et al. 2023 Luo et al. 2023a Toshniwal et al. 2024).
However, these models still fall short, by large margins, of the most advanced proprietary models in
their all- all-around capabilities to tackle a diverse range of challenging problems. We conjecture that this performance gap can be primarily attributed (1) the lack of high-quality alignment data and (2)
the underexploration of preference learning techniques for improving models' complex reasoning capabilities. this paper, we take strides towards bridging this gap by addressing both factors and developing EURUS.
"Equal Contribution. "ooresponndig Authors.

---

Published as a conference paper at ICLR 2025
45
-78
GPT4
40
-708
35
GPT:Series
Eunux-8x28B-NCA
Ours
DeegSeek-Coderr33--nnnttt
Wizard Wizard M-2-0x226
LeenCode
MagiCodee-SSDS--.5
OpenCI-C170R
Mitrral (astral-Bx22e--oo an22e-nntrrut-wo..
6PT-3 S-Turbo
DeepSeek
Eurus-7B-KTO
Op-661-556677
CL-708-ime insttnstt
"gWen1 5-7811---2..-h-
Uama- 3-Ena---------T0
OpenMath CL-70B
Openkaath-istra-77
WicerdMahh778- v1.1
Misiral-76 Hilsifal-7--nstruce 76-nnstrut--
Mistral-S77B- Su7B-inttruct
10
15
20
25
30
35
40
55
TheoremQA
Figure I: Evaluation results LeetCode and TheoremQA, two challenging OOD coding and math benchmarks with only test sets. Our EURUS-7P is comparable with baselines that are 10x larger and EURUX-8X22B is the only one on par with GPT-3.5 Turbo.
EURUS consists of suite of LLMs and reward model finetuned from Mistral-7B (Jiang et al. 2023a), Llama-3 (Metal 2024), and
(Jiang et al.) 2024). Across diverse set of complex reasoning benchmarks that are mostly out-of-distribution (OOD), EURUS achieves state-of- the-art overall performance among all open-source models. In particular, EURUS excels in solving challenging problems that often require sophisticated planning, reasoning, tool integration, and the ability to interact with and learn from the environment and users. As shown in Figure I on university- level STEM questions TheoremQA (Chen al., 2023) and competition-level coding problems LeetCode Contest (Guo et al.) 2024a), EURUS significantly outperforms all open-source models, achieving comparable performance GPT-3.5 Turbo. Besides, reward model, SURUS-RM-77 outperforms baselines with a 5x larger size on various reward model benchmarks and demonstrates superiority on best-of-N and MCTS-guided decoding tasks.
EURUS models are trained on alignment data specifically designed to improve LLMs' reasoning capabilities. ULTRAINTERACT
our newly-curated, large-scale, and high-quality
consists of a diverse set of instructions spanning math, coding, and logical reasoning problems from 12 established datasets. For each instruction, ULTRAINTER ACT collects a preference tree that includes: (1) Diverse planning strategies in a unified pattern, such as sequential processing (Wei et al. 2022) and tool creation (Qian al. 2023), followed by executing step-by-step actions formatted in either text or code, to provide divserse reasoning trajectories. (2) Multi-turn interaction trajectories with the environment and the critique, to improve models' capabilities to learn from feedback and correct previous errors (Wang al. 2023bb. (3) Paired correct and incorrect actions organized in tree structures, to facilitate preference learning. In total, ULTRAINTERACT contains 86K instructions and 220K action pairs, where each pair consists of an instruction, correct response, and an incorrect one. Conceptually, ULTRAINTERACT data resemble imbalanced binary trees as shown in Figure 2
ULTRAINTERACT be used in both supervised fine-tuning and preference learning. Our experi- ments show that, using ULTRAINTERACE along with established datasets in instruction fine-tuning already achieves strong performance. ULTRAINTERA. INTERACT further facilitates preference learning for
reasoning tasks, improving the performance even further with KTO (Ethayarajh et al. 2024) and NCA (Chen et 2024a). Surprisingly, applied to an instruction finetuned EURUS model, DPO Rafailoy et al. 2023) sometimes hurts the performance.
Through careful analysis, we provide evidence that the performance in reasoning correlates with the value of rewards of chosen data higher final reward often indicates a better reasoning capability. Besides, our investigation suggests that DPO may be less suitable for reasoning tasks than KTO and NCA. Inspired by this fresh finding, we devise new objective for reward modeling to augment the Bradley-Terry objective (Bradley & Terry 1952), explicitly encouraging training to increase the absolute rewards of chosen solution and decrease those of rejected data. Furthermore, ULTRAINTERACT" leads to our reward model EURUS-RM RM which achieves a better correlation with human annotators than all existing models on AutoJ (Li et al. 2023a) and MTT-Bench ZZheng et al. 2023). including GPT-4 (OpenAl 2023). EURUS- RM-7B demonstrates especially strong preference modeling performance reasoning tasks.
In short, we compiled this work by first synthesizing both SFT and preference datasets to improve the reasoning ability of open-source models (Section $2) We examined the effectiveness of
2

---

Published as a conference paper at ICLR 2025
datasets by training both policy and reward models (Section We evaluated the performance of policy models in Section $4 during which we observed correlation between reward patterns and benchmark performances. Next, we then evaluated our reward models and validated that insights on the reward-pert wwward-performace correlation can be converted into gains in model training (Section $ Finally, ablate some factors in dataset construction in Section 6
2 ULTRAINTERACT: TREE-STRUCTURED ALIGNMENT DATA FOR REASONING
Solving complex problems often requires the model's capability in planning and reasoning, integrating with tools, and interacting with and learning from both the environment and the users. This is reflected in ULTRAINTERAL ACT's design choices: (1) Its instructions are di- verse, challenging, and of large scale ($2.T): (2) provides multi-turn trajectories that solve the input instruction through multiple turns of interaction with and learning from the envi- ronment and critique. At each turn, it breaks down the problem into smaller ones (3) ULTRAINTERACT includes pairwise data to facilitate prefer- ence learning ($2.)
U
U
User Instruction
Conceptually, ULTRAINTER- ACT collects preference for each instruction, with the instruction being the root and each action a node (Figure 2 root-to-leaf path consisting of sequence of actions. In each preference tree, all nodes of correct actions and all trajectories ending with correct actions
Unpaired
Action
Rejected
Action
trajectory
Osc
Observation
R
Osc
Observation
Critique
be used for SFT. Paired correct and incorrect nodes or trajectories can be used for preference learning.
Figure 2: Left: CodeActInstruct (Wang et al. 2024 and Code- Feedback (Zheng al.) (024);; Middle: HH-RLHF (Bai al.) 2022]; Right: U..TRAINTERACE Each instruction ULTRAIN- TERACT is constructed preference tree.
2.1 INSTRUCTION SELECTION EMPHASIZING COMPLEXITY, QUALITY, AND DIVERSITY
We target three representative reasoning tasks: math problem-solvee code generation, and logical reasoning. The complexity, quality, and diversity of the alignment data erucial to the model's performance (Liu et al. 2023). Following Wang et al. (002b)), we select challenging problems that GPT-3.5-Tuubb fails solve. We intentionally restrict the selection of the datasets those with ground-truth solutions, aiming to ensure high-quality oversight signals rather than relying on LLM- as-a-judge annotation (Weyssow al. 2024). Besides, the gold solutions also serve as references for the critique model to generate feedback. To promote ULTRAINTERACT' diversity, we pick datasets of different categories. For each dataset, we include distinct reasoning patterns based on question categories formulations necessary to solve the problems. Table & summarizes the datasets selected by ULTRAINTERACT. Except for MATH, none the training datasets used in our evaluation.
2.2 DECOMPOSITION INTERACTION AT EACH TURN
Figure 3 provides an illustrative example. In what follows, we connect the actor model with a Python interpreter as the "environment". We use GPT-3.5 Turbo the actor model.
Following Wang et al. 2024). the actor model first decomposes the input problem into several sub-problems and then solves each by generating Python code pieces as actions and using the environment to execute them. To promote solution diversity, the actor model randomly samples reasoning schema in the form of either CoT (Wei et al. 2022) modularization programming (Qian et al. (2023] Yuan et al. 2023). The actor then generates actions in text or code to solve each sub-problem, with each step being marked by explicit notations.
3

---

Published as a conference paper at ICLR 2025
In a classroom, x children have 7 books each. [..] what is the value of unknown variable x?
Generation
Observation
Generation
Observation
<thought>
[-] KaneRrrori name 'x!
<thought>
[-]
Step 1: First, let's calculate the total [-].
Step 1: First, create equation [..].
Out: 10.0
Step 2: Then,
the
not defined.
Step 2: Each has 7 books, so L-J.
</ 3/thought>
Step 3: The sum of the books should [-].
Your andwer
Stop 4: Now we can solve for by [..].
Step The teacher brings (--)-
Your
<ececute>>
</ thought>
total, total_booksschhe child
Step First, let's calculate [-].
Critique
<execute>
Step 5: Setting equatioo:
Critique
Stop 5: Noxt,
[-].
Export Feedback:
equation 17x
Espact feedback:
total books total books, child
Step 6: We can now solve for [-].
Your thought process accurate and your error lies
SStpp Solving for
Good job! You have fixed the error last turn. Now your answer correct. (--]
x (78 8)
</execute>>
your code.
<golution>
<eolution>
define 'x' first I..]
</solution>
</solution>
Figure 3: An illustrative example of ULTRAINTERACT trajectory turns. In each turn, the actor model generates step-by-step reasoning chains, and the environment and the critique model provide observations and textual critique respectively.
Multi-tune turtitttummn interactions with the environment are often necessary to solve challenging problems (Wang et al. 2023b). To improve such capabilities of the models, ULTRAINTERACT collects trajectories in which the actor model interacts with the environment and a critique model (a proxy for user) and refines its action based on their feedback.
The environment receives an action from the actor model along with the interaction history, and then the code interpreter returns two kinds of "Observation": (1) Python execution results, either program outputs or error traceback messages; (2) binary feedback, indicating whether the solution is correct not. Then, the observations along with the history will be passed critique model, which locates the errors and provides suggestions for improvements. To avoid potential bias introduced by self-correction (Wang et al. 2023b Xu et al., 2024), we adopt stronger model, GPT-4 as the critique and ensure critique quality by providing GPT-4 with ground truth answers as references.
This procedure resembles Wang al. (2024). However, we adopt more diverse reasoning patterns to teach LLMs to learn rationales rather than simply memorizing answers (Mitra et al.) [2023). and learn create and use tools QQian al.) [2023] Yuan et al.) [2023] Qin al., (023). Besides, we believe that it is important for LLMs to learn from the feedback provided by the critique rather than solely from observations of the environment.
2.3 PREFERENCE TREES FACILITATES PREFERENCE LEARNING ACROSS MULTIPLE TURNS
Unlike open-ended conversations, where human preference is ambiguous and challenging to specify, many reasoning tasks have clear and objective preferences for correct actions. The preference annotation is therefore an evaluation of the correctness of the solutions conditioning ground truth ones, which come with the datasets in ULTRAINTERACT. This climinates the need for human or LLMM-ased preference annotation and ensures high data quality. To facilitate preference learning, ULTRAANTERACT pairs correct and incorrect actions in each turn.
Sampling Paired Correct and Incorrect Actions at Each Turn. For each instruction in ULTRAINTERACT. we sample, from the actor model, a pair of correct and incorrect actions following 22.2 We follow Cui et al. (2023) to sample the pair from different actor models to ensure response diversity. To prevent models from exploiting shorteuts based on surface features, we exclude instances that fail to pass the Python syntax check.
Certain challenging problems in ULTRAINTERACT pose difficulties in obtaining correct actions, even using strong actors such as GPT-4, with nearly zero pass@ 100 accuracies. To improve the pass rates of the actor models while keeping the expense under control, we sequentially take the following steps. (1) Directly sampling 20 actions and randomly keeping correct one, if any. (2) If no correct action obtained, we repeat the above process up to three times, progressively switching from more cost-effective models to the strong yet expensive GPT-4 Turbo. (3) For the remaining difficult problems where no correct action is acquired after the previous two steps, we provide the actor with
'The majority of data collection and experiments are prior the release of GPT-4o.
4

---

Published as a conference paper at ICLR 2025
ground-truth rationales and answers, and then apply various techniques elicit correct actions. The specific information provided and the techniques applied vary depending on the tasks (Appendix (A.2).
Tree-structured Action Pairs Across Multiple Turns. After each turn, the correct action concludes its trajectory. We expand the incorrect action into the next turn, and have the actor interact with the environment and the critique refine its solution ($2.2) We then repeat the procedures introduced earlier in this section to collect an additional action pair. By expanding the incorrect
action, ULTRAINTERACT can provide data help models learn from feedback, and collect multiple action pairs for preference learning across multiple turns.
Conceptually, for every instruction, ULTRAINTERACT constructs binary preference tree with each action being a node (Figure 2) We cap the tree at maximum of five turns.
Additional Instruction-action Pairs for Challenging Problems. We believe the challenging instructions that make it to step (3) above can provide valuable training signals. Therefore, for a subset of these problems with multiple ground truth solutions, we further sample additional correct actions to cover all ground truths. Accordingly, we further sample incorrect actions to pair with these additional correct actions, so that they can be used both supervised fine-tuning and preference learning.
With preference trees, INTERATT enables comparisons at every turn, in contrast to comparing only at the last turn (Bai et al. 2022), and thus can improve the models' interaction ability. Closing this section, Table |
summarizes statistics of ULTRAINTERACT. Check more details in Appendix
A.4
We show examples of the data in Appendix 0
Table 1: Some statistics of ULTRAINTERACT.
Task
w/ Internction?
Type
w/ Tool?
Instructions
T2
T'Turns per Traj.
T3
T4
T5
# Tokens per Traj.
Avg. Traj
per Ins.
Total Pairs
Answers
Correc
22,928
10,440
1,896
5,564
1,7500
42,780
68,033
Math
2,757
16,151
139.1
5.9
13.217
16,151
x
22,639
10,708
3,521
1,459
723
6,228
1,521.9
1.0
44,750
62,182
x
x
2,083
16,348
538.1
12,624
16,348
Coding
x
/
20,463
8,495
92,618
13,265
2,584
3,248
1,728.5
1,000.4
5.5
78.634
18,106
92,618
22,215
Logle
x
2,086
L,685
296
72
23
1.299.8
1.0
1,750
2,198
1,167
2,153
1,266.7
7.958
7,231
Total
85.918
163.671
12,199
4,756
2,014
15,063
1,201.8
2.3
219.819
286.979
3 EURUS: STATE-OF-THE-ART OPEN LLMS IN REASONING
ULTRAIINTERATT helps us develop EURUS, a suite of L.MMs and a reward model (RM).
Supervised Tine-Tuning. EURUS- 7B- and LLAMA-3 --------UUUS----SSSS are fine-tuned from Mistral-7B (Jang et al.) 2023a) and L-ama-3-8B (Meta) 2024 respectively, and DURUX-8X22B--0
from Mixtral-8x22B (Jang al. 2024). First, we perform SFT using all correct actions (287K) in ULTRAINTERACT. We find it yields better performance to discard interaction history and train only on correct leaf nodes in each tree. To improve general instruction-folloving ability, we include some open-source SFT datasets in our data mixture. Please find mixture details in Appendix B
Preference Learning. Based on SIRUS-SFT models, we explore three preference learning algo- rithms, DPO (Rafailov et al. 2023). ). KTO (Ethayarajh et al. (024), and NCA (Chen et al. 2024a).
Differently from SFT, here we include all multi-turn trajectory pairs in our ULTRAINTERACT (220K) and include all UltraFeedback (Cui et al. 2023) pairs (340K).
Reward Modeling, Similarly to the preference learning, we use all 220K multi-tumn turt trajectory pairs from INTRAINTERACT; it is further augmented with the 240K single-turn action pairs from ULTRAINTERACE UltraFeedback and one pair for each instruction from UltraSafety (Guo et al. 2024b), totaling 3K. EURUS- MURUS-RMM-B is initialized from BURUS-7B-SPE with new lincar layer.
More details are the Appendix B We include all 340K pairs from
Our findings in 4.2 indicate that the absolute values of rewards make big difference in the models' reasoning performance, with decreasing rewards of chosen actions possibly resulting in a perfroamce degradation. We therefore augment the established Bradley-Tee (BT) objective Cer with an additional term CDR to directly increase the reward of the chosen actions for instances from ULTRA INTERACT and decrease those of the rejected ones:
5

---

Published as a conference paper at ICLR 2025
Table Open-source LLM baselines that we compare to.
Type
Models
Misttal-7B-IIsstuucc (Jang et
Zephyr-7B..8 (Tunstall et al. 2023). OpenChat-3.5- (Wang et
General Purpose
2025a L-77ln (2hu nl. 2025], Bextral-8x7--nnttnee Jiang 20230 DecpSeck-LLMM67--
Chat (Deepeek-AA]] 2024 @Wen 1- 3-72 Chat (Bai et 2023)
Magiooda-S-DSS6677 et 2025, OpenCodelnterpetter (OpenCI for short, DS-6.7B/CL-708) (Zheng et
Coding
N024, DeepSeck TreepSek-CCader--5 33B--lsttucl (Guo 20244a and CodeLLaMA-7008 Instruct 2225
Ta-mmooTH--BMMMMt (Yue al. 2023
WizardMath-7B-vlk
Luo al. 2023a OpenMath (Mistral-
Math
MWVCodeLLaMA-7000 (Toshnwwal
2024
LULTRANNTERACT
log (r rol(, v.))) (
log (ro (2)),,),) log
-
Car: optimize rewards
Lou: increase r0(2, ve) and docrease ro yr-)
We train ULTRAINTERACT examples with LULTRANNTERACT while for instances from other datasets, we train with LBT only. 0 denotes the reward model's parameters, (xr, Me) and re (z, Mr) the rewards on the chosen and rejected actions respectively. We ablate the importance of LBr and LDR in $6.2
EVALUATION OF EURUS MODELS
Evaluation Setup. We consider both single-turn and multi-turn reasoning. For single-turn evaluation. we consider HumanEval (Chen et al.) 202T), MBPP (Austin al. 2021). and LeetCode (Guo et al. 2024ab for coding, GSM-Plus (Li al. 2024), MATH, Theorem QA (Chen et al. (222), SVAMP (Patel et al. 2021). and ASDiv (Miao et al. 20200) for math, and BBH- (Suzgun et al. 2022) for reasoning. We evaluate with pass accuracy. We also IFEval (Zoou et al. 2023) the instruction-followingg ability and report the prompt-level loose score. For multi-turn evaluation, adopt MINT (Wang et al. 2023b) and only consider the coding and math problems. We report the success rate at Turn 5. Please find further details on evaluation setups and evaluations beyond reasoning in Appendix C As shown in Table 2 compare our EURUS with general-purpose models, and those specialized in coding and math of various sizes. We also summarize the results of GPT-3.5 Turbo and GPT-4 reported previous works.
Table 3: Overall performance, All test sets except MATH are out catt---diitriuttion to our models and most baselines. MAmmoTH, OpenChat, and Starling-LA. LM have been trained on TheoremQA test sets. We strikethrough the contaminated numbers.
Cading
Math
Reseoning
Ins-Follwing
Multi-Tum
Model
HumanR
MATH
Tha QA
ASD
Code
Avp
29.3
35.8
5.0
7.5
OpenChut-3.51211
64.0
28.1
30.3
Startiag-LM-7 -M-7BBa
8.9
21.5
29.5
30.8
Maglooder S-
23,9
19.9
13.1
62.8
57.0
38.1
OpenC1-DS-66.1
768
66.2
16.1
31.6
19.3
40.5
MAmmo T1--7B--------
26.4
42.4
1.2
36.0
36:3
72.3
34,9
34-4
1
50.0
51.9
67
300
16.5
$7.8
73.5
64
22.6
57.9
OprnMatt-Miitii
33.5
46.6
11.7
39.4
39.1
13.1
83.4
79.3
58.6
15.0
2.9
57.4
78-SS-
55.5
99.1
20/0
$2.1
32.6
20.0
82.3
$4.1
64.6
44.0
154
28.4
46.5
DPO
50.6
$2.1
8.3
$1.0
28.3
20.9
76.7
$3.3
65.0
42.5
20.6
32.4
44.5
XTO
56.1
56.6
18.9
55.0
35.2
20.6
$4.4
$5.0
67.6
43.1
19.1
43.6
45.8
NCA
35.5
60.2
14.4
54.9
34.2
20.9
$4.6
85.4
64.3
42.7
21.3
38.7
48.1
LLAMA-3 LLLAMA--EERUSSS
51.2
57.9
17.2
50.7
32.01
21.3
$2.2
$3.7
72.4
47.1
18.4
24.5
45.6
IPP
43.9
SO.1
11.7
45.3
75.8
21.4
$4.1
67.5
71
56.7
21.3
30.2
42.4
KTO
5188
$6.1
15.6
$4.8
34.3
34.0
80.1
$6.7
71.7
$0.6
26.5
37.4
40.4
NCA
506
60.4
15.6
55.2
34.8
25.4
79.9
$7.5
71.7
56.2
21.3
36.3
49.6
+-40B
Mistral B----88-B-nnnnertt
90.1
20.4
66
42.5
DerpSrrk-Cadeestti
19.5
20.2
12:0
DespSedl-LM167B.
41.0
17.9
53.5
OWen1.5-71B-C 5-71B-Chaat
43.4
18.5
OpmCC1CCL-7U
29.2
19.88
79.4
OpenMach- CL-700)
3900
52.6
45.9
15.9
82.8
Wizarll Wizarl M-2-8x228
72.0
64.2
24.4
$7.0
S(L9
32.0
81.3
62.2
85.3
68.9
13.3
43.3
56.2
76.2
68.7
21.3
51.2
49.6
33.1
88.3
92.4
96.0
67.1
30,0
59.7
61.2
EoBUx-HN22BA. KTO
71.3
68.9
29.4
68.3
48.1
35.3
91.5
9016
83.6
67.1
38.2
57.5
62.5
EUBUX EURUX-8s22B-- NCA
75.0
69.7
35.0
68.1
490
35.5
91.6
92.1
X3.5
67.1
33.1
63.0
63.6
Progrietary Models
GPE-35 Tarbo
7688
83.5
23.5
61.3
37.8
35.6
85.0
90.6
70.1
56.6
20.4
36.9
57.0
GPT-4
65.4
83,5
41.8
85
69.7
$2.4
92.6
86.7
79.7
39A
65.8
748

---

Published as a conference paper at ICLR 2025
4.1 RESULTS
According to Table 3 all EURUS variants achieve the best overall performance among open- source models of similar sizes. EURUS even outperforms specialized models in corresponding domains in many cases. Notably, EURUS-7B and LLAMA-3- --R-USS-UUUU- 88 outperform baselines that are 5x larger, and EURUX-8X22B achieves better performance than GPT-3.5 Turbo. EURUS'S instruction-followii performance is among the best seneral-purpose models, substantially better than specialized ones.
Preference learning with INTRAITT can further improve the performance, especially in math and the multi-turn ability. KTO and NCA consistently improve the models' performance in all five math benchmarks and mult-turn evaluations, while their effects vary in others. Since SFT models only use the single-turn data from ULTRAINTERACT while preference learning uses the multi-turn ones, the improvements in interaction ability should also be attributed to UI.TR ACTTRACTT rather than the algorithms alone (See Appendix I for data mixture ablation results). Surprisingly, we observe that DPO not as effective as its variants. We analyze this phenomenon in $42
Eurus-78 78 DPO
Eurus-7B KTO KTO
Eurus-78 NCA
2-
Value
Value
Value
Marghs
Rewwrck/Ehhsenn
Rewarrck/Cooeen
7
Rewardk/EEosen
Rewards/Rejettes p
4-
Rewards/Rejectes Vel
Rewards/Rajected Seymletuni
0
20
60 Steps (%)
80
100
20
40
6O
100
20
40
60
80
100
Steps (%)
Steps (%)
Figure 4: Reward patterns of EURUS-7B preference learning with DPO, KTO, and NCA. For all algorithms, the rewards of rejected data keep decreasing and the margins between chosen and rejected data keep increasing. However, the rewards of chosen data decrease below zero in DPO while keeping increasing and staying positive in KTO and NCA. The absolute values of the reward in the last step (in red) of the three algorithms positively correlate with their performance in Table 3
4.2 EXPLICIT REWARD AS A PROXY? HYPOTHESIS FOR PREFERENCE LEARNING IN REASONING
We investigate the reason why DPO behaves differently than KTO and NCA. We start by empirically inspecting the rewards throughout the preference learning process, as shown in Figure and Figure 7 Rewards for chosen and rejected data both keep decreasing through DPO, though the rewards for chosen data is still higher hence the loss decreases. KTO and NCA, the rewards of chosen data keep increasing with those of rejected data decreasing.
Therefore, we hypothesize is the distinction the trend of rewards that leads to the performance gap between DPO and the other two algorithms. This distinction can be attributed to that DPO, derived from the Bradley-Terry model, only optimizes the relative differences between chosen and rejected data overlooking the absolute values of the rewards. This is non-issue in alignment with general hu- man values where preference "relative" and there can be many valid answers to the same input. How- ever, in reasoning tasks, the space of correct answers is much smaller than that of incorrect ones. Fur- ther, we notice that the rewards of chosen data in the last training step follow the ranking order of KTT NCA > DPO, positively correlate with their performance trends. Therefore, we believe that increas- ing the rewards of the chosen data is especially beneficial in preference learning for reasoning tasks.
EVALUATION OF EURUS-RM-7E
5.1 REWARD MODELING PERFORMANCE
We evaluate SURUS-RM-7T on three RM benchmarks, RewardBench (Lambert et al., 2024), AutoJ L et al.) 2023a). and MT-Bench (Zheng et al. 2023). Aiming for a more realistic OOD evalation, exclude the "prior sets" split from RewardBench, since many baselines train the datasets that this split contains. We compare with PairRM (Jang et al.] (0236), Starling- RBB73334B (Zhu et al.
2023), UltraRM-13B (Cui et al. 2023). GPT-3.5 Turbo, and GPT-4.
7

---

Published as a conference paper at ICLR 2025
Table 4: Results reward modeling benchmarks. UF: UltraFeedbakk; US: UltraSafety. The best performance in each benchmark is bold and the second best one underlined. Most baseline results from Jiang et al. (2023b) and Lambert al. (2024).
Reward Bench
AutoJ
Model
MT-Bench
Chat
Chad
Safery
Reasoning
Avg.
Code
Math
Others
Overall
PuirRM
90.2
$3.0
31.5
60.0
58.7
58.3
52.8
58.9
59.1
59.0
Starling-MM- RMM-B
98.0
43.4
88.6
74.6
76.2
59.2
47.2
61.4
60.8
56.8
StrrlingRMM -3334B
96.9
59.0
89.9
90.3
84.0
65.8
54.2
62.3
62.6
60.4
UltraRM-13B
96.1
55.3
45.8
82.0
69.8
55.0
GPT-3.5 Turbo
36.6
40.3
41.2
42.7
57.1
GPT -4
69.2
51.4
61.4
61.9
63.9
EURUS-RM-7B
96.5
65.3
80.7
87.0
82.4
87.5
82.5
78.0
80.7
79.4
Crn
96.4
59.9
79.5
77.5
78.3
83,8
78.9
80.7
79.3
w/o CuI
96.8
58.5
83.8
84.2
80.8
88.8
92.5
79.4
81.9
79.6
w/o US
96.5
66.2
67.7
81.7
73.3
87.5
90.0
79.2
81.8
79.2
w/o UI US
95.1
61.1
63.7
73.4
78.0
73.8
80.0
71.7
72.8
73.0
Results. Table summarizes reward modeling performance. Takeaways follows:
EURUS-RM-7B stands out as the best 7B RM overall, and achieves similar or better per- formance than much larger baselines. Particularly, it outperforms GPT-4 in certain tasks. EURUS-RM--7 achieves a better correlation with human experts than all existing models on AutoJ and MT-Bench, and it achieves comparable performance the 5x larger Warling-RM-33P on RewardBench. On RewardBench, EURUS-RE -M-7 outperforms all baselines on the "Chat-Hard" split while achieving very competitive performance on the "Reasoning" split. Across the AutoJ splits, EURUS-RMM RB outperforms all baselines, with the only exception being GPT-4's 's results on Coding.
Our training objective beneficial in improving RM performance on hard problems and reasoning. Table 4 shows that optimizing LR improves RM'ss reasoning ability, but BT modeling is still beneficial in equipping RM with abilities in general chatting as suggested in the "Chat- column, though its effect on reasoning may vary.
ULTRA INTERACT compatible with other datasets like UltraFeedback and UltraSafety, and mixing these datasets can balance different RM abilities. Improving RM's capabilities in reasoning with ULTRA INTERACT does not sacrifice others, which indicates that UL.TRAINTERAS can great ingredient for the training data mixture of reward models.
HumanEval
MBPP
MATH
Parsg Accuracy
Pass00 Rccuracy
PassiP Accuracy
PassiB1 Accurapt
12
55
15.0
86
10.0
36
7.5
16
16
#Respanses Per Instruction
Resspnnsss Per Instruction
#Responses Per Instruction
#Responses Per Instruction
Consistency
Starling
Eurus RM
Figure 5: Results on reranking
responses. Full results in Table 11
5.2 BEST-OF-N AND MCTS DECODING
To further explore EURUS-RM-7BB potential in improving models' performance through reranking, we use it to rerank Mistral-7B- 1ier-B---ntsuutto.222 responses on HumanEval, MBPP, GSM8K, and MATH. We report the results of random sampling, self-consistency. and Starling-RM-34B baselines. Finally, we examine the ability of EURUS-RM-7B to guide decoding on math, compared Starling-RM-7BB RM-7B. Due the inference overhead, we only sample 500 samples from GSM8K and MATH for evaluation.
for MCTS
Results. Figure 5 and Table presents some results with others in Appendix D.I
BURUS-RM--B improves LLMs' reasoning performance by large margin through reranking, It consistently improves pass @ accuracy actoss all tasks and performs better than 5x larger baseline Maaling-RRM344p Also, EURUS-RM-7HB reranking performance scales well with #responses per instruction, except for a slight decrease in HumanEval when increasing the response number from to 16. In contrast, Starling-RM- RM-34B suffers from a severe performance drop on HumanEval and it consistently hurts model accuracy on MATH.
8

---

Published as a conference paper at ICLR 2025
Table 5: Bentral-7--Instruct-ee results with RM-guided MCTS decoding.
EURUS-RM-77 improve the performance of by 81.4% on GSMSK and 43.8% MATH. According to Table 5l the policy model only achieves 25.8% and 6.4% on GSM8K and largely
Model
GSM8K
MATH
25.8
6.4
MATH respectively, but the performances improved with RM-guided MCTS decoding. Compared Marling-RMM-7B EURUS-RM- shows an advantage of 46.8% vs. 44.8% on GSM8K, and 9.2% vs. 6.6% on MATH.
Starling-RMM RM-7B
44.8
6.6
EURUS--RA RM-7B
46.6
9.2
5.3 HOW DOES LDR WORK IN REWARD MODELING?
Results in Table have demonstrated the effectiveness of our proposed reward modeling objective. To figure out the working mechanism, we plot the reward patterns in reward modeling in Figure As can see, LgT only optimize the reward margin, and the reward trend of chosen data is unstable while rewards of rejected data are positive for the most time. In contrast, LDR forces the reward of chosen data to be positive and that of rejected data to be negative as expected. There is a consistent trend of increasing the reward of chosen data and decreasing that of rejected data, which also leads to a widen- ing reward margin. When combining LBT with LDR, the resulting pattern is more similar to using LDr only. These results indicate the importance of the absolute value of rewards in reward model training.
CST
Cok
Car Cox.
20
10
15-
Value
10-
value
value
Rewards/Chosen
Rewards/Chosen
Powards/Chesen Rerwar RewjrrdsRejacted
Neowardk/Rujjettee
1
Reweards/Reeetted
-15
0
40
60
80
100
20
60 Steps (%)
40
80
100
20
60
80
100
Steps (%)
Steps (%)
Figure 6: Reward pattern in reward modeling. The rewards of chosen data and margins increase
regardless of LDR, but the rewards of rejected data decrease to be negative with regularization.
6 ANALYSIS
6.1 ARE PROPRIETARY MODELS NECESSARY FOR HIGH-QUALITY DATA CONSTRUCTION?
The current construction of ULTRAINTERACT largely adopts GPT models. However, using proprietary models raises concerns the non-permissive license and the heavy burden in financial budgets, both of the which can be addressed by open-source models. Therefore, to ablate the use of proprietary models and examine how far we can go with open-source models for data synthesis, we supplement another version of ULTRAINTERACE by substituting all GPT responses with those generated by open-source models, including EURUS itself. We list the statistics, incluidng models used to sample responses and corresponding proportions, in Appendix E We named the original GPT-involved version as ULTRAINTERACT-- and the open-source model generated as ULTRAINTERACT- We retrain LLAMA-3-EURGS presented in Table 6
with ULTRAINTERACT-VV and union of v1 and v2. Results are
Surprisingly, LLAMA-3-EURUSS performances are greatly improved. Compared to v1, training v2 improves model performance on both SFT and preference learning stage, and particularly, LLAMA- L-EURUS-SB-KTE (v2) successfully surpasses the official Slama-3-8b-Instee which it previously failed to, indicating we can train strong models without distilling proprietary models. Further, trained on the mix of v1 and v2, LLAMA-3-EERRSS -BB consistently outperforms Llama-3-8B-Innstee after preference learning. Notably, to the best of our knowledge, this is the first open recipe Llama-3-8B that can outperform the official Llama-3-88 3-8B-Instrucc model reasoning.
model (Meta 2024).
6.2 ABLATION STUDY
We study the impact of ULTRAINTERACT and other open-source alignment data EURUS- 7B- FF'S performance. We consider three settings: (1) With original ground-truth answers, which replaces the generated actions with ground-truth rationales and answers from the original

---

Published as a conference paper at ICLR 2025
Table 6: Performances of
trained with different versions of ULTRAINTERACT.
Model
Coding
Math Thargx
SYAME
ANDKS
Reasaning
Inv-Pallowing
Mann-lurn Maah
Avg:
73.7
15.9
34.7
LLAMK- (01
67.5
-Y7O
5188
58.3
15
58.8
26.9
5001
86.7
31.7
50.5
37.4
NCA
50.6
60.4
15.6
55.2
34.5
25.4
70.0
87.5
71.7
56.2
21.3
36.3
-9.6
353
0503
17.
17-3
7135
RXLT
8530
1522
4510
11.1
75.7
10:3
DPO
57.9
53.9
12:2
40.1
27.6
79.6
76.9
73.1
55.6
22.5
44.0
+9.1
19.1
15.0
53.0
54.4
21.5
89.0
72.2
51.0
42.1
+ -KTO
61.6
59.1
15.6
61.2
39.0
76.8
96.
88.2
72.1
47.7
29.4
19.2
52.3
-NCA
55.7
60.9
12.8
62.
36.1
25.4
67.3
87.7
72.5
40.0
25.0
35.5
50.9
datasets. no rationales are available, we use those from ULTRAINTERACT. (2) Existing data only. (SNTERAT only. We evaluate with the same setting $ and report the averaged scores. See full results in Appendix
In Table EURUS outperforms the "Grouth- truth" model on all tasks, confirming the advan- tage of ULTRAINTERACT'S designs of divide- and-conquer and code-as-actie patterns, in line with conclusions concurrent work (Chen et al.) 2024b) Wang al., 2024). Training only on ex- isting data without INTRAINTERACT greatly hurts the reasoning performance, confirming the effectiveness of mance drop except for BBH, especially in instruction following. We attribute the performance drop to worse -fllowii ability. This suggests the necessity of mixing ULTRAINTERACT with other alignment data for better all-around supervised fine-tuning.
Table 7: Ablation study of SFT data.
Model
Coding
Math
BBH
IFEval
Avg.
EURUS-7B-SFT
44.9
58.5
64.6
44,0
53.6
Ground-truth
42.9
44.0
Existing Data Only
31.2
33.5
65.3
43.6
37.0
ULTRAINTERACT Only
56.2
174
Meanwhile, training only on UL.TRAINTERAC) suffers perfor-
RELATED WORK
Open LLMs in Reasoning, Open-source LLMs have shown remarkable progress in building specialists that excel in mathematics reasoning (Luo et al. 2023a Yue et al. 2023 Toshniwal et al.
2024) or coding abilities (Roziere et al. 2023 Wei
2023 Guo al. 2024a| Zheng al. 2024). On the contrary, mastering general reasoning capabilities still challenges open models, while the most advanced ones (DeepSeek-AI) 2024 Bai al., 2023 Touvron al. 2023 Jiang al., 2024) well
behind proprietary models. More, these cutting-edge open ceneral-purpose models maintain their alignment recipes confidential, which further hinders the replication and development of open-source reasoning models.
Preference Learning for Reasoning, Aligning language models from human or Al preferences has emerged as a prevalent approach in the open-source community (Tunstall et al. 2023 Bai et al. 2023) with the proposal of DPO (Rafailov et al., (202) and high-quality preference datasets (Cui et al. 2023
Zhu et al.) 2023). Different from open-domain chatbots, preference learning is largely underexplored in complex reasoning. Recent research showed performance degradation when applying DPO on reasoning tasks, but some newly proposed algorithms demonstrated positive effect (Ethayarajh ct al. 2024] Chen al. 20244 Mitra et al. 2024] Shao et al. 2024b)). However, deep understanding of preference learning, specifically its efficacy complex reasoning, is not yet established.
8 CONCLUSION
We strive to narrow the huge gap between open-source models and proprictary models from the perspective of alignment. Our work pushes the boundaries of open-source reasoning generalists by (1) releasing a high-quality multi-turn reasoming dataset ULTRAINTERACT with preference trees, (2) introducing EURUS-series LLMs which achieve SOTA challenging reasoning benchmarks and (3) providing insights on preference learning for reasoning through analysis, leading to new reward modeling objectives as well as powerful reward model for reasoning.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China (Grant No. 62106126), Young Elite Scientists Sponsorship Program by CAST (20230NRC001, National Natural Science Foundation of China (No. 62406165)
REFERENCES
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proc. of NAACL-HLT, 2019.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. ArXiv preprint, abs/2 08.7272.. 2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijic Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. ArXiv preprint, abs/2309.16609, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauma Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv preprint, abs/2204.0558 2022.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39, 1952.
Huayu Chen, Guande He, Hang Su, and Jun Zhu, Noise contrastive alignment of language models with explicit rewards. ArXiv preprint, abs/2402.05369, 2024a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel BerberttVosss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
Wenhu Chen, Ming Yin, Max W.F. Ku, Yixin Wan, Xueguang Ma, Jianyu Xu, Tony Xia, Xinyi Wang, and Pan Lu. Theoremqa: A theorem-driven question answering dataset. ArXiv preprint, abs/2305. 125224 2023.
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. volume abs/2403.1288 12881, 2024b.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñchiro Nakano, et al. Training verifiers to solve math word problems. volume abs/21 10. 14168, 2021.
11

---

Published as a conference paper at ICLR 2025
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ulrafeedback: Boosting language models with high-quality feedback. ArXiv preprint, abs/2310.01377, 2023.
DeepSeek-AI. Deepseek I1m: Scaling open-source language models with longtermism. ArXiv preprint, abs/2401.02990 2024.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Conference on Empirical Methods in Natural Language Processing, 2023.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. ArXiv preprint, abs/2402.01 2024.
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. In Proceedings of the International Conference on Machine Learning, 2023.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9, 2021.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming the rise of code intelligence. ArXiv preprint. abs/2401. 14196, 2024a.
Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Jiexin Wang, Huimin Chen, Bowen Sun, Ruobing Xie, Jie Zhou, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Controllable preference optimization: Toward controllable multi-objective alignment. ArXiv preprint, abs/2402. 2024b.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. ArXiv preprint, abs/2310.06882 2023a.
Albert Q Jiang, Alexandre Sablayrolles. Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L'elio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. 2024.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise ranking and generative fusion. In Annual Meeting of the Association for Computational Linguistics, 2023b.
Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hanna Hajishirzi. Rewardbench: Evaluating reward models for language modeling. 2024.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. ArXiv preprint, abs/2310.05477 2023a.
12

---

Published as a conference paper at ICLR 2025
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of Ilms as mathematical problem solvers. ArXiv preprint, abs/2402.192555, 2024.
Rongao Li, Jie Fu, Bo-Wen Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. volume abs/2312. 14852, 2023b.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition- code generation with alphacode. volume abs/2203.0788 04, 2022.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset gpt augmented flan reasoning traces. https://https::/ /OpenOrca) 2023.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? comprehensive study of automatic data selection in instruction tuning. 2023.
Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan
Catanzaro, Chatqa: Surpassing gpt-4 on conversational qa and rag. 2024. URL BRLlhtts:://pp
semanticscholede htt.:/aa....cc/cooo.aa.002000000
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. Proceedings of ICLR, 2023.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. ArXiv preprint, abs/2308.09583, 2023a.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardeoder: Empowering code large language models with evol-instruct, 2023b.
Meta. The Ilama herd of models. ArXiv, 2024.
Shen-yun Miao, Chao-Chun Liang, and Keh Su. diverse corpus for evaluating and developing English math word problem solvers. In Proc. of ACL, 2020.
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. NumGLUE: suite of fundamental yet challenging mathematical reasoning tasks. In Proc. of ACL, 2022.
Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agrawal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah. Orca 2: Teaching small language models how to reason. ArXiv preprint, abs/231
1.1 1..s/211.111 1045, 2023.
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. ArXiv preprint, 148300002.14800 2024.
OpenAl. Gpt-4 technical report, 2023.
Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proc. of ACL, 2015.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
13

---

Published as a conference paper at ICLR 2025
Cheng Qian, Chi Han, Yi Ren Fung. Yujia Qin, Zhiyuan Liu, and Heng Ji. Creator: Tool creation for disentangling abstract and concrete reasoning of large language models. In Conference on Empirical Methods in Natural Language Processing. 2023.
Yujia Qin, Shi Liang. Yining Ye, Kunlun Zhu, Lan Yan, Ya- Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Marc H. Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. ArXiv preprint, abs/2307.16789, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. ArXiv preprint, abs/2305. 18290, 2023.
Baptiste Roziere, Jonas Gehring. Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code Ilama: Open foundation models for code. ArXiv preprint, abs/2308. 12950, 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024a. URL
org/ abs / 2402. 03300
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv preprint, abs/2402.03300, 2024b.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. ArXiv preprint, abs/2210.09261, 2022.
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Git- man. Openmathinsstuct-I1 1: A 1.8 million math instruction tuning dataset. arXiv preprint arXiv: Arxiv-2402.10176, 2024.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Maric-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, 0tss/207..09285 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. ArXiv preprint, abs/2310.169444 2023.
Guan Wang, Sijie Cheng. Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data. ArXiv preprint, abs/2309. 1235,, 2023a.
Xingyao Wang, Zihan Wang. Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating Ilms in multi-tum interaction with tools and language feedback. ArXiv preprint, abs/2309. 10691, 2023b.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li. Hao Peng, and Heng Ji. Executable code actions elicit better Ilm agents. ArXiv preprint, abs/2402.01030, 2024.
14

---

Published as a conference paper at ICLR 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv preprint, abs/2201.11903, 2022.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need, 2023.
Martin Weyssow, Aton Kamanda, and Houari Sahraoui. Codeulrafeedhack: An llm-as-a-judge dataset for aligning large language models coding preferences. ArXiv preprint, abs/2403.09032, 2024.
Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang. Perils of self-feedback: Self-bias amplifies in large language models. ArXiv preprint, abs/2402.11436 2024.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpoiQA: A dataset for diverse, explainable multi-hop question answering. In Proc. of EMNLP, 2018.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: reading comprehension dataset requiring logical reasoning. In Proc. of ICLR, 2020.
Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi Ren Fung, Hao Peng, and Heng Ji. Craft: Customizing Ilms by creating and retrieving from specialized toolsets. ArXiv preprint, abs/2309. 100.17742, 2023.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. ArXiv preprint. abs/2309.05653, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt- and chatbot arena. ArXiv preprint, abs/2306.05685, 2023.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencoddeinterprete:: Integrating code generation with execution and refinement. ArXiv preprint, hass22402.146553 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followin evaluation for large language models. ArXiv preprint, abs/2311.079111, 2023.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7bs 7b: Improving Ilm helpfulness & harmlessness with rlaif, 2023.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang. Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Decpseck-coder--2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint 11..::22200.119111, 2024.
15

---

Published as a conference paper at ICLR 2025
Table 8: ULTRAINTERACT covers diverse set of datasets spanning three tasks.
Task
Datasets
GSM8K Cobbe al.
2021). MATH (Henndyykks et al. 2021b MathQA (Amini al. 2019 NumGlue
Math
Mishra al. 2022 TabMWP (Lu al. 2023
CodeContest Li et al. 2022]. TACO (Li et al. 2023b) WikiTaabeQQussionss (Pasupat Liang 2015
Coding
Magicoder- Instruct (Luo et al. 2023b Wei et al.
2023
Logic
ReClor Yuu al.
2020. HotpotQA (Yang et al.
2018
Strategy QA Geva et al.
2021
A ADDITIONAL DETAILS IN ULTRAINTERACT CONSTRUCTION
A.1 DATASET DETAILS
Math. We adopt GSM8K (Cobbe et al. 2021), MATH (Hendrycks et al. 202 MathQA (Amini al. ). and NumGLUE (Mishra et al. 2022 mathematic reasoning, and include TabMWP (Lu et al. 2023) for tabular processing. We retain all the instructions for all datasets except MathQA, NumGLUE, and TabMWP. MathQA divides problems into different categories according to the topics and annotates the formula that indicates the pattern needed to solve each problem. We apply stratified sampling to sample at most five problems for each pattern and prioritize the problems that come from the long-tail -ail category. Numglue contains eight different reasoning tasks and discard Task 5 (Reading Comprehension + Explicit Numerical Reasoning), Task 6 (Reading Comprehension + Implicit Numerical Reasoning), and Task 7 (Quantitative NLI) due to the simplicity Mishra et al. (2022). For TabMWP, we only keep the questions with difficulty levels 4 and since the rest are too easy for current state-of-the-as tftt models.
Code. We focus on programming with Python for the simplicity of integration of the interpreter. We use CodeContest (Li al. 2022) and TACO (Li et al., 2023b), two competition-level coding datasets collected from various online platforms. We filter out the overlapped questions. Note that part of the questions in TACO only contain ground- truth solutions and do not contain test cases for evaluation, hence we apply GPT-4 to generate 12 test case inputs (4 basic inputs, 4 edge cases, and 4 large numbers) for each question and then execute the ground-truth solution snippets to produce outputs. Given that the two datasets mainly focus competition problems that may deviate from real-world daily uses, we exclusively adopt Magicoder-Bi Evol--rrru--- (Luo et al. 2023b Wei et al.) 2023), the only dataset in our selection that does not contain test cases or ground- -ruth solutions. We employ GPT-4 Turbo to judge the correctness of generated code during interaction, and therefore we do not use this dataset for preference learning since we cannot rigorously construct pairs of correct and incorrect actions limited by the evaluation reliability. We also include Queuutins (Pasupat & Liang 2015) for table processing with code.
Logical Reasoning. we use the multi-hop reasoning datasets HotpotQA (Yang et al. 2018) and
StrategyQA QA (Geva et al.] [222]) and the logical reasoning dataset ReClor Yuu et al. 2020). We
follow the setting of Wang et al. (2023b) and convert HotpotQA to a generation task, removing the contexts and requiring LLMM to search relevant information using Wikipedia API.
A.2 DETAILS ON PREFERENCE TREE CONSTRUCTION
Models Adopted for Incorrect Action Sampling. We randomly sample one model from Mistral-7B- Instruct-v0.2, DeepSeek-Cooe Coleek---oddr-33BB- -onstrcc,, Mixtral-8,, -x--ss--ttt-nsttuuttt and DeepSeck-LLMM-6BBBOO to generate one incorrect action to pair with each correct one.
Correct Action Generation Based on Ground Truth Annotations.
We adopt GPT-3.5 Turbo as the generator to generate correct actions based on ground truth considering abilityy We provide different access to the ground truth information for
the instruction-followe different tasks, specifically: (I) For coding, where test cases are black boxes to reference solutions, we provide full access to the solution codes. The actor model will add step marks and corresponding explanations to the ground-trun code to make it easier to understand, or further refine the code for optimization. (2) For tool- frreffeeeeeeeeeeeeee math problems, avoid the actor model directly copying the answers to pass the correctness checking, we mask the answer numbers in the rationale before providing to L.LMs. This approach can better ensure response quality since encourages LLMSs to generate
16

---

Published as a conference paper at ICLR 2025
Table 9: Stats breakdown
w/ Tool?
Prompis
Answers.
Avg. Length
Anuotation
Has Answer?
Has Rationale?
GSMEK
x
4,522
7,257
10.277
15,752
17,392
1,746.7
1,189.0
Math
X
7,471
25.765
36,005
1.735.0
MuthQA
7,552
15,079
20,328
2,338.5
7,189
17,743
22,500
1.966.3
NEmGLUE
3,020
3
5,717
1,474.6
X
2,835
2,987
4,273
1.056.1
3,117
4,135
CodeContest
2,06177
Coding
TACO
9,016
50,877
58,191
2,143.5
WiliTableeuestionss
1,401
1,544
1.738
1.794.8
Magiccndar-Bool-Insste
10,374
10,238
687.1
Reclor
4,467
7,958
7,231
1,266.7
Logio
HorpotQA
1,182
1,009
1,230
1,333.2
StratogQQA
904
741
968
1,256.2
responses with complete reasoning chains with each step clearly marked. (3) For program-enhanced math reasoning, we first translate the textual rationale into code. Then, we either directly provide the actor model to generate plans, ask the actor model to convert the code into modularization programming and then make plans to create tools to solve problems.
A.3 DATA DECOMTAMINATION
We conduct careful decontamination. Firstly, for LeetCode, we apply the Exact Substring Matching Algorithn to compare with each instruction in the ULTRAINTERACT and find no overlaps. For others, we perform 8-gram exact matching to compare ULTRAINTERACT instructions with test sets of the same task. We remove those instructions that overlap 8 grams with any test sample.
A.4 DETAILED STATISTICS
In total, ULTRAINTERACT has 86K instructions and 220K action pairs. The Total Pairs does not equal Total # Turns in ULTRAINTERACT. since we fail generate sufficient correct actions for every incorrect action in multi-tury turn trajectories mainly due to lack of sufficient ground truth annotations. The total pairs may equal # correct answers, either, because it also difficult and unnecessary to sample incorrect actions for the correct ones for some simple instructions. We present the specific information for each dataset. In particular, we list information on human annotation in each dataset, which plays an important role in correct action generation ($2.3 and Appendix (A.2). All three steps of correct action sampling methods mentioned $2.3 can be applied to datasets that have rationales,
while for datasets only containing answers, only the first two steps applicable. We do not apply any of the three-step methods to generate correct answers for Magicoder, the only dataset without any human annotation, to construct preference pairs.
B ADDITIONAL. DETAILS TRAINING EURUS MODELS
Supervised Fine- Tuning. We finetune base models for 1 epoch with 2e-5 learning rate and 0.1
warmup ratio using a cosine scheduler. For EURUS-7B, B, we mix 32K UltraChat (Ding et al. 2023). 30K ShareGP 3 and 50K OpenOrca([ian al. 2023). For EURUS-70B, we mix 63K UltraChat, 30K ShareGPT, and 70K OpenOrca. For LLAMA-3 -LLAA---EEUUUS-SE EURUS-8B, we mix 50k UltraChat, 18k OpenChat
(Wang al. 2023a). TheoremQA decomtaminated OpenHermes and the entire ChatQA (Liu et al.
2024 and CodeAct (Wang et al, 2024).
decontaminatioon

---

Published as a conference paper at ICLR 2025
Preference Learning. For hyperparameters, all [ is set to 0.1, and -+/X_ in KTO is set to 1.33 as recommended. We finetune models for epoch with a 5e-7 learning rate and 0.1 warmup ratio using cosine scheduler.
Reward Modeling. We train RM for epoch with Ir=1e-5 learning rate. We also use a cosine scheduler with warmup ratio of 0.1.
Regarding pair augmentation, we scale up the pairs by matching every correct action for each instruction with incorrect action of other turns. This leads to NxN pairs of single-turn actions for a trajectory of depth N. We remove the action pairs consisting of nodes at the same turn, as they are already part of the multi-turn trajectory pairs we included. Next, avoid overfitting on the training set, we only select instructions with NxN < 10, and for these instructions, we randomly sample at most 9 pairs with each action occurring no more than times. This leads to an augmentation of 240k single-turn action pairs.
C ADDITIONAL EVALUATION RESULTS OF EURUS
Table 10: MMLU and MTT-Bench
Detailed Setup in $ For math, we test both textual rea-
soning and program-enhanced settings and report the best performance the two. All evaluations are conducted 0-shot CoT with two exceptions: BBH uses shots and IFEval does not use COT. For MINT, we select MATH, TheoremQA, and MMLU-math from "reasoning" as new
Model
MMLU
MT-Bench
Mistral-7B-Imttrudd--1..22
58.9
7,60
Zephyr-7B-8
7,34
OpenChat-355-1100
63.4
7.81
Starling-LM-7BB-
64.0
8,09
"math" split. We also evaluate 5-shot MMLU (Hendrycks al. 2021a) for STEM knowledge and MT-Bench (Zheng et al. 2023) for conversation abilities to study whether
Magloode---DS-6.79
37.1
4.21
OpenCI-DS-6.78
37.2
4,06
MAmmo TH-7t-Mistral
56.2
4.25
WirardMath-7B.r1..
60.3
5.62
OpenMath-Mistral-P7
58.3
2.69
EURUS needs to trade off other capabilities for reasoning.
EURUS-7B-SFT
7.15
DPO
62.4
7.38
Results. Results are shown in Table TO
KTO
62.2
7,38
NCA
62.2
7.38
On MMLU, EURUS outperforms baselines dedicated LLAMA-3-EURUS-BUUC coding and math, and achieves comparable or higher results than Mistral-Instrrut--00.2 and Mixtral-8x22B- Instruct-v0.1, the official aligned versions of our base model built by their authors. Compared to general- purpose baseline models, EURUS-7B and LLAAM-3- EURUS-8B achieve comparable performance with the top-performance OpenChat and Starling- LM, and EURUX- 8x22B also matches the top level of performance among other general-purpose models.
64.6
6.82
KTO
64.9
7.27
NCA
64.7
7.14
408
Mixtral-&<7B-IInste Instrrctt DeepSedl-Coderr33B--
70.3
8.30
40.2
3,83
~000
CodeLLaMA-7UBS -ant--B----struucc
55.1
5.12
Begsseek-LM-67B-hhaa
72.3
8.08
OWenL.5-78B-Chat
72.9
OpenCI-CLL-708
52.4
5.67
OpenMatl--CL-708
2.29
WizardLMM-2-w223B
77.0
9.08
On MT-Bench, we report baseline numbers from the offi- cial leaderboard if available. EURUS matches the perfor- mance of mainstream open-source general-purpose mod- els, and EURUX-8X22H further surpasses the score of
Blktraa--k22B--Isstuutt---
Instruct-v0.1
77.6
8.66
EURUX-8X22B-SF7
KTO
75.9
8,58
NCA
75.6
8.46
Proprictary Models
GPT-3.5 Turbo.
GPT-3.5 Turbo
70.0 86.4
7.94
GFT4
8.96
Reward Pattern on LLAMA-3- LLAMA-3-EURUS-SBE The reward patterns when performing preference fine-tuning to train CAA-MA----UUS---- are shown in Figure The conclusions are consistent with ECRUS-7B Figure
D DETAILED RESULTS ON REWARD MODELING
D.I ADDITIONAL RESULTS ON RERANKING
We present the full results reranking in Table 11 where the conclusions consistent with those drawn from $D (1) Our reward models always achieve the highest accuracy on all test sets across different N, except when N=2 Humanliyal. (2) Both LeT and LDR consistently help
httts::/hugginggeeeeh co/ wttp:/ccccppcpeccccec
arena- 1eaderboaa
18

---

Published as a conference paper at ICLR 2025
Table 11: Detailed results of reranking Mistral-Instruct--ve respooossssssssssssssss responses on coding and math.
Datasets
HvmanEval
MBPP
GSMSK
MATH
N
16
2
16
16
2
4
16
Random
41.5
39.0
40.2
39.6
33.1
33.6
34.3
30.1
45.0
43.
44.5
40.2
11.5
11.3
10.0
8.5
Top Logits
43.3
43.3
43.3
43.3
35.3
35.3
45.7
45.7
12.1
12.1
12.1
12.1
Self-Conistenny
43.3
42.7
42.1
40.9
35.3
36.3
36,6
37.1
45.7
49.5
52.2
52.8
12.1
13.8
15.8
16.8
Starling RM 34B
47.6
47.0
49.4
45.7
37.8
38.8
39.6
40.4
49.1
52.8
56.0
$6.5
6.5
7.2
7.7
7.7
EURUS -M-U-
45.7
47.6
47.0
42.6
43.4
43.9
56.3
57.3
16.2
wlo CoR
45.7
44.5
46.3
50.0
39.3
42.4
42.4
42.1
49.4
53.2
55.4
56.3
14.2
16.1
17.0
16.9
wlo Cgr
47.0
48.2
38.6
57.8
16.3
17.2
17.1
w/o US
45.7
47.0
49.4
50.6
39,3
41.1
41.4
42.9
49.4
53.8
57.4
58.7
14.5
16.6
17.2
17.5
w/o UF US
43.9
43.3
47.0
46.3
36.3
38.1
36.6
35.3
49.4
52.3
54.6
57.2
14.3
16.5
17.4
17.4
Pass@N
73.8
88.4
92.7
42.4
48.1
52.6
73.2
80.4
22.7
28.9
35.5
improve reranking performance on three test sets except for HumanEval, where removing either of the objectives can prevent the accuracy from dropping when increasing N from 8 to 16. (3) Modeling safery hurts reranking performance in reasoning. When removing UltraSafety from the training data, the RM achieves higher accuracies than SURUS-RM-PT except MBPP.
E STATISTICS OF ULTRAINTERACTT-V
We use EURUS- Brruus-BBKYY 7B-KTO,
Baama----/7OBBInsttuue DeepSeek MarsSeek-Chht---ee V2 Zhu et al. (2024), SeepSeek-Coder. V2 (Guo et al. (002a)). DeepSeek-Math- Shao et al. (2024a), etc, to construct the ULTRAINTERACT--V. Particularly, the proportion of EURUS responses is 26.4% in the SFT split and 28. 2% in the preference split, while the Llama responses take the share of 59.0% and 56. 7% in the SFT split and preference split respectively. We present the detailed statistics of ULTRAINTERACT in Table ( and [3]
Table 12: Some statistics of ULTRAINTERACT-22
Task
Type
Instructions
Turns per Traj.
Tokens Traj.
Avg. + TTaj per Ins.
Total Pairs
# Carrect Answers
w/ Internction?
w/ Tool?
T1
T2
T3
T4
TS
20,935
8,018
6,149
2,413
1,991
3,264
1,374.2
1.0
61,738
M0.982
Math
1,279
6,723
x
21,474
10,117
4,707
1,263
4,798
1,155.7
1.0
72,944
143,587
634
4,876
396.0
7.7
2,172
4,876
Coding
9,498
4,323
2,057
942
459
1,717
1,429.4
1.0
71,900
155,429
Total
56.697
43,881
12,913
4.688
2,139
9.779
894.9
3.2
216,799
410.421
Table 13: Stats breakdown for ULTRAINTERACT--2
Task
Dataset
Tool?
Prompts
Pairs
Correct Answers.
Avg. Length
Human Annotation
Has Answer?
Has Rationale?
4,438
14,616
17,797
642.4
GSMBK
X
6,997
10,708
11,893
367.6
6,197
20,056
27.217
681.8
Math
MATH
X
6,799
46,720
115,924
531.7
/
6,046
22,062
MathQA
X
5,898
14,525
16.237
NamGLUE
2,436
4,241
9,113
570.4
X
2,414
3,163
4,409
409.0
X
TabMWP
3,097
5,565
11,516
507.1
X
CodeContest
5,788
39.713
76,272
1,291.7
Coding
TACO
6,459
37,317
86,844
1,062.9
Codeforzes
128
860
2,137
1,480.9
19

---

Published as a conference paper at ICLR 2025
Table 14: Ablation Study.
Model
Coding
Math
Reusoning
Ins Following
Avg.
HumamFval
(SMRKK
ThoraniQA
SVAMP
[FEval
EURLS-7B-SFT
Ground-Truth
46.3
16.4
62.2
75.1
68.8
64.0
42.9
41.0
Exisiting Data Only
38.4
44.1
11.1
45.3
10.8
9.3
527
49.4
65.3
43.6
37.0
ULTRAINTERACT Only
46.3
50.1
15.6
67.6
3099
20.1
80.4
82.0
67.0
17.4
47.7
F DETAILED ABLATION RESULTS
We present the full results of 7 Table 14
with detailed metrics on all coding and math datasets.
G TREE STRUCTURE BENEFITS MULTI-TURN INTERACTION ABILITY
To further demonstrate the effectiveness of the tree structure in ULTRAINTERACT, we decomposed multi-turn tree into multiple single-turn pairs and trained LLAMA-3 1- --RA-----EUUSSDD pairwise data. Results can found Table 15 Compared to training on single-tury pairs, training on multi-turn trees enjoys huge benefits multi-tur interaction ability and slightly improves the overall performance.
single-turn
Table 15: Model performance when trained on multi-turn and single-turn pairs from ULTRAINTER- ACT.
Model
Cuding
Math
Reasuning
Imo Fallowing
Multi-Turs
Ang.
Hamaniival
MBPY
LoctCidde
USMMA
MATH
Theoren1AA
SVAMP
ASDrr
BJl (CoT)
Code
Math
Liama Earus 88 SPT
51.2
57.9
17.2
50.7
32.0
21.3
82.2
83.7
72.4
47.1
18.4
24.5
45.6
(Siegle
71.7
(Singl:
14.4
23.1
48.1
NCA
NCA (Suugl Tura)
55.9
72.2
H THE ABSOLUTE VALUE OF REWARDS Do NoT MATTER MUCH ON GENERAL CONVERSATIONS
In this section, we verify our intuition in that the absolute value of rewards primarily matters in reasoning. We conducted an additional experiment on UltraFeedback (one pair per instruction) with LBT and LULTRAINTERACT (CDR + LBT) respectively. Results in Table I6 show that LDR does not help improve the performance of the reward model on general chat data, indicating that the absolute value of rewards is not important as it is reasoning tasks. This aligns with our intuition of only applying Los to UltraInteract examples.
Table 16: Ablating CDR on general conversations.
Objective
Chat
Chat Hard
Reasoning
Safety
LwT
94.5
44.1 36.0
56.5
52.9
CDR CBT
92.8
45.6
43.0
ABLATION STUDY ON PREFERENCE DATA MIXTURE
We present the ablation study on prefernce data mixture in this section. We perform KTO on Llama- 3-Eurus-8B-SC with different combinations of UltraFeedback and Ultranteract and the results are presented Table 17 From the results, we see that training only on UltraInteract leads to higher overall reasoning performances, which are mainly credited the multi-tum interaction ability, demonstrating the superiority the tree structure of our data. However, we also observe a lower MT-Bench score compared training solely Ulrraeeddack. Nevertheless, this be mitigated
20

---

Published as a conference paper at ICLR 2025
3-Eurus
7.5
5.0
2.5
Value
Value
-5.0
Margins
Margins Rewards/Roeected
Margins Rewarde Rejested wwpht
-7.5
RewerdafRejected War
10.0
Steps (%)
Steps (%)
Steps (%)
Figure 7: Reward patterns of LLAMA-3- --AN-----UUSSB--- preference learning with DPO, KTO, and NCA. Conclusions are consistent with Figure
without hurting reasoning performances by mixing these two datasets together, which indicates that our data is compatible with other datasets, consistent with our conclusions reward modeling.
Table 17: Additional experiment results preference learning with different compositions of UltraFeedback and Ultralnteract.
Model
Coding
Math
BBH
IvEal
Multi-Turn
Avg.
MT-Bench
42.11
53.97
72.40
47.10
21.45
46.60
6.8
+KTO (UF + U1)
41.84
56.14
71.70
50.60
31.92
49,40
7.3
+KOO (Only UF)
44.60
56.26
72.00
50.10
24.03
48.80
7.5
+KTO (Only UT)
40.70
55.86
71.70
50.60
34.49
49.40
7.2
J DATA EXAMPLE
We present an example for SFT data in Table
and an example for preference learning in Table 19
21

---

Published as a conference paper at ICLR 2025
Table 18: An example from the SFT data.
Task: Coding
Instruction: Write Python code to solve the task.
Mike has a string consisting only lowercase English letters. wants change exactly one character from the string so that the resulting one is palindrome. A palindrome is string that reads the same backward as forward, for example strings z, aaa, aba, abecba are palindromes, but strings codeforces, reality, ab are not.
Input
The first and single line contains string Output
Print YES (without quotes) if Mike can change exactly one character so that resulting string palindrome or NO (without quotes) otherwise.
Response:
Step I: This problem asks to determine if is possible change exactly character in the string make palindrome.
Step We can start by iterating through the characters the string and counting number differences between corresponding characters from the beginning and Step 3: If the length of the string is add and there is only one difference, or the length is even and there is exactly one difference, then it is possible to change exactly one character to make the string palindrome. Step 4: We need to consider edge cases such as single character string empty string. "'python Step 5: Take the input string input()
string.
Step 6: Calculate the length of the string
= len(s)
Step 7: Initialize counter to keep track of the number of differences
=
Step 8: Iterate through the first half of the string and count the numher of differences hetween corresponding characters from beginning and end of the string. for in range(n//2):
if sfi] 1= s(n-i-1]
enl +=
Step 9: If the length of the string is odd and there are no differences, or if the length is even and there is exactly one difference, print "YES" (without quotes)
and ent == 0) or (ent == 1):
print( YES") Step 10: Otherwise, print "NO" (without quotes)
else:
print("NO")
22

---

Published as a conference paper at ICLR 2025
Table 19: An example from the Preference data.
Mask:Math
Trajectory:
I{
from: user,
value: Tool available:[1] Python interpreter. When you send message containing Python code
python, it will be executed a stateful Jupyter notebook environment. Solve the following math problem step-hy-step. Simplify your answer as much as possible. Simplify (-3 - 21) (1 + 46).
from: assistant,
41). we need distribute negative sign to both terms inside the parentheses. Step 2: We can then combine like terms by adding the real parts and the imaginary parts separately. # Step 3: Distribute the negative sign to both terms inside the parentheses. (-3)-
value: Step I: To simplify expression (-3 2/)
(-1) + (-2i)- (-4i) # Step 4: Combine like terms by adding the real parts and the imaginary parts separately.
3 ++1+ 4i)
from: user,
value: Out: Cell In[1], line 2 (-3) (-33)((11) (-1)+ (-2i) (-4i) SyntaxError: invalid syntax. Your answer is wrong. Your thought process for simplifying the given complex numbers is correct. However, you made syntax explicitly written out with asterisk (*).
in the code. In Python, the multiplication operation even complex numbers. Remember, the imaginary unit in Python denoted as and not Try correcting syntax and calculate again.
from: assistant,
value: Step 1: To simplify the expression (-3 - -1 -1+++3 46). need distribute the negative sign to both terms inside the parentheses. Step We can then combine like terms hy adding the real parts and the imaginary parts separately. Step 3: Distribute negative sign to both terms inside the parentheses. (-3)-
(-1) + (-2)) (-4j) Step Combine like terms by adding the real parts and the imaginary parts separately.
-3
}.
from: user,
value: Out: (-2+2j). Your answer is wrong. Your thought process for simplifying the given complex numbers is on the right track. However, you made an arithmetic error when adding the real parts and the imaginary parts. You should have combined the like terms correctly to get the right answer -4 and -6j respectively. Please try again with the correct arithmetic calculations.
}
Chosen:
Step 1: Split the expression into real and imaginary terms.
Step 2: Subtract the real terms from each other.
Step 3: Subtract the imaginary terms from each other.
Step 4: Combine the results of the subtraction get the simplified form of the expression.
Step 5: Split the expression into real and imaginary parts and then subtract them.
real part= parts
imaginary-part=
Step 6: Combine the real and imaginary parts to get the result.
result = ppart Banaulaapat(((mgnaarrrrr print(result)
(imaginary part:+d]i
Rejected:
Step 1: To simplify expression (-3 - 21) (1 41). we need to distribute the negative sign to both terms inside the parentheses.
Step 2: We can then combine like terms by adding the real parts and the imaginary parts separately.
# Step 3: Distribute the negative sign both terms inside the parentheses.
(-3) 1- (-2j)- 4j
Step 4: Combine like terms by adding the real parts and the imaginary parts separately.
2j
23


Reviews:
[{"review_id": "DSaOq8Zmac", "reviewer": "Reviewer_xgCx", "summary": "The authors explore improving large language model reasoning through the curation of high quality training data for that reasoning.\nThis data (UltraInteract) consists in preference trees, with nodes splitting on correct/incorrect responses; critique and refinement of rejected responses; and uses different reasoning schemas/ actor models to increase training data diversity\nThe actor used to generate these trajectories is GPT3.5 Turbo, with GPT4 being used as a critique model with access to an interpreter/tools.\n\nThe authors then use this dataset (alongside others) to finetune 3 language models using the following process:\n1. SFT over the correct actions\n2. Preference learning over correct vs incorrect actions using off the shelf preference learning algorithms\n\nAdditionally the authors also use this to derive a reward model:\n3. Train a reward model, adding in terms for the difference in absolute rewards to the normal Bradley Terrey reward model.\n\nIn my view the key contributions of this paper are:\n* introduction and analysis of preference-tree based instruction following data, which is scalable and effective\n* introduction of improved objectives for training reward models", "questions": "1. Did you conduct any comparative investigations over general conversational preference learning using your reward modelling objective? This would help to verify your intuition that this method is effective due to the unique features of reasoning tasks\n2. Would it be possible to use the Eurus reward model for PPO-based alignment? How would this perform in comparison to the existing finetuning methods", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. With regards to soundness, I feel that the necessary experiments have been run to validate the majority of claims, especially where those claims are with regards to methodological contributions. The authors have also taken pains to remove contaminated data from their work in order to make comparisons fair and meaningful, including when reporting others' work.\n2. The presented language models have strong performance, and the data and reward models are in and of themselves useful contributions to the research community, removing some of the limitations of scale and quality from prior works creating preference datasets and reward models\n3. The investigation surrounding the flaws of existing preference learning models is an original contribution.\n4. In my view the largest contribution is the rather detailed study of creating their ultra-instruct dataset albeit moreso as an engineering challenge.\n5. The experiments are run against meaningful baselines: models of similar scale, trained on similar data in similar ways.", "weaknesses": "1. As a minor point the spelling and grammar could be improved; for instance \"Is proprietary models\" (line 470) should be \"Are proprietary models\", and more generally things like \"Perference Learning\" (line 247). More substantially some of the references point to the wrong sections (e.g. the reference to section 5 (replaced with 6) (line 255) -- in this case harming readability (hence my review of the presentation...)\n2. I feel that the modification to the reward model could be better motivated in section 3, for instance by referencing other works that maximise a similar margin loss. At the least it should be explicitly linked to the discussion in section 4.2 that actually seems to motivate it. This might be aided to seperating out the reward modelling section from the finetuning section? Since it seems to follow on more logically from the finetuning investigations\n3. Section 6.1 doesn't really address the section title properly. While the performance itself does suggest that just training on open source data is sufficient (ignoring the instruction following benchmark); the body of the section just talks about mixing in this additional V2 data, and the ensuing performance gains. It would suffice to add a brief comment at the end of line 483 explaining the results of finetuning just on V2\n4. As a general comment I feel that this work feels like three distinct pieces of work rather than a single cohesive one. I.e. the proposal of a new training dataset; a set of models finetuned on this dataset alongside others; and more separetely a reward model trained on a combination of dataset including the one proposed here. One way of mitigating this would be to focus on the contribution of the dataset to the reward modelling phase (using the data from the ablation studies).\n5. Section 2. is a little bit confusing and could be rephrased to make it a little but clearer that it is all just an example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "970fgrslJY", "reviewer": "Reviewer_rmjd", "summary": "The authors emphasize the performance gap between open-source LLMs and the most advanced models, particularly in reasoning capabilities. They attribute this gap to two primary factors: (1) the lack of high-quality datasets and (2) the under-exploration of preference learning techniques. To address this gap, the authors introduce a novel dataset, ULTRAINTERACT, which features a multi-turn, tree-structured format designed to enhance reasoning abilities. Additionally, they offer new insights into preference algorithms and reward modeling. They argue that effective reward modeling should consider not only the margin between rewards but also the absolute value of the reward itself.  Based on this insight, they propose a new reward model that combines two loss functions, L_{BT} and L_{DR}, demonstrating superior performance compared to existing models.", "questions": "If the model is unable to effectively learn from vertical improvements, then it raises the question of why we want to synthesize the dataset with tree structure and why we are providing trajectories to the model.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Authors use a new method to synthesize a dataset for SFT and preference learning, which could potentially enhance model's reasoning abilities. The intuition behind the synthesis method is straightforward and easy to be understood. I think the dataset is cool and it could be a potential approach for model to learn how to improve the response. Plus, the insights on preference learning algorithm is interesting.", "weaknesses": "1). I agree that providing trajectories to guide model improvements is a potential approach. However, during the training process, I believe that the vertical improvement information, sequential refinement across turns, may not be effectively learned. This is because current preference algorithms primarily focus on horizontal comparisons, assessing responses within the same turn. \n\n2). The reasons behind the better performance of EURES are hard to track and some studies will be necessary if authors want to claim that the proposed dataset is the reason. Because the baselines has different scales and training method, for example, their training dataset could have different size and their preference algorithm could be different, etc.. Plus if EURES can beat some larger model, the claim that the dataset is better will be more convincing.\n\n3). There may be some factors contributing to the value differences observed in reward modeling, especially given the varying formulations of alignment methods. It would be valuable for the authors to offer insights into the potential reasons for these differences in the value of rewards.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ljfziG1IvC", "reviewer": "Reviewer_EWRx", "summary": "This paper has several contributions. First, it builds a dataset on reasoning tasks that contain both correct and wrong steps. Second, it proposed a modified loss function for training a reward model that is better suited for reasoning tasks. Lastly, it trains a set of LLMs using the proposed dataset that have competitive performance on reasoning tasks.", "questions": "- Is the reward model used in training the actor model?\n- L148 “the actor model first decomposes the input problem into several problems” How is this done?\n- L181 “we adopt more diverse reasoning patterns” How exactly is this done?\n- Is python the only tool used?\n- Typo in L263 reward notation\n- What is \"prompt level loose score\" in L282\n- I think the tables have too many numbers in them (tab3 has at least a hundred) and not sure if anyone will look at all of them. Instead, average scores can be put there and the detailed table can move to the appendix. This is only a suggestion though.\n- Which GPT-4 model is used? I think there are multiple versions. \n- How is the reward model performance compared to ArmoRM?\n- How is GPT-4 used as a reward model in tab4?\n- Why does self-consistency drop in fig1 left?\n- How is MCTS decoding done exactly in sec5.2?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper is advancing open science by making the training data and model checkpoints public. Given the significant improvements in reasoning tasks, it is likely that these assets will be helpful to other researchers.\n- The paper also proposes a new way of training reward models that is better suited to reasoning tasks. In addition, the training datasets have multi-step attempts that contain mistakes and tool usage, which is unlike other preference datasets.\n- The experimental section is detailed and provides many interesting results, such as comparing three different preference optimization methods. There are many ablations provided, and evaluations are done on many tasks, which makes the results more convincing.", "weaknesses": "- The heavy reliance on GPT responses makes me feel like this is more of distilling GPT. Also, it is not clear what are the usage limitations that will arise from using a proprietary model like GPT4. As shown in tab7, this was crucial for obtaining good performance.\n- The problem of the likelihood of chosen responses going down in reasoning is a known issue and studied prior work [1], which is not cited in the paper (the related work is quite short)\n- The term “multi-turn action” was confusing. It seems that all the tasks require only a single correct response. None of the tasks is truly multi-turn where the model has to do multiple actions. From reading the paper, it seems the term “multi-turn” is used to describe a process where a model can try again if it makes a mistake. Actually, it is not clear how this process works, especially when training the model and evaluating it. Also, the dataset contains observations and judgements, but are they also used when training the actor? What about the python executions? There is very little detail on how the agent is trained on these and evaluated. \n- As mentioned in the previous point, there are certain steps that are not well explained. See the questions for examples. Given that the motivation is to advance open-source LLMs, I think it is important to describe the process of training in more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jFPQokGPMk", "reviewer": "Reviewer_rVp1", "summary": "This paper presents EURUS, a new collection of large language models (LLMs) and a reward model (RM) designed to enhance reasoning capabilities. The authors develop ULTRAINTERACT, a dataset designed for complex reasoning tasks with 12 datasets spanning math, coding, and logical reasoning problems. ULTRAINTERACT employs preference trees, which pair each instruction with reasoning chains, interaction trajectories with feedback, and pairwise responses for preference learning.\n\nThe authors use ULTRAINTERACT to fine-tune several open-source LLMs, including Mistral-7B, Llama-3, and Mixtral-8x22B. They show that EURUS models achieve top performance on multiple reasoning benchmarks, including LeetCode and TheoremQA. EURUS-7B and LLAMA-3-EURUS-8B even surpass baselines 5 times their size, while EURUX-8X22B outperforms GPT-3.5 Turbo on 12 test sets.\n\nThey also create a reward model, EURUS-RM-7B, that excels on several reward modeling benchmarks and introduce a new reward modeling objective that merges the Bradley-Terry objective with an additional term to directly adjust the reward of chosen and rejected action", "questions": "1. In line 90-91, the statement is unclear to me. In 'a higher final reward often indicates a better reasoning capability', whose reasoning capability? Can you elaborate a bit more?\n\n2. About the result remove in Table 3 due to data contamination. For some of the model has data contamination issue, the table suggests the TheoryQA is leaked, what about the rest dataset? If the rest doesn't has data contamination issue, should the result be compared? Without TheoryQA number, OpenChat seems like still a strong candidate.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel dataset, ULTRAINTERACT, designed for complex reasoning tasks. It comprises instructions paired with preference trees, featuring reasoning chains, multi-turn interaction trajectories with feedback, and pairwise positive and negative responses. ULTRAINTERACT emphasizes complex and diverse reasoning patterns, encouraging models to break down problems into sub-problems and use tools to solve them. This dataset is a valuable contribution and can be useful for future research on LLM reasoning.\n\n2. The proposed EURUS models achieve state-of-the-art performance on several reasoning benchmarks, demonstrating the effectiveness of ULTRAINTERACT and the proposed training methods. Notably, the smaller EURUS models outperform much larger baselines, showcasing their efficiency.\n\n3. The paper provides valuable insights into preference learning for reasoning tasks. The analysis of reward patterns during training leads to a new reward modeling objective that improves performance, particularly on challenging problems. The authors highlight the importance of the absolute value of rewards in preference learning for reasoning, as opposed to just focusing on relative differences as in general conversation settings.", "weaknesses": "1. While the authors acknowledge the use of proprietary GPT models in data synthesis, they do not thoroughly analyze the limitations of relying on these models. It would be helpful to discuss the potential biases introduced by GPT models and explore alternative approaches for data generation that rely solely on open-source models. Though, it's worth noting that they attempt to address this by creating ULTRAINTERACT-v2 using only open-source models, which shows promising results.\n\n2. In the paper, a few preference learning algorithms, since the preference pairs are collected in the ULTRAINTERACT, not running RL with the data seems like a big miss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)