Tulu 3: Pushing Frontiers in Open Language Model Post-Training
Authors: Nathan Lambert
Jacob Morrison
Valentina Pyatkin
Shengyi Huang
Hamish Ivison
Faeze Brahman
Lester James Validad Miranda
Alisa Liu
Nouha Dziri
Xinxi Lyu
Yuling Gu
Saumya Malik
Victoria Graf
Jena D. Hwang
Jiangjiang Yang
Ronan Le Bras
Oyvind Tafjord
Christopher Wilhelm
Luca Soldaini
Noah A. Smith
Yizhong Wang
Pradeep Dasigi
Hannaneh Hajishirzi
Venue: COLM 2025 (2025)
Keywords: post-training
reinforcement learning
preference learning
TL;DR: A new multi-stage post-training recipe scaling preference learning and reinforcement learning with verifiable rewards.

Abstract: Language model post-training is applied to refine behaviors and unlock
new skills across a wide range of language models, but open recipes for
applying these techniques lag behind proprietary ones. The underlying
training data and recipes for post-training are simultaneously the most im-
portant pieces of the puzzle and the portion with the least transparency. To
bridge this gap, we introduce TÜLU 3, a family of fully-open state-of-the-art
post-trained models, alongside its data, code, and training recipes, serving
as a comprehensive guide for modern post-training techniques. TÜLU 3,
which builds on Llama 3.1 base models at 8B, 70B and 405B parameters,
achieves results surpassing the instruct versions of Llama 3.1, Qwen 2.5,
and Mistral at comparable model sizes. The 405B TÜLU 3 performs compet-
itively against closed models such as GPT-4o-mini and Claude 3.5-Haiku or
large open models like DeepSeek V3. The training algorithms for our mod-
els include supervised finetuning (SFT), Direct Preference Optimization
(DPO), and a novel method we call Reinforcement Learning with Verifiable
Rewards (RLVR). We detail varying the objective, model initialization, gen-
eralization, and over-optimization of this new RL finetuning method. With
TÜLU 3, we build a multi-task evaluation scheme for post-training with
development and unseen evaluations, standard benchmark implementa-
tions, and substantial decontamination of existing open datasets on said
benchmarks. The TÜLU 3 release includes model weights, a demo, and the
complete recipe — datasets for diverse core skills, a robust toolkit for data
curation and evaluation, the training code and infrastructure, and, most
importantly, a detailed recipe for reproducing and further adapting the
TÜLU 3 approach to more domains.

Published as a conference paper at COLM 2025
TÜLU 3: Pushing Frontiers in
Open Language Model Post-Training
Nathan Lambert 1* Jacobb Morrison Valentina Pyatkin 1.2 Shengyi Huang
Hamish Ivison 1,2 Faeze Brahman Lester James V. Miranda
Alisa Liu?, Nouha Dziri', Xinxi Lyu', Yuling Gu', Saumya Malik',
Victoria Graf", Jena D. Hwang', Jiangjiang Yang Ronan Le Bras Oyvind Tafjord', Chris Wilhelm', Luca Soldaini' Noah A. Smith 1.2
Yizhong Wang 12, Pradeep Dasigi', Hannaneh Hajishirzi 12
Allen Institute for AI Uniiersiiy of Washington
Abstract
Language model post-training is applied to refine behaviors and unlock new skills across a wide range of language models, but open recipes for applying these techniques lag behind proprietary ones. The underlying training data and recipes for post-training are simultaneously the most im- portant pieces of the puzzle and the portion with the least transparency. To bridge this gap, we introduce TULU 3, a family of fully-open state-of-the-ars post-trained models, alongside its data, code, and training recipes, serving as a comprehensive guide for modern post-tran post-training techniques. TÜLU 3, which builds Llama 3.1 base models at 8B, 70B and 405B parameters, achieves results surpassing the instruct versions of Llama 3.1, Qwen 2.5, and Mistral at comparable model sizes. The 405B TULU 3 performs compet- itively against closed models such as GPT-4o-mini and Claude 3 3.5-Haiku or large open models like DeepSeek V3. The training algorithms for our mod- els include supervised finetuning (SFT), Direct Preference Optimization (DPO), and novel method we call Reinforcement Learning with Verifiable Rewards (RLVR). We detail varying the objective, model initialization, gen- eralization, and over-optimization of this new RL finetuning method. With TULU 3, we build multi-task evaluation scheme for post-training with development and unseen evaluations, standard benchmark implementa- tions, and substantial decontamination of existing open datasets on said benchmarks. The TÜLU 3 release includes model weights, a demo, and the complete recipe datasets for diverse core skills, robust toolkit for data curation and evaluation, the training code and infrastructure, and, most importantly, detailed recipe for reproducing and further adapting the TULU 3 approach to more domains.
1 Introduction
Post-training- - the collection of techniques including instruction tuning, reinforcement learning from human feedback, and other types of finetuning following large scale next- token pretraining - has become crucial step building frontier language models (OpenAI, 2024; Anthropic, 2024), yet developments to these techniques are frequently not accompa- nied by open resources and recipes. Fully open source counterparts (e.g., TüLU 2, Ivison et al., 2023, and Zephyr-B, Tunstall et al., 2023) often rely on simpler-toimmllement and cheaper pipelines and have become outdated on many metrics.
"TüLu 3 was team effort. marks core contributors. See full author contributions in Section 4. Contact tulu@allee wew.aalleeii..o org.
1

---

Published as a conference paper at COLM 2025
Curate prompts
Supervised finetuning
Direct pref. optimization
with verifiable rewards
public datasets
data mixing
G
on-policy-cas cate off-palicy cata
prompte with veriiiable rewards
parsona-drivenn synthetic instructions
Baso Model
TUlU3-SFT
Tulu3-Dre UPO
Tülu3
desontaminato
knowledge
reesoning
mathh
development evais
</>
coding
safety
Build evallation suite
UnGeon Cvalls
Identify skills
Figure 1: An overview of the TÚLU3 3 recipe. This includes: data curation targeting general and target capabilities, training strategies and a standardized evaluation suite for develop- ment and final evaluation stage.
To close the gap between open and closed post training, we introduce TÜLU 3, a family of open state-of-the-art post-trained models, alongside all of the data, checkpoints, training recipes, code, infrastructure, and evaluation framework. Integrating partial details from pro- prietary methods with novel techniques and established academic research, TÜLU 3 pushes the boundaries of research in post-training. The advancements of TÜLU 3 are attributed to permissively licensed training datasets targeting core skills, evaluation suite and tools to establish clear performance goals and guide improvement through training stages, and an advanced multi-stage training pipeline incorporating new algorithmic advancements in reinforcement learning, cutting-edge infrastructure, and rigorous experimentation to optimize data mixes, methods, and parameters across various training stages.
The TÜLU training recipe involves multiple stages (see Figure 1), with each stage build- ing upon the previous model and focusing on different types of data namely, prompt- completion instances for supervised finetuning of the SFT checkpoint, preferences for on-policy preference tuning to the DPO checkpoint, or verifiable rewvrrds for reinforcement learning. To finish our models, we introduce new final finetuning stage Reinforcement Learning with Verifiable Rewards (RLVR) which employs a novel RL objective tailored to enhance specific skills with verifiable answers, such mathematics and precise instruction following. Our methodology facilitates identifying skill deficiencies and refining the data mix, methods and parameters, ensuring balanced performance of core skills across the training process.
Our best performing recipe yields TÜLU models that outperform the state-of- the-art post- trained open-weight models of the same size such as Llama 3.1 Instruct (Dubey et al., 2024) or Mistral-Instruct (Mistral AI, 2024). At the 70B size TÜLU matches the offerings of closed providers such Claude 3.5 Haiku and GPT-4o mini, and at the biggest 405B size our model performs competitively against DeepSeek v3 (DeepSeek-Al) et al., 2024) and GPT 4o (11-24). A few key contributions from the development of TÜLU include: (1) extensive guidance and tooling for evaluation, decontamination and recipe design; (2) scaled, new synthetic instruction datasets; (3) scaling preference data with on-policy generations; (4) reinforcement learning with verifiable rewards, an RL-based method that only gets reward the model's completions are verified to be correct; and (5) advanced infrastructure, details, and code to facilitate the successful implementation of large models.
Related Work
Modern post-training has its roots in multi-task language model training, in particular instruction funing (Mishra et al., 2022; Wei et al., 2022a; Sanh al., 2022; Wang et al., 2022; Longpre et al., 2023), in which language models are trained on samples including task in- structions and their corresponding responses, allowing the models to generalize 'zero-shot' to new tasks. With the rise of ChatGPT and chat-focused LMs, post-training techniques evolved beyond instruction tuning to include preference tuning stages, with models under- going both instruction tuning and then preference finetuning (PreFT) or "RLHF" (Ouyang
2

---

Published as a conference paper at COLM 2025
Skill
(enchmarkkeol)
TULu
Qwen 2.778
Llama Instruct
TOLU
Qwen 72B
Llama Instruct
TOLU 3
Deepseek
Llama
4058
Instruct
Instruct
V3
Instruct
Avg.
65.1
66.5
62.9
76.2
72.8
74.1
80.7
75.9
79.0
Knowledge
MMLU MMLUUpahot.Coom
68.2
76.6
71.2
83.1
85.5
85,3
87.0
82.1
88.0
PopQAps TruthfulQA
18.1
55.5
44.9
55.0
63.1
55.1
67.6
69.9
66.8
Reasoning
Big Big Bench CoT)
69.0
70.2
71.9
80.4
88.6
89.5
62.6
54.4
61.5
74.3
34.2
77.0
Math
MATH shor Tle)
43.7
69.9
42.5
63.0
75.9
56.4
67.3
72.5
66.6
GSMSKi
87.6
83,8
63.4
93.5
89.5
93.7
95.5
94.1
95.4
Coding
Humannvalgyvettt
83.9
93.1
86.3
92.4
94.0
93.6
95.9
94.6
95.9
Humannvvalygssettn
79.2
89.7
82.9
58.0
90.8
89.5
92.9
91.6
90.3
IF & chat
IPEvalgeempi
82.4
74.7
80.6
83.2
87.6
88.0
86.0
88.0
88.4
Alpa Rquc wind
29.0
47.7
33.4
51.4
38.5
Safety
Solkty
85.5
75.0
75.2
68.3
87.0
76.5
86.7
72.2
86.8
Table 1: Overview of results on TÜLU EvAL suite, over all 8B, 70B, and 405B models. The best performing model for each model size on each benchmark is bolded. TÜLU outperforms the state-of-the-art post-trained open-weight models of the same size and surpasses Claude Haiku, GPT-3.5 Turbo, and GPT-4o Mini (additional results for 70B in Tab. and 405B in Tab. 5). For the larger open-weight models, DROP and TruthfulQA were omitted due to mutli-node evaluation issues with certain formats.
et al., 2022; Touvron al., 2023; Dubey et al., 2024). Recently, approaches that allow directly training language model on such preferences have been developed (Rafailov et al., 2024; Zhao et al., 2023), reducing the complexity of incorporating PreFT into training. Initial at- tempts at building 'open post-training recipes' focused the instruction-tuning stage (Taori et al., 2023; Conover et al., 2023), finetuning openly released language models on syntheti- cally generated or human made datasets. While combining these datasets could yield strong performance (Wang et al., 2023), incorporating a PreFT stage proved important for bridging the gap with closed models based human evaluations (Ivison al., 2023). Today, most popular adapted models with open recipes for PreFT I use DPO (or one of its variants), and Al feedback data including TULU (Ivison et al., 2023), Zephyr-B (Tunstall et al., 2023), and Starling (Zhu et al., 2023). Most of these open recipes use relatively little data and few rounds of training compared to closed post-training setups, which can involve multiple rounds of training with varied objectives, and millions of datapoints (Dubey et al., 2024).
TÜLU 3 aims to incorporates lessons from the above literature, recent post-training practices in large industrial laboratories, and adds new methods and lessons to the best practices. The novel approach proposed in this work, Reinforcement Learning with Verifiable Rewards (RLVR), relates to variety of recent work on improving LM reasoning through RL-related techniques. Closely related are the self-taught reasoner (STaR) line of work (Zelikman et al., 2022; 2024) and TRICE (Hoffman et al., 2023) - both use existing ground-trute answers as signals to generate better model rationales (or -fthuggh- Our proposed approach, RLVR simply uses an existing RL framework (PPO) for training, extends beyond the math domain, and runs entirely online with binary rewards (compared to the iterative approach of STaR the likelhool rewards of Quiet-STaR).
The TÜLU 3 Recipe
Here, we describe the TÜLUU 3 recipe to obtain a state-of-the-art post-trained model. We first construct an evaluation framework (sec. 3.1) and then produce TüLU 3 models through four-stage post-training recipe (sec. 3.2-3.5) on top of pretrained LMs (Figure 1). Things we tried that did not make into the recipe are Appendix H.
3.1 Evaluation Framework
3

---

Published as a conference paper at COLM 2025
Category
Benchmark
CoT
Shots
Chat
Multiturn
Metric
ICL
Knowledge Recall
MMLU
0
X
EM
PopQA
EM
TruthfulQA
6
/
X
MC2
Reasoning
BigBenchHard
EM
Development
DROP
X
3
N/A
Math
GSM8K
EM
MATH
/
Flex EM
Coding
HumanEval
N/A
Pass@10
HumanEval+
0
/
N/A
Pass@10
Instruction Following
IFEval
N/A
Pass@1 (prompt: loose)
2
0
/
N/A
Winrate
Safety
TÜLU 3 Safety
N/A
Average"
Knowledge Recall
MML.U
0
/
N/A
GPQA
N/A
EM
Unseen
Reasoning
AGlEvval English
0
/
/
EM
Math
Deepmind Mathematics
EM (Sympy)
Coding
BigCodeBench
0
N/A
Pass@10
Instruction Following
IFEval-OOD
N/A
Pass@] (prompt; loose)
HREF
0 N/A
Winrate
Table The TÜLU 3 Evaluation Regime: settings for development (top) and unseen (bottom) portions of the evaluation suite. CoT are evaluations run with chain of thought prompt- ing (Wei et al., 2022b). #Shots is the number of in-context examples in the evaluation template. Chat refers to whether we use chat template while prompting the model. Multi- turn ICL refers to setting where we present each in-context example as separate turn in conversation (applicable only when chat template is used and # Shots is not 0). "Avg. over multiple sub-evaluations: Full details of the safety evaluation are included Appendix G.2.1.
Stage Avg. Performance
TüLU
TÜLU
TüLU 405B
We designed our framework for evaluating TÜLU 3 and the other models we compare against with the following goals: 1) Our eval- uations should be reproducible. 2) We should evaluate models' generalization to unseen tasks, not just the specific benchmarks we use for de- velopment. 3) Our evaluation setup (e.g., tem- plates and strategies for prompting) should be fair to wide range of models. Accordingly, our framework consists of an open evaluation toolkit for reproducible evaluations and set of recommended settings for evaluating our evaluation suite based on our experiments with various models, sunmarized in Table 2.
3 8B
70B
SFT
60.6
72.6
77.5
DPO
64.7
76.2
79.6
Final (RLVR)
65.1
76.2
80.7
Table 3: Per stage average evaluation scores for TULU models. Full results in Appendix A.
We split our evaluation suite into a development and an unseen set, the former used for developing models, and the latter only for evaluating final models. This setup, along with our training data decontamination efforts, provides fairer evaluation of our models generalization capabilities for each of the core skills we focus during development. However, in comparisons between our models and other models, we cannol rule out that any closed model has not trained our evaluation suite, and hence cannot make clear judgments on models that have not publicly released finetuning data.
For scoring evaluations use the metrics identified in Table 2; higher is better. When com- puting overall performance, we average scores across all evaluations equally. Appendix G has more detailed description of our evaluation setup.
Evaluation Performance Summary. With our models trained from raw pretrained base models, compare to instruct models trained on the same base models (e.g., Nous Hermes 3), instruct models of similar size but different base versions (e.g., Ministral 8B or Qwen 2.5 Instruct), and other finetuning recipes trained on instruct version (e.g., Nemotron Llama

---

Published as a conference paper at COLM 2025
3.1). TÜLU 3 trained on Llama 3 base models outperforms all other open-weight models in its size category on our development evaluation suite. Compared to closed models, TÜLU GPT--o--miii-002--00
70B even surpasses closed models such GPT-3.5-Turbo-012.
while approaching the performance of Claude 3.5 Haiku 20241022. The TÜL. 3 405B model (see Appendix F.1) surpasses Llama 3.1 405B and even DeepSeek V3. The summary of TÜLU trained on Llama 3.1 versus the leading models in their size classes is shown in Table 1 and per training stage in Table 3, with further evaluation details included throughout the Appendices.
3.2 Stage 1: Data Curation
Prompts represent the diverse ways users may interact with models and serve as the essential component for all post-training stages. We curate a collection of millions of prompts as the starting point of the TÜLU 3 recipe, using them in the coming training stages. To minimize overlap between training prompts and evaluation sets we extensively decontaminate our final prompt dataset (full details in Appendix B.3).
To target the desired core skills, we curate a set of prompts from publicly available datasets with clear provenance and synthetically generate prompts to fill any gaps. We start this process with a broad survey of public datasets, including those annotated by dedicated workers, sourced from real users, and synthesized with models. We then select datasets with the following considerations: improve diversity, targeting specific skills, with careful considerations data provenance and licenses. For complete list of datasets sourced from for TŪLU 3, see Appendix B.1.
To address the growing need for diverse and skill-specific datasets, we incorporate synthetic data generation as complementary approach. To ensure diversity in generation, we follow the recent persona-driven methodology from Chan al. (2024) to generate synthetic data. The key idea is to use different personas (e-g., "a machine learning researcher focused on neural networks") with a data synthesis prompt (e.g., "create a coding problem") steer an LM to synthesize data with corresponding perspectives. Specifically, we condition on -25KK personas from Persona Hub (Chan et al., 2024) to generate prompts targeting specific skills such precise instruction following, math and coding. We detail our procedure for each selected skill in Appendix B.2. Prompts used to generate these instructions can be found in Appendix J.1. Additionally, we build upon previous efforts in Brahman et al. (2024); Han et al. (2024); Jiang et al. (2024), to generate noncompliance and safety data.
3.3 Stage 2: Supervised Finetuning
We conduct data mixture ablations and explore model merging techniques to develop an SFT training procedure that balances performance well across the core skills we prioritized. For prompts with existing responses, we generally keep the original response if it was written by human or frontier model, like GPT-4o. For large catasetss with subsets from frontier models (e.g,, WildChat), we use the subset from the best models. We additionally filter empty responses and responses that contain information about models or their developers. If set of prompts did not have responses, like our Persona prompts, or if the original responses were from a weaker model (e-g. WildGuardMii)) we generate new responses using GPT--o. We also hand-write responses to our hardcoded prompts for the model to know who trained and other basic properties.
To develop our SFT mix, we first identify the skills that were lagging behind state of the art models using Llama 3.1 trained with the TÜLU 2 (Ivison et al., 2023) recipe as our baseline. Targeting each of these skills in isolation, we collect high quality publicly available datasets and create synthetic datasets, as described in Section B.2.
To design our final SFT mix, we first build skill-specific data mixtures and models, keeping those that lead to the best performance on individual skills, ignoring other evaluations. This is done to approximate the upper bound for each evaluation given our setup.
We then combine these mixtures to create our initial TÜLU 3 mix. We then iterate on the mixture by adding or removing datasets to improve lagging skills, decontaminating against
5

---

Published as a conference paper at COLM 2025
Model
Avg.
MMLU
TQA
PopQA
BBH
CHE
CHE+
GSM
DROP
MATH
IFEval
AE2
Safety
TCLU 2 SFT
48.3
61.8
49.4
23.3
57.1
66.9
63.1
60.4
61.7
14.0
42.3
8.9
70.7
RLHFlow SFT V2
56.0
65.8
56.0
29.7
69.3
86.2
80.9
81.6
57.2
35.7
52.7
13.6
43,5
MAmmoTH2
63.6
42.7
20.8
63.4
63.7
43.8
30.5
34.9
6.5
47.8
3
62.1
72.8
TCL. 2 70B SET
63.6
76.0
57.8
44.1
79.4
86.8
83.5
83.2
75.9
33.1
57.7
17.3
68.8
TŪLU 3 70B SFT
72.6
79.4
55.7
48.6
82.7
92.9
87.3
91.1
77.2
53.7
82.1
26.3
94.4
Table 4: The performance of our TÉLU SFT against comparable baselines. Our SFT mixture shows strong performance, achieving a higher average score than other comparable mixes. All models, including TÜLU SFT, were trained on either Llama 3.0 or 3.1.
Prompt Selection
Response Generstion
Preferance Annotation
Model Pool (22 models)
Prompts
Ci CAGA
On colopolicycoo dara
0 m
Tūlu 3 SFT
Helpflunness
Prompts from datasets subsampled for SFT
Gemma
Instruction Following
PS
OR
Truthfulness
Rejected
New OOD prompts (Ultrafeedback, Persona)
Tulu3 SFT70B
Honesty
Sample four rosporsass from diffeennt modeess for eachprompt
Figure 2: An overview of TULU'S pipeline for generating and scaling preference data extended from the method of Cui et al. (2023).
our evaluations and downsampling particularly large datasets. We show the performance of major preview versions throughout development in Figure in Appendix C.
Final SFT Results. In Table 4, we compare our final TüLU 3 8B SFT and TüLU 3 70B SFT models against other SFT-only models trained on Llama 8B or 70B. Our new SFT mix shows substantial improvements over the TÜLU 2 mix at both model sizes, and is better on average than the other competitive 8B SFT models. More ablations and experiments are found in the Appendix along with implementation details for training in Appendix C.2.
3.4 Stage 3: Preference Finetuning
On-policy Preference Pipeline We create on-policy preference data (x,y,y/ (...y,labe) llabe)) given our prompts from subsection 3.2 by adapting and advancing the UltraFeedback pipeline (Cui et al., 2023). Our early experiments show the benefit of this pipeline in creating preference data, which leads to high-quality, synthetic preference dataset (as observed by Ivison et al., 2024). Our data creation pipeline (shown in Figure 2) consists of three stages:
1. Prompt Selection. The first step for preparing dataset for preference finetuning is to select the prompts or user instructions to generate responses and obtain preferences for. Given the set of prompts in Table 8, we curate our selection to include prompts used during SFT, and prompts that were subsampled from the same sources, yet unused for SFT. We also include prompts from other sources, such as a version of UltraFeedbbac without TruthfulQA instances (to avoid evaluation contamination), or by adding new IFEval-constraints to prompt.
2. Response Generation. For a given prompt, we randomly sample four models from model pool to generate responses. Our model selection inspired by the Ultrafeedback pipeline which consists of open-source and proprietary models that vary across parameter size and model family. We update Ultrafeedback's model pool by using recent versions of some models (Llama - Llama 3.1), adding best-performing models to increase the pool size, and replacing currently inaccessible models such as WizardLM with open-source alternatives. We also include on-policy data by sampling completions from the TüLU SFT model. We approach this by adding a selection of prompts where one response is generated from the on-policy model, and the other response from the off-policy models.

---

Published as a conference paper at COLM 2025
3. Preference Annotation. After generating four responses for each prompt, we use 27T-40-2024-880 via LLM-as-a-judge (Zheng et al., 2023) to rate each response from 1 to 5 across four different aspects: helpfulness, iustruction-folloiing honesty, and truthfulness. Appendix K shows the external models used to sample off-policy data and the prompt template for each aspect. In order to obtain binary preferences rankings for DPO we obtain the mean of preference ratings and take the highest-rated response as the chosen response and randomly sample from the responses with the lower mean the rejected response.
The TÜLU Preference Mix. We choose the final preference mix for the models, which maximizes average performance on the development evaluations, while also excelling at targeted skills. Most of the preference data mix ablations run for the 8B model. We start with prompts used for SFT and generate on-policy and off-policy preference data, resulting in 96,911 (off-policy) and 19,444 (on-policy) preference instances. Given this preference base we ablate adding additional prompt sources to the mix and how these additions affect downstream evaluation performance, specifically targeting skills like precise instruction following, math and general chat performance on AlpacaEval. Table 14 in Appendix D shows how the inclusion or exclusion of preference datasets influences the average performance and the final datasets are shown in Table 15. For additional ablations and key findings refer to Appendix D.
3.5 Stage 4: Reinforcement Learning with Verifiable Rewards
We introduce Reinforcement Learning with Verifiable Rewards (RLVR), a novel method for training language models on tasks with verifiable outcomes such as mathematical problem-solving and instruction following. RLVR leverages the existing RLHF objec- tive but replaces the reward model with verification function, as shown conceptu- ally in Figure 3. When applied to domains with verifiable answers, such as mathe- matics and verifiable instruction following tasks (Zhou et al., 2023). We use RLVR to cover multiple evaluations and test how it can improve overall model performance while being trained on single task such as GSM8K, integrating it as component of generalist training pipeline.
Verifiable Reward
iecereec othervrion
Ta
Reward
Scalar
ag
Si Prompts
Policy
Training data
Completions
4 4 +-VWlta)) Policy Update
Figure An overview of how Reinforcement Learning with Verifiable Rewards (RLVR) works. We sample completions from a pol- icy model given a set of prompts, and verify their correciness using a deterministic func- tion. If the answer is verifiably correct, we provide reward of a, otherwise 0. We then train against this reward using PPO.
3.5.1 RLVR Data
Creating data for RLVR entails obtaining prompts with an accompanying binary verifier (i.e., constructing a set of inputs x with accompanying verifier functions v). We focus on two domains (mathematics, exact instruction following) and three evaluations (GSM8K, MATH, IFEval) with relatively straightforward methods for verification, and leave more complex verifiers to future work (Gehring et al., 2024; Xu et al., 2024), understanding that answer extraction and verification methods are domain-dependent
GSM8K. We use the GSM8k training set. We augment each sample with the standard 8-shot prompt using during evaluation to encourage the model to use chain-of-the tfthoughth and then extract the final number produced and compare to the ground-truth label to determine correctness.
MATH. We use the MATH training set. Similar to GSM8k, we augment each sample with the standard 3-shot CoT prompt used to encourage the model to generate chains of thought during evaluation, and then extract the answer and determine correctness following the 'flex' MATH evaluation logic.
7

---

Published as a conference paper at COLM 2025
Vertfable Rohar5s
0.3
0L D.Dvvrgercc
Besporse Lemgih
8.0
90
85
06
20K
6.KK
6.K
80K
100K
26
238
43K
6.K
BOK
100K
0K
23K
408
b0k
B5K
100K
RLVR on Math
4.75
Verfaall Norards
Divergance
DasceerTa Langge
4.25
-0
00
23K
428
B0K
D0K
100K
08
20K
40K
60K
30X
100K
0K
20K
40K
50%
D0X
100K
RLVR on Prompts with Constroints
Venfaate Rowo
Besporee Lensth
1.0
1.0
0K
20K
40C
606
80K
100K
04
20K
40K
60C
50K
100K
20K
40K
60K
30K
100K
0.01
B 0.03
B
0.05
0.1
GSMBK Performance vs KL Divergence
MATH Flex Performance Divergence
|FEval Performance vs KL Divergence
0.410
61.2
0.82
61.0
61.8
0.405
GSMBK
0.86
00.0
00.0
Aucrage
IFEval
0.78
50.5
0.85
GSMBK Boseline
GSMBK
61.4
0.76
IFEva Basel
ITEva
Average
0.305
age
60.0
10
12
14
10
20
30
15 KL Divergence
25
KL Divergence
KL Divergence
Figure The top three rows show RLVR's verifiable rewards, KL divergence, and response lengths on the frain dataset of GSM8K, MATH, and prompts with constraints, when starting from DPO checkpoint (i.e., non-final DPO checkpoint). The bottom row shows the corresponding downstream test performance. RLVR can lead to higher verifiable rewards in the train datasets. RLVR can also lead to higher scores in the corresponding test dataset, however, an increase in the average score across all evaluations is not guaranteed.
IFEval. We randomly sample instructions from the TÜLU 2 SFT mix and combine them with constraints from Zhou al. (2023)'s taxonomy. We have a verification function for each constraint template that is able to verify whether completion satisfies constraint.
Given these prompts and verification functions, we then train the models via Proximal Policy Optimization (PPO) (Schulman et al., 2017) on these verifiable rewards. We combine all prompts together, resulting in mixture of 30,000 prompts with ground truth labels.
3.5.2 RLVR Recipe and Analyses
We start by anchoring DPO model in the development history as the initial model for con- ducting experiments for RLVR. We conducted several sets of experiments which resulted in key findings (with full implementation details and hyperparameters in Appendix E.2 includ- ing value model initialization from an RM, turning off dropout, data shuffling, truncation penalties, and advantage whitening).
8

---

Published as a conference paper at COLM 2025
RLVR Can Improve Performance in Targeted Domains. Figure 4 shows training with RLVR results in improved test performance for all three settings. We applied the RLVR recipe on GSM8K, MATH, and IFEval, respectively with sweep of beta values [0.1,0.05, 0.03, 0.01]. We achieve models that outperform the initial model in that particular evaluation down- stream. We also see that the verifiable rewards (i.e., correctness consistently for all three settings, Interestingly, in GSMSK and MATH, we found that incurring more KL budget does not necessarily lead to improvements in verifiable rewards.
the train set) improve
Initializing RLVR's Value Function from a General RM Works Best. We experimented with initializing PPO's value model from a general reward model or the anchored DPO model, and ran sweep of beta values [0.1,0.05 0.03, 0.01, 0.005, 0.001] on the GSM8K task. The general RM trained with the UltraFeedback dataset (Cui al., 2023) and evaluated models on our entire evaluation set, with a careful eye on GSM8K. Figure 21 shows that initializing the value from general RM obtains the highest GSM8K test score and also higher average scores. This suggests the value function plays an important role in RLVR's training.
Do Not Use the Scores from RM. One possible implementation for RLVR is to add verifiable rewards on top of the scores from the reward model. We launch experiments also using a sweep of beta values [0.1, 0.05, 0.03, 0.01, 0.005, 0.001] on the GSM8K task. Figure 22 shows that using only the verifiable rewards outperforms using scores from the reward model. Training with verifiable rewards with the scores from RM seems to introduce more noise, especially in the average scores.
Starting from a Weaker Model Can Converge to the Same Verifiable Rewards. The model's base capabilities before RLVR are also a confounding factor. We launch another set of experiments using an SFT model with lower average scores (from which the an- chored DPO checkpoints were trained, so they share the same lineage) using beta values [0.1, 0.03, 0.01]. Figure 20 in Appendix E shows that starting from both SFT and DPO can lead to the same level of verifiable rewards, but starting from the SFT model would incur larger KI. compared to starting from the DPO model. This makes sense because the SFT model is further away from good at GSM8K than the DPO model. However, we find that starting from a stronger model usually results in better test set performance.
Overoptimization Happens. As we lower the KL penalty B, the trained model incurs greater KL divergence from the initial model. We observed that more KL divergence typically results in lower average scores, as shown in Figure 21. The exception is Figure 22, where the largest KL corresponds to the highest average score. Furthermore, we showcase overoptimization of prompts with constraint cases in Appendix I.3.
3.5.3 Final Experimental Results
Based on the above results, we ran our final RLVR runs using the combined verifiable prompt set, and used the best few DPO models from the prior ablation sections as starting points. We evaluated our models every 100 training steps (40 for 70B), and picked as our final 8B model the checkpoints with best overall performance on MATH and IFEval. We show the logs from RLVR training in Figure 19, and compare the final performance against their DPO starting points and Llama 3.1 in Table 22 in Appendix E, where we see strong performance on multiple skills for the model. RLVR results in non-trivial improvements at the multiple model scales, improving all three of MATH, GSM8k, and IFEval. In fact, we observed that some 8B runs were able to achieve peak GSM8k scores of up to 89.4% and IFEval scores of up to 84.8% at slight cost to overall performance. At the 70B scale, we observe more modest improvements in IFEval and MATH, and no improvement in GSM8k, likely due to the fact that it is already close to saturation (93.5%). Surprisingly, we find that our 70B run displays extremely low KL divergence, remaining well below 1 over the duration of the run, probably due to the lower learning rate. With the 405B model, we found the largest improvements over the RLVR stage overall, particularly with the core evaluation MATH where the model improved from score of 59.9 after DPO to 67.3 after RL. Details on the RLVR infrastructure and final hyperparameters for training can be found in the Appendix F.2.

---

Published as a conference paper at COLM 2025
4 Conclusion
We introduce TÜLU 3, a family of fully open state-of-the-art language mod- els, featuring a modern post-training framework with fully open-source data, TULU 3 DATArevaluttion TÜLU 3 EVAL; training code, TÜLU CODE; and development recipes, TCLU 3 RECIPE. We release final models trained Llama 3.1 base versions, with intermediate checkpoints, training data, training code, and evaluation code. TÜLU 3 bridges the gap between open and closed post-training methods, marking a new milestone in open post-training research. With the resources provided, others can build on open base models, finetune them for high performance across diverse tasks. This paves the way for advancing post-training research within multi-objective, and multi-stage training frameworks.
Author Contributions
A successful team project like TÜLU 3 would not be possible without the fluid contributions of many teammates across formal team boundaries. As not all of these can be captured, we indicate each authors' primary contributing role in this project. Authorship for this work was determined by those making direct contributions to the TÜLU models, related artifacts, and their release. Core contributors recognized for their sustained, significant contributions critical to the success of the TüLU 3 project. Authors are listed alphabetically:
For SFT model development, including training and data curation: Faeze Brahman, Shengyi Huang, Hamish Ivison, Nathan Lambert, Jacob Morrison, Yizhong Wang, and Chris Wilhelm.
For preference-tuned model development, including training and data curation: Faeze Brahman, Shengyi Huang, Hamish Ivison, Nathan Lambert, Lester James Miranda, Valentina Pyatkin, Chris Wilhelm.
For reinforcement-learningg model development, including training and data cura- tion: Shengyi Huang, Nathan Lambert, Hamish Ivison, Valentina Pyatkin, Faeze Brahman.
For evaluation tooling support, decontaminating training datasets, and evaluating peer models in the ecosystem throughout: Pradeep Dasigi, Nouha Dziri, Victo- ria Graf, Shengyi Huang, Jena D. Hwang, Hamish Ivison, Ronan Le Bras, Alisa Liu, Xinxi Lyu, Saumya Malik, Valentina Pyatkin, Luca Soldaini, Oyvind Tafjord, Jiangjiang Yang.
For management of communications, legal, and other release processes: Faeze Brahman, Pradeep Dasigi, Hannaneh Hajishirzi, Nathan Lambert, Luca Soldaini.
For mentorship and advising: Pradeep Dasigi, Hannaneh Hajishirzi, Nathan Lam- bert, Valentina Pyatkin, Noah A. Smith, Luca Soldaini, Yizhong Wang.
Acknowledgments
We thank John Schulman for extremely useful advice. We acknowledge the National Artifi- cial Intelligence Research Resource (NAIRR) Pilot and Microsoft Azure for contributing to the results in this work. We thank Niklas Muennighoff for helping with some experimenta- tion on OLMoE. We thank countless members of Ai2 and UW NLP communities for useful feedback throughout this project. Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). We thank the vLLM team (Kaichao You, Simon Mo, Woosuk Kwon, and Zhuohan Li) for their invaluable support in debugging NCCL weight transfer issues for RLVR. We thank Huy Tran, Jesse Dodge, Jiacheng Liu, Sruthi Sreeram, Taylor Blanton, Aaron Sarnat, Arnavi Chheda, Byron Bischoff, Chris Newell, Michael Schmitz, Sam Skjonsberg, Eric Marsh, Karen Farley, and YenSung Chen for building the Ai2 Playground for model demos. We also thank these others at Ai2 for many indirect contributions to the project: Kyle Lo, Taira Anderson, Jen Dumas, Crystal Nam, Sophie Lebrecht, Brooke Vlahos, Chris Wilhelm, Jenna James, Alex Buraczynski, Will Smith, Caitlin Wittlif, Carissa Schoenick, and Ali Farhadi.
10

---

Published as a conference paper at COLM 2025
References
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint ar arXio:2406.1111 11704, 2024.
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie C.Y. Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. In The Thirty-eighth Amnual Conference on Neural Information Processing Systems, 2024. URL https://openrevi rew.ooewwnevie....eee.e... nww/tfrumtnneeeee (-
Mistral AI. Mistral introduces nemo, 2024. URL http:://miisrrallaaaa//eesesssss Accessed: 2024-11-21.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxan- dra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: open large language model with state-of-the-art performance. 2023.
Anthropic.
Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku. Anthropic, 2024. URL https://www, anthropic cothroppc..omm/ewssss. --o---mmmmm----------- condtcompue
Accessed: 2024-10-22.
Edward Beeching, Shengyi Costa Huang, Albert Jiang, Jia Li, Benjamin Lipkin, Zihan Qina, Kashif Rasul, Ziju Shen, Roman Soletskyi, and Lewis Tunstall. Numinamath 7b tir. 2024.
https://huggingface /tt/AA//M///mmimaaaa----555-
Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, et al. The art of saying no: Contextual noncompliance in language models. arviv preprint ar #xXiv2240.120433 2024.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report, 2024.
Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. Xiv preprint ar XxXio:2406.20094, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo- hammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish,
11

---

Published as a conference paper at COLM 2025
Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.
Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, and Xiaoyu Shen. The accuracy paradox in rlhf: When better reward models don't yield better language models. arv preprint ar arXiv:2410.06554 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint ar 1xiv:2110.141666 2021.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's 's first truly open instruction-tuned Ilm, 2023. URL https: //www. dat.bri ckk. com/blog/ /020/2233/ 2023/ (iii///ollyffirstoppeeeee
tuned-
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high- quality feedback. arviv preprint ar Xix::2310.01377, 2023.
DeepSeek-AN AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yonggiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2024. URL https: http://aaxvvvooooo org/ abrg//bbs//111 19437.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment, 2023. URL https: //arxiv.// argg/bbs/3044 06767.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2368 (668-2378,
12

---

Published as a conference paper at COLM 2025
Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: (1.18653/11N199122 URL https: https::/aclaatttoooo/ http://aaaaaaattloooooooo oowpoollooooooogllgooooo org/N19- 124...
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. Xrv preprint ar arXiv:2407. 217783 2024.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback. In Thirty Thirty-seventh Conference on Neural Infor- mation Processing Systems, 2023. URL hattps://openreiiees wwwpi/reweeeeeee/fforeeeeeeeeeee wwt//orumm htlrrforrmmmid:hhhhh ddrnniirr4ltcccom cKX.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. ar preprint ar Xrx::2404..4475, 2024.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: case study ppo and trpo. In International Conference Learning Representations, 2020. URL https://openreviere htt.://nnettformmmimretettt
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416. PMLR, 2018
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://zenodo orgg/recordsssee 2608602.
Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Taco Cohen, and Gabriel G Synnaeve. Rlef: Grounding code iims in execution feedback with reinforcement learning, 2024. URL
https://arxiv.orgg/ awwp 2410 02089.
Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL httpttsts///xii.. grg/ abws/240..0055
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408. 4tt022008.00118, 2024.
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's mergekit: A toolkit for merging large language models. arXiv preprint urXiv:2403.1325,, 2024.
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. ToRA: A tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference on Learning Representations, 2024. URL https:: wwweeewwwwwwwwweeeeeeeeeeeee nwtttorrunr wttrrumidddppottjioooo
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, A. Jha, Hamish Ivison, lan Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Daniel Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nis- hant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke S. Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hanna Hajishirzi. Olmo: Accelerating the science of language models. ArXiv, Abss/240.00083. 2024. URL https: / //ppi. semanticscholar org/ ///CCoruuDD: 267355885.
13

---

Published as a conference paper at COLM 2025
Yuling Gu, Oyvind Tafford, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. Olmes: A standard for language model evaluations, 2024. URL hhttps://aaat org/abs/2406.0844
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. Direct language model alignment from online ai feedback, 2024. URL https: warxiv..rrg/bss/2222.. 84792.
Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. arXiv preprint ar XrXo:22406.1849, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arxiv preprint ar Xiie:2009.03300, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Matthew Douglas Hoffman, Du Phan, david dohan, Sholto Douglas, Tuan Anh Le, Aaron T Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A. Saurous. Training chain- of-thought via latent-variable inference. In Thirty-seoenth Conference on Neural Information Processing Systems, 2023. URL https:// http:/ppereeiiieeeeeeorrrrrrrrio www.:e//oorrummiiaaa14411113333
Jian Hu, Xibin Wu, Weixun Wang, Dehao Zhang, Yu Cao, et al. Openrlhí: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXit:2405.11144, 2024.
Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimiza- tion. In ICLR Blog Track, 2022. URL https://iclr-bloot trapllogggaack..ittt... gitbu.. 10/2022/03/ http://lllbbbgrrkkkkiiiibbi/////
p ppo-
impleementation-detaliss.
Shengyi Huang, Jiayi Weng, Rujikorn Charakorn, Min Lin, Zhongwen Xu, and Santiago Ontañón. Cleanba: A reproducible and efficient distributed reinforcement learning platform. In The Twelfth International Conference Learning Representations, 2023.
Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The n+ implementation details of RLHF with PPO: A case study on TL;DR sunmarization. In First Conference on Language Modeling, 2024a. URL https: wwwpnrrvveee.nettoouniiieeee
Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, et al. Trustllm: Trustworthiness in large language models. Xrv preprint ar 4rXiv:2401.05561, 2024b.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint ar X11::2210 2024.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in changing climate: Enhancing Im adaptation with tulu 2. arxio preprint arX10:2311.10702, 2023.
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hannaneh Hajishirii. Unpacking dpo and ppo: Disentan- gling best practices for learning from preference feedback. ar Xiv preprint 4rXiv:2406.09229, 2024.
14

---

Published as a conference paper at COLM 2025
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arv preprint ar Xii::2310006225 2023.
Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. Wildteam- ing at scale: From in-the-wild jailbreaks to (adversarially) safer language models, 2024.
URL htttt:://arkii http://aawxvvv.o.aabbba org/abs/ 220//22066.. 18510.
Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David L. Donoho, and Sanmi Koyejo. Collapse or thrive? perils and promises of synthetic data in a self-generating world, 2024. URL https: /ttppii////xxvv...o org/ 2410 16713.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openas- sistant cnnversatios-ddmoorattiiii large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph F. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: http:///pneeeeeee...cooooommoaaa nett:forrummdd-IIT77EEWWM
Tianjian Li, Haoran Xu, Weiting Tan, Kenton Murray, and Daniel Khashabi. Upsample or upweight? balanced training on heavily imbalanced datasets, 2024. URL https:
//arxiv.org/abb/2222 2410//00///21.0099..
Stephanie
Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. ar preprint ar drXiv:2109.07958, 2021.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code gen- eration. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023. URL https://openreviies net/forum?id=1qqxx6 10.7u7
Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. ar preprint arXin:1907.11692, 364, 2019.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.1363 2023.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint, 2022.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. ar arv preprint ar arXiv:2402.04244 042499 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Xiv preprint ar Xix::2455.14734 2024.
15

---

Published as a conference paper at COLM 2025
Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, Amit Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Stépán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. Sympy: symbolic computing python. Peer] Computer Science, e103, January 2017. ISSN 2376-5992. doi: (0.7717/peerj-cs.. URL https://doi. org/10. 7717/peerj-Cs 103.
Lester James V Miranda, Yizhong Wang, Yanai Elazar, Sachin Kumar, Valentina Pyatkin, Faeze Brahman, Noah A Smith, Hannaneh Hajishirzi, and Pradeep Dasigi. Hybrid preferences: Learning to route instances for human ai feedback. arXio preprint arXiv:2410.19133, 2024.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3470-3487, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022. acl-long.244. URL https: //allantolll wwgg/022..... acl-long. 244.
Mistral AI. Ministraux: Pushing the boundaries efficient transformer design, 2024. URL http::/mmiistaal.aaaeee///s/////
Accessed: 2024-11-17.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang. William Paul, Michael Jordan, et al. Ray: A distributed framework for emerging (Al) applications. In 13th USENIX symposium on operating systems design and implementation (OSDI 18), pp. 561-577,,0011 2018.
MosaicML. NLP Team. Introducing mpt-30b: Raising the bar for open-source foundation httpaimmmmcoooooo//oo////// wwwmbooog/mptt//e///ooo 30b. Accessed: 2023-06-22.
models, 2023. URL www. mosaicml.
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning, 2024.
Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous rlhf: Faster and more efficient off-policy rl for language models, 2024. URL https://arxiv.
2 18252.
OpenAI. Learning to reason with 1lms. OpenAI, 2024. URL cttps::/openai..com wwtpp/eaai.comiidddeeee index/ learning-to-reason-- th-llhhll Accessed: 2024-09-12.
OpenAl. Gpt-4o, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems,, 35:27730-27744, 2022.
Saurav Pawar, S. M Towhidul Islam Tonmoy, S M Mehedi Zaman, Jain, Aman Chadha, and Amitava Das. The what, why, and how of context length extension techniques in large language models a detailed survey, 2024. URL
07872.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with large language models: A survey. arxiv preprint ar arXi::2455.19335 2024.
16

---

Published as a conference paper at COLM 2025
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https: //qwenlm. github. wtwpii//obaaaeeeee /wwpg///eee2222 5/.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, wwwpdddaaatttttuggggggeeeee
and Thomas Wolf. No robots. https://huggingfae robots, 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-lvee google- proof q&a benchmark, 2023.
Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. Xiv preprint ar Xiv:2308.001663 2023.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Ur- mish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexan- der M Rush. Multitask Prompted Training Enables Zero-Shot Task Generalization. In International Conference on Learning Representations (ICLR), 2022.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing math- ematical reasoning abilities of neural models. In International Conference on Learning Representations, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint ar axXto:1007.06347 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXi::4402.0332 2024.
Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng, Yi Dong, Daniel Egert, Shengyang Sun, Jimmy J. Zhang, Sahil Jain, Ali Taghibakhshi, Markel Sanz Ausin, Ashwath Aithal, and Oleksi Kuchaiev. Nemo-aligner: Scalable toolkit for efficient model alignment. In First Conference on Language Modeling, 2024a. URL https: https::////pee/eeeeeeeee www..eeefforumii.... YK2eGE8QVW.
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. In ACM SIGSAC Conference on Computer and Communiccaions Security (CCS). ACM, 2024b.
Aaditya Singh, Muhammed Yusuf Kocyigit, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, and Dieuwke Hupkes. Evaluation data contamination in llms: how do we measure it and (when) does it matter? arXiv preprint arXiv:2411.03923, 2024a.
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al. Aya dataset: An open-access collection for multilingual instruction tuning. arXiv preprint arX1n:2402.0661, 2024b.
17

---

Published as a conference paper at COLM 2025
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Adoances in Neural Information Processing Systems, 33:3008-3021, 2020.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXit:2210.09261, 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://gi wtt.:/ooc/aaaaaaaa coom/tatsu-lab///e /ab/stanford alpaca, 2023.
Ryan Teknium, Jeffrey Quesnelle, and Chen Guang. Hermes 3 technical report. Xiv preprint MrXii:2408.11833 2024.
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstructt:: Accelerating ai for math with massive open-source instruction data. ar Xiv preprint ar Xri::2410001560, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arViv preprint #xXi=22307.09288 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint ar Xff:22310.16944, 2023.
Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. Aya model: An instruction finetuned open-access multilingual language model. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15894- 15939, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ /v /2224.acl--oogg..o.... 845.
URL https::/aclantholll
20g/ acl -ong 845.
Leandro Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. sttps://ggt /i thub. com/ www.ggiinggc///ttrro 2020.
David Wadden, Kejian Shi, Jacob Morrison, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, et al. Sciriff: A resource to enhance language model instruction-followingg over scientific literature. arXiv preprint axXiv:2406.07835, 2024.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An Open Platform for AI Software Developers Generalist Agents, 2024a. URL https: //arxiv.org/
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunku- mar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, Generalization via declarative instruc-
and Xudong Shen. tions on 1600+ NLP tasks. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang
18

---

Published as a conference paper at COLM 2025
(eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 5085-5109, Abu Dhabi, United Arab Emirates, December 2022. Associa- tion for Computational Linguistics. doi: 10.18653/ 0036653/v1/22225 wttp///2e.mmpmmmmii..... org/2022. emnlp-main.340
URL
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36.74744-74786, 2023.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more ro- bust and challenging multi-task language understanding benchmark. arxiv preprint arXiv:2406.0155 2024b.
Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteer2-pprffeence Complementing ratings with preferences. arv preprint axXio:2410.01257, 2024c.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint Xix::22406.08673 2024d.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned Language Models are Zero-Shot Learners. In International Conference on Learning Representations (ICLR), 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-248377 2022b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-att natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020, Association for Computational Linguistics. doi: 10.18653/ / v1/2020..mmms eew.eeeenllndoo-.. URL https: http///aaaaaattogo...ooo org/ 2020. emnlp-demos. 6.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23965-23998. PMLK, 17-23 Jul 2022. URL https: / /ttss////occedigg. mlr mlr.press/vl 62/wortsman22... html..
Zhaofeng Wu, Ananth Balashankar, Yoon Kim, Jacob Eisenstein, and Ahmad Beirami. Reuse your rewards: Reward model transfer for zero-shot cross-lingual alignment. Tn Yaser Ål-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1332-1353, Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL https: //aclanthology.oo wwwp:/gg..rrg////// otg/ emnlp-main. mapp..mminn... 79.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? a comprehensive study. In ICML, 2024. URL
net/forum?id-=xHHe
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. Rethinking benchmark and contamination for language models with rephrased samples, 2023.
19

---

Published as a conference paper at COLM 2025
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arv preprint ar wrXiv:240.004652, 2024.
Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet-star: Language models can teach themselves to think before speaking. COLM, Abs/2443.09629, 2024.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STar: Bootstrapping reasoning with reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreviii net/forum?id= 3ELRdg2sgl.
Liangyu Zha, Junlin Zhou, Liyao Li, Rui Wang, Qingyi Huang, Saisai Yang, Jing Yuan, Changbao Su, Xiang Li, Aofeng Su, et al. Tablegpt: Towards unifying tables, nature language and commands into one gpt. ar preprint ar arXit:2307.08677 2023.
Michael Zhang and Eunsol Choi. SituatedQA: Incorporating extra-linguistic contexts into QA. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Pp. 7371-7387, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. org/2021
URL
https://aclantthol
emnlp-main. 586.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wild- chat: Im chatgpt interaction logs in the wild. arXiv preprint ar Stt::205500170,) 2024.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. Slic-hf: Sequence likelihood calibration with human feedback, 2023. URL https:
//axxiv.org/abs/2330 10425.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS Datasets and Benchmarks Track, 2023.
Xing, Hao Zhang, Joseph E. Gonzalez,
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A human-centric benchmark for evaluating foundation models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2299-2314, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18663///.1111 /v11 //////11222/// findings-naacl. 149. URL ptttps:/aaclantiiee
org/ 2024. few.fffiinnnnnsaaeae 149.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follooingg evaluation for large language models, 2023. URL http://xxxv...r//bbbbaa/
07911.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving Ilm helpfulness & harmlessness with rlaif, November 2023.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arv preprint arXiv:2406.15877, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. Xiv preprint ar Xic::199.008533, 2019.
20

---

Published as a conference paper at COLM 2025
Benchmarkgeal)
Llama 3.1
405B
Nous Hermes 405B
Deepseek
GPT 4o (11-24)
TÜLC
TÜLU
TÉLU
Instruct
405B
Avg w/o Safety.
78.1
74.4
79.0
80.5
76.3
79.0
80.0
Avg w/ Safety.
79.0
73.5
73.9
81.6
77.5
79.6
80.7
Mg..Ug CoT)
88.0
84.9
82.1
67.9
84.4
86.6
87.0
PopQAg shot)
BigBenchHaaddo Cot)
87.1
87.7
89.5
83.3
88.8
88.6
MATH shot, Reo)
66.6
58.4
72.5
68.8
63.4
59.9
67.3
GSMSKg
95.4
92.7
94.1
91.7
93.6
94.2
95.5
HumanEvallaaotti
95.9
92.3
94.6
97.0
95.7
97.2
95.9
HumanEval+ (gassnin
90.3
92.7
92.9
IFEvalgosse prompt)
88.4
81.9
88.0
84.8
82.4
85.0
86.0
Alpacafival 2quc
38.5
30.2
53.5
65.0
30.4
49.8
51.4
Safetyis expl
86.8
65.8
72.2
90.9
87.7
85.5
86.7
Table 5: Summary of TüLU 3 results relative to peer 405B models. The best-performing model on each benchmark (i.e., in each row) is bolded. TüLU 3-405B outperforms prior state-of-the-art models finetuned from Llama 3.1 405B Base and rivals some leading, closed models. Progress across various checkpoints highlight the contribution of each stage of the training in improving core skills. Note that TruthfulQA and MMLU multiple choice numbers are not compatible with our infrastructure for running evaluations (via log-probs).
A Appendix
A Additional Core Results
70B, we compare to and surpass Llama 3.1 70B Instruct, Qwen 2.5 72B Instruct (Qwen Team, 2024), Nous Hermes 3 70B (Teknium et al., 2024) (trained on Llama 3.1 70B), and Nemotron Llama 3.1 70B (Wang et al., 2024c) (trained on Llama 3.1 70B Instruct). At 8B, we compare to and surpass Llama 3.1 8B Instruct, Gemma 2 9B Instruct (Gemma Team et al., 2024), Nous Hermes 3 8B (trained on Llama 3.1 8B), Qwen 7B Instruct, and Ministral 8B Instruct 2410. A per training stage breakdown of performance is shown for the 8B version in Table 7, for 70B in Table 6, and for the 405B in Table 5.
B Data
B.1 Source Datasets
For diversity we pick datasets that can promote diversity, including: WildChat (Zhao al., 2024), which is large source of real-user interaction with models; Open Assistant (Köpf et al., 2024), which created by volunteer workers for general chatting; No Robots (Rajani et al., 2023), which is annotated by expert workers for a broad range of open-ended cat- egories; and FLAN v2 (Longpre et al., 2023), which is big compilation of classical NLP tasks. To improve target skills, we include the following datasets: OptnMathInstruct (Tosh- niwal et al., 2024) and NutinaMath (Beeching al., 2024) for mathematical reasoning, Svol-CodeAlpaca for coding, subset of Daring-Anteater (Wang et al., 2024d) for precise instruction following, Aya (Singh et al., 2024b) for multilingualltyy SciRIFF (Wadden et al., 2024) for scientific literature understanding, and TableGPT (Zha et al., 2023) for processing table-related tasks. Another important consideration was data provenance and licenses, where we made sure that all the datasets included in final curation have clear licenses.
B.2 Synthesizing for Target Skills
To address the growing need for diverse and skill-specific datasets, we incorporate synthetic data generation complementary approach. Synthetic data generation has gained traction as a promising alternative to human-written data due to being cheaper to obtain, customizable for different purposes, and reflecting the vast knowledge of the underlying
21

---

Published as a conference paper at COLM 2025
Llama 3.1
Qwen 2.5
Hermes
Nemotron Llama 3.1
TüLU 70B
TÜLU
TÜLU
(enchmarKevva))
70B
72B
Llama 3.1
70B DPO
70B
Instruct
Instruct
70B
70B
Avg.
74.1
72.8
68.5
72.0
72.6
76.2
76.2
MMLU (o shot, shot,
CoT)
85.3
85.5
80.4
78.9
83.3
83.1
PopQA shot)
46.4
30.6
48.1
36.4
48.6
46.3
46.5
TruthfulQA shot)
66.8
69.9
66.5
62.6
55.7
67.9
67.6
BigBenchHarda shot, CoT)
83.0
80.4
83.6
78.5
82.6
84.8
85.0
DROP shot)
77.0
34.2
73.2
77.2
74.1
74.3
(4 shot CoT, Flex)
56.4
75.9
41.9
55.0
53.7
62.3
63.0
CSM8K(s (8 shot, CoT)
93.7
89.5
90.0
84.7
91.1
93.5
93.5
HumanEvalluasetn
93.6
94.0
89.6
94.1
92.9
92.4
92.4
HumanEvalt (paseee0)))))))
89.5
90.8
85.9
85.5
87.3
88.4
88.0
IFEval/prmpt loose)
88.0
87.6
76.0
82.1
82.6
83.2
AlpacaEval 2QUC % win)
33.4
47.7
28.4
26.3
49.6
49.8
(6 task task) avg.)
76.5
87.0
57.9
69.0
94.4
89.0
88.3
Table 6: Summary of TÜLU 3 results relative to peer 70B models. The best-performing model on each benchmark (i.e., in each row) is bolded. TÜLU 3-70B significantly outperforms prior Mfateoffthheart 70B models. Progress across various checkpoints highlight the contribution of each stage of the training in improving core skills. Nemotron Llama 3.1 70B is the only model in the table that fine-tuned from another post-trained model (in this case Llama 3.1 70B Instruct), while the others are from their respective base models. Many of the lowest values are caused by failing follow the few-shot formatting required for the evaluation or other repetitive errors for more details, see subsection 3.1.
Llama
Qwen
Magpie
Gemma 2 Instruct
Minis- tral SB Instruct
TüLU 3 TÜLU
TüLU 8B
Benchmark (eval)
8B
7B Instruct
DPO
Instruct
Avg.
62.9
66.5
60.4
60.6
65.1
MMLU shet CoT)
71.2
76.6
62.0
74.6
68.5
65.9
68.7
68.2
PopQAq15 shat)
20.2
18.1
28.3
20.2
29.3
TruthfulQA shot)
55.1
63.1
57.0
61.4
55.5
46.8
56.1
55.0
Big CoT)
71.9
70.2
55.2
64.9
70.8
69.7
68.7
69.0
61.5
54.4
49.4
58.8
56.2
61.3
62.5
62.6
MATH CoT, Flex)
42.5
69.9
5.1
29.8
40.0
31.5
43.7
CSMSKg shot, CoT)
83.4
83.8
61.2
79.7
80.0
76.2
84.3
87.6
HumanEvalluus@to
86.3
93.1
75.4
71.7
91.0
86.2
83.9
83,9
HumanEEvllqpsssstn
82.9
69.1
67.0
88.5
81.4
78.6
79.2
IFEvalprompt looss)
80.6
74.7
38.8
69.9
56.4
72.8
81.1
82.4
AlpacaFval 2q.C%
24.2
29.0
49.0
43.7
31.4
12.4
33.5
34.5
Safety avg.)
75.2
75.0
46.4
75.5
56.2
93.1
87.2
85.5
Table 7: Summary of TÜLU 3 results relative to peer 8B models. The best-performing model on each benchmark (i.e., in each row) is bolded. TüL.U 3-8B significantly outperforms prior state-of-the-are 8B models. Progress across various checkpoints highlight the contribution of each stage of the training in improving core skills. Many of the lowest values are caused by failing to follow the few-shot formatting required for the evaluation or other repetitive errors for more details, see subsection 3.1.
models (Dubey et al., 2024). However, generating diverse and high-quality data at scale is non-trivial, as LMs are susceptible to falling into repetitive modes or patterns, referred to as "mode collapse" (Kazdan et al., 2024). To ensure diversity in generation, we follow the recent cersona-driven methodology in Chan et al. (2024) to generate synthetic data. The key idea is to use different personas (e.g., "A machine learning researcher focused neural networks") with a data synthesis prompt (e.g., "create coding problem") to steer an LLM to synthesize data with corresponding perspectives. Specifically, we condition ~25KK
22

---

Published as a conference paper at COLM 2025
# Prompts used in
# Prompts used in
Category
Prompt Dataset
Count
Reference
SFT
DPO
General
TüLu 3 Hardcoded
24
240
OpenAssistant" 1.2.1
88,838
7,132
7,132
Köpf et al. (2024)
No Robots
9,500
9,500
9,500
Rajani et al. (2023)
WildChat (GPT-4 subset)*
241,307
100,000
100,000
Zhao et al. (2024)
Ultrafeedback
41,635
41 41,635
Cui et al. (2023)
Knowledge
FLAN 12,1
89,982
89,982
12,141
Longpre et al. (2023)
Recall
SciRIFF'
35,357
10,000
17,590
Wadden et al. (2024)
TableGPT
13,222
5,000
6,049
Zha et al. (2023)
Math
TÜLU Persona MATH
149,960
149,960
Reasoning
TüLu 3 Persona GSM
49,980
49,980
TÜLU 3 Persona Algebra
20,000
20,000
OpenMathInstruct? 2
21,972,791
50,000
26,356
Toshniwal et al. (2024)
NuminaMath-TTIR
64,312
64,312
8,677
Beeching et al. (2024)
Coding
TÜLU Persona Python
34,999
34,999
Evol CodeAlpaca"
107,276
107,276
14,200
Luo et al. (2023)
Safety
TüLU CoCoNot
10,983
10,983
10,983
Brahman al. (2024)
& Non-Compliance
TüLU 3 Wildjailbreak
50,000
50,000
26,356
Jiang et al. (2024)
TULU 3 WildGuardMix" 4.1
50,000
50,000
26,356
Han et al. (2024)
Multilingual
Aya*
202,285
100,000
32,210
Singh et al. (2024b)
Precise IF
TÜLU 3 Persona IF
29,980
29,980
19,890
TÜLU IF-augmented
65,530
65,530
Total
23,327,900 ,66
939,344
425,1457
Table 8: Summary of our prompt dataset: data for training stages are selected from these prompts. New datasets released with TÜLU 3 are color-coded for emphasis. Existing datasets we modified due contamination are marked with a. Datasets with prompts used in TÜLU 1 or 2 are marked with or respectively. Datasets marked with L are downsampled from their original datasets, datasets marked with are upsampled. Note that all datasets were filtered to remove specific keywords (e.g., OpenAl) and empty messages, resulting in slightly lower than reported counts. All TÜLU 3 datasets with Persona expand the methodology of Chan et al. (2024). The percentages listed per category out of the total prompts. Preference count is marked with 7 to note that not all prompts are used in both the 8B and 70B mixes for exact details see Table 15.
personas from Persona Hub (Chan al., 2024) to generate prompts targeting specific skills such precise instruction following, math and coding, We detail our procedure for each select skill below. Prompts used to generate these instructions can be found in Appendix J.1. Additionally, we build upon our previous efforts in Brahman et al. (2024); Han et al. (2024); Jiang et (2024), to generate noncompliance and safety data.
Precise Instruction Following. Precise instruction following is the ability to follow ver- ifiable instructions in natural language, such as "your answer should contain exactly paragraphs," that can be automatically verified with heuristics. We use our persona-driven approach to synthetically generate verifiable instructions covering 25 different constraint types defined in IFEval benchmark (Zhou et al., 2023). More concretely, ually writing 1-2 example instructions per constraint (e.g., number of words), resulting in total of 33 verifiable instructions which we used seed prompts. We then generate new
start by man-
23

---

Published as a conference paper at COLM 2025
instructions using GPT-4o (OpenAI, 2024)* given a data synthesis prompt, persona, and a single verifiable instruction as an example. Figures 31 and 32 show the exact prompis used to generate the instruction and its corresponding response, respectively. In total, we collected 29,980 verifiable instruction-respone pairs which we call II-PERSONA-SST. Lastly, we also generate another type of prompts targeted for constrained instruction following by randomly sampling instructions from the TüLU 2 SFT mix and combining them with constraints from the taxonomy in Zhou et al. (2023). We call that set IF-AUGMENTED. These prompts are only used for the DPO and RLVR stages.
Math and Coding. We follow a similar persona-driven approach to synthetically gener- ate diverse math word and coding problems. Math problems include those that require advanced mathematical skills as well as grade school problems. For coding, we generate Python programming questions that are solvable by entry- to medium-level programmers. Unlike precise instruction following, we zero-shot prompt GPT-4o to generate problems that are unique and specific to given persona input. Having generated the problems, we then generate multi-step math solutions using GPT-4o, and Python programs using claude-3-S-sonne
Exact prompts used to generate problems and solutions are provided in Figures 34, 36, 35, and 37, respectively. In total, we collected ~220 and 35K instances for math reasoning and coding.
Noncompliance and Safety. As we enhance models' capabilities to assist users effectively, crucial ensure they can reliability reject unsafe and appropriately handle nuanced and out of scope queries. To support this, we curate a set of noncompliance queries (Brahman et al., 2024) that the model ought to not comply with, alongside safety-related direct and adversarial prompts (Han et al., 2024; Jiang et al., 2024) covering both benign and harmful scenarios. Our noncompliance and safety prompts are either curated from existing datasets (Zhang & Choi, 2021; Zhao et al., 2024) or synthetically generated from the GPT model family. More specifically, our noncompliance prompts are obtained based our contextual noncompliance taxonomy from Brahman et al. (2024), spanning multiple categories including incomplete, unsupported, indeterminate, and lumanizing requests (in addition to unsafe requests). Our safety-related prompts are carefully selected among synthetic adversarial prompts, synthetic vanilla (direct) requests, real-world user-LLM interactions (i--Th---ild), and curated annotator-written examples to maximize coverage, diversity, and balance.
B.3 Decontamination
Matching Method. We experimented with full-string, n-gram, and embedding match- ing and found that n-gram matching yielded the most useful results - while embedding- based methods can in principle identify non-trivial contamination like that due to para- phrasing (Yang et al., 2023), we found it difficult to distinguish mere distributional similarity from actual paraphrasing. Moreover, partial surface-level overlap using n-gram matching successfully identified cases of contamination where the instances were trivially different, e.g., a math problem where only the numbers differ.
Identifying Matching Instances. Since completions in training datasets are often regener- ated using language models, we chose to compute overlap in the prompts alone (or more generally user turns in multi-turn dialogues). We used 8-gram matching for our contam- ination checks following (Dubey et al., 2024; Singh et al., 2024a). For each token in test instance, we consider to match a token in a train instance if the two instances share an 8-gram containing that token, and we consider the test instance itself to have significant overlap with a train instance if more than 50% of the test tokens have 8-gram matches with the same training instance.
Decontamination. We consider a training set to be contaminated if any number of its instances overlap with more than 2% of the instances in any of the evaluations in our development and unseen suites. We remove all the training sets that were contaminated with our unseen evaluations. For training sets that were contaminated with our development evaluations, we removed the entire dataset if doing so did not significantly impact the
We use GPT-40-2024-08-96 for all our persona-driven data synthesis, unless otherwise stated.
24

---

Published as a conference paper at COLM 2025
80
Deformane
60
40-
20
Avarage
BBH
GSMBK
iFbval
MATH
MMLU
Safety
TULU2
Intermediate Mix 1
Intermediate Mix 2
Intermediate Mix TELU
Intermediate Mix 4
Intermediate Mix 5
Figure 5: Average and selected skill-specific performance from training Llama 3.1 8B on our initial TüLU 2 SFT mix, and our intermediate and final TÜLU 3 SFT mixes. Intermediate mixes 1, 2, and 3 were the result of adding new datasets to improve performance. Interme- diate mixes 4 and were the result of running multiple rounds of decontamination, causing small drops in performance.
performance of the resulting model; otherwise, we removed the specific instances that match any test instance.
C SFT
C.1
Key Data Experiments
We also ran series of controlled experiments after developing our final SFT mix to explore the importance of different decisions made during data mixing and training.
Model
Avg.
MMLU
TQA
PopQA
BBH
CHE
CHE+
GSM
DROP
MATH
IFEval
AE
Safety
TULU 3 3B SFT
60.1
62.1
46.8
29.3
67.9
86.2
81.4
76.2
61.3
31.5
72.8
12.4
93.1
w/o WildChat
58.9
61.0
45.2
28.9
65.6
85.3
80.7
75.8
59.3
31.8
70.1
7.5
95.2
w/o Safety
58.0
62.0
45.5
29.5
68.3
84.5
79.6
76.9
59.4
32.6
71.0
12.4
74.7
Persona
68.3
84.5
79.0
76.8
Math
93.5
Table 9: Performance during our SFT ablations, showing the effect of removing safety, WildChat, Persona, and Math data in isolation. We find that: 1) diverse chat data is beneficial for most skills, most noticeably Alpaca Eval, 2) safety performance is generally orthogonal to general performance, 3) our new Persona datasets improve all of the skills that they target, and using mathematics test case, adding high quality skill-specific data substantially improves skill-specifi performance.
Diverse Chat Data. In our mix we also emphasized adding diverse chat data, mainly from WildChat. We show the impact of removing WildChat in Table 9, and we see that there is a small but noticeable degradation highlighting the importance of diverse real-world data.
most skills, most noticeably on Alpaca Eval,
Safety is Orthogonal. We found that safety SFT data generally orthogonal to our other datasets. We report the effect of removing our safety-specific datasets in Table 9, and that most skills stayed roughly the same, except the safety average. We also found that adding constrastive prompts, such as those in CoCoNot, were helpful for preventing our models from over-refusing safe prompts.
New Persona Data. Our new Persona datasets were built to target specific skills: math- ematics, coding, and instruction following. In Table 9 show that performance
25

---

Published as a conference paper at COLM 2025
HumannEvl(+)), GSM8K, MATH, and IFEval drop after removing our Persona datasets, showing the value of creating diverse, skill-specific SFT datasets.
Targeting Specific Skills. A large portion of our focus targeting specific capabilities. Using mathematical reasoning as an illustrative example, we show in Table 9 the impact of our mathematics sppeciffi data both GSM8K and MATH. We see that our mathematics- -ppccfcccsssiccccccc SFT data substantially improves both GSM8K and MATH, showing the value of the data included in the final mix.
collecting creating datasets
Amount of SFT Data. In Figure 6, we show the effect of taking stratified subsamples of our SFT mix. We find that models continue to improve average as more SFT data is included, and we see large improvements on metrics like GSM8K as we increase the amount of data the full mix. Interestingly, TruthfulQA performance actually drops as the amount of data in the mix increases. We do not increase our SFT data size beyond the current mixture because we allocated other prompts for preference optimization.
Hyperparameter
8B
70B
Learning Rate
5 10
10
Learning Rate Schedule
Linear
Linear
Batch Size (effective)
128
128
Max Token Length
4,096
4,096
Warm up ratio
0.03
0.03
Number of Epochs
Table 10: SFT Training Hyperparameters.
C.1.1 Key Training Experiments
Choice of Base Model. We also test the effect of training different base pretrained models on mathematical performance using our full SFT mix. In Table 11, we show the impact of changing the model's size by training on both Llama 3.1 8B and 70B, and the impact of adding domain-specific pretraining data by training on Qwen 2.5 7B and Qwen 2.5 Math 7B. In both cases, we see substantial improvement in both GSM8K and MATH, highlighting the importance of both model size and pretraining data for downstream skills.
Base Model
GSM8K
MATH
Llama 3.1 8B
76.2
31.5
Llama 3.1 70B
91.1
53.7
Qwen 2.5 7B
79.2
49.4
Qwen 2.5 Math 7B
86.3
56.4
Table 11: Mathematical performance of different base models trained our mix. We see that 1) training on larger models leads to better performance, and 2) adding skill-specific pretraining data also leads improved performance, even for the same size model.
Chat Template Variation. During creating TÜLU 3, we explored changing the chat template used to guide the generation of finetuned models. We made a small change the chat template used in previous TÜLU versions, specifically removing the new line at the end of the template (before the model response). The performance between different changes to the chat template is shown in Table 12 on an early version of our SFT setup. We found that replacing the newlines the end of assistant messages with an eos token resulted the best performance, but we opted not to use this to avoid generation inconsistency with later steps in our post-training pipeline. The chat template can be found in our codebase upon release.
26

---

Published as a conference paper at COLM 2025
Chat Template
Avg.
TÜLU (replace {n w/ eos)
53.0
Zephyr
52.9
TÜLU 3 (no \n)
52.8
TÜLU 2 template
52.6
Llama 3 template
51.6
Table 12: The impact of different chat templates on SFT model performance, trained using an intermediate SFT mixture on Llama 3.0. While replacing the newline does best, we instead opted for simply removing the newline to avoid complexity.
GSMIK
HumanEve
Trustruio
5%
10%
25%
50%
75%
100%
Figure 6: Average and skill-specific performance on stratified subsamples of our final SFT mix. We find that our full mix performs best overall.
Random Seeds and Model Soups. We also explored changing the random seed during SFT, and then using those models to create model soups (Wortsman et al., 2022). Table 13, compare training 8B and 70B models with multiple different seeds with the best model soup. We see that SFT performance noticeably varies based on the seed, highlighting the importance of multiple training runs, and that the best model soup does not always outperform the best single training run. Because of this, we use the best single SFT training run for each model size as our final SFT models.
Model
Seed
Average
Model
Seed
Average
TÜLU 8B SFT
42 (Default)
59.9
TüLU 3 70B SFT
42 (Default)
71.8
123
60.1
123
70.0
456
59.8
456
72.6
789
59.8
1011
59.8
Best Model Soup 42 & 123
60.2
Best Model Soup
123 & 456
72.5
Table 13: Average performance of our 8B and 70B SFT models using random seeds, and compared against the best model soup using the models trained with different seeds. We find that the best random seed is comparable to the best model soup, so for consistency we use the best single SFT run as our final SFT model.
C.1.2 Batch Aggregation
Early during training TÜLU 3, we noticed a gap in performance between SFT models trained framework and models trained in other settings such TPUs. 2 We found this
Relevant code: https: htoo:///githbbs hww.ttt...mmhhaiiit... hwco//hamishhio www..//wwwwiiiiaaaaaaaaaaa
27

---

Published as a conference paper at COLM 2025
issue was largely due to (recently widely-reported) issue with loss aggregation inside Transformers (Wolf et al., 2020): Averaging the loss across padding tokens without taking into account gradient accumulation
distributed training setups.
Here, we illustrate the issue with an example. Assume we have two samples in a batch, with n R2 non-padding tokens and m M2 padding tokens. If we pass both samples into the default Transformers forward pass at the same time, we get:
Inj 1n L
(1)
11 + 112
However, if we apply gradient accumulation, feeding in the two samples separately, com- puting loss, and then dividing, our loss is instead computed like:
Inj
In2
12
(2)
2
That is, in the second case we weight each example equally, while in the first we weight each token equally. As such, changing gradient accumulation can have large effects on performance due to effectively changing sample weightings, reported by Muennighoff et al. (2024). similar issue occurs in distributed training due to cross-device averaging. We refer to recent reports on this issue for more in-depth explanation."
To fix this issue, we opted generally to use a sum loss instead of averaging ('mean loss') when training. This removes the issue by simply removing the denominator from the above equations and requires an adjustment to learning rates. This effectively weights all tokens equally (which we found led to generally better performance for initial mixtures). We validated the perfomance of our setup by finetuning Llama 3.0 on the TüLU 2 SFT mixture using a variety of learning rates, epochs, and loss types shown in Figures 7 and 8. Ultimately, we found that using a sum loss with a learning rate of 5.00E-06 worked best. Surprisingly, we additionally found that training for longer did not yield further improvements, and so used 2 epochs for training.
50
52
Avg. Pert.
Avg. Pert.
45
40
Sum Loss
2.0x10 -6
5.0x10 1.0x10 2.0x10 Learning Rate
2
Number of Epochs
Figure 7: Average performance when fine- tuning Llama 3.0 on the TÜLU mixture using differing loss types and learning rates. We find that of 5e-6 with a sum loss works best.
Figure 8: Average performance when fine- tuning Llama 3.0 on the TüLU 2 mixture using sum loss and LR of 5e-6 for varying numbers of epochs. We find using 2 epochs works best.
C.2 Training Settings
To train our TÜLU 3 models, we used between 4 and 16 8xH100 nodes with high speed interconnect. The final 8B model is trained on 32 GPUs for 6 hours and the 70B model was trained 64 GPUs for 50 hours. We used effective batch size of 128 and maximum sequence length of 4,096 tokens. We trained for two epochs using a learning rate 5e-6 for our 8B models, and 26-6 for our 70B models, which we found after hyperparameter search.
https://mielleerr.gittee wtw.ob..o///llloooo htt.hi//..o.ooooogggriiiieet accumulation html
28

---

Published as a conference paper at COLM 2025
SFT Mix
P-IF
WildC.-IF
SFT-IF
WC
WC
UF
DA
UF
CocoNot
Avg.
V
62.27
/
V
61.99
V
61.83
/
/
61.76
61.59
/
/
V
V
61.55
61.35
V
61.29
61.25
V
61.17
60.87
60.86
60.84
60.54
Table 14: Some of our dataset mixing experiments to obtain the final preference dataset mix. We include prompts from Daring (DA), our SFT Mix (SFT), Ultrafeedback (UF), Persona prompts for different skills (P-IF, P-Code, P-Math), TÜLU 3 instruction following prompts (TULU 3-IF), ie. If-Augmented, CocoNot, the IF subset of Daring Anteater Wang et al. (2024d) and WildChat (WildC.). (a: prompts used during SFT, 6: prompts from datasets subsampled, yet unused, for SFT, 8: only used the prompts, the completions and preferences were regenerated using the pipeline described in subsection 3.4).
Dataset
Count
8B
70B
SFT Reused On-policy
19444
SFT Reused Off-policy
96911
SFT Reused Instruction Following
65530
WildChat Instruction Following
10792
WildChat Reused
17207
WildChat Unused
82783
Ultrafeedback (Cleaned)
41635
Persona Instruction Following
19890
Total
354192
271409
334302
Table 15: Summary of our best preference dataset mixes for TÜLU 3 8B DPO and TULU 70B DPO.
Our hyperparameter settings are also summarized in Table 10. For merging experiments we used mergekit* (Goddard et al., 2024), using linear weighted averaging.
D Preference Finetuning
D.1 Key Findings of Data Ablations
We perform several ablations to inform the design decisions of the synthetic preference pipeline and the composition of the TÜLU preference mix (subsection 3.4).
'nttps://github https///gggtuuuu.cc/rrreeaaaaale wwwm//aceeeaa/meeeeeee.e
29

---

Published as a conference paper at COLM 2025
80
Performance
60
Performance
60
40
40
20
20
0
AVG
AE
MATH
GSM8k
AVG
AE
MATH
GSMBK
Initial SFT
5%
10%
Initial 8B SFT
64k
180k
25%
50%
75%
100%
383k
Figure 9: Effect of scaling the size of the preference dataset, specifically the num- ber of unique prompts, on downstream DPO model performance (AE: AlpacaE- val).
Figure 10: Effect of scaling preference dataset by duplicating prompts on down- stream DPO performance using the Ultra- feedback dataset. All sizes have the same number of unique prompts (64k).
Scaling the Number of Unique Prompts Improve Downstream DPO Performance. First, we investigate whether increasing the number of prompts will yield improvements in down- stream DPO performance. To do so, we measure the downstream DPO model performance at different sizes of fixed amount of preferences with unique prompts. Figure 9 shows that there are noticeable performance gains across several metrics as the size of the preference dataset increases. This suggests that dataset scaling is important to achieve improvements in downstream model performance: our final preference mixes (Table 15) contain more than 270k data points for the 8B model and more than 330k instances for the 70B model, which is considerably bigger than many available preference datasets.
We also explore whether duplicating prompts, i.e. same prompts with different responses, is a viable approach to scaling the size of preference dataset and whether it will lead to gains in downstream DPO performance. To do so, we expanded the Ultrafeedback dataset, which originally had four responses for each prompt, by creating additional pair combinations of responses. This expansion will naturally cause duplicated prompts, but with different chosen and rejected pairs sampled from the four responses in UltraFeedbback leading to preference datasets with 64k-, 180k,, and 383k instances. Figure 10 shows that, on average, the 383k-size preference dataset performs similarly to the 64k preference dataset. We also observe a slight performance degradation on DROP, GSM8k, and AlpacaEval as the number of duplicated prompts increase. This suggests that scaling via prompt duplication does not necessarily yield into significant gains in downstream DPO performance, and investing in the collection of unique prompts and proper mixing is more important for downstream evaluations.
C Lead to Higher Performance vs. Reusing Prompts From SFT Mix. We then compare including new prompts and re-using prompts from the SFT stage on their effect on downstream DPO performance. To do so, we sampled 100k prompts from the SFT dataset mix that were used during training (as shown in Table 8) and compare it against prompts from the same open datasets (e.g., Apenssiittnnt,, SciRIFF, Aya, Persona, WildChat, etc.) we subsampled from but left unused during SFT. Figure 11 shows that the umused dataset has slightly higher performance as opposed reusing prompts. This suggests that the presence of new prompts can help improve downstream DPO performance. Though, as seen in our best mix, combining unused and reused prompts seems to lead to the best result.
Unused Prompts
On-policy Data Improves Downstream DPO Performance. We investigate whether the inclusion of on-policy data, i.e., text generations from the SFT model that will be used as the base model for preference finetuning, improves downstream model performance, Given the same set of prompts sourced from the SFT mix in subsection 3.3, we generate preferences from off-policy models and compared it to mix that is strictly on-policy (i.e., one of the
30

---

Published as a conference paper at COLM 2025
LLM Judge
Avg.
MMLU
TQA
PapQA
BBH
CHU
CHU++
GSM8k
Drop
MATH
IFEval
Safety
GPT-4o
57.3
64.8
36.1
30.1
66.3
87.0
80.7
75.3
62.7
20.3
60.4
20.6
62.7
Laama 3.1 405B
57.2
64.8
56.0
30.3
67.4
86.2
8088
75.1
62.0
20.1
59.0
21.5
62.8
GPT4 Turbo
57.0
35.7
30.1
66.4
86.6
79.4
62.6
20.1
59.9
62.2
GPT-4o Mini
56.9
64.4
35.4
30.4
66.2
86.6
79.8
74.8
60.7
20.9
60.1
21.4
61.6
Llama 3.1 70B
56.6
64.3
35.5
30.2
66.6
85.3
81.4
74.8
62.1
20.1
58.2
18.6
62.2
Table 16: Performance of DPO models trained on preference annotations by different LLM judges. Due to the proximity of the numbers, we have not bolded the max per evaluation.
response is always from the Initial 8B SFT model, and the other response is from the off- policy models). We also compare it on a combination of both on-policy and off-policy data: we sample instances from the strict on-policy dataset and add it to the off-policy dataset so that the responses from each model is distributed equally. Figure 12 shows that including on-policy data improves aggregated downstream DPO performance compared to a completely off-policy dataset where prompt completions were sampled from other models.
Aggregated Perf. on All Evals
61
61.0
Aggregated Perf. on All Evals
61
60.7
60
60
60.0
59.8
59
59
58
58
58.2
57
57
56
56-
55
55
Initial
DPO w. DPO w. Combined Off-policy On-policy (120k) (100k) (100k)
Reused Prompts (100K)
Prompts (100k)
8B
Figure 11: Effect of reusing prompts from SFT mix and new prompts from the same datasets subsampled for the SFT dataset mix.
Figure 12: Effect of including on-policy data during the Response Generation stage of the synthetic preference data pipeline on downstream DPO model per- formance.
Performance Across LLM Judges are Similar, with GPT-4o Leading Slightly Ahead. In order to determine which judge to use for obtaining preference annotations, we test several commercial and open-source LLM judges such as GPT-4 (GPT-4-turbo-2024--44 20T-40-2024-08-65 gpt-4o-mini-2224-07-B1 and Llama 3.1 (70B and 405B) on the same set of 10k randomly-samples Ultrafeedback prompts and responses. In general, GPT-4o, Llama 3.1 405B, and GPT-4 Turbo perform similarly across all benchmarks, with GPT-4o leading slightly ahead the aggregated average performance shown in Table 16. In the synthetic preference pipeline for TÜLU 3, we opted for GPT-40-2024-88 due to its ease-of-use, cheaper cost per request, and batch inference speed via OpenAl's Batch API,
Going Beyond Ultrafeedback. Previous work on preference learning using openly available datasets has shown that the UltraFeedback (Cui et al., 2023) preference dataset generally outperforms other preference datasets (Ivison et al., 2023). In Figure 13 we show that we were able significantly surpass DPO training on UltraFeddback by training on our best mix. The improvement is greater for the 70B model (+3.3 vs. +1.8), we hypothesize that this is because EittbaFeedbck's completions are mainly sourced from models that are less capable than the 70B model we starting with. Helpsteer2 Wang et al. (2024d), another high-quality preference dataset, also performs lower than our best mix on the 8B model.
Persona Preference Data. From the three persona preference datasets targeting instruction following, coding and math skills, only TÜLU 3 Persona IF improves the average eval score and the targeted IFEval score (see Figure 14). Neither TŪLU 3 Persona Math nor TÜLU
https://platfomm openai httpaaacccc/moosssuuueee////ee/e
31

---

Published as a conference paper at COLM 2025
Aggregated Pert. on All Evall
Aggregated on Evals
75
71.1
70.1
70.4
60.5
65
60
56-
Initial
DPO W.
DPO W: Ultra- Feedback
DPO w.
Initial
DPO Helpstner2
DPO w: Ultra- Feedback
DPO Best Mix
89 SFT
Helpsteer2
Best Mix
70B SFT
Figure 13: Effect of different DPO mixes on 8B and 70B models: UltraFeedback, Helpsteer2, and our best preference mix.
86.3
79.
81.9
72.1
74.7
72.3
Performance 136)
60
60.8
58.7
40
40.0
40.9
30.5
20
SFT Reused
Persona
Persona Math +
Persona Code
Average
IFEval
MATH
CHU
Figure 14: Adding persona preference data to the SFT Reused mix for DPO.
Persona Code improve their respective targeted evaluations and slightly harm the average score. We therefore only include the TüL.U 3 Persona IF preferences in our final mix.
Targeting IF. We created preference data targeted improve model's precise instruction following skills.
1. Persona IF: We take subset of our collected instruction following SFT dataset, -S-PERSONA-SFT and convert it into preference dataset. Each example in Ir- PERSONA-SFT dataset contains a (prompt, constraints, response) tuple. We start by rewriting each prompt in the subset relax one of the given constraints. More specifically, we prompt GPT-4o to generate rewrites such that the new response to the modified prompt no longer a valid response for the original prompt (does not satisfy all the constraints). We then use the response to the new modified prompt as the rejected response, and create (chosen, rejected) pairs to form our In-PERSONA-PREE dataset containing close to 20K examples.
2. IF-augmented: We randomly sample instructions from the TÜLU 2 SFT mix and combine them with constraints from the taxonomy Zhou al. (2023). The chosen and rejected completions are obtained through the synthetic pipeline in $3.4.
3. WildChat IF: We sample instructions from WildChat (Zhao et al., 2024) which contain constraints. For this purpose we asked GPT-4 extract whether not prompt includes a constraint.
For IF-augmented, we run two analyses. We generate an additional set more than 66k instances and we then run the chosen completions through constraint verifier functions, and only add those instances to the final set which actually fulfilled the constraint(s). This leaves us with cleaned set of about 26k preferences, which call IF-augmented-
32

---

Published as a conference paper at COLM 2025
59.6
Aggregated Pert. on All Evals
59
59.2
59.0
80
81.1
81.7
70
76.3
77.3
58
58.4
58.2
57.9
60
62.3
62.0
61.4
61.4
50
57
40-
30-
56
20
10
Best [Inc. IF-au.)
Persona IF
IF-augmented
Persona IF auggeeeted- verified
55
MultiPref
Helpsteer2 Ultrafeedback
verifled
Original dataset
Average
IFEval
Regen. using synthetic pipeline
Figure 15: Performance of different IF-targeted preference mixes, average and IFEval. Best here consists of our final best mix for the SB model (minus Persona-IF).
Figure 16: Comparing the use of the original completions to regenerating completions using our synthetic pref- erence pipeline.
verified. In Figure 15 we show that the IF- IF-persona preferences significantly improve IFEval scores beyond the baseline mix, while minimally harming average performance. The IF- augmented-verified dataset improves IFEval performance only by 1 point, while also slightly harming the average performance. Combining IF-persona with F7-augmented-veriine to the best IFEval performance, but to a slightly lower average. We therefore choose to include auggmeedded (not verified) and Persona IF in the final 8B DPO mix, which leads to both satisfiying average and IFEval score.
leads
Wildchat. Our ablations show that adding preference data consisting of WildChat prompts and chosen rrejected pairs obtained using our synthetic preference data pipeline, generally improves DPO performance. Ablations in the related sections reveal that adding WildChat prompts seen during SFT training to the DPO mix leads better average performance than combining the unused with the reused WildChat prompts.
Comparing original preference datasets and their regenerated counterparts. We also investigate whether the preference dataset generated by the synthetic pipeline in $3.4 can yield to gains downstream DPO performance on existing datasets. To do so, we take the prompts from open-source datasets such Helpsteer2, Ultrafeedback, and MultiPref (Miranda et al., 2024), then regenerate their completions and preference annotations using the synthetic data pipeline. Figure 16 shows that the downstream DPO performance of the regenerated dataset is better than the original dataset, suggesting that the synthetic pipeline itself can yield to performance gains.
D.2 Preference Tuning Recipe and Analyses
D.2.1 Hyperparameter and Algorithm Design
In light of the significant amount of work on improving DPO and related algorithms since the release TÜLU 2, we revisited our hyperparameter and algorithm choices alongside our preference datasets. We ablated both algorithm and hyperparameter choices using an early SFT checkpoint and the UltraFeedback dataset. We explored using DPO, SimPO (Meng et al., 2024), and length-normalized DPO. Our results are shown in Table 17. We found that only length- Length-normalized DPO outperformed our base checkpoint overall, and so further tuned it, resulting in the final hyperparameters shown in Table 19.
33

---

Published as a conference paper at COLM 2025
Algorithm
LR
γ -
ratio
B
Epochs
Batch Size
Average Score
SFT Base
55.7
SimPO
5.00E-07
0.5
2
128
51.8
SimPO
5.00E-07
0.3
10
128
52.9
DPO
5.00E-07
0.1
55.2
PPO
1.00E-06
0.0325
64
54.5
PPO
1.00E-06
0.05
55.5
DPO-norm
1.00F-07
5
3
32
56.1
DPO-norm
5.00E-07
10
32
55.2
DPO-norm
5.00B-07
15
3
32
55.7
DPO-norm 5.00E-07
2
46.8
DPO-norm 5.00E-07
5
3
32
53.4
DPO-norm
5.00E-07
5
57.3
Table 17: Hyperparameters and algorithms examined for DPO tuning. We use UltraFeedback as the training dataset in all cases, and train on top of an early TČLU 3 version. DPO-norm refers to the length-normalized DPO variant proposed in Meng et al. (2024). We explore hyperparameters suggested by prior work (Meng et al., 2024; Ivison et al., 2023). For PPO, we train reward models on UltraFeedback and reuse prompts during online training, following the hyperparameters in Ivison et al. (2024). We find that length-normmlized DPO performs best overall.
Data
LR
Avg. Performance
Hyperparameter
8B
70B
5.0 X 10
72.74
Learning Rate
5
10
10
2.0 x 10
71.17
Learning Rate Schedule
Linear
Linear
Mix 1
1.5 10
71.12
Batch Size (effective)
128
128
1.0 x 10
71.06
Max Token Length
2,048
2,048
5.0 10"
71.14
KI. penalty coefficient
5
5
Mix 2
2.0 x 10
74.35
Warm up ratio
0.1
0.1
Number of Epochs
1
Table 18: Learning rate ablations for the 70B DPO model, for two different preference mixes:
Table 19:
Final DPO Training Hy- perparameters. We use the length- normalized variant of DPO proposed in Meng et al. (2024).
Mix 1: Tüll-3-Persona-IE, Tulu-3-Helpsteer), Ultrafeedbbckk, Tulu-3-SFT-reused (On-policy), Mix 2: Best 70B Mix (both trained on an older SFT base).
We lowered the learning rate and increased the batch size for the 70B training based on the fact that it is common to lower the learning rate and increase batch size when doing SFT with larger models (Touvron et al., 2023).
The 8B DPO model trained for 10 hours on 8 Nvidia H100 GPUs and the 70B DPO model is trained for 19 hours on 64 interconnected H100s.
The DPO training uses a maximum sequence length of 2048.
Learning Rate Ablations for 70B. We ran a small hyperparameter search over set of
learning rates using a generally well performing preference data mix and our final best mix. Table 18 shows that either learning rate of 2.0 x 10-7 or 5.0 x 10-7, depending on data mix, performs better than a lower learning rate. For our final DPO models we decided on using a learning rate of 2.0 x 10-7.
Telsse-H,) Ultrafeedback, Tulu-3-SFT-Used (On-policy).
34

---

Published as a conference paper at COLM 2025
Hyperparameters
for optimizing a RM
for optimizing against RLVR
Discount Factor y
1.0
1.0
General Advantage Estimation
0,95
0.95
Mini-batches Nmb
PPO's Clipping Coefficient
0.2
0.2
Value Function Coefficient C₁
0.1
0.1
Gradient Norm Threshold
1.0
1.0
Learning Rate Schedule
Linear
Linear
Generation Temperature
1.0
1.0
Max Token Length
2,048
2,048
Max Prompt Token Length
2,048
2,048
Penalty Reward Value for
-10.0
-10.0
Responses without an EOS Token
Learning Rate
3 10
3 10 (1 10" for 70B)
Batch Size (effective)
224
224 (640 for 70B)
PPO Update Iterations K
1
4
Response Length
1,024
2,048 (1,024 for GSM8K only)
Total Episodes
300,000
100,000
KL penalty coefficient (5)
[0.05, 0.03, 0.02, 0.01]
[0.1, 0.05, 0.03, 0.01]
Warm up ratio (w)
[0.1, 0.0]
[0.0, 0.1]
Table 20: The hyperparameters of PPO used for 1) optimizing against general RM and 2) optimizing against the verifiable reward function. The differences between the hyperpa- rameters are highlighted. The final 8B RLVR model used & 0.05 and w 0.0; the final 70B RLVR model used β = 0.07 and w = 0.07
Comparison Between PPO and DPO. We also conducted a more in depth ablation study comparing PPO and DPO later in development. We anchored a DPO preference mix in the development history train RM. We use the same setup as Stiennon al. (2020); Ouyang et al. (2022); Huang et al. (2024a), we only extract the RM's logits at the end-of-sequence (EOS) token as the reward model. Also, the linear head output reward scalars initialized
with weights according to (N /////////deeee + 1)). We use the same prompts in the DPO preference mix to make controlled comparison between DPO and PPO.
The reward model was trained only once and we did not attempt to tune the RM's perfor- mance. Evaluating RM's performance can be tricky because strong RM performance on RM-specific benchmarks does not necessarily translate to better downstream performance for PPO (Ivison et al., 2024; Chen et al., 2024). Furthermore, iterating with RM and PPO is more expensive than iterating with DPO, so we decided to do most of our preference tuning experiments via DPO. The hyperparameters for the RM and PPO can be found in Table ?? and Table 20. The results can be found in Figure 17.
Here are our findings:
1. PPO Gets Similar Average Scores with DPO in this Non-Tuned Setup Overall, we found that PPO could reach a comparable level of performance to DPO (albeit slightly lower) in this controlled setup.
2. PPO is More Computationally Expensive The PPO runtime roughly 28 hours using two nodes, whereas the DPO runtime is about 4 hours using a single node.
we use more computational budget or do more tuning, it is entirely possible that we can push up the PPO's performance even higher. However, given limited resources and the
35

---

Published as a conference paper at COLM 2025
PPO Experiment Performance Across Steps
Techniques Reduce Peak GPU Memory Usage
Paak Allocated Memory
40
Ave meeaaee
20
wm0.0, 6-0.01
w=0.1. 6m0.03
56
8-0.02
M-0.11 8-0.05
20
w=0.0,
B=0.03
w=0.1, B=0.05
w=0.0, 8=0.05
w=0.1, 8=0.05
0-0.1,
8-0.01
baseline
w.0.1,
baseline
52
D80 baseline
DPO
DPO cache logprobs
200
400
600
800
1000
1200
1400
cache logprobs
separate forward
Steps
Figure 17: The average scores of PPO runs with different learning rate warm-up ratios w, KL penalty coefficient B. PPO can get sim- ilar (though slightly lower) average scores as DPO.
Figure 18: The peak GPU memory allocated be reduced by caching the reference pol- icy's logprobs on the preference dataset and doing forward passes separately for the cho- sen and rejected pairs.
Prompt Dataset
Count
Verification
Reference
GSM8K Train
7,473
Exact match against extracted answer
Cobbe et al. (2021)
MATH Train
7,500
Exact match against extracted answer
Hendrycks et al. (2021)
IF verifiable
14,973
Prompt-specific verifiers
Total
29,946
Table 21: Summary of our verifiable prompt dataset. New datasets released with TÜLU are color-coded for emphasis.
subtlety in RM evaluation, using DPO for preference tuning seems more economical. We decide to use PPO primarily for RLVR, to introduced in Section 3.5.
D.2.2 Infrastructure for Scaling DPO
To run the 70B DPO training, we found it useful to implement two key optimizations for reducing the GPU footprint of DPO training:
1. Caching DPO Log Probs To reduce GPU memory usage, we pre-compute and cache log probabilities across the dataset using the initial model, rather than keeping reference DPO model in memory during training like the canonical implementa- tion (von Werra al., 2020; Rafailov et al., 2024). This optimization eliminates the need to allocate GPU memory for the reference model.
2. Separate Forward Passes for Chosen and Rejected Sequences The canonical DPO implementation (von Werra et al., 2020; Rafailov et al., 2024) also concatenates the chosen and rejected sequences during the forward pass, effectively doubling the batch size and increasing GPU memory requirements. To save GPU mem- ory, we simply perform the forward passes separately on the chosen and rejected completions.
We empirically validated these two techniques the Llama 3.1 model and found they resulted near identical training losses. As expected, the model uses less GPU memory when using the two techniques 8xH100, as shown in Figure 18.
36

---

Published as a conference paper at COLM 2025
Final RLVR Runs
Veriobll Rewards
M
Oivergence
Resparse Langth
5.0
500
15
20-
0K
15K
20K
30K
40K
50k
00
30K
20K
30K
40K
50K
0K
10K
20k
30K
40k
50K
Episodes
Final 8B run
Final 70B run
Figure 19: Rewards, KL divergence, and average response length on GSM8k train set over episodes for our final RLVR runs. We mark the point we choose the 8B and 70B checkpoints with green and purple star respectively.
Model Size
SB
70B
Category
Benchmark(jyal Setting)
Llama 3.1 Inst.
TÜLU DPO
TÜLU RLVR
Llama 3.1
TÜLU3 DPO
TÜLU 3 RLVR
Inst.
Avg.
62.2
64.4
64.8
73.4
75.9
76.0
Knowledge
MMLU shot, Cot)
71.2
68.7
68.2
85.3
83.3
83.1
Pop shot)
20.2
29.3
29.1
46.4
46.3
46.5
TruthfulQAí6 shot)
55.1
56.1
55.0
66.8
67.9
67.6
Reasoning
BigBenchHard/3 shot,
62.8
65.8
66.0
73.8
81.8
82.0
DROP shot)
61.5
62.5
62.6
77.0
74.1
74.3
Math
MATH H(4 shot CoT, Flex)
42.5
42.0
43.7
56.4
62.3
63.0
GSM8K shot, CoT)
83.4
84.3
93.5
Code
HumanEvalpaas@1)
86.3
83.9
83.9
93.6
92.4
92.4
HumannEva-+ipssseo0
82.9
78.6
79.2
89.5
88.4
88.0
IF & Chat
IFEval(trictt
80.6
81.1
82.4
88.0
82.6
83.2
AlpacaEval 2 % win)
24.2
33.5
34.5
33.4
49.6
49.8
Safety
Safetys task aas. avg.
75.2
87.2
85.5
76.5
89.0
88.3
Table 22: Final performance of RLVR-trained TÜLU 3 models compared to Llama 3.1 and DPO starting points. The best-performing model on each benchmark (i.e., in each row) and of each size is bolded.
E RLVR
E.1
Additional Results
The training curves for our final SB and 70B model checkpoints are shown in Fig. 19, where the best checkpoint is not always the final saved checkpoint of the RL run. The final RLVR results relative to the DPO model starting points are shown in Fig. 20.
E.2 RLVR Implementation Details
Implementation Details. RL. and RLHF have many subtle implementation details that can significantly impact training stability (Engstrom et al., 2020; Huang et al., 2022). As we use PPO to train our models against our verifiers, we adapt these effective implementation details from Huang et al. (2024a):
1. Initialize the Value model from General RM We initialize the value model from the reward model for the RLVR setup (following the standard setup in Ziegler al. (2019); Ouyang et al. (2022)).
37

---

Published as a conference paper at COLM 2025
RLVR on GSMBK
100
verpsole Rewonts
Response Length
Divergence
95
90
2.
20K
40K
60K
80K
100K
0K
20K
40K
GOK
8OK
100K
20k
40K
60K
80K
100K
Episodes
start DPO, B= 0.03
start = DPO, B 0.05
start DPO, ( 0.01
start DDO, = start =ontt=Spp SFT, $
start SFT, 0.03
start SFT, 3 0.05
start SFT, 6 0.01
Figure 20: The comparison of RLVR's performance on GSM8K between starting from a DPO checkpoint and starting from a weaker SFT checkpoint. We see that starting from both SFT and DPO can lead to the same level of verifiable rewards, but starting from SFT would incur a larger KL compared to starting from DPO when using the same B.
2. Disable Dropout We set the dropout probability to be during RM and RL training (similar to Ziegler et al. (2019)). This ensures the token log probabilities can be computed deterministically during the forward passes of the policy model and ref- erence model, allowing more accurate estimation of the KL penalty. Furthermore, PPO calculates the token log probabilities two phases: during the rollout phase and the learning phase. It is important to ensure the token log probabilities match up during these two phases: they produce probability ratio of 1 during the first PPO epoch, PPO can clip the ratio and apply the surrogate objective properly. If the log probabilities differ drastically due to dropout, all ratios could potentially be clipped, resulting in zero gradient.
3. Train with the SFT Dataset and Shuffle Between Epochs As pointed out in in Huang et al. (2024a), PPO can train for more episodes than the total available prompts, effectively training for multiple epochs. In our RLVR ablation experiments, we train for roughly 100, 000/7, //,7477 473 = 13 epochs. We shuffle the prompts in between epochs. For our final runs, we examine model checkpoints every 40-100 steps and choose the best checkpoint on our development evaluation set.
4. Non End-of-Sequence (EOS) Penalty During training, PPO typically sample fixed amount of maximum tokens. If the sampled response does not end with EOS token, we give a -10 penalty to encourage the model to always complete its responses.
5. Advantage Whitening/ Normalization Like done in standard PPO implementation details literature (Engstrom et al., 2020; Huang et al., 2022; 2024a), we normalize the advantages by subtracting its mean followed by dividing its standard deviation.
For hyperparameters, for 8B models, used the hyperparameters from Table 20, but tested higher KL. penalty coefficients (up to 0.15) based on previous 8B RL. development runs.
For 70B models, we used the hyperparameters from Table 20, but with a 10" learning rate, 0.1 warmup ratio, 2048 response length, 400,000 episodes, 640 effective batch size, and 0.7 based on previous 70B RL development runs. We initialize our value model from reward model trained on the same dataset as the best DPO model (the TüLU 3 8B preference mixture) starting from TÜLU SFT, using the same hyperparameters as in our ablation experiments (Table ??).
E.3
RLVR Infrastructure
Our PPO setup follows best practices implementation details (Huang et al., 2024a). To enable our implementation to scale to models up to 405B parameters, we also adapted model allocation techniques from existing distributed RLHF frameworks (Hu et al., 2024). Furthermore, we accelerate throughput by making RL training asynchronous (Noukhovitch et al., 2024).
38

---

Published as a conference paper at COLM 2025
Performance Metrics of RLVR vs KL Divergence
Performance Metrics RLVR vs KL Divergence
0.88
62.50
0.66
0.87
62.25
CSMBK Score
62.25
0.08
GSMBK
Awerage
0.86
62.00
0.85
0.05
61.75
0.84
0.84
81.30
10
20
Divergence
KL Divergence
Init value from SFT (G5MBK)
-X- Init value from SFT (Average)
Init value from RM Init valua from RM (CSS882)
Init value from RM
Init value from RM (GSM8K)
Init value from RM (/verage)
and (Awerage)
Baseline
Average Baseline
-
Init valkae from RM (Average)
GSMBK Baseline
Average Baseline
Figure 21: The performance of RLVR vs KL Divergence under different value model ini- tialization. Both initializations could im- prove the models compared to an initial DPO baseline model. However, as the model diverges more from the initial model, overoptimization happens as the average scores drop significantly.
Figure 22: Similar Figure 21, but this is com pariisioon of using scores from on top of the verifiable rewards and 2) using only the verifiable rewards. We found using the ver- ifiable rewards performs better in GSM8K, and using scores and verifiable rewards to be more noisy.
The final 8B reward model is trained in 9 hours on 8 H100 gpus, while the final 8B RL run takes 65 hours on 8 GPUs, the final 70B RL run takes 60 hours 48 GPUs, and the final 405B RL run takes 46 hours on 256 GPUs. Note, for all of these models we took an earlier than final checkpoint from the run.
Distributed Setup. Our PPO infrastructure leverages Zero Stage 3 (Rajbhandari et al., 2020) to fit the models and applicable optimizer states into the memory. In RLVR, have 3 models: the policy model, the reference policy model, and the value model. The policy and value models need be trained, but the reference policy model only performs inference. Often inference time is a bottleneck in RLHF infrastructure, so we allocate dedicated GPUs do inference like done in OpenRLHF (Hu al., 2024). In particular, we use Ray (Moritz et al., 2018) to allocate dedicated GPUs to run Paged Attention via vL.LM (Kwon et al., 2023). Attention helps reduce GPU memory fragmentation and redundant duplication leveraging virtual memory and paging techniques. As result, it helps our PPO infrastructure run LLM inference using a much larger batch size and speed up inference. This setup allowed us to scale PPO policy training to the 405B scale. We share more 405B scaling details in Section E.1.
Asynchronous RL Training. Furthermore, our PPO setup uses asynchronous RL training to improve training efficiency (Noukhovitch et al., 2024). The existing PPO frameworks (Hu et al., 2024; Shen et al., 2024a) are typically synchronous: the inference GPUs would first collect policy rollout data, and then the learner GPUs would train on that rollout data. This setup would utilize GPU relatively well under a typical setup with Atari (Mnih al., 2015). However, under the RLHF setup, inference computation typically requires different optimizations (e.g., PtteedAteetiio), thus requiring different GPU memory allocation strategies. As a result, synchronous RL training means inference GPUs could be idling while the learner GPUs run, and vice versa. An alternatiee implementation is to use the same set of GPUs for training and inference, but it could incur additional overhead such an additional copy of the policy parameters and compiling inference engine in the training GPUs (Shen et al., 2024a).
Our setup allocates GPUs specifically for inference and training, alleviating the need to compile an inference engine and save a copy of the policy parameters in the training GPUs. Furthermore, we run the inference computation concurrently with the training computation, thus reducing the GPU idle time. However, asynchronous RL training can introduce stale data, in cases where the inference can generate data much faster than training consumes (Espeholt et al., 2018), which can introduce reproducibility problems (Huang
39

---

Published as a conference paper at COLM 2025
Hyperparameters
405B RLVR
Learning Rate
1 x 10
Discount Factor y
1.0
General Advantage Estimation A
0.95
Mini-batches Nmb
1
Hyperparameter
405B SFT
405B DFOO's Clipping Coefficient &
0.2
Learning Rate
2 10
2 x 10
VValue Function Coefficient 61
0.1
Learning Rate Schedule
Linear
Linear
Gradient Norm Threshold
1.0
Batch Size (effective)
256
256
Learning Rate Schedule
Linear
Max Token Length
4,096
2,048
Generation Temperature
1.0
KL penalty coefficient B
Batch Size (effective)
1,856
Warm up ratio
0.03
0.1
Max Token Length
2,048
Number of Epochs
2
Max Prompt Token Length
2,048
Penalty Reward Value for
Table 23: Hyperparameters for training TÜLU 3 405B. We use larger batch size due to the increased number of GPUs, and lower the SFT learning rate.
Responses without an EOS Token PPO Update Iterations K Response Length
-10.0
1
1,024
Total Episodes
300,000
KL penalty coefficient (B)
0.05
Warm up ratio (w)
0.0
Table 24: The hyperparameters of PPO used for optimizing against a general RM.
MATH Train Split
MATH Test Split
Final 405B Checkpaint
4058 RL. Run
Vertable Reweres
Wny Mypmirr Anple
Rosp Rosp-anze Length
Oberggncc
4:00
3.50
3.00
3K
50k
100K 150K
0K
50K
100K
150K
50K
100K
150K
Episodes
Figure 23: Rewards on MATH, KL divergence, and the average response length for the final 405B training run. We mark the point with the final checkpoint with star. We note that this was the last checkpoint saved we intended train longer but hit compute constraints. Note that technically the metrics in MATH test split is accuracy, but here multiply the accuracy by 10 to convert to verifiable rewards.
et al., 2023). To help make training more reproducible, our setup always trains the policy using the second latest inference data (Huang al., 2023; Noukhovitch al., 2024).
F Discussions
F1 Scaling TÜLU 3 Recipe to Llama 3.1 405B
Here, we demonstrate the scalability and effectiveness of our post-training recipe applied at 405B parameter scale. Scaling to this size required several engineering efforts and posed number of challenges:
40

---

Published as a conference paper at COLM 2025
Compute Requirements: Training TÜLU 3 405B demanded 32 nodes (256 GPUs) running in parallel. While most of our codebase scaled well, we occasionally en- countered NCCL timeout and synchronization issues that required meticulous monitoring and intervention (especially with RL training). Using more GPUs in- creased the chances of encountering hardware failures, necessitating semi-frequent run restarts.
For inference, we deployed the model using vLLM with 16-way tensor parallelism, while utilizing the remaining 240 GPUs for training. After each iteration of RLVR update, the weights are synchronized to the vLLM engine using NCCL broad Inference typically takes -550 seconds, weight transfer takes -25 seconds, and training takes ~1,500 seconds. To reduce computational cost during the RLVR stage, we utilized an 8B value model. Future works can benefit from exploring larger value models or alternate value model-free RL algorithms such as GRPO (Shao et al., 2024).
RLVR Training:
Hyperparameter Tuning Challenges: Given the computational costs, hyperparam- eter tuning was limited. Following prior Tülu and Llama work, we opted to lower the LR for larger models, training with a "lighter touch".
Our training recipe for the 405B model followed very similarly to that of the 8B and 70B models, but with different training dataset for RLVR. Given the model's saturation of GSM8K from SFT and DPO training alone, we removed the GSM8K data, and we additionally found that the IFEval data did not help much in initial RLVR runs. As such, for TULU 3 405B RLVR we only used the MATH train set. Surprisingly, we found that even with as few as 25 RLVR steps, MATH performance improved by over points, and continued to increase with more training.
With the challenges of scaling the asynchronous RL infrastructure, we only trained for 75 steps, fewer than our smaller models. The RL reward, KL divergence, and response length per training batch are shown in Fig. 23. The training hyperparameters for the SFT and DPO stages are shown in Table 23. The hyperparameters for RL with verifiable rewards at this scale are shown in Table 24.
In Table 5, we compare TüI.U 3 405B with prior state-of-the-art models finetuned from Llama 3.1 405B as well as DeepSeek-Ve (DeepSeek- AT et al., 2024) and GPT-4o. Generally, TüLU 405B results are improved compared to TÜLU 3 70B.TÜLU 3 405B achieves competitive or superior performance to both Deepseek v3 and GPT-4o, while also surpassing prior open-weight post-trained models of the same size including Llama 3.1 405B Instruct and Nous Hermes 405B (Teknium et al., 2024) on many standard benchmarks. We note that we ended RLVR training early due to compute constraints, and further training may further improve performance. In particular, we did not observe that MATH performance had saturated during training and testing (see Figure 23).
F2 Testing Generalization to Target Evaluations
Throughout the report and in Sec. 3.5 the final report of RLVR training runs reported. An important metric for verifying that it is performing as expected to check the evaluation of the model on the target evaluation at multiple intermediate checkpoints during training (given that RLVR is training on similar training data, but not the exact data in the evaluation, over-fitting can occur). Training runs on GSM8K, MATH, and IFEval prompts only are shown in Fig. 24 initialized with both DPO trained models and SFT trained models as value functions. Across all of our training configuration, generalization can be seen, but the exact best configuration is still to be found.
F33 RM Training Hyperparameters
We detail the hyperparameters used to train TÜLU 8B RM (used for initializing the value model for RLVR training) in Table ??.
41

---

Published as a conference paper at COLM 2025
SFT
DPO
10.0
5.0
010
130
180
(a) GSM Performance and Generalization
HITT
010
400
200
10K
(b) Math Performance and Generalization
Raal
200
305
10.6
15.8
2030
30C
100K
130K
536
130K
1996
206
Colodes
(c) IF-Eval Performance and Generalization
Figure 24: Performance and generalization of RLVR training on three specific prompt sets GSM8K, MATH, and IFeval on downstream evaluations and internal RLVR training metrics.
Core Skill Knowledge
Development
Unseen
(MLU(em)
MMLU-Pro(em)
POPQA(EM)
GPQA(em)
TruthfulQA@Mc2em
Reasoning
BigBenchHard(em))
AGIEval English
DROP
Math
MATH (flex em)
Deepmind Mathematics
GSM8K<em)
Coding
HumanEvallpass@10
BigcodeBenchhuasee0n
#umann
Instruction Following (IF)
IFEval
IFEval-OOD
AlpacaEval (wwiratt))
HREF
(REF(winrate) (winrate)
Safety
TÜLU Safety
Table 25: TÜLU 3 EVAL consists of development and unseen splits to evaluate core skills. With TüLU EVAL, we release unified standardized evaluation suite and a toolkit to decontaminate training data against benchmarks. The subscript shows the metric we use for evaluation. TüL.U 3 Safety collection of safety evaluations taking the average score across them (avg"), see Sec. G.2.1 for details.
42

---

Published as a conference paper at COLM 2025
G
Evaluation
G.1 Open Language Model Evaluation System (OLMES)
In an effort to make evaluations more standardized and reproducible, we are sharing the code base used to produce the evaluations in this work. The evaluation system supports:
A wide range of models and tasks, leveraging existing work in the Eleuther Al LM Evaluation Harness (Gao et al., 2024),
Flexible configuration options for each task,
Direct access to the specific task formulations used in this work (as well as in earlier work such as OLMo (Groeneveld et al., 2024) and the OLMES standard (Gu et al., 2024)), and
Detailed instance-level output data for analysis model predictions, confidences, etc
E.g., to reproduce our Llama-31-8B-Instruuct numbers for MMLU-Pro, one would simply run something like "olmes --task mmlu mmlu.pro: : pro::tulur tulu3 --model 1lama3. seb-instruct".
G.2 TÜLU 3 Evaluation Suite Development
We design the evaluation setup for the development partition of TüLU 3 EVAL based on existing practices in current literature as well insights during development. Where ap- propriate, we adapt the evaluation setup based on the nature of the task and take additional care robustify our answer extraction and comparison approaches as described below. For generative evaluations our output length is 4096.
MMLU (Hendrycks et al., 2020) is heterogeneous with regard to the type of reasoning skills required to answer the questions, containing instances that require basic factual recall as well as those that demand logical reasoning and problem-solving skills. We design a zero- shot CoT setting that asks the models to "summarize" its reasoning before answering the questions (see Table 40 for the prompt used). We find that among the various CoT settings, including prompting the model to think "step-by-step" and using no CoT, the prompt yields a systematic performance improvement over standard 5-shot multiple choice setting across tested models (See Table 42) and maximizes the number of subject (knowledge categories) that the CoT benefits. This indicates that our "summarize" prompt is an effective strategy for dealing with the heterogeneity of the benchmark. See appendix G.6 for details. We compute macro average over all the subjects in MMLU as the final task metric.
PopQA (Mallen et al., 2022) is an entity-centric question-answering ates language models' tendency to forget information about long-tail entities. We prompt the models in a 15-shot setting (as recommended in the dataset paper) without any additional instructions, with each QA demonstration presented in different conversation turn, setting that we refer as Multiturn ICL henceforth, and use greedy sampling to obtain model predictions.
benchmark that evalu-
TruthfulQA (Lin et al., 2021) contains questions that humans tend answer incorrectly due to common misconceptions. We use the multiple-choice version of this benchmark where the models being evaluated are presented with questions and options containing multiple correct answers (the MC2 setting).
HumanEval (Chen al., 2021) and HumanEval+ (Liu al., 2023) evaluate models' ability to complete Python code given docstrings. HumanEval+ uses more rigorous evaluation procedure than the original HumanEval benchmark with additional tests. We use samples from the models at temperature of 0.8 and use pass@10 the evaluation metric.
GSMSK (Cobbe et al., 2021) contains grade school math word problems. We use the 8-shot chain-of-thoughh prompt from Wei et al. (2022b), formatted for multiturn ICL evaluation. We obtain model responses using greedy sampling and extract the last number in the model response the predicted answer.
43

---

Published as a conference paper at COLM 2025
MATH (Hendrycks et al., 2021) contains problems from mathematics competitions spanning various categories such as algebra and calculus. We use a 4-shot multi-turn setup with CoT from Lewkowycz et al. (2022), formatted for multiturn ICL, and greedy sampling for model completions. For determining the correctness of predictions, we use a 'flex' scheme that attempts to extract the answer in three different ways: (1) following the minerva format (Lewkowycz et al., 2022); (2) finding the last instance of | < ans "); (3) taking the text between the last two ' tags. This is due to issues we found during development wherein models would often not follow the correct output format despite the few-shot examples, necessitating varied answer extraction strategy. We find that moving from the minerva format alone to our 'flex' strategy can sometimes improve reported scores by up to 10 points, highlighting the need for this flexible strategy. We compute macro average across the subsections to obtain the final task metric.
BigBench-Hard (Suzgun et al., 2022) contains challenging reasoning problems for which models benefit from step-by-step reasoning. We follow the setup described in the original paper and use 3-shot CoT prompts, formatted for multiturn ICL. We use greedy sampling for obtaining model predictions.
DROP (Dua et al., 2019) is reading comprehension task that requires discrete reasoning. We draw 3 random few-shot examples from the train split following the setup used for Llama 3 (Dubey et al., 2024), and use greedy sampling to get model predictions.
IFEval (Zhou et al., 2023) evaluates instruction following ability of models in a setting where each instruction corresponds to constraints such that the it can be programmatically verified whether the outputs satisfy those constraints. We use greedy decoding to generate model outputs give the instructions in the dataset, and measure the prompt-level accuracy of satisfying constraints in the loose evaluation setting.
AlpacaEval 2 (Dubois et al., 2024) contains set of prompts sourced that reflect real human usages of LMs, and compares model outputs to GPT-4 turbo responses with an additional length control to avoid longer answers being unfairly favoured. We generate responses using greedy decoding up to 8,192 tokens in length, following Ivison et al. (2023).
G.2.1 Safety Evaluation
We follow Han et al. (2024) and Jiang et al. (2024) to define our safety evaluation suite using the following benchmarks.' 7 Each of these benchmarks evaluates whether models refuse to respond to unsafe requests, and in the case of XSTest and WildJailbreak, additionally evaluate whether they comply with benign requests. We use greedy sampling to get model responses for the prompts in each of the benchmarks, and compute the accuracy (at refusal or compliance as appropriate). We report the macro average of the scores over all the benchmarks as our final safety evaluation metric.
XSTest (Röttger et al., 2023) consists 200 unsafe prompts and 250 prompts which are safe but superficially resemble unsafe prompts: these prompts use vocabulary similar to that of unsafe prompts. Categories include homonyms, figurative language, safe targets, safe contexts, definitions, real discrimination/ /nonsense group, nonsense discrimination/ real group, historical events, public privacy, and fictional privacy. We report the overall accuracy score based on whether WildGuard (Han et al., 2024) classifies the response as refusal or compliance.
criminaation/noneensee
HarmBench (Mazeika al., 2024). We evaluated on a subset of harmful prompts which 8 categorized into Functional and Semantic categories. Func-
consists of 321 harmful prompts tional category includes two types of behavior: Standard behaviors, which are modeled after
"Built on A12 Safety Tool, forked focus only on safety evals: https: thub. com/ nouhazurraziri/ safety-eval-fort Note that while WildGuard and Wildjailbreak are popular training datasets, we use generations over the test set prompts as test of compliance with a response filter such as the WildGuard model.
"https://ed htt///gtttbbbb www.e//wwwf..tieeeeeeeeeeeeeeaaa harmbench behaviors text test.csv
44

---

Published as a conference paper at COLM 2025
Benchmarks
Llama 3.1 8B
Ministral 8B Instruct
Qwen 2.5 7B
TüLU 3 8B SFT
TüLU 8B
TüLu 3 8B
Instruct
Instruct
DPO
HarmBench
82.8
53.4
84.1
98.4
94.4
94.7
XSTest
92.7
85.6
91.8
90.4
92.4
93.3
WildGuarddeest
86.2
68.1
85.0
99.2
98.9
98.5
Jailbreaktrigger
78.8
63.3
71.0
95.8
87.0
85.5
DoAnythingNow
45.0
16.0
61.7
88.3
69.7
62.0
WildjailbreakTest
65.6
50.7
56.2
86.7
81.1
78.8
Overall
75.2
56.2
75.0
93.1
87.2
85.5
Table 26: Breakdown of safety scores by benchmark of TÜLU 3 8B models compared with similarly sized open weight models.
existing datasets harmful behaviors such as AdvBench and the TDC 2023 Red Teaming Track dataset, and Copyright behaviors, which test the handling of copyrighted content. The semantic category comprises seven types of harmful behaviors: cybercrime, unauthorized intrusion, chemical biological weapons drugs, copyright violations, misinformation/ dis- information, harassment/ harahiinahaasmme/bbullliii /ulllying illegal activities, and general harm. We use WildGuard to evaluate the model's refusal to assist with harmful prompts.
Do-Anything-Now (Shen et al., 2024b) consists of jailbreak prompts that were created by combining the jailbreak templates from DAN with harmful behaviors from HarmBench and subsample 300 of them for testing. We report accuracy using the WildGuard classifier.
SuubbreakTrigger (Huang et al., 2024b) incorporates prompts based 13 distinct jailbreak
attack methods. In total, the dataset consists of 400 examples, split evenly between two categories: "Questions about Bad Behaviors" and "Instructions to generate Toxic Content". This dataset serves to evaluate the effectiveness of LLMs'' defenses and measures the toxicity of responses under jailbreak scenarios. The reported metric is RTA measured by WildGuard.
WildJailbreakTes. (Jiang et al., 2024) is an adversarial evaluation set which contains subset for adversarial benign queries (210 examples) and subset for adversarial harmful queries (2000 examples). The adversarial benign queries are used to measure models' exaggerated safety behaviors and the adversarial harmful queries are used measure models' safeguards regarding adversarial attacks. We measure RTA using WildGuard for both categories. For benign queries, RIA is expected to be (1) and for harmful queries, the RTA is expected to be (T).
WildGuardTest (Han et al., 2024) contains 1725 items for prompt harm, response harm, and response refusal classification tasks. 55% are vanilla prompts, and 45% are adversarial. The prompts are collected based on adversarial synthetic data and in-the-wild user-LLM (In-the-wild) interactions. We report RTA using WildGuard.
G.3 TÜLU3 3 Evaluation Suite Unseen
For the unseen evaluation suite, the task formulations were decided through an independent design process from that of development suite. One goal of the unseen suite is to evaluate instruction-tuned models in ways that are closely aligned to realistic usage. Specifically, we follow these general principles:
Formulate tasks similar to how humans interact with the models. E.g., avoid few- shot examples presented as dialog, or precise chain-of-thought (CoT) examples for how the model is "supposed" to think.
Prompt models with clear instructions that set the context, encourage concise reasoning, and specify how the final answer should be formatted.
45

---

Published as a conference paper at COLM 2025
Benchmarks
Llama 3.1
Qwen 2.5
Hermes
Nemotron Llama 3.1 70B
TüLU 70B SFT
TüLu 70B
TüLU 70B
70B
72B
Llama 3.1
DPO
Instruct
Instruct
70B
HarmBench
80.6
86.3
54.7
84.4
98.8
97.8
97.8
XSTest
87.1
93.6
89.3
92.0
91.1
94.9
92.4
WildGuarddTest
81.3
93.1
66.6
84.9
99.1
99.2
98.9
Jallbrakktriggrr
71.0
89.8
56.3
60.3
95.3
87.0
86.7
DoAnythingNow
80.0
93.3
26.7
36.3
93.7
69.0
67.7
WildjailbreakTest
59.2
66.0
53.8
56.1
88.6
86.3
86.2
Overall
76.5
87.0
57.9
69.0
94.4
89.0
88.3
Table 27: Breakdown of safety scores by benchmark of TÜLU 3 70B models compared with similarly sized open weight models.
Model
Llama 3.1 prompt
Ours
Gemma 9B Inst
51.6
52.6
Gemma 2 9B Inst-SimPO
52.6
51.8
Llama 3.1 8B Inst
49.2
48.7
Llama 3.2 3B Inst
39.1
39.7
Ministral 2410 8B Inst
43.8
44.3
OLMo 0724 7B Inst
26.1
22.9
OLMoE 0924 1B 7B Inst
20.7
20.3
Qwen 2.5 7B Inst
56.2
54.2
Tulu DPO 7B
25.4
22.2
Table 28: Comparing evaluating MMLU-Pro using our O-shot CoT prompt and the 5-shot prompt used in Llama3.1 evaluations, using macro average over tasks both cases (the Llama3.1 evaluation used micro average).
Apply reasonable heuristics for answer extraction and comparison to gold answers, avoid penalizing models that do not exactly follow a syntax implied by the instructions (based on examining outputs from a wide set of existing baseline models).
We first apply these principles to some of the tasks in the development suite using set of exploratory models (instruction-tunee models predating TüLÚ 3). On the exploratory models, we find that following the above principles, in particular being more aligned to how human usage, generally does not degrade performance and often times allow most models to perform better on tasks (despite, e.g., removing few-shot examples). We did not update the formulation of the development tasks based on this, but carried the principles over to formulating the unseen tasks. More detailed analysis can be found in Appendix G.6.
The task formulations in the TÜLU 3 unseen suite are as follows. For all benchmarks which include multiple sub tasks, we compute the average over the sub tasks (i.e., the "macro" average) to be consistent.
AGIEval English (Zhong et al., 2024) includes the English language subset of the AGIEval benchmark, specifically these multiple-choice tasks: aqua-rat, logiga-en, Isat-ar, Isat-Ir, Isat-rc, sat-en, sat-math, and gaokao-english. We do not include the weteen-wihhout-psseeeee find these questions are typically severely underspecified without access to the passage. We formulate the task using the simple "zero-shot CoT" prompt which encourages concise reasoning ending with clearly stated answer choice. In Appendix G.6, Figure 26, we provide the 0-shot reasoning prompt, used for the multiple-choice tasks the unseen evaluation suite, including AGIEval English. The model's answer choice is extracted by
task as we
46

---

Published as a conference paper at COLM 2025
Model
Llama 3. 1 prompt
Ours
Gemma 2 9B Inst
35.7
35.5
Gemma 9B Inst-SimPO
35.0
35.7
Llama 3.1 8B Inst
29.5
29.5
Llama 3.1 70B Inst
46.2
44.0
Llama 3.2 3B Inst
33.5
27.7
Ministral 2410 8B Inst
31.0
31.5
OLMo 0724 7B Inst
27.2
27.9
OLMoE 0924 1B 7B Inst
24.6
24.8
Qwen 2.5 7B Inst
32.1
36.8
Tulu DPO 7B
28.4
27.5
Table 29: Comparing evaluating on GPQA using our 0-shot CoT prompt and the prompt in Llama3.1.
Model
base-adapted,
context examples
CoT prompt
Gemma 2 9B Inst
18.0
45.9
Gemma 9B Inst-SimPO
19.3
45.3
Llama 3.1 8B Inst
20.0
39.4
Llama 3.2 1B Inst
11.6
13.1
Llama 3.2 3B Inst
19.2
32.6
Ministral 2410 8B Inst
18.8
36.7
OLMo 0724 7B Inst
3.2
5.8
OLMoE 0924 1B 7B Inst
9.0
4.2
Qwen 2.5 7B Inst
21.2
54.7
Tulu 2 DPO 7B
9.6
6.0
Llama 3.1 Tulu 8B
21.7
13.6
Llama 3.1 Tulu 2 DPO 8B
18.6
14.5
Table 30: Comparing evaluating on DeepMind Mathematics using different prompts de- signed evaluation practices for instruction-tured models, with minor variants. Bolded numbers indicate cases where applying principles more aligned with real usage leads to better performance on models. our evaluation suite, the chain of thought (CoT) prompt is referred to as chat-v3.
first matching to the requested format, with fallback patterns if the format was not followed precisely. Specifically, we first look for the exact phrase indicated in the prompt ("Therefore, the answer [ANSWER]") and grab the last such match. If that fails, we look for a sequence of softer variants, like "answer is [ANSWER]" or "answer: [ANSWER]" before falling back to the last letter in parenthesis found, and that fails, the last stand-alone capital letter.
MMLU-Pro (Wang et al., 2024b) is 10-way multiple-choice extended version of the MMLU dataset. We use essentially the same prompt and answer extraction as used for our AGIEval setup, just adjusting for the number of answer choices. We generally find our formulation to be as effective as the traditional 5-shot CoT prompt (e.g., used in Llama 3.1 evaluations (Dubey et al., 2024)) despite being much shorter, more realistic, and easier for users to create (see Table 28 for a comparison across exploratory models).
GPQA (Rein et al., 2023) is set of very challenging multiple-choice questions written by domain experts biology, physics, and chemistry. We use the same zero-shot prompt and answer extraction for AGiEval. This is similar to the approach used in Llama 3.1
47

---

Published as a conference paper at COLM 2025
evaluations, but with less prescription on how the reasoning should be structured. Figure 29 shows that across exploratory models, that our approach leads to comparable scores.
Deepmind Mathematics (Saxton et al., 2019) is dataset of 56 categories of math questions, testing mathematical and algebraic reasoning skills. We devised a "zero-shot CoT" prompt that sets the context of the task, explains the format expected for the answers (e.g., "x*22" for powers), and for each category included three example answers to specify the answer format. The prompt used and answer extraction heuristics were refined by examining example outputs from our exploratory models. As seen from Table 30, applying evaluation principles aligned with real usage ("chat" versions), often performs better than using setup involving presenting few-shot examples adapted from base model evaluations ("base- adapted"). We provide the zero-shot reasoning prompt for the Deepmind Mathematics task in Appendix G.6, Figure 27. Aligned with the instructions for formatting in the prompt, when extracting the answer, we first look for an answer in the format "Therefore, the final answer is [answer]. hope it is correct." We look for this in inseiinseneiivee way, process the [answer] by stripping away any trailing period, and known math delimiters surrounding the answer (e.g., "$") based on outputs from exploratory models. For answer comparison, we first compare the raw strings of the processed generated answer and gold answer to check if they are equal. In addition, we parse both using the SymPy (Meurer et al., 2017) package and compare the equivalence of the parsed outputs to check their mathematical equivalent. We also consider the generated answer be correct if it is equal to the parsed gold answer after this parsing.
"
BigCodeBench (Zhuo et al., 2024) is a set of coding challenges. We focus on the "hard subset" of 148 (out of the total 1140) instances, using the "instruct" formulation of each task and the "calibrated" score. We follow the setup used for the original leaderboard for our implementation.
G.3.1 New Evaluation: IFEval-OOD
In order to test precise instruction following abilities of LLMs and whether they are able to follow constraints that go beyond the 25 constraints included in IFEval (Zhou et al., 2023), we developed IFEval-OOD (IFEval Out-of-Distrbbution) IFEval-OOD consists of 52 constraints across six broad categories such as the examples in Table 31. A complete list of the constraints can be found in Appendix G.7. A portion of the constraints were sourced by asking group of researchers for constraint ideas and others were written by authors of this paper. One of the six categories ("custom")) comprises manually written verifiable prompts to cover specific sub-skills, such as CSV generation. For the remaining five categories, the verifiable constraint was combined with 10 unseen prompts from WildChat. To select the final prompts, these constrained prompts were human annotated for quality and compatibility with the constraint (e.g. a prompt to paraphrase a one-sentence reference text that doesn't 't contain any names to begin with would not be compatible with a constraint to mention at least 23 different person names in the response). To ensure constraint coverage, additional unseen WildChat prompts were manually paired with constraints that did not have at least five selected prompts. Our evaluation dataset emphasizes broad coverage of constraint types to differentiate constraint-folloiing abilities from overfitting on the existing IFEval constraints.
to
G.3.2 New Evaluation: HREF
We constructed an automatic evaluation of instructability of language models, called Human Reference-guidded Evaluation of instruction Following (HRÉF). HREF focuses on 11 instruction following tasks that language models are typically trained on, namely, Brainstorminng Open QA, Closed QA, Extraction, Generation, Rewriting, Summarization, Classification, Numerical Reasoning, Multi-document Synthesis, and Fact Checking. We obtained high quality human- written prompts and responses in these categories from professional instruction-tuning data creators, and developed a reliable automatic evaluation procedure. Following Alpaca- Farm (Dubois et al., 2023), we win-rate against a fixed baseline model as the evaluation metric. However, since we have a larger number of tasks in our benchmark than Alpaca- Farm, and also because have access to human-written references, hypothesized
48

---

Published as a conference paper at COLM 2025
Instruction
Instruction
Description
Group
count
person names
Mention at least {N} different person names in the response.
format
emoji
Please use an emoji at the end of every sen- tence.
ratio
stop_words
Ensure that stop words constitute no more than {percentage) % of the total words in your response.
sentence
keyword
The response must include keyword (keyword) in the (N)-th sentence.
words
alphabet
Each word in your response must start with the next letter of the alphabet, looping back to 'A' after 'Z'.
custom
csv csv.special csv.special.charactee www.aaaacaaaaaeeeennneea CSV data: The column names are ["ProductID", "Category", "Brand", "Price", "Stock"], the data should be comma delimited. Please generate 14 rows. Add one field which contains a special character and enclose it in double quotes.
Table 31: Examples of IFEval out-of-distribution constraints. Constraints are added to unseen WildChat prompt to form the final prompt except for in the " "custom" instruction group. A complete list of constraints is provided in Appendix G.7.
that the setup used for win-rate computation may not be directly applicable to our new evaluation. So we experimented with various win-rate computation methods, and their details along multiple dimensions:
Choice of LM Judge. We tried GPT-4, GPT-4 turbo, and Llama 3.1-Instruct models at 7B and 70B sizes as LM judges.
LM-as-a-Judge vs. Direct Comparison with Human References. Using large language model e.g,, GPT-4 Turbo for AlpacaFarm, may not be appropriate for tasks where the responses are expected to be short and factual. We compared this setup with directly comparing the similarity of the responses from the target and the baseline models with the human- references, according to simpler embedding model (RoBERTa-Large (Liu, 2019)).
Use of Human-Written References When Evaluating with LM-as-a-Judge. Since we have access to human-written references, we experimented with including those as additional context while prompting the LM judge.
To make decisions about the evaluation setup, we collected human judgments comparing responses from a diverse set of 16 models, when prompted using the instructions taken from subset of our evaluation dataset. We collected four human judgments per each model response pair, and compared the agreement of each evaluation setup with majority human judgments. We repeated this procedure for each task category to identify the best setup for that category.
Summary of the Final Evaluation Setup. We used Llama 3.1 70B Instruct as our LM judge since its agreement with human judges was the highest of the pool of models we considered. Our baseline model we used Llama 3.1 405B Instruct. We found that using LM as judge results in higher human agreement (compared to the embedding- based method) in all subtasks except Open QA and Fact Checking. We use embedding similarity with human-written references the way of computing win-rates in these two subtasks. Of the remaining 9 subtasks, evaluation in Brainstorming and Summarization did not benefit from using human-written references while prompting the LM judge, and hence prompt
49

---

Published as a conference paper at COLM 2025
Skill
8B SFT
8B DPO
8B Final
70B SFT
70B DPO
70B Final
Dev:
Uns.
Uns.
Dev.
Uns.
Dev.
Uns.
Dev.
Uns.
Uns.
Avg.
64,9
29.9
68.3
31.9
68.8
32.4
78.1
41.0
80.5
44.4
80.7
44.4
Knowledge Recall (MMLU GPQA)
65.9
31.9
68.7
31.2
68.2
35.7
78.9
43.3
83.3
48.0
83.1
48.0
Reasoning (BBH - AGIEval)
67.9
56.2
61.8
66.0
59.3
82.7
73.2
81.8
75.0
82.0
75.0
Math (MATH DM Mathematics)
31,5
32.3
42.0
33.0
43.7
35.4
53.7
49.7
62.3
49.4
63.0
49.8
Coding (HumanEval BigCodeBench)
86.2
11.5
92.9
23.0
21.6
Inst. Following (IFEval IFEval-COD)
72.8
17.6
81.1
23.9
82.4
24.3
82.1
26.8
82.6
26.4
83.2
27.8
Table 32: Comparison of the performance of TÜLU 3 SFT and DPO checkpoints and the final models on development and unseen tasks corresponding to each of the core skills. Table shows that our pipeline generalizes well to unseen evaluations.
Know. Recall
Reasoning
Math
Coding
Inst. Follow.
Model
Avg
Dev.
Avg.
Uns.
MMLU
GPQA
BВH
AGIE
MATH
DMM
CHE
BCB
IFE
IFEO
TOLL 3 SFT
64.1
29.9
62.1
31.9
67.9
56.2
31.5
32.3
86.2
11.5
72.8
17.6
WildChat
62.8
28.8
61.0
31.5
65.6
53.1
318
31.2
85.3
7.4
70.1
20.8
w/o Safety
63.7
29.7
62.0
31.9
68.3
55.6
32.6
32.6
84.5
10.8
71.0
17.6
w/o Persona
29.4
68.3
84.5
10.8
53.6
18.0
w/o Math Data
62.2
27.4
62.2
32.6
68.9
54.1
23.5
23.3
86.0
8.8
70.6
18.3
Table 33: Comparison of the performance of data-source ablated SFT models with that of the final TÜLU 3 8B SFT checkpoint. Numbers in bold reflect the best performance per column. AGIE is AGIEval, DMM is Deepmind Mathematics, CHE is Codex HumanEval, BCB is BigCodeBench, IFE is IFEval and IFEO is IFEval-OOD.
LM judges with human references only for the remaining subtasks. On the subset for which we collected human judgments, our composite evaluation procedure resulted in an agreement of 69.4% with humans, which comparable to the inter-human agreement of 67%.
G.4 Evaluating the Development Process Using the Unseen Suite
To evaluate how much and in what ways over-fit to our development evaluations, measure the performance of our models on unseen tasks that still correspond to the same set of core skills. We evaluate the checkpoints corresponding we made in the development process on the unseen suite to check whether our decisions overfit the development evaluations, and summarize our findings in Section G.4.1. In Section G.4.2, we show a comparison between our final checkpoints and comparable public models.
various design decisions
C.4.1 Evaluating the design decisions
Training pipeline. Table 32 shows the performance of the SFT, DPO, and the final check- points of TULU3 3 at 8B and 70B sizes on one development and one unseen evaluation for each core skill. We see that our pipeline generalizes well to unseen evaluations, with the final checkpoints obtaining the best average performance on both the development and unseen evaluations. For Reasoning and Coding, where the SFT checkpoints have the best performance on development evaluations, the subsequent training stages still improve model performance on harder unseen evaluations.
Data mixing for SFT. To assess whether the data mixing choices we made for SFT generalize to unseen evaluations, we evaluate the performance of various data-ablated SFT models with the final SFT checkpoint, as shown in Table 33. For each core skill, we compare the model performance trends development and an unseen evaluation. We see that the data choices generalize on average, as indicated by the best average performances on both development and unseen evaluations by the final SFT checkpoint. In individual skills,
50

---

Published as a conference paper at COLM 2025
Llama 3.1 SB
Hermes Llama 8B
Llama
Hermes Llama 3.1
TULU
TCLU
Skill
Benchmarkigval)
3.1 70B
SB
70B
Instruct
Avg.
36.4
30.7
34.2
51.3
43.1
47.2
Knowledge Recall
GPQAgo CoT)
28.8
32.8
35.7
43.8
42.6
48.0
MML.U Projo shot, CoT)
49.1
40.9
44.3
68.3
60.3
65.8
Reasoning
AGIEval Englisho shot, Cot)
64.2
58.1
59.3
77.8
73.3
75.0
Math
DeepMind Mathga shot, CoT)
39.3
28.3
35.4
62.4
50.0
49.8
Coding
BigCodeBench-FFe Handeuus--Haduussstt))
15.5
9.5
7.4
26.4
14.2
21.6
Instruct Following
IFval OPD loooe))))
26.1
19.4
24.3
34.5
24.6
27.8
(IESavinsste)
38.5
26.2
32.7
45.6
36.8
42.3
Table 34: Evaluation a selection of open-weight and TÜLU 3 models on our unseen evaluation suite. It is important to note that without open training data for any of the other models that we cannot verify that they are not training on any of the unseen benchmarks.
see that our choices overfit to the development evaluations in Precise Instruction Following, and to some extent in Knowledge Recall and Reasoning.
5G
20
266 (Devi
MATH
Initial 8B SFT
5%
10%
25%
50%
75%
100%
Figure 25: Effect of scaling the size of the preference dataset, specifically the number of unique prompts, downstream DPO model performance in development and unseen evaluations. AVG are the averages of all the tasks in development and unseen suites. Also shown are the trends in MATH and Deepmind Mathematics.
Data scaling for preference tuning. We show the effect of scaling DPO data development and unseen evaluations Figure 25. We see that the scaling trends generalize on average to unseen evaluations. Of note is the trend we observed in the development and unseen Math evaluations, indicating that our development process overfit MATH to some extent. We hypothesize this is mainly due to formatting differences between MATH and Deepmind Math. The former often requires solutions and answers to be output in LaTeX format, while the latter does not. We found that trained models have the tendency to format the chain-of-thought reasoning and the final answers in LaTeXX even for the questions in the Deepmind Math dataset where this is not required. This often interfered with the intermediate reasoning, and also made our answer extraction logic fail.
G.4.2 Comparison with public models
Table 34 shows comparison between TÜLU 3 models, Llama 3.1 Instruct models, and Hermes 3 Llama 3.1 models at 8B and 70B sizes, and Table 44 shows subtask-level break- down of the performance of these models on HREF. is important to note that while all these evaluations are unseen for the TÜLU 3 models, we do not know if GPQA, MMLU-Pro, AGIEval, DeepMind Math, and BigCodeBench were used for developing the two other models. We summarize below qualitative analysis of the outputs of TŪLU 3 models on these datasets:
key takeaways from this comparison and from our
TÜLU generalizes well to unseen evaluations. In almost all the evaluations, TÜLU 3's performance is generally comparable to that the two other models we evaluate, often falling between the performance numbers of the two models. This suggests that recipe
51

---

Published as a conference paper at COLM 2025
Categories
Llama 3.1 8B
Ministral 8B
Qwen 2.5 7B
TüLU 3 8B
TüLU 8B
TüLU 8B
Instruct
Instruct
Instruct
SFT
DPO
Chemical & Biological Weapons/ Drugs
97.6
57.1
97.6
100.0
95.2
97.6
Copyright Violations
75.0
56.3
60.0
100.0
100.0
100.0
Cybercrime & Unauthorized Intrusion
76.9
42.3
88.5
96.2
92.3
96.2
Harassment & Bullying
100.0
90.5
100.0
100.0
100.0
100.0
General Harm
88.9
66.7
94.4
94.4
83.3
83.3
Illegal Activities
96.2
64.2
96.2
98.1
100.0
Misinformation & Disinformation
66.7
27.8
81.5
100.0
85.2
79.6
Table 35: Breakdown of model refusal rate by risk categories of the HarmBench benchmark.
of choosing representative evaluations for each core-skill and curating training datasets targeting those evaluations can lead to models that generalize well other tasks that require the same skills.
Models generally overfit to IFEval. We find that there significant difference between performance on IFEval and IFEval-OOD of all the models, even though we created the latter to be structured very similar to the original dataset, just with a disjoint set of constraints. We observe that instruction following with verifiable constraints is a challenging skill for models to learn effectively, and hypothesize that those models that do well on IFEval are likely overfitting to the specific set of constraints included in the dataset, making it hard for them to generalize to new constraints.
Generalization knowledge recall may be dependent the post training recipes. As one might expect, the performance of models on MMLU and MMLU-Pro is correlated. We see that their performance on GPQA shows a different trend - all three models we compare here are post-trained from the same base model, suggesting that the post-training recipe may affect the generalization in knowledge-
Instruction following performance varies across categories. We observe that the relative performance of TÜLU models on AlpacaEval is different from that on HREF. This may be explained by the fact that instruction following is a highly diverse task, and the distributions of HREF and AlpacaEval may differ, with some categories of instructions not necesarily transferring well to others, leading to the shift in relative performance. We do note that TÜLU 3 70B outperforms Llama 3.1 70B Instruct on out of 11 subtasks, as seen Table 44 in the Appendix, which shows breakdown of model performance across subtasks in HREF. Future work will explore how different behaviors of instruction following can be measured with more diverse instruction following evaluations.
G.5 Detailed Safety Results
We provide detailed breakdowns of the safety scores of TÜLU and peer 8B models by risk type on the HarmBench (Table 35) and WildGuard'eest (Table 36) benchmarks.
G.6 Evaluation principles
Experimenting Principles for unseen suite on development tasks
We compare evaluating instruction-tune models using an evaluation setup adapted from base model evaluation with our designed evaluation practices that align more with real usage. See Tables 37, ,3, 39 for example results.
Unlike the case for base models 4-shot (MATH) or 8-shot CoT (GSM8K), few-shot in-context examples goes into issues like few-shot as multi-ture turn could be seen as putting words in the mouth of the model, sticking all examples in the prompt togetherr could confuse models into answering all questions. We observe based on the exploratory models used for testing our setups, that natural instruction is more important when dealing with instruction-tuned
52

---

Published as a conference paper at COLM 2025
Table 36: Breakdown of model refusal rate by risk categories of the WildGuarddeest bench- mark.
Categories
Liama 3.1 8B Instruct
Ministral 8B
Qwen 2.5 7B Instruct
TÜLU 3 8B
TÜLU 3 8B
TÜLU 8B
Instruct
SFT
DPO
Sensitive information government
93.9
81.6
87.8
100.0
100.0
100.0
Social stereotypes/ discrimination
88.2
71.1
90.8
98.7
100.0
100.0
Toxic language hate speech Defamation
78.0
46.3
75.6
100.0
95.1
95.1
82.6
43.5
69.6
100.0
100.0
100.0
Private information individual
97.5
91.4
96.3
98.8
100.0
98.8
Cyberattack
64.6
54.2
68.8
97.9
97.9
95.8
Misleading information
51.2
27.9
72.1
100.0
97.7
95.3
Sexual content
93.6
68.1
85.1
97.9
100.0
100.0
Mental health
93.3
80.0
93.3
100.0
100.0
100.0
Violence and physical harm
90.0
45.0
72.5
97.5
97.5
97.5
Fraud assisting illegal activities
86.7
66.7
81.7
98.3
95.0
95.0
Causing material harm
71.1
55.6
80.0
100.0
100.0
100.0
Others
99.0
90.8
99.0
100.0
100.0
100.0
Model
MATH: bdse-adedsdd
MATH: : chat-v1
MATH: chat-v2
Gemma 2 9B Inst
1.57
42.84
38.07
Gemma 2 9B Inst SimPO
0.00
23.12
29.05
Llama 3.1 8B Inst
40.39
44.97
42.67
Llama 3.2 1B Inst
19.10
23.90
19.70
Llama 3.2 3B Inst
36.89
40.80
40.13
Ministral 2410 8B Inst
21.12
47.32
48.60
OLMo 0724 7B Inst
4.35
4.21
3.23
OLMoE 0924 1B 7B Inst Qwen 2.5 7B Inst
0.04
9.07
7.88
0.05
34.23
67.17
Tulu 2 DPO 7B
4.20
2.69
3.63
Table 37: Comparing evaluating instruction-tunne models on MATH using an evaluation setup adapted from base model evaluation with our designed evaluation practices for instruction-tunet models. Bolded numbers indicate cases where applying principles more aligned with real usage leads to better performance on models.
models. In the tables chat-vl refers to using zero-shot CoT prompt and chat-v2 refers to using a 1-shot CoT prompt. Given that providing an additional example does not always lead to better scores, we stick to zero-shot CoT, keeping the prompt simple and avoid unintentionally steering the model to favor any answer due to the single example. A similar story holds for DROP where 1-turn zero-shot CoT prompt (more similar to users would prompt an LM) performs better than the setup where in-context examples are stitched together the 1-turn (as adopted from Llama 3's base model evaluation setup).
Prompts used for unseen evaluation tasks
In Figure 26 we provide the 0-shot reasoning prompt used for the multiple-choice tasks in the unseen evaluation suite, which includes AGIEval English, MMLUUPro, and GPQA. We provide the 0-shot reasoning prompt for the Deepmind Mathematics task in Figure 27.
MMLU Chain-of-Thought Prompting
For MMLU, we experiment with various CoT settings that prompts models to provide reasoning before answering the question. We experiment with four CoT settings (see Table
53

---

Published as a conference paper at COLM 2025
Model
DROP:
: adpta
DROP: :hat-v2
Gemma 2 9B Inst
14.13
55.78
65.01
Gemma 2 9B Inst SimPO
14.67
54.57
63.80
Llama 3.1 8B Inst
14.41
53.62
54.25
Llama 3.2 1B Inst
7.50
24.24
17.53
Llama 3.2 3B Inst
14.57
48.81
45.57
Ministral 2410 8B Inst
20.55
43.90
48.76
OLMo 0724 7B Inst
33.35
16.59
11.99
OLMoE 0924 1B 7B Inst
33.43
15.33
13.22
Qwen 2.5 7B Inst
8.68
49.62
63.07
Tulu 2 DPO 7B
42.05
27.70
17.31
Table 38: Comparing evaluating instruction-tunde models on DROP using an evaluation setup adapted from base model evaluation with our designed evaluation practices for instruction-tuned models. Bolded numbers indicate cases where applying principles more aligned with real usage leads to better performance on models.
Model
GSM8K: bww.bbase--pppede
GSMBK:
:SMBKP::chat-
Gemma 2 9B Inst
79.45
86.66
84.15
Gemma 9B Inst SimPO
79.30
87.64
87.04
Llama 3.1 8B Inst
83.55
84.15
81.65
Llama 3.2 1B Inst
44.88
46.17
37.91
Llama 3.2 3B Inst
75.74
76.95
76.12
Ministral 2410 8B Inst
79.76
84.46
83.47
OLMo 0724 7B Inst
23.81
17.89
17.36
OLMoE 0924 1B 7B Inst
47.46
39.50
39.42
Qwen 2.5 7B Inst
84.08
91.13
90.07
Tulu 2 DPO 7B
8.72
27.90
20.62
Table 39: Comparing evaluating instruction-tuned models on GSMSK using an evaluation setup adapted from base model evaluation with designed evaluation practices for instruction-tune models. Bolded numbers indicate cases where applying principles more aligned with real usage leads to better performance on models.
54

---

Published as a conference paper at COLM 2025
0-shot reasoning prompt for multiple-choice unseen tasks
Answer the following multiple-choice question by giving the correct answer letter in parentheses. Provide CONCISE reasoning for the answer, and make sure to finish the response with "Therefore, the answer is (ANSWER where (ANSWER LETTER) is one of (A), (B), (C), (D), (E), etc.
Question: {question] (A) {choice_A}
(B) {choice B}
(C)
Answer the above question and REMEMBER finish your response with the exact phrase "Therefore, the answer is (ANSWER LETTERY" where (AN- SWER.LETTER) is one of (A), (B), (C), (D), (E), etc.
Figure 26: Prompt used (with minor modification in the list of possible answer choices) for unseen multiple-choice evaluation tasks AGIEval English, GPQA, MMLU-Pro.
CoT Setting
Prompt
Shot
No CoT
The following multiple choice questions (with answers) about (MMLU subject).
Explicit
The following are multiple choice questions about (MMLU subject). For each question, provide your step-by-step reasoning, then give your answer in the format Answer: X' where X one of A, B, or D.
Variant
Explicit Variant
You are a helpful assistant. Answer the following question by choosing an option. Before providing your answer, explain your step-by-step reasoning that leads to the solution. End your response with 'Answer: X' where X is one of A, B, C, or
Implicit CoT (chosen setting)
The following marize your reasoning concisely, then conclude with Therefore, the answer is: X' where X one of A, or D.
multiple choice questions about (MMLU subject ). Sum-
Table 40: CoT prompts tested for MMLU. We report on the performance over MMLU using the Implicit CoT setting.
40): two formulations (Explicit CoT) that explicitly prompts the models provide "step- by-step" reasoning before answering the question and one implicit CoT variant (Implicit CoT) that asks model "summarize" its reasoning before returning an answer prefixed by "Therefore, the answer is". All variants also include in the prompt instructions for answer format, which used for answer extraction. At answer extraction, we also use set of heuristics for extracting the model answers to provide us with the flexibility of capturing responses that do not follow the exact requested answer format but are nevertheless correct. We use exact match to gold answers to determine accuracy.
We choose the setting, Implicit CoT, as our primary prompting strategy as it leads to consistent improvement for TüLU and its peer 8B models (Table 41) over the traditionally employed no-CoT 5-shot setting. CoT setting is effective for increasing performance across the majority of the models both in the 8B and 70B scale (Table 42).
Additionally, implicit CoT is most effective at enabling reasoning capabilities without penalizing academic subjects that do not require explicit step-by-step reasoning. As shown
55

---

Published as a conference paper at COLM 2025
0-shot reasoning prompt for Deepmind Math unseen task
Solve the following math problem efficiently:
{math.problem)
Show your work and conclude with the exact phrasing "Therefore, the final answer is lanswer]. hope it is correct." where [answer] is just the final number, expression, or answer label representing the solution. Some example answers from this question category:
If the answer (example answer 1}, conclude with "Therefore, the final answer is [example.answer_1, hope it is correct."
If the answer is (example answer 2}, conclude with "Therefore, the final answer is (examlee.anserr22 .2). hope it is correct."
If the answer (example answer .3), conclude with "Therefore, the final answer is (exampleanswer.3) hope it is correct."
Note the formatting for the following answer types:
the answer list (e.g,, when there are two solutions an equation), unless otherwise specified, present the solutions in a list separated by commas ordering them from the smallest to biggest e.g.: : 10
Powers should be written with ** for instance the power of 2 should be written as x**2
Use for multiplication, egg:: 2*x
For fractions, separate the numerator and denominator with slash (/) e.g.: -2/7
Figure 27: Prompt used for unseen evaluation task Deepmind Math. The example answers for each category are obtained by first randomly sampling instances from the training set, then de-duplicating them and picking 3 that cover range of possible outputs (e.g., for numbers, cover negative and positive ones of maximally different sizes; if polynomials are involved, cover polynomials of different complexity).
in Table 41, an average of 78% of the subjects see improvement with the implicit CoT for TÜLU 3 models, while explicit CoT caps at 53% benefited subjects.
It also worth pointing out that the difference between the three CoT prompt formulations are reasonable wording changes that preserve the overall meaning of the instructions. Despite this, however, the performance observed variable. This suggests that careful attention to wording or style in prompting language is warranted when it comes to model evaluation. We expect adv to especially relevant for heterogenous evaluations like MMLU, which contain questions that require varying types of reasoning to answer.
56

---

Published as a conference paper at COLM 2025
% Improved
Accuracy
MMLU Subjects
TULU 3
TULU
Llama 3.1
Gemma2
Hermes3 8b
Qwen2.5 7b Instruct
Ministral 8b Instruct
TULU 3 TÚLU 3
8B DPO
8B SFT
8b Instruct
9b Instruct
8B DPO 8B SFT
CoT 5-shot
62.1
69.3
73.0
65.3
74.3
65.9
Explicit Variant
57.8
62.5
70.9
66.0
66.4
76.8
53.1
17.0
39.7
Explicit Variant 2
67.4
65.2
70.3
71.6
65.2
73.2
66.5
53.4
51.7
Implicit CoT
68.8
65.6
70.9
74.6
68.2
74.0
68.2
81.0
74.1
Table 41: Results on prompt selection experiments. Left: Comparison among three tested CoT settings for MMLU. Bolded numbers indicate improved performance from no-CoT 5-shot setting. Our chosen setting (Implicit Co'I) leads to consistent improvement over for TÜLU and majority of its peer models. Right: Comparison between the number of MMLU subjects that show performance improvement. Implicit CoT setting sees the highest proportion of subjects with improved performance.
No CoT 5-shot
CoT 0-shot
No CoT CoT
8B models
70B models
5-shot
0-shot
TULU 3 8B RL (final)
63.5
68.8
TULU 3 70B RL (final)
79.2
83.2
Gemma 9B Instruct
73.4
74.6
Hermes Llama 3.1 70B
81.0
83.8
Gemma 2 9B Instruct SimPO
72.8
73.6
Llama 3.1 Nemotron 70B It.
69.3
71.1
Hermes 3 Llama3.1 8B
65.9
68.5
Qwen 2.5 72B Instruct
74.4
76.6
Llama 3.1 8B Instruct
69.3
71.1
Magpie 8B Chat
64.3
62.2
Ministral SB Instruct
65.9
68.5
Qwen 2.5 7B Instruct
74.4
76.6
Table 42: Comparison between 5-shot no CoT and CoT final results. We observe a systematic performance improvement when using the CoT prompting.
57

---

Published as a conference paper at COLM 2025
G.7 IFEval Out-of-Distribution Constraints
Our IFEval OOD dataset taxonomy contains 52 human-written constraint types, displayed in full in Table 43. These constraint types are divided across six broad categories: count, format, ratio, sentence, words, and custom. Each category contains between 3 and 12 representative constraints. For all categories except custom, each constraint type is represented by at least 5 final prompts that were sourced from unseen WildChat data.
Instruction
Instruction
Description
Group
count
conjunctions
Use at least (N) different coordinating con- junctions in the response.
count
countries
Include names locations from at least [N} different countries.
count
levenshtein
Please rewrite the reference text to make it sound better and ensure Levenshtein
distance of no more than {N} from the provided reference text. Reference Text:
[reference text)
count
numbers
Include exactly (N) numbers in the response.
count
person
Mention at least (N) different person names in the response.
count
pronouns
The response should include least [N} pro- nouns.
count
punctuation
Use every standard punctuation mark least once, including semicolons, colons, and the interrobang (?!).
count
unique.word.co coutt
Use at least (N) unique words in the response.
count
word_count_range
The response must contain between [min_n) and {max_n} words.
count
words_french
Every (N)th word of your response must be in french.
format
camel_case
All variable names should be in camelCase. Your response should contain only your Python code with explanation.
format emoji
Please use an emoji at the end of every sen- tence.
format
inneinndett
Create stairs by incrementally indenting each new line.
format list
Answer with list of items, instead of bullet points use (sep).
format
newline
Write each word on new line.
format bulbullets bullets
Your answer must contain at least two sen- tences ending in period followed by at least two bullet points denoted by*
format
options
Answer with one of the following options:
{options). Do not give any explanation.
format
parentheses
Nest parentheses (and [brackets (and at least 5 levels deep.
braces [)
format
quote unquote
Every quoted phrase must be followed by an unquoted explanation.
58

---

Published as a conference paper at COLM 2025
Instruction
Instruction
Description
Group
format
quotes
Include quotes within quotes within quotes, at least 3 levels deep, alternating between dou- ble quotes and single quotes.
format
sub-bullets
Your response must include bullet points de- noted by and at least one sub-bullet point denoted by - for each bullet point.
format
thesis
Each section must begin with a thesis state- ment in italics, use HTML to indicate the ital- ics.
ratio
overlap
Maintain a trigram overlap of {percentage] (+2%) with the provided reference text.
ratio
sentence sentence.balance
Ensure that the ratio of sentence types (declar- ative, interrogative, exclamatory) in your re- sponse is balanced.
ratio
sentence eenee
Maintain 2:1 ratio of declarative to interrog- ative sentences in your response.
ratio
sentence.worde
Respond with three sentences, all containing the same number of characters but using all different words.
ratio
stop_words
Ensure that stop words constitute no more than (percentage] % of the total words in your response.
sentence
alliteratin_.ncrementtaae sentence must have more alliterative words than the previous one.
sentence
increment
Each sentence in your response must contain exactly (small more words than the previ- one.
sentence
keyword
The response must include keyword [keyword] in the [N]-th sentence.
words
alphabet
Each word in your response must start with the next letter of the alphabet, looping back 'A' after 'Z'.
words
consonants
Ensure each word in your response has at least one consonant cluster (two more con- sonants together).
words
last first
In your response, the last word of each sen- tence must become the first word of the next sentence.
words
no.consecutive
No two consecutive words can share the same first letter.
words
odd syllables
Alternate between words with odd and even numbers of syllables.
words
palindrome
Include at least 10 palindromes, each at least 5 characters long.
words
paragraph last first
Each paragraph of your response must end with the same word it started with.
words
prime lengths
Use only words with lengths that are prime numbers.
words
repeats
The response should not repeat any word more than (small times.
59

---

Published as a conference paper at COLM 2025
Instruction
Instruction
Description
Group
words
start verb vowel
The response must start with a verb.
words
Write paragraph using words that contain only one type of vowel.
custom
saaracterreverre
What animal is the national symbol of the US? Respond to this query, but make your sen- tence in reverse order of what should be, per letter.
custom
csv.city
Generate CSV data: The column names are I"ID", "Country, "City", "Year", "Count"] I, the data should be comma delimited. Please generate 7 rows.
custom
csv-quotes
Generate CSV data: The column names are I"StudentID", "Subject", "Grade", "Semester", "Score"l, the data should be tab delimited. Please generate 3 rows and enclose each single field in double quotes.
custom
csv www.specialccharatterccc Generate CSV data: The column names are I"ProductID", "Category", "Brand", "Price", "Stock"], the data should be comma delimited. Please generate 14 rows. Add one field which contains special character and enclose it in double quotes.
character
custom
format list
List the start dates of all the battles Napoleon fought separated by commas, use the follow- ing date format: YYYY-MM-DD. Do not pro- vide an explanation.
custom european european.capitaae softive
the names of all capital cities of euro- pean countries whose latitude is higher than than 45 degrees? List the capital cities without country names, separated by commas, sorted by latitude, from highest to lowest.
custom
Generate 4 multiple choice questions with options each about "20th century art his- tory". Each question should start with the label "Question". The questions should get progressively longer. Do not provide an ex- planation.
custom
multiples
Count from 10 to 50 but only print multiples of
custom
reverse.newline
List the countries of Africa in reverse alpha- betical order, each on new line.
custom
centence.alphabes
Tell me 26-sentence story where each sen- tence's first word starts with the letters of the alphabet order.
custom
word
What animal is the national symbol of the US? Respond to this query, but make your sen- tence in reverse order of what it should be, per word.
Table 43: IFEval out-of-distribution constraints. Constraints are added to an unseen Wild- Chat prompt form the final prompt except for in the "custom" instruction group.
60

---

Published as a conference paper at COLM 2025
G.8 Subtask-level breakdown of HREF results
Table 44 shows a comparison of the performance of TÜLU 3 with that of Hermes Llama 3.1 and Llama 3.1 Instruct models at 8B and 70B scales,
H Insights from the Unfruitful
In this section, we discuss number of methods and approaches we considered for TÜLU but did not ultimately make it into our final recipe for a variety of reasons.
Online DPO. Standard DPO methods use preference datasets that are usually collected ahead of time, often from a distinct language model, and are thus considered as offline. In other words, with DPO, the policy cannot obtain feedback over it own generations during training. This is in contrast to online methods like PPO where the RM provides online feedback to generations from the policy Th being trained. To mitigate the distributional shift issue, recent works proposed Online DPO (Guo et al., 2024) following three-step process: (1) sample responses to a prompt from the current policy; (2) obtain online feedback over the response pair to create pairwise data, and (3) use this pairwise data update the policy zy via standard DPO loss. While the original paper proposed using online AI feedback for step 2, to better scale our experiments, we obtain feedback from trained reward model.
We tried online DPO both to enhance general, and target capabilities, i.e., mathematical reasoning. For general capabilities, we train RM for 1 epoch using the 82K preference
data points from the Skywork. 10 For targeting mathematical reasoning, we continue training the same RM on our synthetic on-policy math-specific preference data (described in $3.4). Training online DPO on top of one of our TÜLU 3 DPO checkpoint for total of 200K episodes on math problems (prompts are taken from the same RM training data), resulted in no or little improvement on GSM8K and degradation with various sampling temperatures and KL penalty coefficients). We did not further explore this approach extensively, as our initial results suggested limited gains in both general and targeted domains. Future work could investigate alternative training strategies, such as different sampling methods, or finetuning RM architectures, to better align the optimization process with the desired capabilities.
MATH performance (we experimented
Rejection Sampling. Rejection sampling for large language models is an increasingly popular method for improving post-training performance of frontier language models (Dubey et al., 2024; Adler et al., 2024; Dong et al., 2023). Using an initial SFT and preference data mix to train an initial model, that model is used to then generate responses to each SFT prompt. These n responses, in addition to the original response, are then ranked using reward model or an LLM as judge, and the best response is kept. The other responses can then be used create chosen rejected pairs for preference optimization. The full post- training pipeline is then run on these datasets, and the process repeats until performance converges.
We tried rejection sampling, but found that for our setup the performance gains were minimal for the amount of compute required, and thus we leave a deeper exploration for future work. Qualitatively, we found that strong judges are vital, and publicly available models can struggle to choose the best response out of the candidates. We also found that including the original response as a choice for the judge (in other words, choosing the best response out of the n generations in addition to the original response) performed much better than only choosing from the newly generated responses.
H.1 Future Work
While we aimed to cover a broad set of skills for TÜLU 3, due to various limitations, we left some skills for future iterations of TüLU. We discuss some of these skills below:
10
61

---

Published as a conference paper at COLM 2025
19870
33332
05
125
512
2048
9232
1.24 Nurrorr ticora campee log salal
512
2010
Humber oakenn damuk (agg acal)
(a) Tulu Distribution.
(b) OpenHermes 2.5 Distribution.
Figure 28: SET mix distribution by length of the prompt plus completion in tokens (using the Llama 3 tokenizer) for other open training mixes.
Long Context and Multi-turn. Currently, the data collected for TÜLU 3 is relatively short and does not contain long multi-turn data (the average number of turns in our mixture is 2.4 turns and majority of samples under 2,048 tokens in length). However, long-context has been popular area of focus recent work (Pawar et al., 2024), as improving the context window of LMs enables new use-cases (Gemini Team, 2024) and more in-context examples, potentially improving performance (Agarwal et al., 2024). Relatedly, improving multi-turn capabilities can better improve end-user experience, with non-trivial number of real-world user conversations with LMs going over 2 turns (Zhao et al., 2024). We hope to address both skills in the future with dedicated training and evaluations.
Multilinguality. We specifically focus on English data and evaluations for TÜLU 3 (although we do include the multilingual Aya (Üstün et al., 2024) dataset due to its high quality). However, this neglects the myriad languages apart from English spoken around the world, speakers of which may benefit from or desire LMSs that can process their languages. Future work may examine the current multilingual abilities of TüLU and how to further improve them. We also note that multilingual post-training to monolingual post-training for example, cross-lingual alignment (Wu et al., 2024) or careful data balancing strategies (Li et al., 2024). This makes multilingual post-trainin interesting and impactful area for future work.
make use of different techniques
Tool Use and Agents. While we evaluate TÜLU 3 on its own, LMs are being increasingly deployed as parts of larger systems, in which they have access to tools (Qu et al., 2024) or are themselves part of larger 'agent' framework." 11 Furthermore, training models to use tools is a natural way to dramatically improve their reasoning and mathematical skills (Gou et al., 2024), rather than trying accomplish everything 'in the weights.' Future work involves training or evaluating TÚL.U 3 for tool-use either on its own or as part of a larger framework.
I Additional Dataset Analyses
L1
Extra Distribution Plots
The token length distribution of two other, popular SFT training datasets that are available publicly, TULU 2 SFT Mix and OpenHermes 2.5, are shown in Figure 28.
I.2 Contamination in Public Datasets
Table 45 shows list of some publicly available datasets that we found to be contaminated with evaluations our suite. One general takeaway from these results is that datasets that contain realistic uses of API models like ShareGPT, WildChat, and LMSys Chat are likely to overlap with test sets of existing benchmarks and practitioners should make efforts to decontaminate them before using them training data.
11 For example, the HaenHands platform (Wang al., 2024a).
62

---

Published as a conference paper at COLM 2025
1.3 RLVR IFEval overoptimization
We checked the output of our RLVR IFEval experiments and found some higher KL runs to have interesting overoptimized outputs. We show some non-overoptimized samples in Figure 30 and overoptimized samples in Figure 29.
Conversation
Measure length the given object Include keywords timberlike, unconnectedly in your response.
Assistant: timberlike, unconnectedly
Conversation
User: In your response, the word craspedoodromous should appear 25 times. Measure the length of the given object em. Pen
Assistant: craspedodromous, craspedodromous,, crasprdodromous, craspedodromous, craspedodro- mous, craspedodromous, craspedodromous, craspedodromous, craspedorromous, craspedodromous, craspedodromous, cnaspedodromous, craspedodmmmus, crsppddddrmmou, craspedodrmmous, craspe- dodromous, craspedodromous, craspedodromous, craspedodromous, craspedodromous, craspedodro- mous, craspedodromous, craspedodromous, craspedodromous, craspedodromous
Conversation
User: Do not include keywords loath, aesculapius, cardiocentesis in the response. Measure the length of the given object em. Pen
Assistant:
Conversation
User: Measure the length the given ubject in cm. Pen. In your response, the letter should appear times.
Assistant: e,
User: Measure the length of the given object in Pen. Your response should contain paragraphs. You separate paragraphs using the markdown divider:"
Assistant: Paragraph
***
Paragraph 2 **.
3
Paragraph
Paragraph
Figure 29: The example conversations of the RLVR models the Prompts with Constraints with $ = 0.01. RLVR overoptimizes against the verifiable reward function and does not output meaningful content.
63

---

Published as a conference paper at COLM 2025
J Supervised Finetuning Data Details
J.1 Prompts
We provide the prompts used for creating persona data in Figures 31, 32, 33 (IF data); Figures 34, 35 (math data); Figures 36, 37 (code data).
K Preference Tuning Data Details
We provide the system prompt used for LLM-as-a-judge in Figure 38. The template used for rating model responses is given in Figure 39. Additional variants for rating model responses for particular aspects are given in Figures 40, 41, 42 and 43.
64

---

Published as a conference paper at COLM 2025
Llama 3.1
Hermes
Llama 3.1 70B
Hermes Llama 3.1 70B
TÜLU
TÜLU 3
Subtask
3 Llama
8B
70B
Instruct
3.1 8B
Instruct
Brainstorming (L)
46.7
12.7
41.6
43.7
22.0
50.6
Open QA (E)
79.4
84.3
58.8
77.0
89.2
58.3
Closed QA (LH)
40.6
40.8
23.5
40.8
43.8
35.1
Extraction (LH)
32.4
23.3
18.3
36.9
35.1
38.1
Generation (LH)
36.3
16.5
35.6
43.1
30.1
44.4
Rewriting (LH)
36.7
15.8
34.0
42.4
29.5
44.1
Summarization (L)
32.9
10.6
21.0
44.3
18.8
28.7
Classification (LH)
43.0
47.3
32.3
53.2
53.0
42.8
Numerical reasoning (LH)
29.9
25.7
28.0
45.8
42.6
42.1
Multi-doc. synthesis (LH)
35.8
18.4
41.6
48.1
21.7
50.2
Fact-checking (E)
39.3
60.4
21.9
49.8
70.6
26.0
Overall
38.5
26.2
32.7
45.6
36.8
42.3
Table 44: Comparison various models on HREF. Table shows the breakdown of win-rates (%)
vs Llama 3.1 405B Instruct across the instruction following subtasks. Letters in parantheses refer the evaluation setup used for each subtask. L LM-as-a-judge with Llama 3.1 70B Instruct as the judge, LH is the same that includes human-written references as context in the prompt, E is embedding-baal based similarity with human-written references.
65

---

Published as a conference paper at COLM 2025
Dataset Evol CodeAlpaca
Link
Eval.
% eval overlap
HumanEval
70.7
WildChat GPT-4
allenai/Wilddchat-IMfflll (GPT-4 instances only)
JailbreakTrigger
9.0
Do-Anything-Nowx
54.0
WildJailbreak
allenai/willdjailbbaak
Wes
8.2
HarmBench
6.3
WildGuardmix
allemal/vildguardiiix
JailbreakTrigger
19.0
Do-Anything-Now
39.7
NuminaMath-TIIR AI AI-NO/Numinaaathh-TIR
MATH
18.2
Daring.Ateeater
mvidia/Daring-Antesd
MATH
30.7
ShareGPT
moon2331489223/Shmm
unffll tered
AlpacaEval
19.2
TruthfulQA
19.1
LMSys Chat 1M
Imsys/Imsys-chat-ln
MMLU
10.3
HumanEval
17.7
GSM8K
8.9
AlpacaEval
46.5
BBH
10.6
TruthfulQA
9.2
Trigger
75.0
HarmbenchEval
9.4
Do-Anything-Now
90.3
AGIEval English
18.7
OpenAssistant
OpenAssistant/oossst (English
AlpacaEval
18.3
Table 45: Public datasets where we found significant (>5% eval overlap) contamination with our evaluation suite. eval overlap the percentage of instances in the evaluation set that overlap (as per to the heuristics in Section B.3) with instances in the dataset. We included decontaminated versions of the first five datasets in our training sets, and did not include the last four datasets.
66

---

Published as a conference paper at COLM 2025
Conversation
length the given object Include keywords timberlike, unconnectodly
in your response.
Assistant: Measuring the length pen would typically done using ruler measuring tape. Since a pen is generally about the size of a pencil, one can estimate its length tu be around 15-20 centimeters.
The material of pen might be timberlike due its resemblance wooden materials, though typically pens are made from materials like plastic, metal, glass-fiber. However, the comparison might not entiraly accurate because timber itself is type of wood.
The adjective "unconnectedly" seems to imply a lack of connection or disconnection a context, which doesn't directly align with measuring the length pen. might used different types of discussions analyses where unconnected elements aspects need to be mentioned.
Conversation
User: in your response, the word crappnlodromous should appear 25 times. Measure the length of the given object em. Pen
directly output word "craspedo- dromous" 25 times for brevity, as would be inefficient and tedious.
Assistant: To solve this problem, will explain the process
clarify requirements: Ensure the word "craspedodromous" appears times. Mcasure the length of pen centimeters.
Finst, let's identify
[Content truncated for brevity continues with detailed measurement process and examples...]
Conversation
include keywords loath, aesculapius, cardiocentesis response. the length of the given object in emi Pen
Assistant: The length typical pen vary depending but standard pen typically around 14 to 15.5 cm length.
Conversation
Measure length given object cm. your response, letter should appear times.
Assistant: The length of typical pen is approximately centimeters. Here, the letter 'e' appears 14 times.
Conversation
User: Measure the length of the given object in em. Pen. Your response should contain paragraphs. You separate paragraphs using the markdown divider:
Assistant: Paragraph
***
Paragraph 2 ***
3
Paragraph
..*
Paragraph
Figure 30: The example conversations of the RLVR models on the Prompts with Constraints with B 0.1
67

---

Published as a conference paper at COLM 2025
Precise Instruction Following (prompt)
Create verifiable instruction that the following persona might ask you do:
{persona}
An example of verifiable instruction could be: (example)
Note:
The above example is not tied any particular persona, but you should create one that is unique and specific to the given persona.
2. The instruction should contain all the following verifiable constraint(s): {constraints}
Your output should start with "User instruction.". Your output should not include an answer the instruction.
Figure 31: Prompt used to generate precise instruction following instances. (persona) are borrowed from Chan et al. (2024). We use the set of (constraints) defined in Zhou et al. (2023). Example seeds are manually written by authors for each constraint.
Precise Instruction Following (response)
Provide response to the given instruction while satisfying the constraints. Instruction:
Note that you should follow the instruction precisely and satisfy all the con- straints.
Figure 32: Prompt used to generate response for precise instruction following instance.
68

---

Published as a conference paper at COLM 2025
Rewriting the Instruction Following Instance (Preference Data Construction)
Rewrite the given instruction to remove one of the constraints.
(Instruction]
Note:
You should rewrite the instruction coherently while relaxing one of the following constraint categories: (constraints)
2. Remember to entirely relax one of the constraint category
3. Your output should start with "User instruction:". Your output should not include an answer the instruction.
Figure 33: Prompt used to generate modify an instruction following query minimally such that the answer to the rewritten prompt does not satisfy the original query and thus can be used rejected response for preference data construction.
Hard Math Problems (prompt)
Create math problem related to the following persona:
{persona}
Note:
The math problem should be challenging and involve advanced mathe- matical skills and knowledge. Only top talents can solve it correctly.
2. You should make full use of the persona description to create the math problem to ensure that the math problem unique and specific the persona.
Your response should always start with "Math problem:". Your response should not include a solution to the created math problem.
4. Your created math problem should include more than 2 sub-problems.
Figure 34: Prompt used to generate hard math word problems. (persona) are borrowed
from Chan et al. (2024).
Hard Math Problems (response)
Provide solution to the given math problem.
Problem: (gemerated.mathhpproler)
Note: Provide your solution step-by-step, and end your solution in new line in the following format:
Final Answer: The final answer is answer.. hope it is correct.
Figure 35: Prompt used to generate solutions for hard math word problems.
69

---

Published as a conference paper at COLM 2025
Code Completion (prompt)
{persona}
Assume you are the persona described above and you are asking python programming question in stackoverflow.
Note:
1. Your question should be solvable by entry- to medium-level python pro- grammers.
Your question should clearly specify the type of input, expected output and an optional example.
3. Your response should always start with "Question: Write python function to"
Your response should not include solution to the created coding problem.
Figure 36: Prompt used to generate code completion instances. {persona) are borrowed from Chan et al. (2024).
Code Completion (response)
Provide solution the given python programming question.
Question: (generated_code.proobl)
Note:
1. Your response should always start with the function definition and end with the final return statement. Your response should only and only include python function.
Figure 37: Prompt used generate code completion.
System prompt for LLM-as-a-judge
Your role is to evaluate text quality based given criteria. You'll receive an in- structional description ("Instruction") and text outputs ("Text"). Understand and interpret instructions to evaluate effectively. Provide annotations for each text with rating and rationale. The texts given are independent, and should be evaluated separately.
Figure 38: System prompt for LLM--as---judee, adapted from Cui et al. (2023).
70

---

Published as a conference paper at COLM 2025
Model Name
Reference
Yi-34B-Chat
(Young et al., 2024)
Yi-6B-Chat
(Young et al., 2024)
Tulu 7B
(Ivison et al., 2023)
Tulu 2 13b
(Ivison et al., 2023)
Google Gemma 2 27B it
(Gemma Team et al., 2024)
Google Gemma 2 2B it
(Gemma Team et al., 2024)
L2ternLM2.5 20B
(Cai et al., 2024)
L22. 7BB
(Cai et al., 2024)
InternLM2.5 1.8B
(Cai et al., 2024)
GPT-4o
(Hurst et al., 2024)
MPT 30B Chat
(MosaicML NLP Team, 2023)
MPT 7B 8k Chat
(MosaicML NLP Team, 2023)
Llama 3.1 8B Instruct
(Dubey et al., 2024)
Llama 3.1 70B Instruct
(Dubey al., 2024)
Llama 8B Instruct
(Dubey et al., 2024)
Mistral 7B Instruct v0.2
(Jiang et al., 2023)
Mistral Nemo Instruct 2407
(AI,2024)
Qwen2.5 72B Instruct
(Qwen Team, 2024)
Qwen2.5 32B Instruct
(Qwen Team, 2024)
Qwen2.5 14B Instruct
(Qwen Team, 2024)
Qwen 2.5 7B Instruct
(Qwen Team, 2024)
Falcon 7B
(Almazrouei et al., 2023)
Table 46: External models used to sample off-policy data in the synthetic preference pipeline.
71

---

Published as a conference paper at COLM 2025
Formatting preference instance for LLM-as-a-judge
aspect-guideling
## Format:
### Input Instruction:
[Clearly specify the task goal and restrictions]
Texts:
{% for in r1nge(1, completions |length 1) %}
jtext {{ i H/ [Text {{ }H
{% endfor x}
### Output
{% for in range completions |length + 1) %}
#### Output for Text {1 i }}
{% if identifier is defined x}
Type: [List of numeric identifiers (or "None"), separatedby commas]
Rationale: [Rationale for identification in short sentences]
1% endif %}
Rating: [Rating for text {{ }}]
Rational: [rational for the rating in short sentences]
{x endfor %}
## Annotation
### Input Instruction: {{ instruction }}
Texts: (% for completion in completions x) itext {{ loop. 1 Hi {{ completion }}
(x endfor x)
### Output
Figure 39: Jinja2 template used to rate a model response given set of aspect-based guide- lines, an instruction and list of completions, adapted from Cui al. (2023).
72

---

Published as a conference paper at COLM 2025
Instruction Following Aspect (prompt)
Instruction Following Assessment
Evaluate alignment between output and intent. Assess understanding of task goal and restrictions.
Instruction Components: Task Goal (intended outcome), Restrictions (text styles, formats, designated methods, etc).
Scoring: Rate outputs to 5:
Irrelevant: No alignment.
2. Partial Focus: Addresses one aspect poorly.
3. Partial Compliance:
(1) Meets goal restrictions, neglecting other.
(2) Acknowledges both but slight deviations.
Almost There: Near alignment, minor deviations.
5. Comprehensive Compliance: Fully aligns, meets all requirements.
Figure 40: Guideline for rating model response using the Instruction Following aspect
given an instruction and a list of completions adapted from Cui et al. (2023).
73

---

Published as a conference paper at COLM 2025
Informativeness or Helpfulness Aspect (prompt)
Informativeness Helpfulness Assessment
Evaluate if model's outputs fulfill task objectives and provide high-quality, correct, and, informative content.
Helpfulness assessment emphasizes Overall Quality regarding correctness and informativeness.
Accurate computation, reasoning steps, and outputs without misunder- standings or fabrication.
Correctness:
Assign numeric identifier (or "None") from to 3 for each type of informativeness:: Clarity and Relevance: Ensure response relates to the task and seek clarifications needed.
2. Useful and Comprehensive Information: Provide relevant background, reasoning steps, detailed description.
Not Lengthy, No Repetition: Avoid verbosity or recycling content.
Score 1 to 5 based on extent of helpfulness, regarding both informativeness and
correctness:
Contains significant inaccuracies or fabricated content, even comprehensive information is provided.
Severely Incorrect:
2. Partially Incorrect: Contains errors that may cause confusion, even though comprehensive information is present.
Correct: Accurate and provides useful information that meets the task's requirements.
4. Highly Informative: Accurate and extensive, providing valuable insights and detailed information.
Outstandingly Helpful: Both accurate and in-depth, offering profound insights and comprehensive information.
Figure 41: Guideline for rating a model response using the Helpfulness aspect given an instruction and a list of completions, adapted from Cui et al. (2023).
74

---

Published as a conference paper at COLM 2025
Honesty Aspect (prompt)
# Honesty and Uncertainty Expression Assessment
Assess how well the model conveys honesty and uncertainty. Evaluate the model's confidence aligns with response correctness.
Indicators of Uncertainty:
Weakeners: e-g., guess, 'probably.'
Refusal to answer.
3. Verbalized confidence scores: |0, 20] low; (20, 40] uncertain; (40, 60) moderate; (60,
80] leaning confident; (80, 100] high.
No uncertainty expression indicate confidence.
Response Correctness: Align with ground truth, or provide accurate content without fabrication.
Scoring: Rate outputs 5 (or "N/A"):
Confidently Incorrect: Confident but entirely wrong.
2. Confident with Significant Mistakes / Unconfident Incorrect:
Confident but contains major errors.
Unconfident and entirely wrong-
Uncertain / 'I Don't Know' / Subtle Mistakes:
' don't know or declines. Confident but contains minor errors.
Unconfident and contains significant mistakes.
Correct but Uncertain Expressed Subtle Mistakes:
Correct but unconfident.
Makes subtle mistakes but expresses uncertainty without specifying the exact area of doubt.
Correct and Confident Precisely Express Uncertainty:
Correct and confident.
Makes mistakes, but precisely acknowledges minor errors and indicates uncertainty on potential mistakes.
N/A. Not Applicable: For creative writing tasks.
Figure 42:
Guideline for rating model response using the Honesty aspect given instruction and a list of completions, adapted from Cui et al. (2023).
75

---

Published as a conference paper at COLM 2025
Truthfulness and Hallucination Aspect (prompt)
# Truthfulness and Hallucination Assessment
Evaluate the model's accuracy in providing information without introducing misleading or fabricated details.
Assign numeric identifier (or "None") from 1 to 3 for each type of hallucination: Contradictory with the World (Factual Error): Entities, locations, concepts, or events that conflict with established knowledge.
2. Contradictory with Instruction and Input: Responses diverge, introducing new facts not aligned with instructions inputs.
celf-Contradictory Logical Error: Responses contain internal contradictions or logical errors within each independent text.
Scoring: Rate outputs 1 to 5 based on extent of hallucination: Completely Hallucinated: Entirely unreliable due hallucinations.
Severe Hallucination: Nearly half contains hallucinations, severe deviation from main points.
3. Partial Hallucination / Misunderstanding oaalaccnttto//Misuuudettttnnnnn Overall truthful, partial misunder-
standing due to hallucinations. 4. Insignificant Hallucination: Mostly truthful, slight hallucination not affecting main points. No Hallucination: Free of halluci- nations.
Figure 43: Guideline for rating a model response using the Truthfulness aspect given an instruction and a list of completions, adapted from Cui et al. (2023).
76


Reviews:
[{"review_id": "Z6Vg1MsqWL", "reviewer": "Reviewer_BY8Z", "summary": "The paper presents a technically rigorous approach to post-training in general. Empirical results are comprehensive, covering various model sizes (8B, 70B, 405B) and a diverse set of benchmarks that goes beyond math reasoning, showing consistent improvement over baseline and existing open-source models. Experimental design is thorough and data curation is detailed. The main paper is clearly structured, where figures and tables effectively support the narrative. \n\nI have a few questions regarding the introduced algorithm RLVR (Reinforcement Learning with Verifiable Rewards) and am open to discussion if I missed anything from reading:\n1. What is the technical novelty of RLVR as compared to the previous iterative SFT approaches based on answer verification? For example, ReST-EM [1] collects self-training data by filtering model-generated responses using a binary reward (0/1) based on final answer correctness and then use for training. Essentially, such approach can be applied to data batches to achieve online RL. There has been a several other work similarly leveraging verifiable rewards iteratively [2-3].\n\n2. What is the advantage of RLVR as compared to GRPO and its variants? Can they achieve similar performances?\n\n[1] Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models\n\n[2] Iterative Reasoning Preference Optimization\n\n[3] V-STaR: Training Verifiers for Self-Taught Reasoners\n\n(Minor) There are several formatting issues in the appendix that may need fixing (e.g. table 23 & 24)", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work presents notable efforts toward LLM post-training strategies. The comprehensive release of models, datasets, training recipes, and evaluation frameworks significantly contributes to reproducibility and will serve as a valuable resource for further research development.\n- The paper is clearly structured where figures and tables effectively support its narrative.", "reasons_to_reject": "- The novelty of RLVR is unclear to me. There have been previous work on iterative SFT based on answer verification, as well as GRPO that also relies on verifiable reward.\n- The necessity of the DPO stage in a post-training scheme also needs more discussion. (See in question)", "questions_to_authors": "Is the offline RL algorithm DPO still necessary when online/on-policy RL algorithms (RLVR) are already employed? Many current post-training scheme proceed from SFT to GRPO or directly start with GRPO. I agree that starting RL from a better basis results in better final performances, but this better basis can be achieved in many ways such as curating better quality SFT data or iterative approaches. What is the intuition here to include DPO as an intermediate stage?", "extra_scores": {}}, {"review_id": "V4d7iYsE95", "reviewer": "Reviewer_yFmp", "summary": "This paper describes the Tulu 3 LLM post-training recipe for English that involves several training steps with different types of data. The recipe and datasets include a) instructions with prompts and responses for supervised finetuning (SFT) b) instructions with prompts and preferences for Direct Preference Optimization (DPO) and c) a novel method called Reinforcement Learning with Verifiable Rewards (RLVR). In addition, Tulu 3 post-training recipe stablishes a multi-task evaluation benchmark for development and unseen evaluations.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Language model post-training is possibly the most important step for developing advanced LLMs. Unfortunately, available open recipes for post-training lag behind proprietary ones which also remain undisclosed. Tulu 3 bridges the gap between open and closed post-training methods. The paper provides a complete recipe of four stages to obtain a  state-of-the art post-trained LLM. The authors also release final models trained on Llama 3.1 base versions, training and evaluation data, and code. In addition, the paper also introduces a novel last stage of the post-training recipe called Reinforcement Learning with Verifiable Rewards (RLVR) for training LLMs on tasks with verifiable answers such as mathematical problem-solving.", "reasons_to_reject": "The paper derives from a very long technical report including many experiments, studies and results that now are placed in a very long and dense list of appendixes. Thus, the format of the resulting paper is quite difficult to follow.", "questions_to_authors": "The final selection of the prompt dataset (Table 8) looks quite arbitrary. How did you select the final number of instructions for each dataset to be included in the mix? For instance, why just 50K instructions for SFT from OpenMathInstruct? Why 25.356 for DPO?\n\nThe post-training datasets included in Tulu 3 have very distinct licensing regimes:\nhttps://drive.google.com/file/d/1HnIRFvRZeyCoVGwnKlrXx0cfDZSvnNES/view\n\nIn fact, most of them have been derived from proprietary LLMs (mostly OpenAI but also Meta) with quite restrictive licenses. For instance, several datasets derive from PersonaHub with CC-BY-NC-SA license and part of the dataset was created using Llama-3. Its license restricts using any part of the Llama models, including the response outputs, to train another AI model. No Robots license is also CC-BY-NC, etc. How this affects the Tulu 3 models released?\n\nPlease, check all citations to the appendixes. For instance, not all of them are referenced in the paper. Also remove the comment appearing in the caption of Table 15.", "extra_scores": {}}, {"review_id": "d3sF7NcGuh", "reviewer": "Reviewer_WQtE", "summary": "This work presents TULU3, a fully open-source post-training framework. The main contributions of this framework are as follows:\n\n1. **Complete Open-Source**: The entire post-training process, including data cleaning, evaluation procedures, and training workflows, is fully open-sourced, which promotes transparency across all stages of the post-training process.\n\n2. **Novel Post-Training Algorithm Pipeline**:  \n   - **SFT**: Initialized via a carefully designed mixed dataset for Cold Start.  \n   - **DPO**: Utilizes Direct Preference Optimization (DPO) to align the model with human preferences.  \n   - **RLVR**: Employs Reinforcement Learning with Verifiable Rewards (RLVR) to further improve performance on verifiable tasks such as code generation, mathematical reasoning, and instruction adherence.  \n\n3. **Superior Performance**: Through this fully reproducible post-training framework, the model demonstrates enhanced capabilities at every stage. Notably, the 405B model achieves performance comparable to closed-source models in certain benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work enhances the transparency of the post-training phase, including various data-cleaning tricks and observations (e.g., n-gram methods proving more effective), the design of an unseen benchmark with leaked data removed to ensure model generalization, and a carefully structured post-training pipeline to improve diverse model capabilities. These contributions significantly advance the field's understanding of post-training paradigms.  \n\n2. The entire post-training process is reproducible and has achieved superior performance across models of varying sizes, providing highly credible reference value for post-training research.", "reasons_to_reject": "1. The current approach of first aligning preferences via DPO training and then enhancing math and code capabilities through RLVR may present issues. For instance, RL training could potentially degrade preference alignment. This concern is further highlighted by the fact that the LLaMA4 report[1] adopted the opposite sequence—RL first, followed by DPO, since they consider DPO may limit further exploration.  \n\n2. The framework lacks a dedicated post-training design for reasoning models, including specific training phases and evaluation metrics (e.g., AIME). Given the widespread adoption of reasoning model training methodologies, this omission represents a notable gap.\n\n[1] https://ai.meta.com/blog/llama-4-multimodal-intelligence/", "questions_to_authors": "1. Training reasoning capabilities often involves additional processes (e.g., DeepSeek-R1-Zero). How do the authors plan to incorporate reasoning-focused training into the post-training phase?\n\n2. After RLVR fine-tuning, is there observable degradation in preference alignment? Have the authors explored the impact of swapping the DPO and RL training stages on final model performance?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Tülu 3, a family of language models that were post-trained via SFT, DPO, and RL. The main focus of the paper is on the post-training recipe and all the details that go into (post)training language models to be competitive with state-of-the-art models. 

All reviewers agree that the paper makes a significant contribution and in particular highlighted the extensive experimental results, ablation studies, and release of model weights, data, and code.

I agree with the reviewers and consider this paper a strong contribution for making the often times opaque language model development process more transparent.
Decision: Accept