The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains
Authors: Scott Geng
Hamish Ivison
Chun-Liang Li
Maarten Sap
Jerry Li
Ranjay Krishna
Pang Wei Koh
Venue: COLM 2025 (2025)
Keywords: Preference tuning
LLM post-training
synthetic data
weak-to-strong generalization
TL;DR: We show that preference pairs of weak data can be leveraged to improve a stronger language model beyond the strength of each individual point, an insight that enables new state-of-the-art post-training recipes that work without strong supervision.

Abstract: Improvements in language models are often driven by increasing the quality of the data we train them on, which can be limiting when strong supervision is not readily available. In this work, we show that paired preference data consisting of individually weak data points can enable gains beyond the strength of each individual sample. We formulate the **delta learning hypothesis** to explain this phenomenon, positing that the relative quality _delta_ between points suffices to drive learning via preference tuning—even when supervised finetuning on the weak data hurts. We validate our hypothesis in controlled experiments and at scale, where we post-train 8B models on preference data generated by pairing a small 3B model's responses with outputs from an even smaller 1.5B model to ensure a meaningful delta. Strikingly, on a standard 11-benchmark evaluation suite (MATH, MMLU, etc.), our simple recipe matches the performance of Tülu 3, a state-of-the-art open model that was tuned from the same base as our model while relying on vastly stronger supervisors (e.g., GPT-4o). Delta learning thus enables simpler and cheaper open recipes for state-of-the-art post-training, highlighting that models can learn a surprising amount from data that might typically be considered weak.

Published as a conference paper at COLM 2025
The Delta Learning Hypothesis:
Preference Tuning on Weak Data can Yield Strong Gains
Scott Geng* Hamish Ivison +
Chun-Liang Li Maarten Sap
Jerry Li
Ranjay Krishna
Pang Wei Koh
Unniversity of Washington
Allen Institute for AI Carnegie Mellon University
sgengêcs, sgengges.wwshhington edu
titb Repo
Abstract
Improvements in language models are often driven by improving the quality of the data we train them on, which can be limiting when strong supervision is scarce. this work, we show that paired preference data consisting of individually weak data points can enable gains beyond the strength of each individual data point. We formulate the delta learning hy- pothesis to explain this phenomenon, positing that the relative quality delta between points suffices to drive learning via preference tuning even when supervised finetuning the weak data hurts. We validate our hypothesis in controlled experiments and at scale, where we post-train 8B models on preference data generated by pairing small 3B model's responses with outputs from an even smaller 1.5B model to create meaningful delta. Strik- ingly, on a standard 11-benchmark evaluation suite (MATH, MMLU, etc.), our simple recipe matches the performance of Tülu 3, a state-of-the model tuned from the same base model while relying on much stronger su- pervisors (e.g., GPT-4o). Thus, delta learning enables simpler and cheaper open recipes for state-of-the of-the-rtttttttt learning, we prove in logistic regression that the performance gap between two weak teacher models provides useful signal for improving a stronger student. Overall, work shows that models can learn surprisingly well from paired data that might typically be considered weak.
open
better understand delta
Weak Training Data
Student Model
Finetuned Students
Synthesized with Llama3 3B (75.7% GSM)
SFT
70.3%
Weak Data
(- (-5.7%)
Tulu3
Quality Delta
DPO
81.5%
(+ 5.5%
76.0% GSM
SFT
Weaker Data
Synthesized with 1B (44.4% GSM)
Figure 1: The delta learning hypothesis posits that paired preference data enables language models to learn from relative differences in data quality, driving gains beyond the absolute quality of each individual data point. Example: Tuning Tül--3-SB-SFT to prefer greedy responses from Llama3 3B over those from Llama3 1B improves Tülu's GSMSk accuracy, even though both Llamas are weaker than Tülu GSM8k. SFT on the weak data hurts.
1 Introduction
Common wisdom in machine learning holds that strong data builds strong models: improving performance typically requires training on data that exceeds a model's current capabilities. This principle has driven progress across the language model pipeline- from pretraining

---

Published as a conference paper at COLM 2025
corpus curation (Li et al., 2024; Penedo et al., 2024; OLMo et al., 2024), to rejection sampling for finetuning data (Dong et al., 2023; Adler et al., 2024), and to preference tuning, where human annotators identify the best model outputs as targets to tune towards (Ouyang et al., 2022; Bai et al., 2022). However, this wisdom also implies an inherent limitation: model capability may be upper- upper-bounded by the strength of supervision available. Many desirable tasks are difficult to support with strong data, either because of high collection costs (e.g., synthesizing scientific literature at PhD level) or because the task exceeds current human expertise (e.g., formulating a unified theory of physics). Thus, we ask: how might we build models that exceed the capabilities demonstrated in their training data?
In this paper, we show that preference pairs consisting of individually weak data points (e.g., responses from weak models) can be leveraged to improve a stronger language model beyond the strength of each individual sample. Our study is motivated by preliminary evidence in the literature (Yao et al., 2024; Zhu et al., 2024) and an intriguing pilot result: preference tuning modern 8B Llama 3 (Dubey et al., 2024) large language model (LLM) using paired outputs from weaker, past-generation models consistently leads to performance gains, even when supervised finetuning on those same weak responses directly results in degradation.
We formalize these observations as the delta learning hypothesis (Figure 1), which posits that data with high absolute quality is not strictly necessary to improve language models. Instead, the relative quality difference- the "delta" " delta"-between between paired samples can provide sufficient supervision to guide improvement through preference tuning, even if neither sam- ple alone is stronger than the model being trained. Intuitively, the delta defines a meaningful direction of improvement; a strong model may learn to generalize along this direction and improve beyond the absolute quality of the preferred example. We systematically test our hypothesis in two controlled experiments by explicitly constructing preference pairs with limited absolute quality but a clear delta, and find consistent empirical evidence in support.
Our hypothesis enables new open recipes for state-of-the-art language model post- training without requiring any strong supervision. To test the limits of "delta learning," " we preference-tuned Tulu-3-8B-SFT, the instruction-finetuned precursor to Tülu 3, state- of-the-art openly post-trained model (Lambert et al., 2024). In contrast to typical open recipes (Lambert et al., 2024; Ivison et al., 2023), which heavily distill from strong supervi- sors (e.g., GPT-4o) to generate high-quality chosen responses, we generate chosen responses with a single small model (egg,, Qwen 2.5 3B Instruct) that is not stronger than T-l--3-8B-SFT itself. We pair these responses with outputs from an even smaller model (e.g., Qwen 2.5 1.5B Instruct), thus creating a delta for learning. Strikingly, on a standard 11 11-benchmark evaluation suite, our simple recipe matches Tülu 3's performance, despite using vastly less supervision. Our analysis explains our recipe's 's success; we find that the chosen response only needs to meet surprisingly low quality threshold (i.e., not significantly worse than base model), beyond which the size of the quality delta becomes the primary determinant of downstream preference tuning performance. Delta learning offers simple, cheap, and performant post-training, reducing the reliance of open recipes on strong model distillation.
To further illuminate why delta learning works, we theoretically study a logistic regression setup where a student model is trained to prefer synthetic pseudo-labels from a (possibly weak) teacher model over those from an even weaker one. We prove that even when both teachers provide misleading supervision individually, the performance gap between them ensures that the delta between these signals is still directionally correct. Learning from this delta can improve an already-strong student with high probability.
Overall, our work shows that models can learn surprisingly well from weak data, provided the data is paired to expose informative deltas. We find these deltas are often readily obtainable- eeein even simple heuristics like model size differences suffice to capture them. Thus, we optimistic that weak, currently unused data may be revitalized into valuable supervision. Furthermore, curating pairs of weak data may offer a more scalable alternative to finetuning in settings where strong supervision is limited-for example, by generating targeted corruptions to existing data or collecting lightweight human edits of weak model outputs. Finally, curating paired data may potentially enable training of superhuman models with preference labels on human-level outputs (Burns et al., 2023; Bowman et al., 2022). We leave these directions for future work.
2

---

Published as a conference paper at COLM 2025
Model / Training
MMLU
AE2
Full Avg.
Table 1: We tune Llama 3 Instruct models on the ULTRAFEEDBACK- -WAK preference dataset, generated by models weaker than Llama 3. Training with prefer- ence learning (DPO) prefer "weak responses" over "weaker responses" - -ields gains, while SFT directly
62.9
18.7
57.8
UF-WEAK SFT
61.8
12.3
54.0
+ UF-WEAK DPO
64.0
22.4
59.0
LLAMA-3.1-8DA
71.8
24.9
63.9
UF-WEEK SFT
65.7
56.1
on the weak preferred responses hurts performance.
+ UF-WEAK DPO
72.0
26.3
64.5
Blue indicates gain over baseline, orange degradation.
2 A Warm-up Case Study
We begin our investigation with an intriguing empirical finding: ence data generated by weak models can improve stronger model's performance, even when finetuning directly on the weak models' outputs hurts.
training on paired prefer-
Data. We start with ULTRAFEEDBACK, a popular preference dataset (Cui et al., 2023) consisting of preference pairs (x, ycr yr), where y, and y, are an LLM-generated chosen and rejected response (respectively) to a prompt x. We filter to explicitly exclude all responses from models that have an LMSYS Chatbot Arena ELO score near or above Llama-3.2-3B- Instruct. Hence, the chosen response Y/ now derives from a model weaker than the Llama models, although it is still higher-quality than the rejected response yr. We call the resulting filtered dataset ILTRAFEEDBACK-VE WEAK. See Appendix G.2 for details.
Training and evaluation. We finetune Llama-3.2-3B- Instruct and Llama-3.1 1aae-33.-8---nntteee (Tou- vron et al., 2023) on ULTRAFEEDDACK--WEEEE the DPO algorithm (Rafailov et al., 2024) on the preference pairs Vc.)). We compare to (2) supervised finetuning (SFT) directly on the chosen responses (x. We evaluate models on 8 standard benchmarks that measure knowledge recall, mathematical reasoning, instruction following, truthfulness, general reasoning, and coding. See Appendix G.2 for full details of benchmarks used, along with training and hyperparameter details.
in two ways. One, we (1) preference tune with
Results. We show a representative subset of results in Table 1, deferring the rest to the Appendix (Table A2). SFT on the chosen responses significantly hurts performance- because the models are finetuned to imitate weaker outputs. Yet surprisingly, preference tuning with the same weak preference pairs improves overall performance across bench- marks. Thus, regardless of absolute data quality, there may exist valuable learning signal in the pairwise contrast between chosen and rejected responses, which preference tuning can leverage. We will now develop this intuition into our central hypothesis.
3 The Delta Learning Hypothesis
We hypothesize that training on paired responses Yer Vr) enables learning from the relative quality difference-the delta- Ye and yr. Even if both responses ye, yr have low absolute quality compared to the model we aim to improve, as long as ye is better than y, along some informative axes, the model can learn from this delta and improve.
Formally, let μxx,y) y) be the utility of a response y to some prompt x. In practice, h may repre- sent human preference, or simply some arbitrary function we wish to optimize. Suppose we wish to improve model M. The delta learning hypothesis posits that there exist natural preference pairs (x, yc.),, where u(x,,y) ye) > pxx, such that two conditions hold:
1. Low absolute utility: The utility x(x, ye) of the chosen response V/ is no higher than the current capability of model M, and therefore supervised finetuning on (x) explicitly hurts the model or at best does not help.
2. Extrapolated gain: Preference tuning on the pair improves model M beyond μ(x,
We now present experiments with language models in controlled settings where we explic- itly manipulate μ and construct responses yr of varying utility to test the delta learning hypothesis; find consistent evidence in support. Later in Section 6, we theoretically study delta learning in logistic regression to better understand why delta learning can work.
3.1 Controlled Experiment: Stylistic Delta in Number of Bold Sections
We start with a toy setting where we explicitly define μ(x,y) y) to be "the number of Markdown-denoted bold section headers in y" (e.g., **xammle header**), measurable and
3

---

Published as a conference paper at COLM 2025
Model/Algorithe
Chosen Res.
Rejected Res.
Section A
# Sections Generated
LLAMA-3.2-30-INST. (Baseline)
5.9
+ SFT
9 sections
24.6 (+ 18.7)
+ SFT
3 sections
4.4 (-5.)
SFT
2 sections
2.9 (- 3.0)
DPO
3 sections
2 sections
+1
81.1 75.2)
DPO
2 sections
3 sections
-1
1.1 (- 4.8)
+ DPO
3 sections
sections
6.1 0.2)
Table 2: We train Llama-3.2-3B-Instruu with DPO on preference pairs with responses containing a varying number of bold sections, and compare to SFT on the chosen response directly. When responses contain fewer sections than the model's baseline (i.e., < 5.9 sections), SFT decreases the number of sections generated. In contrast, preference tuning can leverage the delta between responses, increasing the number of sections generated even when each response is individually suboptimal.
controllable metric. Our hypothesis predicts that we tune M on preference pairs (x,yt, yr, Vr) where y/ contains, say, 3 sections and y,, contains 2, then M should learn to produce more sections- even though sections (the "better" response) is fewer than M's current capability, and hence would hurt when used as SFT data. As shown below, this is indeed observed.
Vy,,.. Setup. We build dataset of prompts x matched with responses 1/2, ....,,,, Vk,, containing varying numbers k; of bolded sections (details in Appendix G.3). We then tune Llama-3.2- 3B-Instruct with DPO on preference pairs (x, yk, Mxx)) formed by selecting a response Чλ, with more sections (k; k,) as chosen. To isolate potential confounding effects associated with preference tuning, we also consider two control settings: (1) reversing the preference pairs (k; < kt) and (2) tuning with responses containing an equal number of sections (k; = kj). We compare against SFT on the chosen responses VA,- See Appendix G.3 for hyperparameter details. We evaluate by measuring the average number of bolded sections generated before and after training in response to set of held-out test prompts.
Results. Results in Table strongly support our hypothesis: SFT only helps when the training responses are higher quality than the model's baseline "capability" (i.e., response contains more sections than what the model generates), and otherwise hurts. In contrast, even when responses are individually weak according to our defined jt, pairing them together with a positive delta massively boosts section generation, extrapolating beyond the number of sections contained in the chosen response (see Figure A1-the model learns to make nearly every single word a new section header!). Our negative controls (preference tuning with negative delta zero delta) do not yield gains; the positive delta is thus critical.
3.2 Controlled Experiment: Semantic Delta from a Weaker Model
We test whether our hypothesis extends beyond a one-dimensional style feature to general semantic quality by tuning a model M to prefer its own self-generated greedy responses (which, by construction, do not exceed its capability) over responses from weaker (smaller) model. Please see Appendix D.3 for results and analysis that further support our hypothesis.
4 Post-training Language Models with Delta Learning
Having validated our hypothesis in controlled settings, we now test its applicability in realistic, large-scale setting: 8B LLM post-training. Current open recipes extensively rely on strong LLMs (e-g., GPT-4o) to generate preference data with high-quality chosen responses to learn from (Lambert et al., 2024; Ivison et al., 2023). However, the delta learning hypothesis suggests that preference tuning can be effective even when chosen responses are not high quality, provided we can construct meaningful delta to weaker rejected response. Pushing this idea to its logical extreme, we propose a simple preference tuning setup that explicitly eliminates the use of any strong LLMs (i.e., larger than 3B parameters) for either response generation or preference annotation from an existing state-of-the-an synthetic preference data recipe. Surprisingly, we find our simplifying changes incur little
4

---

Published as a conference paper at COLM 2025
performance trade-off, enabling significantly cheaper and more accessible post-training recipe that reduces the reliance of open recipes on strong model distillation.
Generate Responses
LLM Quality Judge
Preference Data
SOTA Open
Recipe (Tülu
Strong Model Pool (22 models)
Prompt
DPO
8.1 708
Chosen:
Best Quality Response
Owan 25 728 Instruct
GPT-4o
Rejected: Lower Qualiy Response
Prompts
GPT-4o
Gemma
Cenerate Responses
Preference Data
Tülu 8B SFT
Weak Model Pool (2 models)
Quality Heuristic (Model Size)
Prompt
2.5
Small Model
Our Simple Recipe
2.5 1.5B Instruct
Rejected:
Smaller Model Response
We simplify the Tülu preference data recipe (top half). Instead of using G.T--4o judge to pick the best response from many strong models as chosen, our recipe (bottom half) uses single small model generate all chosen responses, relying on the implicit delta to an even smaller (and thus weaker) model's responses to drive downstream learning.
Figure 2:
4.1 A State-of-the-Art Existing Setup: The Tülu 3 Recipe
To contextualize our simplifications, we first detail Tülu 3 (Lambert et al., 2024), the current
ffateoof-theare recipe in open-source post-training. Tülu 3 comprises series of 8B and 70B language models post-trained on top of the Llama 3 base models, achieving performance that matches exceeds equivalently-sizee proprietary models. Hence, adopt the Tülu 3 8B preference tuning recipe (Figure 2, top half) as an ideal starting point for our exploration.
Tülu 3 preference data is constructed starting from 271k diverse prompts. Responses are generated using strong modern LLMs (e.g., -lamm-3.1-700-Instii Instruct (Yang al., 2024a), etc.). A frontier LLM (GITT-4) then scores these responses; preference pairs are formed by selecting the highest-scoring response as chosen and lower- scoring one as rejected. This data is then used DPO tune an intermediate instruction- finetuned model (Tülu-3-SB-SFTT) yielding the preference tuned model Tülu-3-8B-DPP.
Qwen-2.5-72B-
Besides the substantial cost of GPT-4o annotation (~$10,000 USD), the Tülu preference tuning recipe fundamentally assumes access to supervision sources stronger than the model being trained (i.e., an 8B model), both for (1) generating high-quality responses (i.e., using 70B models) and (2) annotating response quality (using GPT-4o). As we demonstrate below, this assumption can be entirely eliminated.
4.2 Our Simple Recipe: Constructing Preference Data without Strong Supervision
Our recipe, illustrated in Figure 2 (bottom half), simplifies the Tülu 3 preference tuning recipe while keeping the starting model checkpoint (Tülu-3-SB-SFT) and initial prompts fixed to isolate our changes. We intervene by removing all use of strong models:
Chosen response generation. Starting from the same set of prompts as the Tülu 3 dataset, we generate all chosen responses with single small model (e.g., Qwen 2.5 3B Instruct) that is near or below the capability of Tülu-3-8D-SFT (as measured by downstream evaluations, see below). With a 3B chosen model, this change reduces the FLOPs needed for data generation by over an order of magnitude (~66 of the original).
Forming preference pairs. We eliminate GPT-4o quality annotations entirely. Drawing from our findings in Section 3.2, simply use model size proxy for quality. We pair every chosen response with response from the next-smallest model in the same model family (e.g., pair Qwen 2.5 3B Instruct with Qwen 2.5 1.5B Instruct). While this heuristic is noisy-the smaller model might occasionally generate better responses -uu previous controlled experiments show that learning can still occur with such noisy semantic deltas.
With our simplified pipeline, we construct three preference datasets, generating chosen responses with either (1) Qwen-2.5-3B-B Instruct (paired with (wwen-2.511.B--Instt (2)
5

---

Published as a conference paper at COLM 2025
Model/Preference Data
MMLU
PopQA
MATH
CSM
AE2
IFEval
BBH
DROP
TQA
HEval HEval+
Avg.
LLAMA-3.2-18 B.--BBBB.-1B-B-- I-ISTUUTT
46.1
13.9
21.1
44.4
8.8
54.5
40.2
32.2
40.0
64.8
60.0
38.7
1LAMA--.2-3B-ISSIIE
62.9
19.4
39.6
75.7
18.7
76.5
61.6
48.5
50.6
79.7
76.8
55.5
46.2
10.1
27.2
39.2
3.3
28.8
32.2
25.3
45.4
60.5
58.9
34.3
WWEN-2.5-1.5B-INSTTE
59.7
15.4
41.6
66.2
7.2
44.2
45.9
14.1
46.5
83.0
79.8
45.8
QWEN--.5-3B-INSTTUCT
69.5
15.7
63.1
77.7
17.8
64.0
57.6
31.5
57.2
90.5
87.4
57.5
TÜLU-3-8B-SFT
66.1
29.6
31.2
76.0
12.2
71.3
69.2
61.2
46.8
86.2
79.8
37.2
Lllama 3.2.38 over
30.3
40.9
81.5
24.9
75.0
70.0
60.7
54.2
84.7
81.1
61.1
Qwen 2.5 1.5B over 0.58
67.4
29.9
39,9
79,8
15.8
72.5
70.8
61.8
52.1
83.7
78.1
59.3
QQwnn 2.5 3B over 1.58
69.4
31.7
42.6
83.4
36
78.6
69.4 4
62.0
57.7
84.4
81.7
63.4
Tülu 3 Preference Dataset
69.8
30.3
42.6
84.2
32.8
80.4
69.2
62.3
56.1
84.7
80.8
63.0
We train Tulu-3-8B-SHF with DPO on preference data constructed with our simple recipe, which pairs outputs from a weak model (chosen response) with outputs from an even weaker model (rejected). Strikingly, our best setup matches the original Tülu 3 preference data, which requires vastly stronger supervision (e-g., from GPT-4o). We generate our data using models that are near or below Tülu-3-8B-SFT in average performance (top half).
Table 3:
(paired with Llama- 3.2-1B-Instruct). We choose these exact models because the original pool of data-generating models in Tülu 3's preference data recipe explicitly p includes larger models from both the Owen 2.5 and Llama 3 model families, while excluding these small models. Hence, the chosen responses in the original Tülu 3 preference data of significantly higher absolute quality compared to our setup (i.e., 4.44/5 absolute quality points as judged by GPT-4o, versus 3.98/5 5 in our setup with -wennn25--B-Innttuuct generating chosen responses). Our setup thus relies more on the delta between chosen and rejected responses to drive learning, See Appendix for qualitative examples of these deltas, as well as additional statistics of our datasets (e.g., response length, vocabulary diversity, etc.).
D.ren-25-1.5B-Innttree (paired with 0.5B), (3) Llama-3.2
4.3 Matching Tülu with Delta Learning
We preference tune Tülu-3-8B-SFT on our weak preference datasets using DPO, tuning hyperparameters following Lambert et al. (2024) (see Appendix G.5). We evaluate our models on all benchmarks from Section 2 as well as three additional benchmarks measuring model capabilities for consistency with Tülu 3's evaluations (see Appendix B). We show further results on six safety evaluations in Appendix C. To compare our simple preference data against the Tülu 3 preference data, we evaluate the official Tülu-3-8B-DPO model.
Our main results in Table 3 reveal a striking finding: tuning with our simple weak prefer- ence data recipe matches the original Tülu 3 recipe in performance, achieving a +0.4 point average gain over the Tülu 3 preference data when using Qwen-2.5-3B Instruct to generate chosen responses. Even though Tülu preference data uses strong model supervision to synthesize chosen responses of significantly higher absolute quality, the quality delta be- tween chosen and rejected responses in our weak pairs still suffices to produce comparable gains it is possible to learn a surprising amount from the quality delta alone.
Consistent with our hypothesis, we observe gains when tuning with any of our three weak datasets. For instance, tuning with chosen responses from
11.4 points worse than Tülu-3-8B-SFT on average- still yields a +2.1 point gain in average performance. The delta learning phenomenon also holds on the level of individual tasks; for example, tuning on 1.maa.3.----Insttte
chosen responses boosts GSM8K accuracy by 5.5 points, despite Llama being weaker than Tülu on GSM8K. Finally, our preference data recipe yields gains when using either Llama or Qwen models to generate preference data, suggesting that it is not reliant on idiosyncrasies of specific model family.
5 Analysis
We identify four factors in our simple recipe that may impact preference tuning perfor- mance and study each: (1) the magnitude of the quality delta between chosen and rejected responses, (2) the absolute quality of chosen responses, (3) our model size-based reward heuristic, and (4) our choice of Tulu-3-8B-SFT as the base model to tune from. We use the

---

Published as a conference paper at COLM 2025
Average Perf. (11 Evals)
(7,
(72, 1.5)
62
[3, 1.5)
(32, 1.5)
(3, 0.5)
(14,
(72, 0.5)
(7.3)
(32.3) (72,
(14, 1.5)
(7. 0.5)
(32. 0.5)
60
(14, 3)
Model Size Preference Dataset Tülu 3 Preference Dataset
(14,
(32, 71 (72.
(1.5, 0.5)
Tülu BB SFT
[32, 14)
56
(72, 14)
(72, 32)
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Chosen-Rejected Response Quality Delta (GPT-4o Score)
The size of the quality delta between chosen and rejected responses a strong predictor of downstream preference tuning performance. Performance improves as the delta increases, up to approximately A = 0.55, beyond which gains plateau. Each dot in the plot represents a model preference-turee 3 data or data generated by pairing two Qwen 2.5 Instruct models following our simple recipe. Numbers in parentheses indicate the parameter counts (B) of the paired models.
Figure 3:
from Tülu-3-8B-SFT, using either the original Tülu
same 11-benchmark evaluation as in Section 4. Unless noted otherwise, all models are tuned from Tulu-3-8B-SFT with DPO. We defer full training details to Appendix G.6.
5.1 How does the magnitude of the chosen-rejected quality delta affect learning?
Using our simple recipe, we construct 21 preference tuning datasets containing chosen and rejected responses with varying absolute quality (and thus varying deltas). Starting with Tülu 3 prompts, we synthesize preference pairs with all 21 possible model pairs from the Qwen 2.5 Instruct family, which we select for its wide variety of model sizes (0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B). Following our recipe, we select the larger model's response as chosen (e.g., 72B over 7B). To quantify the absolute quality of these responses, we apply the GPT-4o annotation method from Lambert et al. (2024) to score 10k responses from each model on 1-5 scale. We plot average performance after preference tuning against the average pairwise delta between chosen and rejected responses in Figure Our results suggest that:
The magnitude of the delta strongly predicts downstream preference tuning performance, with a minimal delta required for optimal gains. We observe a strong positive correlation between the magnitude of the delta and downstream performance, up until approximately s 0.55, after which performance peaks and plateaus. Beyond this threshold, further increases in the delta do not yield additional downstream gains. Interestingly, the Tülu 3 preference dataset falls in line with this same trend, despite being generated via significantly different recipe. This alignment provides some evidence for why our simple recipe can match Tülu 3 in performance: both the Tülu 3 dataset and our Qwen-2.5-3B- Instruct-generated weak data exhibit deltas that are above the saturation threshold, after which nearly all datasets perform comparably.
An outlier to this trend is the dataset with 1reen-25-1.5---nsstuut---ee sponses. The gain over Tülu-3-8B-SFT from tuning on this data is positive, but smaller than what its delta size alone would predict. We conjecture that this is because Qwen 1.5B is the only chosen model considered that is substantially weaker than the Tülu-3-8B-SFT base.
chosen re-
Not all positive deltas drive learning. Tuning on preference datasets generated by pairing the 72B Qwen model with 32B or 14B-sized models hurts performance-thess are the only two points that fall below the Tülu-3-8B-SFT dashed line in Figure 3. We observe that the log-likelihooss of both the chosen and rejected responses decrease during DPO training on these datasets (an occasional phenomenon DPO training, see Razin et al. (2024)). We hypothesize that this effect may be particularly harmful when both chosen and rejected responses are much stronger than the base model being tuned; the model is optimized to downweight good behaviors. It remains open question to what properties of the
7

---

Published as a conference paper at COLM 2025
Model/Prelerence Data
MMLU
PopQA
MATH
GSM
AE2
IFEval
BBH
DROP
TQA
HEval
HEval+
Avg.
TCLU-3-8B-S3
66.1
29.6
31.2
76.0
12.2
71.3
69.2
61.2
46.8
86.2
79.8
57.2
Model size heuristic
69.4
31.7
42.6
83.4
36.1
78.6
69.4
62.0
57.7
84.4
81.7
63.4
CPT-4o judge reward
69.9
31.5
40.6
83.9
29.9
79.5
66.5
61.2
62.4
85.7
80.8
62.9
OLMO-2-7B-SFT
61.4
23.6
25.3
73.5
66.5
49.3
59.6
48.6
70.0
63.8
50.0
Qwen 2.5 over 1.5B
62.9
23.6
30.0
80.6
71.5
59.3
56.3
55.0
Oo. Preference Dataset
61.9
23.5
30.3
83.1
27.7
72.3
50.9
60.2
56.0
70.7
66.2
54.8
Table 4:
Top half. We ablate our use of model size as a heuristic to label preference pairs in our simple recipe, and find that it is good proxy for GPT-4o judge reward. Bottom half. We ablate our choice of tuning from Tül----8B-SFT. We find that our setup generalizes to OLMo-2-7B-SFT, matching the original OLMo 2 preference recipe's performance.
delta-beyone magnitude-ant are necessary to drive effective learning, and how these factors interact with the choice of preference tuning algorithm.
5.2 How does the chosen response's absolute quality affect learning?
We group the 21 preference datasets described above (Section 5.1) according to the strength of the Qwen 2.5 model used to generate the chosen response (i.e., 1.5B, 3B, 7B, 14B, 32B, or 72B parameters). For each group, we selected the dataset that yields the highest downstream performance after preference tuning as a best-case measure for the performance achievable when tuning on chosen responses generated by model of given strength. For comparison, we also evaluated supervised finetuning directly on the responses generated from each of the Qwen models. We defer results and full analysis to Appendix D.4. In summary, we find that SFT only yields gains when tuning responses from Qwen models stronger than our base Tulu---8B-SFI model, with gains scaling monotonically as the data-gent eeta-generating model's strength increases. In contrast, preference tuning chosen responses of any quality yielded strong gains even even when generated by models weaker than or equal to our base model.
5.3 Ablations
We ablate our choice of (1) using model size as preference heuristic (as opposed to annotating preferences with an LLM judge) and (2) tuning from Tül.-3-SB-SFT. We report results in Table 4 and highlight key takeaways, deferring full experimental details and discussion to Appendix D.5. First, our simple recipe generalizes well to tuning OLMo-2- 7B-SFT, matching the performance of the Olmo 2 Preference Data (constructed similarly to the Tülu 3 data). Second, the model size heuristic is surprisingly accurate proxy for GPT-4o preferences, with an 80.5% agreement rate. Finally, learning from the delta between weak responses can succeed independently of the specific preference signal used. See Appendix D.6 for a further ablation on our choice of preference tuning algorithm; we find that delta learning also succeeds with SimPO (Meng al., 2024).
Delta Learning in Logistic Regression, Provably
We have shown empirical evidence of our hypothesis (Section 3) and utilized it for perfor- mant language model post-training (Section 4). Now, we study why delta learning works. We study intercept-free binary logistic regression where a student is preference-tuned to prefer pseudo-labels from one teacher over pseudo-labels from weaker one. We prove that this teacher strength gap alone can guarantee that the student improves with high probability, even when both teachers are weaker than the student.
6.1 Problem Setup and Notation
Suppose inputs drawn x~~ Ia) and that labels y* E {0,1) are as- signed via some ground-truth unit-norm parameter #a Rd y 1((0", x) 0}. Because the input distribution is an isotropic Gaussian, the population 0-1 loss incurred by any model
Preliminaries.
0 E Rd depends only on its angle with 0*: Lon == Pr Prz-)(0.1)) [sgn (0.x) x) # sgn (0", x)]
8

---

Published as a conference paper at COLM 2025
Figure 4: We train student model 8g to prefer pseudo-labels ye from a (possibly weak) teacher model 9r over pseudo-labels yr from a weaker teacher 8,. Individually, the gradient updates Ve log pe(ve|x;;0) Ve log Pe(y. (x; (dashed arrows) induced by the weak labels can be harm- ful in expectation and steer the student away from the ground-truth parameters 6*, In con- trast, the delta learning gradient updates fol- low their difference vector Vo log po (ye x;; 0b) V log (e(y)1x;;)) fo), which is positively aligned with 0 whenever le outperforms 0,. Pairing weak-and-weaker teachers thus yields a learn- ing signal that can improve a stronger student.
E V p fn)
E EV po
Io
A- L-Laanning Update
0*
Thus, the classification accuracy- or strength any model @ is proportional to its cosine similarity with 0", denoted cos(0,0*).
arccos (0.0")
10l
Delta learning training. Fix a student model 0g Rd and fix two teachers Oz, fy. Write No, &c, Ry to denote the cosine similarity of each model with 0*, respectively. Assume that ae > ür, so that teacher 0. is stronger than 0, in expectation over the data. We have no assumption on the strength ao of student 8g. For any input x, we assign chosen and rejected pseudo-labels as y/ = (0c,, x) 2 0),y, 1 x),, 2 0}, forming preference pair (x, yr, yr)
) )1 and poly= 1|x) = ((0,x)) we update student to with SGD on a naive preference loss: 0,41 + 0, y Σー V Vrerrer (x(1), ye yr (0), ;0;), (x, ye, Vr;
annotated as y/c > yr- Given a batch of B preference pairs {xx(1, ye yr
(log (ycl log (o (x)) The loss pre can be seen unnormalized SimPO-style loss (Meng et al., 2024); drop the normalization for theoretical simplicity.
6.2 Delta Learning Succeeds with High Probability
Our central claim is that in sufficiently high dimensions, delta learning in logistic regression works with high probability. At high level, we show that given any student model fl and under mild conditions, most pairs of teacher models le, 0, exhibiting a performance delta (i.e., 0g has higher accuracy than 0,) suffice to generate preference data that will improve the student model, even beyond the performance of each teacher.
Intuitively, preference tuning pushes the student's parameters towards the stronger teacher 6e and away from the weaker teacher 6,. Since 9g is (by assumption) better aligned with the ideal parameters 6* than 6,, the difference vector 0. 6, is itself positively aligned with 0", regardless of how low the absolute alignment of 9. may be. In other words, the delta between the two teachers yields directionally correct signal, even when both teachers are individually weak (Figure 4). As long as this useful signal is not swamped out by other spurious signals arising from the teachers' errors, tuning will improve the student. In particular, the spurious signals are most problematic when they align with and amplify the student's existing errors. Fortunately, the teachers' errors are essentially orthogonal to the student's errors in high dimensions, so such amplification rarely happens. This high-dimension effect creates a training length "sweet spot," where the student can improve from the useful signal without overfitting to the teachers' errors. We formalize this below.
Theorem 6.1 (Delta Learning for Logistic Regression). Fix a failure probability 5 E (0, 1), and let 0 Rd be an arbitrary student model whose initial cosine similarity with the (unobserved) ground-trus truth model 8* is ag := cos cos(fp, 8*) < 1, 6 + 6*. We train 6g with delta learning following the setup of Section 6.1: given two teacher models tc, 0r satisfying
cos(O,, 0*) &cr cos(er, ar,
ac Uri
9

---

Published as a conference paper at COLM 2025
update to to prefer teacher 0e's pseudo-labels over the weaker teacher 0,'s via SGD on the naioe preference loss prof.. Then if the teachers and student satisfy
(ac-ar) ag) ao (Proi (ro
O,
(C1)
useful signal from delta
sparioos noise arthogonal to 0"
VA = (0e/ (10ell2) - (0,/ (1r/1)),
B 1h: 0// |100|12
(1)
and the ambient dimension exceeds threshold of d 2 In [Kx Vall total steps with batch size (d) and learning rate Lj where
(6) || [3]] training for
1 ② xx |00ll2 min (1/Vd, x/ |1al)))
T= [x / (41 ||=
(2)
yields (with probability at least 1 - 8) student iterale Oy satisfying
cos(0 ) > cos(B), 8t) + O
(3)
Hence, the trained model 0r incurs strictly smaller population 0-1 loss than the initial student Bg-
Note that the right-hand side of Condition C1 can be made small regardless of the teachers' performance level; learning can succeed even when the initial student already outperforms both teachers, #g > 0g > ay, In fact, most teacher pairs satisfy Condition C1 in high dimensions, yielding our main result:
Corollary 6.2 (Informal). In high dimensions, most pairs of teacher models with performance gap suffice to improve student Gg via della learning. Randomly sample two teacher models 6., O, uniformly from all unit vectors, conditional on their cosine similarities de, Rr with 6", dc > ay. Then so long as the dimension d exceeds mild threshold, with high probability Condition CI holds and by Theorem 6.1 training strictly improves On with high probability.
We defer all formal proofs and further discussion to Appendix F.
7 Related Work
Learning from preference feedback. Early preference tuning used reinforcement learning from human feedback (Ziegler et al., 2019; Ouyang et al., 2022; Bai et al., 2022), which involves training a reward model on human-annotated rankings of model outputs that is then optimized against with algorithms like PPO (Schulman et al., 2017). Recent work has simplified this approach by (1) removing the reward model in favor of direct policy updates (Rafailov et al., 2024; Meng et al., 2024; Ethayarajh et al., 2024) and (2) replacing hu- man annotations with strong LLM judges (Cui et al., 2023; Lee et al., 2023). While the source of supervision has evolved, its use remains largely unchanged: modern approaches (Dubey et al., 2024; Lambert et al., 2024) still often rely on strong judges to select high-quality responses. We show that learning can succeed even when the chosen response is weak.
Weak-to-strong generalization. As models advance, the field exploring ways to supervise them beyond human capability (Burns et al., 2023). Prior work has focused on eliciting behavior from base models (Hase et al., 2024; Burns et al., 2023) or enabling models to iteratively improve their own training data (Yang et al., 2024b; Wu et al., 2024). We show alternative approach: leveraging relative differences in weak data to guide generalization.
8 Conclusion
In this work, we have shown that models can learn surprisingly well from the delta between paired weak data points. We further characterized key factors of paired data that drive learning, such as the delta's magnitude and the chosen response's absolute quality. We find that not all deltas are equally useful: some fail to drive gains, and gains saturate as chosen response quality improves. Natural questions arise: what makes delta informative? How can we effectively scale basta-based learning? And to what extent are these dynamics dependent on the specific task or tuning algorithm? We leave exploration to future work.
10

---

Published as a conference paper at COLM 2025
9 Reproducibility
To ensure clean reproducibility, we provide extensive details on our training codebase and setup (Appendix G.1), our evaluation codebase and setup (Appendix B), our compute usage (Appendix H), and experiment-speeciic details for all experiments in our study (Appendix G), such as training hyperparameters and details in dataset creation. We encourage the reader to review the referenced sections.
Acknowledgements
We thank Marco Tulio Ribeiro for for insights on an earlier version the work. We further thank (alphabetically) Victoria Graf, Stella Li, Rulin Shao, Rui Xin, and Zhiyuan Zeng for useful discussions and proofreading. We further thank Stella and Victoria for their artistic expertise in iterating supported by the NSF GRFP. This work was supported by the Singapore National Research Foundation and the National AT Group in the Singapore Ministry of Digital Development and Information under the AI Visiting Professorship Programme (award number AIVP- 2024-001), the A12050 program Schmidt Sciences, and the Google ML and Systems Junior Faculty Award.
our visualizations. We thank Ananya Harsh Jha for solidarity. SG is
References
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint ar arX::22406. 11704, 2024.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxan- dra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large language model with state-of-the-art performance. 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. ar preprint arXiv:2204.05862, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilè Lukošiate, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. Xiv preprint ar Xiv:2211.0035222 2022.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold As- chenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to- strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.093990 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained code. ar preprint ar Xiv:210: 033774 2021.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL https:///mssy.org///loo//222 wunaa
11

---

Published as a conference paper at COLM 2025
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichrr Nakano, et al. Training verifiers to solve math word problems. arviv preprint ar Xit:22100114163 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high- quality feedback. 2023.
Tri Dao. FlashAttention-2:
Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. ar Xiv preprint ar Xiv:2305. 14233, 2023.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. ar preprint ar #rXiv:2304.06777 2023.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. Xivev preprint arXiv:1903.00161 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. Xiv preprint or Xiv: 217273 2024.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039-300 300099,-0099,.00 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: simple way to debias automatic evaluators. arXiv preprint ar Xixx:2444.04475 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint ar arXio:2402.01366 2024.
Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. Olmes: A standard for language model evaluations. arXiv preprint
arXiv:2406.08446. 2024.
Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of lims. arXiv preprint ar arXio:2406.18490 2024.
Peter Hase, Mohit Bansal, Peter Clark, and Sarah Wiegreffe. The unreasonable effectiveness of easy training data for hard tasks. arxiv preprint ar Xrxx:2401.00751. 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Xiv preprint ar arXii:2009.0333 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arviv preprint arXiv:2103.00877. 2021.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Open- rlhf: An easy-to-use scalable and high-performance rlhf framework. ar arviv preprint ar Xri::24005.1114 11144, 2024.
12

---

Published as a conference paper at COLM 2025
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Jang, David Wadden, Noah Smith, Iz Beltagy, et al. Camels in changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint arXio:2311.10702, 2023.
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lam- bert, Noah A. Smith, Yeji Choi, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback, 2024.
Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, et al. Wildteaming at scale: From in-the-wild jailbreaks (adversarially) safer language models. Advances in Neural Information Processing Systems, 37:47094-47165, 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. 2024.
Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. Annals of statistics, pp. 1302-1338, 2000.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-Im: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 37:14200-14282, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. Xiv preprint arXiv:2109.07958 2021.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang, Is your code gener- ated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558-21572. 2023.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. Xrv preprint ar arXiv:2212.10511 2022.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, al. Harmbench: standardized evaluation framework for automated red teaming and robust refusal. ar Xiv preprint ar arXiv:2402.04249, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-122225, 2024.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXit:2501.0055 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3ystems,.5:27730- 11277730-27744 2022.
13

---

Published as a conference paper at COLM 2025
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811-30849 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System
optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 3505-3506, 2020.
Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference opti- mization. arXiv preprint ar 0rXiv:4410008847, 2024.
Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arv preprint ar Xiv:2333 002663, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithmss. Xiv preprint ar 1rXic:1707.06347 2017.
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. " doo anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communica- fions Security, pp. 1671-1685, 2024.
Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large language models. Xiv preprint ar 0ttt::2401.05661,, 3, 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arv preprint arXi::220.092611 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://git htw.ggtthuu....m wow.//ttst-lab//teeeeeee wtt.aoor.alpacaaa......a 2023.
MosaicML NLP Team. Introducing mpt-30b: Raising the bar for open-source foundation models, 2023. URL www.mosaicm...cm//llooooeeoo
Accessed: 2023-06-22.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumyy Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arviv preprint vrXiv:2307.092883 2023.
Joel A Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics, 11:89-4344 2012.
Lewis Tunstall, Nathan Lambert, Nazneen Rajani, Edward Beeching, Teven Le Scao, Leandro von Werra, Sheon Han, Philipp Schmid, and Alexander Rush. Creating a coding assistant with starcoder. Hugging Face Blog, 2023. https:/ https://huggingface.com /loog htt/:/logg//aaaa galog//tarchat.. starchat.
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason
Weston, and Sainbayar Sukhbaatar, Meta-rewarding language models: Self-improving alignment with aam-as-a-meta-jun
Xiv preprint ar Xiv:24407 19590, 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. ar preprint arXtto2304.12224, 2023.
14

---

Published as a conference paper at COLM 2025
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXio preprint mrXiv:2412.15115 2024a.
Yuqing Yang, Yan Ma, and Pengfei Liu. Weak-to-strong reasoning. arXio preprint a/Xin:2407.13647, 2024b.
Jihan Yao, Wenxuan Ding, Shangbin Feng, Lucy Lu Wang, and Yulia Tsvetkov. Varying shades of wrong: Aligning llms with wrong answers only. arXiv preprint ar arXiv:2410.11055, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follovee evaluation for large language models. ar preprint (rXi:22311.07911 2023.
Wenhong Zhu, Zhiwei He, Xiaofeng Wang, Pengfei Liu, and Rui Wang. Weak-to-strong preference optimization: Stealing reward from weak aligned model. arXiv preprint WrXiv:2410.186400 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. ar Xiv preprint ar Kiv:1909.08593, 2019.
A Limitations
As empirical study with limited compute, our results are largely based on in-depth analysis over a single preference tuning algorithm (DPO) and few base models (Llama Tülu OLMo 2). Moreover, our evaluation does not capture all potential downstream model behaviors. For example, we do not evaluate multilingual capabilities or domain- specific use cases such as scientific writing. As such, extending our findings (1) other preference tuning algorithms, (2) larger model scales and different base models, and (3) new tasks are all interesting directions for future work.
B Evaluation Details
Unless otherwise noted, evaluate all models the following core set of 8 standard bench- marks. We provide the skill that each benchmark measures as well as abbreviations used in parenthesis. Following Tülu 3 (Lambert et al., 2024), we evaluate with the OLMES (Gu et al., 2024) implementation of these benchmarks, with the exact same evaluation configurations (e.g., for prompts, metrics, few-shot examples, etc.) for all benchmarks. We defer readers to the above references for further details.
MMLU (knowledge recall) (Hendrycks et al., 2020)
MATH (mathematical reasoning) (Hendrycks et al., 2021)
GSM8k (GSM; mathematical reasoning) (Cobbe et al., 2021)
IvEal (instruction following) (Zhou al., 2023)
AlpacaEval 2 (AEval2, AE2; instruction following) (Dubois et al., 2024)
TruthfulQA (TruthQA, TQA; truthfulness) (Lin et al., 2021)
BigBenchHard (BBH; general reasoning) (Suzgun et al., 2022)
Codex HumanEval+ (HEval+; coding) (Liu et al., 2023)
For our post-training experiments (Section 4 and Section 5), we extend our evaluation suite to include three additional benchmarks to maintain evaluation consistency with the full suite from (Lambert et al., 2024). The added benchmarks are:
PopQA (knowledge recall) (Mallen et al., 2022)
DROP (general reasoning) (Dua et al., 2019)
Codex HumanEval (HEval; coding) (Chen et al., 2021)
15

---

Published as a conference paper at COLM 2025
Unsafe Prompt Refusal
Jailbreak Resistance
Aggngate Scores
Model Preference Data
XSTest
HarmB. WildGuard
DAN
JaitTrig. Wildjail. Avg. Refusal
Avg. Jailbreak Avg. All
LLAMA-3.2-1B- B-INSTRUCT
81.1
65.0
78.1
57.0
74.5
61.8
74.7
74.4
74.6
11AMA-3-2-3B-B INSTRCCTT
OWEN-2.5-0050- -58-INSTRUCT
72.2
70.3
72.5
64.7
84.8
50.9
71.7
66.8
69.2
QWFN-2.5-1.5B-INSTTH
79.8
77.7
2WEN-225530-IISTTUUT
87.8
91.2
83.2
49.0
67.8
56.0
87.4
57.6
72.5
TULU-3-8B-SFT
90.7
98.8
99.2
87.7
96.0
86.6
96.2
90.1
93.2
Lllmma 38 over 1B
93.1
97.2
99.2
73.0
88.0
85.1
96.5
62.0
89.3
"wwn 1.5B over 0.5B
90.9
98.1
99.3
84.7
94.8
88.0
96.1
89.2
92.6
Qwen 38 over 1.5B
93.6
96.2
99.1
62.3
83.0
78.8
96.3
74.7
85.5
Tulu 3 Preference Data
92.9
95.3
98.5
68.7
87..2
81.3
95.6
79.1
87.3
Table A1: We evaluate the safety of the models post-trained with our weak preference data from Tülu-3-8B-SFT (Section 4) on six benchmarks measuring (a) whether the models refuse unsafe requests or (b) whether the models are robust to jailbreaking prompts. On all benchmarks, higher score is better. The safety performance of the models we used to generate data are shown in the top half of the table.
C Additional Safety Evaluations for Post-trained Models
Following (Lambert et al., 2024), we further evaluated the models from our main post- training experiments (Section 4) on six safety benchmarks measuring either (a) whether models refuse to respond to unsafe requests or (b) whether models are robust to jailbreaking prompts. We list the benchmarks below, with skill measured in parenthesis:
XSTest (refusal) (Röttger et al., 2023)
HarmBench (refusal) (Mazeika et al., 2024)
WildGuard (refusal) (Han et al., 2024)
Do-Anything-Now (abbreviated DAN; jailbreaking resistance) (Shen et al., 2024)
JailbreakTriggee (jailbreaking resistance) (Sun et al., 2024)
WildJailbreakTest (jailbreaking resistance) (Jiang et al., 2024)
We show results of evaluation in Table A1. Overall, both strongly-upprrvsed Tülu preference data and delta learning with our weak preference data tend to slightly hurt average safety compared to the base SFT model. Thus, conjecture that these drops are due to characteristics of the Tülu 3 prompt distribution (Lambert et al., 2024) shared between these data rather than an inherent limitation of delta learning. Intuitively, we speculate that if the prompts do not expose useful deltas, then one cannot expect gains. Nonetheless, models trained with delta learning degrade less, and hence are generally more safe than Tülu-3-8B-DPO. An exception is the model trained with 2.5-en2.553B-neeee over 1.5B responses. While it correctly refuses more often than Tülu-3-8B- jailbreak. We conjecture that this is because Qwen 3B itself is easier to jailbreak than Qwen 1.5B (Table A1; see also (Biderman et al., 2023; OLMo et al., 2024) which suggest that model size sometimes inversely correlate with safety), and hence the delta between Qwen 3B and 1.5B in a negative direction. How to curate prompts and deltas that effectively improve safety remains exciting open question.
it is easier to
D Extended Results and Discussions
D.1 Pilot Study on U.TRAFEBDEACK-WEAN
See Table A2. Results consistent with findings in Section 2.
D.2 Controlled Experiment: Stylistic Delta in Number Bold Sections
Figure A1 shows examples of model generations before and after DPO training.
16

---

Published as a conference paper at COLM 2025
Model/Training
MMLU
MATH
GSM
ABval2
IFEval
BBH
TruthQA
HEval+
Avg.
LLAMA-3.2-3B-IN
62.9
39.6
75.7
18.7
76.5
61.6
50.6
76.8
57.8
+ UE-WEAK SFT
61.8
34.3
73.2
12.3
68.0
60.7
46.5
75.4
54.0
+ UFF-WEAK DPO
64.0
42.2
76.4
22.4
76.2
61.3
53.6
76.0
59.0
LLAMA-3.1-BB-INTT.
71.8
43.0
83.7
24.9
78.2
72.7
55.1
81.6
63.9
UF-WEAK SFT
65.7
34.6
77.9
8.9
59.9
71.8
49.1
80.8
56.1
+ UF-WEAK DPO
72.0
83.9
26.3
81.1
72.3
56.2
80.4
64.5
Table A2: We tune Llama 3 Instruct models on the ULTRAPEEDEACK-WEII preference dataset, generated by models weaker than Llama 3. Training with preference learning (DPO) to prefer "weak responses" over "weaker responses" yields gains, while SFT directly the weak preferred responses hurts. Blue indicates gain over base model, orange degradation.
D.3 Controlled Experiment: Semantic Delta from a Weaker Model
To test whether our hypothesis extends beyond one-dimensional style feature to general semantic quality, we study the delta between self-generated outputs and outputs from weaker model. Specifically, suppose we wish to improve model M. Given set of prompts {x}, we can use M to greedy decode responses уM = M(x). By construction, the quality Mxx).. Consequently, we would not expect SFT on (x, yM) to improve M's overall performance. In contrast, our hypothesis predicts that creating quality delta by pairing self-generated responses yM with semantically weaker responses Ym may provide sufficient signal for preference tuning to improve M. A simple way to obtain such weaker responses is use a smaller (weaker) model m from the same model family as M and greedy decode Ym m m(x.
of these responses exactly match M's capability, y(x,yM)
While similar in spirit to our pilot experiment (Section 2), this setup guarantees by construc- tion that M never observes any single responee of higher quality than it can currently produce. Our experiment also studies whether we can learn from noisy delta, as even though xxx, M(x)) > [xx, mxx) on average, some responses yn from the smaller model may surpass corresponding responses yM from the larger model on individual prompts.
response
Setup. We randomly sample 50k prompts from the Tülu 3 SFT dataset (Lambert et al., 2024). We greedy decode chosen responses !M with Llama-3. Llama-3.1-8B-Insiie Insttruutttt (the model we later train) and rejected responses ym with Llama-3.2 3-3...---B-Innttrcttt We tune Llama-3.1 1.mma-3.1-8B. 1-8-- Instruct with DPO to prefer its own responses YM over outputs y/m from its smaller 3B sibling. We compare to SFT on FM- As negative control, we also preference tune to prefer ym over VM- See Appendix G.4 for training details. We use the same evaluations from Section 2.
Results. Results in Table A3 further support our hypothesis. SFT on self-generated greedy responses reduces average performance by 1.2 points, possibly due to overfitting on these outputs at the expense of broader ability. In contrast, pairing self- eelf-generated responses with weaker responses creates a positive delta that drives learning beyond the baseline model's performance. This approach yields small but consistent gains on nearly all benchmarks, with a 0.4-point gain on average. Our negative control-flippigg the preference order of self- generated and weaker cispones--llminne (-0.7 points). Thus, the improvement comes specifically from the positive delta created by pairing with weaker responses, rather than general effects of preference tuning.
these gains and worsens overall performance
D.4 Analysis: How much does Chosen Response Quality Matter?
We group the 21 preference datasets described in Section 5.1 according to the strength of the Qwen 2.5 model used to generate the chosen response (i.e., 1.5B, 3B, 7B, 14B, 32B, or 72B parameters). For each group, we selected the dataset that yields the highest downstream performance after preference tuning as a best-case measure for the performance achievable when tuning chosen responses generated by model of given strength. For comparison, we also evaluated supervised finetuning directly on the responses generated from each of the Qwen models. Results in Figure A2 show that preference tuning generally outperformed
17

---

Published as a conference paper at COLM 2025
Model/Training Selup
MMLU
MATH GSM
ABval2
IFEval
BBH
TQA
HEval+
Avg.
-LAMA-3.2-3B-INSTTEE (Weaker)
62.9
75.7
18.7
76.5
61.6
50.6
76.8
57.8
LLAMA-3. LLAMA-3.1-8B-INSSS 1-8B-INSTRUCT (Baseline)
71.8
43.0
83.7
24.9
78.2
72.7
55.1
81.6
63.9
+ SFT (sell-generated responses)
72.2
42.2
82.9
24.3
76.3
72.1
53.4
78.0
62.7
+ DPO (sell-generated over weaker)
72.0
43.5
84.2
25.7
80.0
71.4
55.6
82.2
64.3
+ DPO (weaker over self-generated)
70.9
42.3
83.4
22.9
78.6
72.1
54.6
80.5
63.2
Table A3: We train Llama-3.1 1-..-88---snnsueee using greedy responses generated by itself and by its weaker sibling, definition, equal the model's current capability-does not yield gains. In contrast, training with DPO prefer self-generated responses over weaker ones can exploit the delta between them and yield consistent improvement, Reversing the preference order hurts performance; the positive delta is critical, not generic effects of preference tuning.
SFT on self-generated responses-which, by
SFT; preference tuning with 3B chosen responses yielded higher gains than SFT on even 72B responses'. Moreover, we observe distinct qualitative trends for each tuning approach:
10.0
Figure A2: Impact of chosen data qual- ity: We plot the performance of the model used to generate chosen responses (x- axis) against performance change after tuning on those responses (y-axis). SFT only yields gains when training sponses from models stronger than the base model tune from; gains scale with chosen quality. DPO yields gains even when the chosen responses come from models no stronger than the base (green box); increasing chosen response quality gives quickly diminishing returns. Each dot represents a model tuned on chosen responses generated by Qwen 2.5 Inst. model (parameter size labeled).
Average Pert. Change Evals)
7.5
7B
728
5.0
7 28
32B
74..
38
-5.0
775
Preference Tuning (DPO)
Supervised Finetuning
1.58
-10.0
-10
-5
10
15
20
Chosen Data Quality (Chosen Base Model Perf.)
Supervised finetuning: performance scales with chosen quality. Performance gains (above y=0) only occur when tuning on responses from Qwen models stronger than our base Tülu- 3-8B-SFT model (to the right of x=0). Gains scale monotonically with the chosen responses' absolute quality, increasing as the data-generating model's strength increases.
Preference tuning: performance is less dependent on chosen quality, but saturates. Tuning on chosen responses of any quality yielded gains gains-even even when generated by models weaker than equal to our base model (green box in figure). Still, chosen quality matters; tuning on chosen responses from Qwen 1.5B (much weaker than our base model) yields smaller gains than using stronger responses. However, once response quality reaches that of the tuned model (x=0), further improvements in chosen quality do not significantly improve performance. This saturation effect further explains why our weak preference data matches Tülu 3, despite using chosen responses with lower absolute quality.
D.5 Extended Ablation Studies Analysis
Model size preference heuristic. We ablate our choice of using model size as a preference heuristic as opposed to annotating preferences with LLM judge. Using Tülu 3's GPT-4o judge method, we re-labeled our best-performing weak preference dataset (responses from Qwen 2.5 3B Instruct paired with 1.5B responses). We find that the model size heuristic is surprisingly accurate proxy for GPT-4o preferences. GPT-4o agrees with our model
Tilu-3-8BBSFT has alredy been finetuned with even stronger supervision (e.g., human-written or GPT-4o responses), potentially diminishing the benefits of additional SFT.
18

---

Published as a conference paper at COLM 2025
size heuristic on 80.5% of the data; for comparison, GPT-4's agreement rate with human annotators has been estimated at around 65% (Dubois et al., 2023). Preference tuning with either GPT-4o preference labels or model size heuristic labels yielded comparable performance (Table 4, top half), except on AlpacaEval 2 (see discussion in next paragraph). Overall, our findings (1) validate our approach of using model size to ensure chosen- rejected quality delta, and (2) show that learning from the delta between weak responses can succeed independently of the specific preference signal used.
AlpacaEval 2 discrepancy. The observed low performance on AlpacaEval 2 when using GPT-4o as reward signal (Table 4) is likely because of the length-correetton applied by the AlpacaEval 2 benchmark. LLM judges are known to have bias towards preferring longer responses (Dubois al., 2024); this bias is likely present when re-annotating our responses using the GPT-4o judge from Tülu 3. Hence, DPO training on preferences annotated by GPT-4o may increase the average generation length of the model, which is then penalized by the length-correction term that AlpacaEval 2 uses when computing winrate. Empirically, the model trained with GPT-4o preferences generates outputs that are around 200 characters longer average compared to the model trained on the model size heuristic reward in response to the AlpacaEval 2 test prompts.
Choice of Base Model. To test the generality of our preference tuning recipe, we use it to -. tune a different base model, OLMo-2-7B-SFT. Using prompts from the OLMo 2 Preference cataset--cnssruutter in nearly identical manner to the Tülu data-we generate chosen and rejected responses using Qwen 2.5 3B Instruct and 1.5B, respectively. We choose this pair as it was the best configuration from Section 4. We tune OLMo--27B-SFT on both our simple data and the original OLMo 2 data. We show results in Table 4 (bottom half). Consistent with our earlier comparison against Tülu 3, our simple recipe matches the OLMo 2 preference data (+0.2 points average), which utilizes significantly stronger supervision.
D.6 Additional Ablation Study: Preference Tuning Algorithm
We further ablate our choice of using DPO as the preference tuning algorithm in our main post-training experiments (Section 4) by instead tuning with SimPO (Meng et al., 2024) while keeping data and base model fixed. Specifically, we use SimPO to tune Tülu-3-8B-SFT on our best weak preference data (Qwen-2.5-3B-Instf largely follow the same hyperparameters as our other analysis experiments (Appendix G.6); we additionally grid-sweep the following hyperparmaters:
responses paired with 1.5B responses). We
Dataset size: (100000,200000)
Learning rate: (5e-8, 7e-8, 1e-7,
SimPO (,,7), roughly following the ranges tried in (Meng et al., 2024): ((10, 3.0), (5, 1.5), (2.5, 1.25), (2, 1.0))
Results of training with SimPO are reported in Table A4. Overall, using SimPO to tune weak data also yields strong gains, with 5.2 point gain in average performance over the base SFT model. Consistent with (Lambert et al., 2024), the gains with SimPO are slightly less than with DPO (-1 point on average).
Model/Preference Data
MMLU PopQA MATH CSM
AE2
1HEval
BBH
DROP
TQA
Héval HEval++
Avg.
TÜLU-3-8B-SFT
66.1
29.6
31.2
76.0
12.2
71.3
69.2
61.2
46.8
86.2
79,8
57.2
+ DPO (Qwen 3B over 1.58)
69.4
31.7
42.6
83.4
36.1
78.6
69.4
62.0
57.7
84.4
81.7
63.4
SimPO (Qwen 3B over 1.5B)
67.8
31.1
40.6
81.8
28.4
77.8
70.6
61.4
57.9
86.0
82.6
62.4
Table A4: Top half. We ablate our use of DPO as our preference tuning algorithm, and find that tuning with SimPO can also yield gains with weak preference data.
E
Weak Preference Dataset Details
Here, we provide generation details and statistics for the weak preference datasets used in our main post-training experiments (Section 4). To better illustrate what deltas exist between
19

---

Published as a conference paper at COLM 2025
the chosen and rejected responses, also provide qualitative examples of preference pairs from our best performing weak dataset (Qwen-2.5-9--nnstrau over 1.5B responses).
E.1 Dataset Creation Details
We find that the original Tülu dataset contains approximately 6000 duplicated prompts (but not duplicated preference pairs, as Tülu 3 uses large pool of models to generate responses and hence can form multiple distinct pairs for each prompt). Because our setup uses the same model to generate chosen responses for every prompt, repeated prompts, leaving 264806 remaining preference pairs for each our weak datasets. These are the seed prompts are used for our simple recipe in Section 4.
de-duplicated the
E.2 Qualitative Examples
To better understand the delta between weak and weaker responses, manually inspected pairs from our best post-training preference data (i.e., chosen responses generated by Qwen- 2.5-3B-Instruct and rejected responses by 1.ven-2.51.55-Innttrur
Overall, there were universal axes along which the Qwen 3B responses were better than the Qwen 1.5B, but we did observe several interesting deltas. We summarize them here, and showcase qualitative examples of these differences in Figure A3, Figure A4, and Figure A5.
1. On prompts with verifiable answers (e.g. math, code), we find pairs where Qwen 3B responds correctly but Qwen 1.5B does not (Figure A3).
2. On knowledge-seeking prompts, we find pairs where Qwen 3B responds with more detail (Figure A4).
3. On prompts that admit brief answers, we find pairs where Qwen 3B generates chain-of-thought, while Qwen 1.5B responds with just the answer (Figure A5).
Note that these deltas are not exhaustive; we simply highlight a few here as interesting examples to motivate future work. In particular, we believe it would be exciting to further characterize what semantic deltas exist in preference data and how the deltas translate to downstream model behavior after training.
E.3 Quantitative Statistics
We computed the following statistics for each weak dataset used in our core post-training ex- periments (Table 3) using the Tülu-3-8B-SFT tokenizer (i.e. the base SFT model's tokenizer):
Average token length of chosen and rejected responses
Vocabulary diversity, measured as the number of unique 1-gram and 2-gram tokens in chosen and rejected responses
Cosine similarity between chosen and rejected response embeddings, computed using OpenAl's text-embede rext-embedding-1 3 API
As a reference point, we also report these statistics for the original Tülu 3 preference dataset. Results are shown in Table A5. Overall, we observe that the chosen responses in our weak preference data are generally (a) shorter and (b) more diverse in vocabulary than the paired rejected responses. In contrast, chosen and rejected responses in the Tülu 3 data exhibit largely similar statistics. We hypothesize that the longer length rejected responses in our data may be due to degenerate outputs from the small weak models; they sometimes repeat tokens until reaching the maximum generation length. The cosine similarity between chosen and rejected responses is relatively small for all datasets; qualitatively, chosen and rejected responses often differ significantly on surface semantic level, which may contribute to this low embedding similarity.
20

---

Published as a conference paper at COLM 2025
Chosen Responses
Rejected Responses
Preference Dataset
Avg. Len.
Uniq. I-gram
Uniq. 2-gram
Avg. Len.
Uniq. 1-gram
Uniq. 2-gram
Cosine Sim.
Llama 3.2 3B over 1B
779.3
112,553
6,858,616
1,041.3
109,090
5,884,941
0.117
Qwen 1.5B over 0.5B
776.5
110,380
5,107,761
1,317.8
106,900
4,626,119
0.112
Qwen 2.5 3B over 1.58
709.1
114,183
6,637,312
776.5
110,380
5,107,761
0.115
Tūlu Preference Data
443.9
119,269
10,980,913
441.3
119,320
10,8663352
0.117
Table A5: Statistics of the chosen and rejected responses in our weak preference datasets, with statistics of responses from the Tülu 3 preference dataset shown for referenc
Delta Learning in Logistic Regression
In this section, we give the full proofs of Theorem 6.1 and Corollary 6.2 along with all intermediate propositions and lemmas.
E1 Additional Preliminaries
We begin by stating additional useful results that we take as preliminaries.
Proposition F.1 (Population Gradient for Naive Preference Loss). Take covariates x ~ ( Ia) and assign pseudo-labele yc, Yr using two teacher models 0z 0, Id via the following rule:
yr 1 ((),,)) 0}, yr = 1 1 (0,,, 2 0}.
Then in expectation over the covariate distribution, we have
0, oe (x, ye. V2T 11eell2
0g
1|0/112
Cov (VeLpnf (x, yo. (c.y))))))) 3 la-
Proof. The naive preference loss Spref is (xre( (x, yer Yri f) = - [log po (yelx) log Po (y]]xx)),
poly= 1/x) = x((,x)))
For any single fixed preference pair (x, yr),,, the gradient with respect to
V VeLnaive yc, yri
[V log po (yc(x)) Vo (a x)]
[Aye o ((, x)) (yr- ( x)) )x]
(ye - (/)-)r)xx
Taking expectation of the gradient over the covariate distribution. .N (0, L2),
TE (x, E lyc x] lyr x]
[[ (0., x) 2 0} x] E[(1((0.,)) 2 03 x0]]]]] x1]
Since the function (xx) (( 03 is weakly differentiable, by Stein's Lemma we have
E [1( 20 -x] = V 2 0}}
E [[0-5((0,x)) 0 [((((())))
0
22
1001
where 5 denotes the Dirac delta function. The final equality holds since Z == (6, x) ~
(O [[0]2), and hence E [0]0((0,))) x (0( 1/ (1
We now bound the covariance Cov Cov(VeLpree Cov((ye - vr)x).. Let D denote the set where 0, and 0, disagree on decision
D fx: sgn (0c, + sgn (0.,x))
21

---

Published as a conference paper at COLM 2025
Then since (ye- yr)x x for all D and 0 otherwise, we have
(VoCrre ( orept))) | By( xxx] = E Exxx D E[xx"] = 11 Cov(Veepre) HE( erel (Velpttt) E V * pret) L prett) T 3 l4
as E. pree IB| pref! is positive semi-definite. The equality [xxx²] 17 holds from
the assumption that our data drawn from isotropic Gaussian.
Lemma F.2 (Vector Bemnseiin-Freddman) Let (F) k20
be filtration and let
{Yr k 0, 0,1,
be -aluud martingale adapted it. Denole the difference sequence
X₂ = Y2 Y 1,
and assume that the difference sequence wniformly bounded:
|Xxl12 S R almost surely for every k 2 1
At any finite horizon define the predictable quadratic variation as
# Σ
Ex-1
Let Sn denote the partial sum of differences up to the horizon, Sn =EK Xx. Then for every 2
-12/2
Pr []|S2112 2 S (d 1) exp 0 + Rt/3"
Equivalently, for any failure probability 5 E (0, 1), then with probability at least 1 -
d d+1
2R
d+1
|| < V 20 In
Proof. This is restatement of Theorem 1.6 from Tropp (2012) specialized to d x 1 vectors in
R To get the equivalent high probability bound statement, simply set Pr Sz]1 2 f] and solve for !.
F2 Full Proof of Theorem 6.1
We build up to the final proof via a series of propositions.
Proposition E3. Take a, b Rd and take 0* to be unit vector in Rd. Define the function
= (a + bt, 0*)
f(t) = cos(a + bl,0")
(4)
a|a + btll
Then the derivative at I 0 salisfies the following identity:
f' (0) ||al|2 = (Proj, e*)
(5)
This can also be expressed as
f(()) ||a||2 = (b,0*) (1 (a, 222
(a,0")
(Projj
(a)) (a), (b)))
(6)
||a||2
||al[2
Hence, (0) is equinalent to either of the above expressions being positive.
22

---

Published as a conference paper at COLM 2025
Proof. This proceeds with direct calculation. Write e(t) + bt. By quotient rule, the derivative of fis
(8(t),0") (()) dt [((()))
|(((0)2 (b,0") (E(1), (E(1),b)
(((())l
Evaluating at = gives
|a| (b,
07) (b,
(a, 0*) (a, (a,
(0)
|al12
Now write the projection of onto the hyperplane perpendicular to a as
Proja- (b) = b (a, b) ||a 2
so then
(a, b) (a, 0%) = f'()) ||a||2
(Proj,,(b), (b),
||al|
yielding the first identity. We further decompose a, b into their components parallel to and orthogonal to f* as
(a, 0 0000000000000000 + (ro) (a),
b == 06, + -Proj (b).
Simplifying (a, b) with this decomposition, we have
(a, b) = (a, (b, 0*) + (e) (a), (ro (b)).
Substituting this into the expression for ( (Proj, (b), yields that f(( ||al|2 (Proj, (b),
(b, 0*) (1 (a, 0x22
(a,0")
(a),
||al12
||al|
yielding the second identity.
Proposition F.4. Take f as defined in Proposition
F3, f() == cos(a + bt, 0"). Then for all
2
||b]|
(1)
v3 ||a btll
Proof. This bound can be shown by some careful algebra. We rewrite f in terms of two intermediary functions defined as
r(t) [1(())22
u(t) = ((1)r(t),,
so that f = (u(t), 8*). Then since 6* is a fixed unit vector we have
(1) (u" (u"(1), (1),0) = (f"(1) (0)l2
So it suffices to bound (f). With some calculus, we compute u' n'(1) br(t) b ((1)
r(t)3
Differentiating again with quotient rule,
(#()) r(t)3 (2b(1(t), b(I(t), ((f b)) - ((t))()(( (())()))))))))))()))))))))))))))
r(1)6
-2((((),b)r(t) (())(),)))) + ())(
r(1)6
b. -2((())bb)
+ +((t). (-
(b,b)
3(I(1),b2
r(t)3
r(t)3
r(t)5
23

---

Published as a conference paper at COLM 2025
To simplify further, decompose b into constituent parts parallel to and perpendicular to ((t):
(1(1),b)
b Projec)) (b)
I(f),
b_ b b- b--
r(t)2
Then since b llj + b we can reduce u² ut) as
(f)) = ((t)) b)
r(t)5 + bb -2(1(1),b)
(b,
b(1(1),b2 r(t)3
+ 1(f)
r(t)3
r(t)5
2(1()),b)
1(t). (b,b) (1(1),))2/r(1)
=b1 r(1)3
1(()
r(t)3
So now we've expressed u" in terms of two orthogonal components, bj T ((t). As such, the squared norms of the components add:
(11b13 ((()),b)(/<()))}
(1)11 611(t),6)211b11 + r(t)6
(1((t))1 =(t)²)
r(1)$
Now let (t)) denote the angle between and [(t), and observe that c(t) = cos( = ((()),) b
Then we have
4c(1)2 11612 2 (1 c(1)3) ||b12 + (19b1 c(t)2 [1b113) 2
||u" (0112
r(1)#
r(t)+
|bb||
(cc()))(1 - -(())) + +(1 (1- c())))]
r(t)
||b|12
+ |3cc(1)++c<(()] 2c(1)2+1]
r(t)#
By construction, for any & have ctt) 1,,1), One can check that
(F)()) =1n")>) (0
| r(t)
[-3x" + 2x2 +1] - r(f)* (4/3).
|bbl
sup xE[-1,1]
Taking square roots of both sides gives the desired bound.
Proposition F.5. Assuiee the delta learning setup of Section 6.1 and Theorem 6.1, and suppose we train with the population update rule
0 1111
02 - 17 Ex-N((L ( yr: 0i)
(8)
Then if Condition C1 holds so that x > O (as defined in Theorem 6.1), training with learning rate for total update horizon of
x 110
H H² nT
4 |
yields a final iterate Or whose cosine alignment with the ideal parameters 0* improves over the alignment of the initial student Gg by a margin of at least I:
I x/50 (xx),
(10)
cos(0r: (0, > cos (on, 0*) +T.
(11)
Proof. As discussed in our sketch in Section 6.2, by Proposition F.1 training with population gradients amounts to tracing various points along a parametric ray é: R20 + IR4,
E(A) 00 + Au
UA 0./ ||0:ll2 0,/ |10/112-
(12)
24

---

Published as a conference paper at COLM 2025
Define to be map measuring the student's performance over the course of training:
(e(a),0")
f(A) = cos() [[e())]l2
(13)
Specializing Proposition F.3 to our setup, we have that f000) = (Projøj (va),0*)
(14)
(ac )r)(1 ag) ao (Proj (0p/ (100l1)),Prro(eee) (DA))
(15)
So (0) - < <> 0, and (0) > 0 is sufficient condition for training with population
updates to yield an improvement. This is exactly Condition C1 from Theorem 6.1. Assuming this holds, we can quantify the magnitude of the gain after training for some I steps. By second-order Taylor expansion of f, the total improvement can be bounded as
== cos 0*) - cos(le, 0*) = f(//T) f(0) 2 n)T - - (1
(16)
where L is bound on the second derivative [A)) derived in Proposition F.::
2
|
L sup Ac(0,,yT]
1
(17)
v3 1100 + A
To simplify L further, observe that |x| s 2 2///2all2
|x| < (us, 00) - ag) + (ro(Pro (0o/ (0d/|10bll2). Prool)),PPro(pe (vA))
(18)
- ||Aal2 + Proj (0 (ex)][
S 2 2
(19)
So then we have for all A [0, T] that
2 (2 L/llll ll 1100112
100 + AAull2 2 |10oll2 A ||alll 2 |10ol12 -
||
1100112
(20)
4 ||
8
| 12
(21)
v3
110012
Now, note that (0) = x/ |0ll2 (Equation 15), and the total training horizon as in the proposition statement:
| 2
||0pll
H = -T= (0)
(22)
4
4 |a
Combining this training horizon with Equation 16 and Equation 21, we have 1/T (V))))
r (0)
(0)
f())
/3
2
2
x²
V3L
2 3L2
= V3 3 L
8 V3
3
||
(23)
Simplifying with ||vall2 = |00ll2 + |10,112 = 2 yields the desired claim.
Proposition F.6. Assume the delta learning setup of Section 6.1 and Theorem 6.1. Then the student
iterates 6, trained with empirical mini-batch SGD do not deviate too much from the exact iterates 0 trained with population gradients (Equation 8). Formally, for any failure probability 8 (0, the ambient dimension satisfies d In(2BT/8) 8), then with probability at least
2dT
d+1
d+1 =nō =nō (yO(V/T/++V))
|0 0rl12 5n
In
8/2 + 14
(24)
8/2
25

---

Published as a conference paper at COLM 2025
Proof. This follows via martingale concentration argument. Write the empirical mini-batch gradient at each timestep as
1 B
(6.)) V pre((( yyee.) yr (
gt
(25)
B
and define the error vector
3 = 8 - Elgl
(26)
where E[g+] = Va up to a constant scaling factor absorbed into n, Each & is a random
variable (over the draws of the mini-batch) that measures how much the empirical gradient g: differs from the population gradient. Introducing & allows us to rewrite the empirical
SGD updates terms of the exact iterates 0y:
0++1
78: (0) - 17 Elgr)) n/: 8,41 1751-
(27)
Unrolling these updates across T steps,
0T = θT 11 Σ Se
(= deterministic "backbone" + stochastic deviation).
(28)
We now bound the stochastic deviation E G1. By construction, the sequence {5 {r} martingale difference sequence with E[Gr] = 0. Hence, their cumulative sum can be bounded via vector Frenstein-FFredman inequality (Lemma F.2). We verify the necessary assumptions for Lemma F22 by bounding ||Erl12 and Σ EE 112:122. To bound ||Gr|1,, observe that x(I.)) (0, Ia), then
[x()))) follows chi-squared distribution with d degrees of 2
freedom. Standard tail bounds due to Laurent & Massart (2000) give that
Pr (11=10001
2 4va) S 4d
(29)
By a union bound, the event that any covariate observed throughout training exceeds this bound occurs with probability at most 81 = TB exp(-4d). So if d exceeds the stated threshold, up to a 6, = 5/2 failure probability we can condition the rest of our argument on the "good" event
ع 8= (1
< 4vd VIAI
(30)
Applying triangle inequality then gives
1 B
| < || + ||aal22 -
Σ
+ 1|llal22 < 4vd+ ||all2 S 6vd.
(31)
B
2
The final inequality holds because ||0^|12 S 10:112 + 110/112 2. We next bound the cumula- tive second moments E IE ||211 Observe that
1 Cov (Vo (2 30 (2))
B
1 Ia. 1
Cov(3r) Cov(g:)
(Proposition F.1)
B2
B
(32)
so then the second moment bounded as
E
[|]
21 <
d
dT Σ E [13
(33)
B
Then by Lemma F2, for any 82 E (0,1) we have with probability at least 1 - &2
2
d+1
d+1
Σ 7
< V
In
+
(34)
B
82
82
2
Combining this with Equation 28 and setting 82 5/2 yields that the desired claim.
26

---

Published as a conference paper at COLM 2025
We are now ready to put everything together.
Proof of Theorem 6.1. We want to show that the early-stopped T-th student iterate Or achieves higher performance than the initial student 0o,
cos( 8*) - cos(Ob, (b, 0*) O(x²) 0.
Using triangle inequality, we break this down as cos(Or, 8*) - cos( 8*) 2 cos((r,, 8*) cos (8,, 8*)
cos(@r, cos((r,00)))00oo 0*)]
(35)
deterministic ideal gain
stochastic deviation from ideal
By Proposition F.5, if Condition C1 in the theorem statement holds and we the total training horizon H* = .yT = (x || (4 || then we are guaranteed a deterministic gain in cosine similarity of least I (x²)). . To bound the stochastic error, we control [ - - (Proposition F.6) and rely on the continuity of cosine similarity. straightfor- ward gradient computation shows that for all fixed > O, the map fu(0) (0, u) |@|| is
(1/R) Liipschzs on the domain {0 R ||0||2 2 R}. In our setting, training for horizon H already guarantees that ||0 > ||0ol12 (Equation 20), so
1 |10b| l2
||07 l
|| ||rr 0r ll
||0, 0T ll
(36)
Thus, if we control the distance such that
||0, 0rl12 - r 1100l
(37)
8
then we have |00112 2 1/ ||oll2 (as 0 < T S 1) and consequently
r
cos(0r,8") - cos 8*) <
||00112 ||0,
0r
(38)
2
So then training &o with delta learning yields an improvement in cosine similarity of at least r/2 ((x2) as claimed. We end by analyzing what the training hyperparameters
n1, T, B need to be set as to control this distance. Rewriting T = H* /n and combining Proposition F.6 with Equation 37,
O 07112 $ "
1
2dH B
In (d/8) + 4Vdlnndd1) 2/2 by < r 010ll22-
(39)
We can rewrite Equation as quadratic inequality in = VT:
bs2 as 10oll S 0.
(40)
Because both a, 0, 17 = satisfies Equation 39 so long as we simultaneously have
1 r
1 r |0ll22
b <
[|00l12
and as <
16
16
Both are immediately satisfied we set y as
r T|
2 ||
1 min
(41)
16b
256a²
Plugging in the definitions of a, b, I into Equation 41 and choosing B - O(d), we have (modulo some fixed scaling constants)
x² 110l122 ||ool2
} --²) (x² |100l12 min { A//Vxx//laal (42)
" min vdin(d/5) In(d/8))((
27

---

Published as a conference paper at COLM 2025
So then the number of training steps T can be expressed as
T 6 x x>|la|l2
max { {va, (1
(43)
Finally, our earlier use of Proposition E6 requires that
d 4 In(2B7/8).
(44)
But we set = (d) and T T= o(va), so the right-hand side grows only logarithmically with d; this condition is trivially satisfied for any reasonably large d. One can check this with some algebra:
Va, |0 12 /x]
1
2d max
In (2BT/5)
-
-In(d/0)
(45)
4
4
||
2d
vā
+ ||0 /x)
( [ 1
]
-
S
3 3ln(d) + In
(46)
82
| 2
82x Fall
+
|| 2
<
d +
S d.
(47)
x2x
|| | 11
So as long as
x | 3 82x
d ス In
(48)
then the total failure probability is at most 8, and running delta learning with SGD improves 8o by (xx) as claimed.
F3
Full Proof of Corollary 6.2
The desired corrolary can be formally stated as follows:
Corollary E7 (Formalization of Corollary 6.2). In high dimensions, most pairs of teacher models with a performance gap suffice to improve the student via delta learning. Suppose we randomly sample two teacher models 0z, O, uniformly over the unit sphere, conditional on their cosine similarity with the optimal model:
0, Uniform (0 s cos(0,00) acJ, 0, Uniform {0 s |cos(0,0*) ar),
and train student 0 following the setup from Theorem 6.1. For any ò E (0,1), define the threshold
d' - f22nnn 4 (Iao) [|0oll (VI (VI- 11-a6 a? 11-1a a?
2
+1.
(49)
(ac ar) (1 - a
Then whenever > d", with probability at least 1 8 Condition C1 holds and by Theorem 6.1 training strictly improves the student with high probability.
Remark F.1. The expected improvement, on the order of (xx2), grows with the magnitude of the teachers' performance delta (a. a, but shrinks as the student's initial performance ag increases. This theoretical result aligns with our empirical results in Section 5: the quality delta between chosen and rejected responses a strong predictor of downstream preference tuning performance (albeit only up point beyond which gains saturate; language model tuning is more complex than the logistic-regression setup assumed here).
Remark F.2. The dimension threshold d' is mild. Say the initial student has unit parameters 6n and is 80% accurate. If the teachers 6,, 8, are sampled to be 70% and 60% accurate respectively, then with d > 2000, at least 90% of all such teacher pairs suffice to improve fg-
28

---

Published as a conference paper at COLM 2025
To prove Corollary H7, we rely on standard concentration bound of inner products on sphere:
Lemma E8 (Sphere-concentration Bound). Fix an arbitrary vector vo E E C R and draw
Ne uniformly and independently from the unit sphere s C CT. For some constants de, E R, set
Vc / 1/1-az a? Mer
V 11-at
a7 Up,
DA Uc Uy.
Then for any E (0, with probability at least - 5,
(vo. va>l S ||val12 (V1 aL 11-a7 a²
2
4
V
In
d
We decompose the inner product and apply Lévy's concentration on each term. We have
Proof.
(00. va) = /I (2 un) V a7 (v u,).
For either {Me, ] the map f(u) (vo, is - in the Euclidean metric. Since (( u) O, Lévy's concentration on the sphere gives, for any e > 0,
Pr(1(vn, u>} 2 c) S 2exp( (d 1)e²
21100| 2
Now set e so that the right hand side equals 8/2; explicitly,
2 Inz
E = |lol| V
By union bound, with probability at least 1 - 5 we have {I uc)l S c) and (I(Po.u)) s
e). So by a triangle inequality we have
((00, vaYl S
/ 1-1aa a² (vo, ue)l 11- (00, ur) s (V1- at v a7 E.
Substituting the specified e gives the displayed bound.
Proof of Corollary F.7. The claim is that Condition C1 holds with high probability in suf- ficiently high dimensions. Since both 0, 0, are randomly drawn and uncorrelated in all components orthogonal to 8", standard sphere concentration bounds show that the "bad" noise
(Proj (0)), Proj (vA))
(50)
vanishes in high dimensions. particular, given any fixed failure probability 8 (0, 10,1), direct application of Lemma E88 yields that with probability at least 1 - 8
(Vi a? V1 a?
2
4
(Proj (B6), (r -)()))
$ |10ol12
1
In
8
(51)
Hence Condition C1 holds with the same probability whenever
(ac- a,) -aa) az)
110ol12 / - a? v1+a7
In
(52)
R
d -
8'
Or equivalently, whenever
a |10ll2 ( (V1 - a? + v1+a7 < (a: 21) - -a/)
d-1
(53)
21n(4/8)
In particular, for any fixed 8, the right-hand side grows like Vd as d co, so Condition C1 is easily satisfied. Specifically, we simply need
|Rol |160l2
(VI (VI-a3 a7 / a?
2
> d" = =21n(4/0)
+1.
(54)
(a. )1) a3)
29

---

Published as a conference paper at COLM 2025
G Experiment Details
G.1 Shared training and hyperparameter details
For all training, we generally sweep hyperparameters near the defaults suggested by recent work (Lambert et al., 2024; Hu et al., 2024). We use a cosine annealing learning rate schedule with a warmup ratio of 0.03. We use an A optimizer with (B) = 0.9, $2 = 0.95) and no weight decay following (Ivison et al., 2024). For all DPO tuning, normalization in the loss, which Lambert et al. (2024) suggests to generally work better. We use batch size 32 for DPO tuning and batch size 256 for SFT. We train all DPO models for exactly one epoch, sweeping learning rate and DPO B. For all SFT experiments, we sweep epochs and learning rate; see each experiment subsection below for exact ranges. We use gradient checkpointing, DeepSpeed (Rasley et al., 2020), and FlashAttention2 (Dao, 2024) to improve training efficiency. We use code from the OpenRLHF Github repository (Hu al., 2024) to train all of our models.
use length-
G.2 Pilot Study on ULTRAFEEDEACK-WER
Data and filtering. The original ULTRAFEEDBACK dataset (Cui et al., 2023) is popular preference dataset constructed by prompting set of LLMs with diverse prompts and then scoring the responses using a much stronger judge model (GPT-4). For each prompt we form preference pairs (x, by selecting the highest-scoring response ye and one lower-scoring response y/-
As of March 28 2025, Llama-3.22 3.2-maa-3.--BB----. 3.2--33-Insttuccc achieves a LMSYS Chatbot Arena ELO score of 1103 and Llama-3.1 1-St--nsttuctt achieves 1176 ELO. We filter out all responses from the original ULTRAFEEDBACK dataset that were generated by models with higher ELO than 1100. This excludes GPT-4-0613 (1163 ELO), GPT-3.5-Turbe (1106 ELO), and WizardLM-70B (1106 ELO). The best remaining model Vicuna-33B (1091 ELO); see Table A6 for a full list of remaining models.
Model
Reference
Alpaca-7B
Taori et al. (2023)
Bard
https:/ http//badddgggl.cooo
Falcon-40B-Instruct
Almazrouei et al. (2023)
1lama-2-13B-Chmr
Touvron et al. (2023)
Llama-2-70B-CIn
Touvron et al. (2023)
Llama-2-7B-Chat
Touvron
(2023)
MPI-30B-Chat
Team (2023)
Pythia-12B
Biderman et al. (2023)
StarChat
Tunstall et al. (2023)
UltraI.M-13B
Ding et al. (2023)
UltraLMM-6B
Ding et al. (2023)
Vicuna-33B
Chiang et al. (2023)
WizardLM-7B
Xu et al. (2023)
WizardLM-133
Xu et al. (2023)
Table A6: Models used to generate the responses in our ULTRAFEEDBACK WEAK dataset, constructed by filtering out all responses generated by models with a LMSYS Chatbot Arena ELO score above 1100 (i.e., near Jlama-3.2-3B-Inste ELO) from the original ULTRAFEEDBACK dataset.
Evaluation. We evaluate the eight core benchmarks detailed in Appendix B: MMLU, MATH, GSMSk, IFEval, AlpacaEval 2, TruthfulQA, BigBenchHard, and Codex Hu- manEval+.
Training and hyperparameters. We follow the setup described in Appendix G.1, and further sweep learning rate in ( (1e-7, 3e-7, 5e-7, 7e-7) and & E (5, 10) for DPO training. For SFT, we sweep learning rate in (1e-5, 5e-5, 1e-6) and epochs in {1,2).
30

---

Published as a conference paper at COLM 2025
G.3 Controlled Experiment: Stylistic Delta in Number of Bold Sections
Data. Each prompt x is formed by appending "Include bolded sections in your re- sponse." to a prompt from the Tülu 3 SFT dataset (Lambert et al., 2024). To generate each response yı,, we modify the appended instruction into hard constraint: "Include exactly k bolded sections in your response." " We generate responses with Llama-3.2-3BB Instruct, and use regular expressions to guarantee correctness. We collect exactly 16384 training data points.
Hyperparameters. We follow the setup described in Appendix G.1, and further sweep learning rate in (1e-7, (e-7, 5e-7, 7 and & E (5, 10} for DPO training. For SFT, we sweep learning rate in (1e-5, 5e-5, 1e-6} and epochs in {1,2}. We select the best hyperparameters based on a held-out validation set.
G.4 Controlled Experiment: Semantic Delta from a Weaker Model
Hyperparameters. We follow the setup described in Appendix G.1, and further sweep learning rate in {3e-7, 1e-7, 5e-8, 1 and & E 10) for DPO training. For SFT, we sweep learning rate in {5e-6, 1e-6, 5e-7, le-7} and epochs in [1,2]. We select the best hyperparameters based only on GSM8K accuracy, keeping the remaining evaluations in this controlled experiment held-out.
The learning rates swept here are slightly lower than those used in prior work (Lambert et al., 2024); we found in our preliminary experiments that lower learning rates generally performed better across the board for both DPO and SFT in this setting.
Evaluation. We evaluate the eight core benchmarks detailed in Appendix B: MMLU, MATH, GSM8k, IFEval, AlpacaEval 2, TruthfulQA, BigBenchHard, and Codex Hu- manEval+.
G.5 Post-training with Weak Preference Data
Data. See Appendix E for details on dataset generation, dataset statistics, and qualitative examples.
Hyperparameters. We follow the setup described in Appendix G.1. Hyperparameter tuning is crucial for performant post-training; we carefully tune hyperparameters for each dataset independently, being careful to sweep the same number of hyperparameters for each setting. Following best practice (Lambert et al., 2024; Ivison et al., 2024), sweep DPO learning rate in (5e-7, 1e-7, 7e-8, 5e-8) and B E (5, 10) and select the best checkpoint for each dataset. We further swept dataset size in {100000, 150000, 200000, 264806), and find that training on subset of the full dataset (264806 samples) was typically slightly better. Finally, Tülu (Lambert al., 2024) finds that performance can depend on random seed initialization and hence picks the best run out of multiple seeds; we follow this practice and sweep 5 random seeds on top of our single best hyperparameter and dataset configuration. This yields the numbers for our best setup in Table 3.
Evaluation. We evaluate on all eleven benchmarks detailed in Appendix B: MMLU, PopQA, MATH, GSM8k, IFEval, AlpacaEval 2, TruthfulQA, BigBenchHard, DROP, Codex HumanEval, and Codex HumanEval+. We compare directly against the officially released Tülu-3-8B-DPG model and re-run evaluations with the exact same version of the codebase we use to evaluate our own models. We find that, despite using the same evaluation configuration and overall evaluation codebase as Tülu 3 (Appendix B), re-running with the current version slightly improves Tülu-3-8B- DPO's performance numbers compared to the numbers reported in the original paper (Lambert et al., 2024).
31

---

Published as a conference paper at COLM 2025
G.6
Analysis Experiments
Data. To construct the 21 preference datasets used in our analysis experiments on delta magnitude and chosen response quality, use 100k prompt random subset of the full Tülu 3 dataset.
Hyperparameters. We follow the setup described in Appendix G.1. For our analysis experiments on delta magnitude and chosen response quality, tune learning rate in {1e-7, 7e-8, 5e-8} and 5 E {0) when training with DPO. When training with SFT for our chosen response quality experiments, we sweep learning rate in (5e-6, le-5, 1e-6) and epochs [1,2}. For our experiments with GPT-4o annotations and OLMo---7B-SFT, we tune hyperparameters described in Appendix G.5.
Evaluation. We evaluate on all eleven benchmarks detailed in Appendix B: MMLU, PopQA, MATH, GSM8k, IFEval, AlpacaEval 2, TruthfulQA, BigBenchHard, DROP, Codex HumanEval, and Codex +vma.
H Compute Details
All models are trained on either single H100 or A100 nodes. Training an 8B language model with DPO on 100k preference pairs takes approximately 4-6 hours on one H100 node. Supervised finetuning an 8B model on 100k data points takes approximately 2-4 hours on one H100 node.
32

---

Published as a conference paper at COLM 2025
Mllma-3.2-88-nnstrue (no training baseline)
...dssuminng graph Fulerian, determine number of possible clused trails length (Le,, the number of Fulerian circuits of length 7) that the guide can choose from, if possible. (Note: A graph Eulerian an Eulerian circuit, and it must satisfy certain conditions). the tour guide decides to add additional attraction the itinerary, represented by adding with edges existing nodes, analyze how this change affects the Eulerian properties of the graph and determine the new possible number of closed trails of length 7, any exist.
node and connecting it
Include bolded sections in your response.
Assistant: ""Problem Analysis*
The problem asks for number of possible closed trails (Fulerian circuits) in graph with 50 nodes and 100 edges, and then analyzing the effect of adding a new node and edges on the Eulerian properties graph.
"Initial Graph Properties"'
(text ommitted without sections for space)
"Initial Number Eulerian Circuits"*
(text without sections ommitted for space)
*'Adding a
Edges**
(text without sections ommitted for space)
*Conclusion"
The addition new node and edges graph does not allow any Eulerian circuits of length 7, since the node introduces constraint that cannut be satisfied. Therefore, the number of possible closed trails of length 7
After DPO training (3 sections chosen response 2 sections rejected response)
User: ...assuming the graph Eulerian, determine the number of possible closed trails of length (ie., the number of Fulerian circuits of length 7) that the guide can choose from, if possible. (Note: A graph Fulerian circuit, and must satisfy certain conditions). If the tour guide decides
is Eulerian to add an additional attraction to the itinerary, represented by adding new node and connecting it with edges existing nodes, analyze how this change affects the Eulerian properties of the graph and determine the new possible number of closed trails of length 7, if any exist.
Include bolded sections in your response.
Assistant: "Initial Conditions "The graph "hhas nodes" "100 edges**, "*atisfyinn con- ditions" of Eulerian graph "with degree** of least *2*** at *tall" nodes "except" possibly "one""
"Eulerian Circuit Conditions"*
For graph have an Eulerian circuit "(i.e.,* "*an Eulerian path "Ithat" "llooss* "back" "*o"" "tth"* "*starting** ""node* "exactly" *oonce* "9," "all"" ""nodss* "mmss" "have"* "even" "degrees** "oo"" "the" "numberr* "of" "edges" "going" "into" uy "t"" "node" "must"* "be" "equull" "Ito"* ""th"" "number" "of"" "edges" "leavvng" "ii* "exactly" "once" "forr" "each" "node" "except" "the" "starting" "and "ending" "node" wy "in" "the "case" "of" "a* "closed "trai" "up"* "path" "uf "length" "greater" "than" 12n sepos..
"New Graph Conditions"
"After" "addinng "*"" "new ""ode" "and" "eedgos" "to" agg "existing" "nodes" "the" "graph" "will!* "noll""""o" "nno" "longer" "be" "Bulerian" "since" "hoot" "all" "nodes" "wil!"" "have" "even" "degrees" *any" "more" "the" "new" wode" "will whave" "degree eius and "two** "existin"" "nodes" "willts "have" "'dggee" uqn "and rthe "remaining" 146" "modes" "wilk "have" "degree* ugu "or" "mor" "than" ugg
"Conclusion"*
The "inittial* "graph" "had" "Ithe" "otential" "ffor" "Eulerian" " "circuits"* of "lengthi" wyn "but"s whhh "new "aeeww""saaae "graph" "does" "nut" "batisfy* othe" "cunditions" "for "an" "Fulerian" "graph" "any"* "mor"" "the" "number" "*of "closed" "trails" of "length" segu eight mqga s0
Figure A1: DPO training massively increases the number of sections generated by the model (from 5 to 89 in this example). Most notably, the increase extrapolates beyond the number of sections (i.e, absolute quality) of the chosen response (3 sections).
33

---

Published as a conference paper at COLM 2025
Prompt:
Dani brings two and half dozen cupeakes for her 2nd-grade class. There are students (including Dani), teacher, and teacher's aid. students called in sick that day, how many cupcakes left after Dani gives one to everyone in the class?
Nate: the correct answer
Chosen Response (Generated by Qwen-225-3BB Insssrrrttttt
First, let's determine the total number of cupcakes Dani brought. Since dozen equal 12, two and half dozen cupcakes is calculated as follows:
2.5 x 12 30
So, Dani brought 30 cupcakes.
Next, we need determine the total number people in the and teacher's 's aid, but 3 students called in sick. Therefore, the number of students present is:
day.
students, teacher,
27 37-3- 24
Including the teacher and the teacher's aid, the total number of people in the class is:
24 24+1- 26
Dani gives one cupcake to each person in the class. Therefore, the number of cupcakes given out 26. The number of cupcakes left after giving one to each person
30 26 =
So, the number of cupcakes left
Rejected Response (Generated by Qwwen=2.--..8--nssruuch
To determine how many cupcakes left after Dani gives to everyone the class, need to follow these steps:
1. Calculate the tutal number of cupeakes Dani brought. students who attended the class.
Determine
3. Calculate the total number of cupcakes given out. Subtract number of cupcakes given out from the total number cupcakes brought.
First, let's calculate the total number cupcakes Dani brought. She brought two half dozen cupcakes. Since dozen equals cupcakes, two and half dozen
2.5 12 30 cupcakes
Next, we need to determine the number of students who attended the class. There are 27 students in tutal, but 3 students called sick. Therefore, the number students who attended
27 students
Now, calculate the total number cupcakes given Since Dani gives cupcake each student, the total number of cupcakes given out is:
cupcakes
Finally, subtract the number cupcakes given out from the total number of cupcakes brought to find out how many cupcakes are left:
30 24 cupcakes
So, the number of cupcakes left after Dani gives everyone in the class
Figure A3: Qualitative example of a preference pair from our Qwen 3B over 1.5B weak preference data. On this math prompt iground-truth answer is 4 cupcakes), the Qwen 3B-generated chosen response contains a verifiably correct answer, while the Qwen 1.5B- generated rejected response is incorrect cupcakes).
34

---

Published as a conference paper at COLM 2025
Prompt:
when first settled by humans
Chosen Response (Generated by -wen-22553B-Innsite
The first humans settle in what now known as the world are believed be the Homo sapiens, who originated Africa around 300,000 years ago. However, the first modern humans to leave Africa and settle in other parts of the world are thought to have occurred around 50,000 to 70,000 years ago.
Specifically:
1. Out of Africa: The first major migration of modern humans out of Africa occurred around 50,000 70,000 years ago. This migration involved groups moving into Europe, the Middle East, and eventually Asia.
2. Europe The first known settlement in Europe by modern humans is believed to be in the Levant (modern-day Syria, Lebanon, and Israel), around 50,000 years ago.
3. Asia: The first known settlement in Asia by modern humans is believed to be in the Altai Mountains in Siberia, around 45,000 to 50,000 years ago.
Americas: The first known settlement in the Americas by modern humans is believed to be around 15,000 to 17,000 years ago, with the earliest evidence found in the Beringia region, which is now part of Alaska.
So, while first humans originated Africa, the first major settlements outside of Africa occurred relatively recently in human history, with the earliest known settlements in Europe, Asia, and the Americas occurring around 50,000 to 17,000 years ago.
Rejected Response (Generated by (wwen-2.5-1.5B-Innsrrcce
The exact date when humans first settled Earth is not known, there is no evidence of human presence on the planet before the formation of the solar system. However, scientists believe that humans originated in Africa and then spread to other parts of the world over period of millions of years, The earliest known human fossils date back to around 2. 5 million years ago, and the earliest known human settlements date back around 100, 000 years ago.
Figure A4: Qualitative example of preference pair from our Qwen 3B over 1.5B weak preference data. On this knowledge-seeking prompt, the Qwen 3B-generated chosen re- sponse contains (1) more detail and (2) more structure compared to the Qwen 1.5B-generated rejected response.
35

---

Published as a conference paper at COLM 2025
Prompt:
In this task, you're given context, question, three options, and an answer. Your task is to classify whether the given answer is correct not by providing 'Yes' or 'No', based on the context with commonsense reasoning about social situations.
Input:
Context: Kai wanted to save money so he wen to the car dealership
and haggled and bought used car.
Question: How would you describe Kai?
Options: (A) wasteful (B) extravagant (C) thrifty
Answer: C
Output:
Chosen Response (Generated by Qwen-2.5-3B- 2.cen22.53B-nnstrruch:
The given answer C (thrifty) is correct based on the context provided. Kai's actions of wanting save money and haggling to buy used car demonstrate frugality and careful financial management, which aligns with the definition of being thrifty
Rejected Response (Generated by (wwen-2.5-1.5B-Immee
Yes
Figure A5: Qualitative example of preference pair from Qwen 3B over 1.5B weak preference data. On this prompt which admits brief answer, the Qwen 3B-generated chosen response contains chain-of- offthhugght justification, while the Qwen 1.5B 1.5B-generated rejected response contains just an answer.
36


Reviews:
[{"review_id": "uTxkPwtZoL", "reviewer": "Reviewer_nk1W", "summary": "This paper introduces the \"delta learning hypothesis\", demonstrating that preference tuning using weak supervision (even when both examples are individually “weak” relative to the model’s capabilities) can drive substantial performance gains via preference tuning. The authors empirically demonstrate this in controlled experiments (e.g., synthetic bold-header generation), semi-controlled setups (e.g., preferring 8B model outputs over 3B ones), and large-scale post-training (e.g., tuning Tulu-3-8B using responses from Qwen 2.5 3B vs. 1.5B).\n\nThe authors show that tuning with preference pairs based on weak models can match or even exceed the performance of existing state-of-the-art pipelines like Tulu 3, which rely on strong supervision from GPT-4o and other large models. They explore key dimensions of the hypothesis, such as the magnitude of the delta, the absolute chosen quality, and the generalizability across base models (e.g., OLMo). The work is well-executed and positions delta-based learning as a compellingly simple and cheap alternative to high-resource preference tuning.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper clearly formulates delta learning as a novel hypothesis, challenging the assumption that absolute data quality must exceed model capability.\n2. Provides strong empirical results across baselines showing consistent performance gains using only weak models for both response generation and preference labelling.\n3. The authors have laid out well-defined ablations, i.e, the relative quality delta, absolute quality magnitude, saturation effects and other heuristic validity, providing ample insights into when and why delta learning works.\n3. Overall, the proposed recipe is more accessible, much cheaper and eliminates the reliance on GPT-4o while retaining SOTA performance.\n4. The paper is well-written, well-described and has extensive experimental details, and clear metrics. Also provides detailed appendices on training, hyperparameters.", "reasons_to_reject": "1. The gains are mostly reported on standardised LLM benchmarks. It remains unclear how delta learning performs in safety benchmarks, or bias assessments or domain-specific downstream tasks. \n2. Though I enjoyed reading the paper, and it presents strong empirical evidence for delta learning, the paper can stand out with a discussion on the mechanistic explanation of why delta learning works, beyond high-level intuition.\n3. While delta magnitude is studied, the paper leaves some gaps and open questions on what semantic properties (beyond size proxies) make a delta most informative.", "questions_to_authors": "1. How does delta learning perform on tasks outside the 11‐benchmark suite, particularly safety‐critical evaluations? Are there tasks or datasets where delta learning fails to improve or actively harms model performance?\n2. If one has limited access to expert data or strong instruction datasets, would delta learning from weak relative signals be a more scalable or reliable strategy than collecting absolute SFT examples? In other words, does delta learning act as a complement to instruction tuning or can substitute for it in some regimes?\n3. What is the performance degradation (if any) when the delta is incorrect (i.e., when the smaller model's output is actually better)?\n4. Tuning with weak data might amplify undesirable patterns in weak responses, so how do you ensure that preference tuning avoids learning from spurious or poor-quality responses (maybe) present in weaker models?", "extra_scores": {}}, {"review_id": "q0AQc6uHMw", "reviewer": "Reviewer_vyR3", "summary": "The paper validates an interesting hypothesis: can a preference dataset created with weaker models improve the performance of a better (e.g., larger) language model? The authors demonstrate that this is indeed possible as long as there is a clear quality delta between the preferences (i.e., the chosen sample needs to be clearly better than the rejected sample). First, the authors carry out a pilot study leveraging preferences from the UltraFeedback dataset, keeping only preferences generated by models that are weaker or equivalent to a Llama 3B model. They show that tuning a Llama 8B and a Llama 3B on that datasets can lead to performance clear improvements. Then, the authors carry out two controlled studies. The first one generates a toy dataset where the model is trained to prefer generation with more bolded headings. They show that the model can learn to generate outputs with substantially higher number of headings, although the preference dataset only contains 3 headings in the chosen sample and 2 headings in the rejected sample. Instead, training the model in a supervised fine-tuned manner over the chosen samples does not make the model generate more headings. In the second controlled study, they generate a preference dataset using a Llama 8B for the chosen samples and Llama 3 B for the rejected samples, and show a Llama 8B model can improve from using this data. Finally, the paper explores their hypothesis against a state-of-the-art preference tuning data recipe, which is used to generate the Tulu 8B DPO model, and leverages preferences generated from a diverse set of LLMs, which are labelled using a frontier model such as GPT-4o. The experiments demonstrate that a simpler and cheaper data recipe leveraging different smaller LLMs from the Qwen 2.5 family, can generate a DPO preference set that can yeld a better improved Tulu 8B model. Additional sensitivity and ablation analysis sheds more light into the delta learning hypothesis.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is very well written and the research is well motivated.\n- Strong experimental results over a range of datasets and language models of different sizes, which answer the main research question, a preference dataset generated from weaker models can improve a stronger LLM baseline.\n- Interesting sensitivity analysis, showing that performance improvements plateau after a certain point of a preference delta.\n- The paper presents a valuable finding. Other researchers and LLM practitioners can leverage this way of generating preference datasets from weaker models in various tasks.", "reasons_to_reject": "- Missing details on the generated preference data. It would be interesting to show some statistics on the different preferences generated by the different Qwen models (e.g., average length, vocabulary diversity, similarity across different models).\n- Missing information regarding the original preference dataset used for the Tulu 8B-DPO model. The details of that dataset may be published in their original paper, but it would be good to present them here too. Is the number of preferences in that dataset comparable to the one generated by the authors using the Qwen 2.5 models?\n- Some preliminary results such as the controlled study for the number of sections generated and a similar controlled study with the Llama 8B and the Llama 3B models (using a fewer number of datasets) have been previously published.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IELaUea2bB", "reviewer": "Reviewer_Mjit", "summary": "This paper introduces the *delta learning hypothesis*, arguing that DPO using paired weak data, where both chosen and rejected responses are generated by models weaker than the one being trained, can yield performance improvements beyond directly SFT the model on the weak data. The authors validate this hypothesis using a toy setting and present a simple and cost-effective pipeline for preference tuning. They demonstrate that models can benefit significantly from learning the *relative quality difference* (\"delta\") between responses, even when the absolute quality of each response is low.\n\nThe paper is well written and easy to follow. The proposed idea is efficient and effective. The ablation study also supports the claims well. Overall, this is a high-quality work and might have a broader impact on the CoLM’s community.", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel and impactful hypothesis: Rather than SFT the model uses the weak responses, creating supervisory signals by comparing weak and weaker responses is a more natural way to conduct weak-to-strong learning.\n- Extensive empirical verifications: The paper contains experiments on different scales and datasets. The demonstration of a toy example is also inspiring.\n- Cost efficiency: The proposed method is simple and effective, reducing the huge cost of using strong models to generate responses.", "reasons_to_reject": "N/A", "questions_to_authors": "- As mentioned by the authors, we might not expect that all responses generated by weaker models are worse than the weak model. Hence, the scope of this approach might not be so general as expected. However, as shown by the experiments, the model can still learn useful information when the delta is noisy.\n- Maybe showing how weak responses are better than weaker responses via some examples would be helpful.\n- Maybe more discussions about what scope the proposed methods would bring more benefits (e.g., reasoning, truthful QA, general alignment, etc.) would be helpful.", "extra_scores": {}}, {"review_id": "uBTmgLguZR", "reviewer": "Reviewer_wem6", "summary": "The paper proposes the ​​Delta Learning Hypothesis​​, arguing that the relative quality difference (delta) in preference data is more important than absolute quality—even weak paired data can improve model performance through preference tuning (e.g., DPO). Experiments show that training a stronger 8B model with preference data generated by weak models (Qwen-3B) and weaker models (Qwen-1.5B) matches the performance of SOTA methods relying on strong supervision. This approach reduces dependence on strong supervision, offering a low-cost, high-performance alternative for model tuning.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Reduce data cost. This method requires less human labeling.\n2. Experiments shows promising results.\n3. Paper compares the effectiveness of preference tuning under different model size deltas.", "reasons_to_reject": "1. The algorithm described in the paper sounds promising, but it remains unclear whether the experimental results on the 8B scale can be generalized to larger models. For instance, in the case of Tulu-405B, the weak models would be the 70B class. Can models of 70B paired with a even weaker model truly teach the 405B model something?\n2. In Table 4, the selected weak model Qwen-3B has better average performance than the base model Tulu-8B. Would this affect the conclusion? Is there an upper limit to the capability gap between the weak model and the base SFT model?", "questions_to_authors": "1. Can we take the difference between the model itself and a smaller model? Why taking the difference between two smaller models?\n2. Do the two models used for differencing need to be from the same series? For example, the paper uses LLama-3B over LLama-1B, or Qwen-3B over Qwen-1.5B. What would happen if we used Llama-3B over Qwen-1.5B?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes and validates the delta learning hypothesis, i.e., preference tuning with paired data created from weaker models can improve the performance of a better language model. The authors show that this is indeed possible as long as there is a clear quality delta, hence the name, between the preference pair (i.e., the chosen sample needs to be clearly better than the rejected sample). 

The authors first study the hypothesis in somewhat synthetic setups but more importantly show that this approach can generate a DPO preference set that can yield a better Tulu 8B model, compared to the original model which was trained on a  preference set that relied heavily on annotations from GPT-4o. 

Overall, reviewers highlighted the novel and interesting contributions and point out that the work is well motivated, well executed, and provides interesting analyses as well as extensive experimental results.

This paper makes an intriguing and significant contribution to language model post-training and I recommend accepting it.
Decision: Accept