SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning
Authors: Tianjian Li
Daniel Khashabi
Venue: ICML 2025 poster (2025)
Keywords: Language Models
Alignment
Preference Optimization
RLHF
Primary area: deep_learning->large_language_models
TL;DR: Combining on-policy and off-policy preference data enhances language model alignment, outperforming single-source and more complex methods.

Abstract: Aligning language models with human preferences relies on pairwise preference datasets. While some studies suggest that on-policy data consistently outperforms off-policy data for preference learning, others indicate that the advantages of on-policy data are task-dependent, highlighting the need for a systematic exploration of their interplay.

In this work, we show that on-policy and off-policy data offer complementary strengths: on-policy data is particularly effective for reasoning tasks like math and coding, while off-policy data performs better on subjective tasks such as creative writing and making personal recommendations. Guided by these findings, we introduce SimpleMix, an approach to combine the complementary strengths of on-policy and off-policy preference learning by simply mixing these two data sources. Our empirical results across diverse tasks and benchmarks demonstrate that SimpleMix substantially improves language model alignment. Specifically, SimpleMix improves upon on-policy DPO and off-policy DPO by an average of 6.03 on Alpaca Eval 2.0. Moreover, it surpasses prior approaches that are much more complex in combining on- and off-policy data, such as HyPO and DPO-Mix-P, by an average of 3.05. These findings validate the effectiveness and efficiency of SimpleMix for enhancing preference-based alignment.

SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning
Tianjian Li
Daniel Khashabi
2022), and produce responses aligned with human values Aligning language models with human pref- erences relies on pairwise preference datasets. While some studies suggest that on-policy data consistently outperforms off-policy data for pref- erence learning, others indicate that the advan- tages of on-policy data may be task-dependent, highlighting the need for systematic exploration of their interplay.
Abstract
(Ouyang et al., 2022).
The ongoing debate (Ivison et al., 2024; Tajwar et al., 2024; Tang et al., 2024; Xu et al., 2024c) in the literature that compares alignment with on-policy data (data sampled from the LM to be aligned) with off-policy data (data is sampled not from the LM be aligned). The debate arrives at mismatched conclusions, with some works reporting on- policy outperforms off- off-policy (Tajwar et al., 2024; Xu et al., 2024c) while others reporting that the gains from on-policy training are minimal, and sometimes even underperform off- policy training (Ivison et al., 2024; Ahmadian al., 2024; Lambert et al., 2024a). Existing work (Ivison al., 2024; Xu al., 2024c) that compares on- and off-policy data does not control using the same algorithm under different data sources, and does not investigate the impact of task types (Tang et al., 2024).
In this work, we show that on-policy and off- policy data offer complementary strengths in pref- erence optimization: on-policy data is particu- larly effective for reasoning tasks like math and coding, while off off-policy data performs better on open-ended tasks such as creative writing and making personal recommendations. Guided by these findings, we introduce SIMPLEEMIX an ap- proach to combine the complementary strengths of on-policy and off-policy preference learning by simply mixing these two data sources. Our empirical results across diverse tasks and bench- marks demonstrate that SIMPLEMIX substantially improves language model alignment. Specifically, SIMPLEMIX improves upon on-policy DPO and off-policy DPO by an average 6.03% on Al- paca Eval 2.0. Moreover, it outperforms prior approaches that are much more complex in com- bining and off- off-policy data, such HyPO and DPO-Mix-P, by average of 3.05%.
To address these weaknesses, we vary only the data source and break down the performances by different task types to answer the following research questions:
(Q1) Under what circumstances on- vs off-policy data offer different strengths?
(Q2) Can we leverage their complementarity for more effi- cient alignment?
With regards to (Q1), we observe that on-policy data and off polipoliy data offer complementary strengths in preference learning of LMs: On-policy data is most effective in tasks that tend have objective answers and require reasoning skills, e.g., math and coding. Off-policy data is effective in more open-ended tasks, e.g., creative writing and making personal recommendations, where is common to have multiple differing preferred responses ($3).
1. Introduction
Alignment of Language Models (LMs) has bestowed them with the ability to leam better from demonstrations (Brown et al., 2020), extrapolate from reasoning chains (Wei et al.,
To address (Q2), we propose SIMPLEMIX: to mix on- and off off-policy data for preference learning. Our approach con- sistently improves upon only using
'Center for Language and Speech Processing, Johns Hop- kins University, Baltimore, US. Correspondence to: Tianjian Li tlil04@jhu.edu
off off-policy data solely in data-constrained setups where the same amount of responses is used for each method. Furthermore, we show that SIMPLEMIX is able to outperform prior approaches that mix on- and off- off-policy data in more complex ways (Song et al., 2024b; Shi al., 2024), showcasing the effectiveness
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
Research question:
($3) On- and off-policy data are complementary
(54) SIMPLEMIX: Mixing on- and off-palicy data improves alignment
collection on- and off-policy data, what approach will lead to the best aligned model?
Exemplese
42.13
Comparison DPO Methods
coslig
35.09
Jo
20
26.41
The target policy model:
His
Length Cont rentreted Winn Rate
22.85
Kaay hoar rosls
27.47
On-policy
17:34
off poiiy DPQ ansprtiyy DPD
Wade for to 120 pages?
17.76
26
accommenstaioon
21.05
Daneni bagiaay dy Hore ranr on incinslay danse rapdder
Other frontier models off-policy oracles:
24. 29
creative writing
15
Off-policy
Data
Longth Controlled Win Rato
Dripo ky DPD Offiollcy
Simpleetii
We show that on-policy data and off-policy data are complementary: on-policy data mostly improves the model's 's performance on reasoning tasks that are objectively correct incorrect (e.g.. Math and Coding) while off oiflpoliyy data improves on sub-tasks where humans might disagree with each other (c.g., creative writing and personal recommendation) ($3). Right panel: Our proposed method SIMPLEMIX mixes on and off-policy data, outperforming solely using either on or off-policy data. (84)
Figure 1. Left panel: Our work studies data origin in preference optimization of LMs. Middle panel:
of our method despite its simplicity ($4).
Preference Optimization (PO) While SFT is a step for- ward in aligning LMs to follow human instructions, the resulting model not aligned human preferences. For example, humans often prefer structured, concise yet com- plete responses. To align TSFT human preferences, we perform Preference Optimization (PO) on top of TSFt with a pairwise preference distribution: D = {(x,, Mes 91)...). where there are two responses (Ve,, 30) for a single prompt. yo is preferred by human annotators (usually referred to as the "winning" or "chosen" response), and yı is dispre- ferred (usually referred to as the "losing" or "rejected" re- sponse). In essence, PO aims to steer TSFT towards the winning responses that are preferred by humans. Formally, the objective is defined as (Rafailov et al., 2024):
Figure illustrates our work: The complementary nature of on- and off-policy data in LM preference learning: on-policy DPO (Rafailov al., 2024) outperforms off-policy DPO in reasoning tasks, e.g., solving math problem, while under- performs in open-ended tasks, e.g.. describing an imaginary city(&3); SIMPLEMIX: Mixing on- and off-policy data in DPO can outperform DPO with a single data source ($44. SIMPLEMIX achieves an average improvement of 6.03 over on-policy and off-policy methods on Alpaca Eval 2.0 and outperforms existing hybrid approaches by 3.05 while main- taining simplicity.
To sum up, our contribution is two-fold:
(1) We show that on- and off-policy data in preference leamn ing are complementary: on-policy data excels at tasks re- quiring reasoning and objective verification, such as math and coding, whereas off-policy data is more effective for open-ended tasks, such as creative writing and personal recommendations.
To(y x)
E (( log
B BBog
TSFT (Y/w
x))
3 log To(y)1 (t)) TSFT (y1 (r) ].
(I)
(2) We demonstrate that a good balance between on- and off-policy data consistently outperforms approaches relying solely on either data source.
where a a non-decreasing function. Maximizing (1) should maximize the increase in log-likelihood of yo and the decrease of yt Ideally, the model learns to generalize characteristics that make Voe preferred. For example, if yoe are constructed that are factually correct and yı contains hal- lucinations, the hope is that by steering the policy towards few factually correct responses, the policy generalizes so that assigns high probability to all responses that are fac- tually correct. We differentiate between on- and off- toficolicy PO depending D: from which yo and y sampled.
2. Preliminaries and Notations
We consider post-training of LMs. Given a user prompt in natural language, the language model क (-12x) takes the prompt as input and outputs a probability distribu- tion over natural language responses y. The Supervised Fine- Fine-Tuning (SFT) stage of LM updates pre-trained LM Thase on dataset with (prompt, response) pairs: DSFT {(11.11 v1).... (IN, } with the following maximum like- lihood objective: TSFT E arg max, [ (3s 2z).
On-policy PO Assuming that the LM π is parameterized by B. If the responses y are sampled from a policy that is derived from either TSFT or wa, we describe y as on-
2

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
policy data: where D = Dnn {(,,y) y He or ] Notable examples of on-policy PO include Proximal Policy Optimization (PPO) (Schulman et al., 2017) that optimizes a mathematically equivalent objective as equation (1) using on-policy generations ~ To, and "on-policy DPO" (Guo et al., 2024) that uses a stronger language model as a judge to label pairs of on-policy generations and optimizes equation (I) with yw, y1" TT.
3.1. Experiment Setup
Models and Dataset We use two language models as TSFT: meta-llama-3... 1at-a-33---B--nsttre (Dubey 1attaa-----uuu-----------
et al., 2024) and (Lambert et al., 2024a). We experiment on the Ultra- Feedback dataset (Cui et al., 2024) that consists of 60k prompts. For each prompt, two responses were collected from a model pool. We treat the two responses as off ooo-py data. For each prompt, we sample N responses on-policy data. We use the best reward model S Skywork/Skyworz wewooorr-ewwwdd-eeeeeeee in RewardBench (Lambert et al., 2024b)² as a proxy of human judgment to score the pair of responses. We refer to this model as the oracle reward model. The response with the highest reward is labeled as yo and the response with the lowest reward labeled as yt- More detailed hyperparameters are reported in Appendix A.
Off-ppolicy PO Recall that on-policy data are sampled from a policy that is derived from either TSFT or no, we describe data that are not on- on-policy data as off- off-po data and PO performed on top of an off-policy dataset D = Don Usually, off-policy data are sampled from a collection of open-sourced or proprietary models (Cui et al., 2024; Wang et al., 2024; Bai et al., 2022a). A notable example of off- policy PO is Direct Preference Optimization (Rafailov et al., 2024) (DPO) which directly optimizes equation (I) with the responses sampled from an off- off-policy dataset.
-2 2-27--22--B--- 227B-v0. 2
Evaluation To test our hypothesis, we break down the prompts in Alpaca Eval 2.0 (Li et al., 2023) into different categories. A detailed breakdown of prompt category and distribution can be found in Appendix B. We select four representative tasks: two are objective tasks that can be verified: mathematical reasoning or calculation (Math) and Coding, and the other two tasks are tasks that require modeling human open-ended preference: Creative writing and making personal recommendations (Recommenda- tion). We report the length controlled win rate (Dubois et al., 2024) of cach selected task for models trained with on- and off off-policy DPO.
3. On- vs. Off-policy Data are Complementary
In this section, we vary the data source in DPO, whether D DD or D = Doff in equation I to answer (Q1). Our setup ensures a consistent setup by only varying the data source while fixing the alignment algorithm, in contrast to prior work that does not control the alignment algorithm by comparing on- vs. off-policy data.
Our Hypothesis We found that in Ivison al. (2024), PPO shows most gains on GSM8k (Cobbe et al., 2021), dataset consisting of grade-school math questions, while being similar in performance to DPO on general tasks such as MML. (Hendrycks et al., 2020). In Tang et al. (2024), the performance gap between on- and off-policy data is smallest on chatbot arena side by side (Chiang al., 2024a), which also attests to the policy's ' performance on general human queries. We thus make the following hypothesis:
3.2. Results
When does on-policy sampling help? Figure 2 shows the win rate for the 4 representative categories. We plot the detailed performance of the performance of on- and off- policy DPO on Math and Coding prompts in Alpaca Eval 2.0 at left of Figure 2, and we plot the results in Creative writing and Recommendation queries at Figure 2.
We observe that on-policy DPO helps in objective and verifiable tasks. Specifically, in our experiments of meta-llama 3. ---B-Innstrue outperforms off off-po DPO in math and coding, improv- ing the length-controlled win rate by +5.72% and +7.02% respectively. However, on open-ended tasks: creative writ- ing and making personal recommendations, on-policy DPO underperforms off-policy DPO by -2.55% and -3.29% re- spectively.
Hypothesis: On-policy data mostly helps in tasks that are objectively correct or not (e.g., math), whereas the difference between on- and off- off-policy data open-ended tasks (e.g., creative writing) is minimal.
on-policy DPO
We describe our experiment setup in $3.1 and report our results that validate our hypothesis in $3.2.
'Our definition of "on- "oonppolcy"" coincides with Lambert et al. (2024a). Note that minor differences exist between our definition of on- vs. off-policy and the traditional Reinforcement Learning literature, which treats responses (actions) that are not sampled from अe as off-policy (Song et al., 2024b; Xie al., 2024; Xiong et al., 2024).
Is the improvement about length? One might argue that possible explanation is that tasks requiring longer genera-
"Samling Parameters: temperature r 0.7, top-P p 0.9 "Best performing as of 2024-11-27
3

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
Coding
Writing Recommendation
Tulu on-policy
62:11
TJlu on-
27:14
60-
aff-policy
25
off-palicy.
Liama on-pelicy
Liama on-pdiity
35
Llama aff-policy
Liama off-polity
Winn logainst GP
37.09
25-
22.85
Win (againnt
15
15-
14.41
11:43
7.34
9.22
5:26
2,93
Math
Coding
Creative writing
Recommendation
Figure 2. Comparison of win rates (against GPT-4-turbo) across different prompt categories in Alpaca Eval 2.0 for (left) objective tasks that have a groundtruth answer and (right) open-ended tasks where humans have individual preferences. On-policy DPO improves performance in math and coding, while off-policy DPO demonstrates better performance in creative writing and making personal recommendations.
Math and Coding
Creative Writing and Recommendation
50
Uama polpyyy off-policy
Liama on-policy off-policy
Win Rate (agairst GPT GPT-4-turbo)
40-
$2.52
42.10
77
41.41
Wn Rate tagairet GPT-4-turbo)
30-
27.89
28.13
26,24
35,63
M.41
15.15
T
25.04
24-28
30-
20
15-
10
0-600
600-12000
1200-1800
1800-2400
2400
0-600
600-1200
1286-1808
1800-2400
2400+
Generation Length
Generation Length
Figure 3. Comparison of win rates (against GPT-4-turbo) by the length of generation. On-policy training does not significantly outperform off-policy training as generation length increases in both math and coding tasks (left) and crcative writing well recommendation tasks (right). Error bars show 95% confidence intervals from bootstrapping.
tions benefit more from on-policy data and that the improve- ment in math and coding can be attributed to the improve- ment in length. To investigate this, we plot the histogram of the length controlled win rate by the length of the generated sequences Figure 3.
not be explained by generation length. In the next section, we demonstrate how combining the strengths of on-policy data with the efficiency and abundance of off-policy data available on the web can lead to improved LM alignment.
We observe that on-policy DPO does not outperform off- policy DPO the generation length increases. This indi- cates that the improvements in on-policy training on math and coding cannot be solely attributed to increasing the length of the generations. The type of task is the primary factor in determining whether on-policy or off-policy train- ing more effective.
4. SIMPLEMIX: Simply Mixing On- and Off-policy Data Improves Alignment
In this section, we show the benefits of combining on- and off- oof-cpoicy data in preference optimization. Specifically, we employ the data source D Don U Dou in equation answer (Q2). We describe our experiment setup and basclines at 1, and report our main results $4.2.
We conclude that on-policy DPO mostly improves ob- jective tasks (c.g., math and coding) while off off-policy DPO excels in improving open-ended tasks (e.g. creative writing and personal recommendation and that this distinction can-
4.1. Setup
SIMPLEMIX Our method SIMPLEMIX combines on- and off-policy data, where the winning you and losing responses

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
Method Name
Sampler of Vie Ut
Loss Function
Non- HonnHy Methods
Off-policcy DPO On-policy DPO
Yw Vi D
Dr yt) DPO y1)
Dw V1 SSFT
Hybrid Methods
DPO-Mix-P (Shi al., 2024)
Ywu y1 [mg
3/2 1/2 1/2 1/2 "SPT "00" "SFT
yor yt)
HyPO (Song al., 2024b)
Ywu yt
D, 1 no
DPO yt) log(y 2x) .sg
(g)
5o (ylar))
SIMPLEMIX (Ours)
You, y1
(D, TTPT)
DPO vt
Table 1. A comparison between SIMPLEMIX and other baselines we compare in this work. "sg() denotes the stop gradient operation.
Y1 are sampled from TShr and the off- off-policy dataset D with equal probability. An in-depth study on the effect of differ- ent sampling probabilities is at $5.1.
Evaluation To evaluate the aligned policy, we follow isting literature on LM alignment and use the same 7 bench- marks in the Win (Penedo et al., 2024) evaluation suite, containing Commonsense QA (CQA; Talmor et al. (2019)), Hellaswag (HS; Zellers et al. (2019)), Openbook QA (OQA; Mihaylov et al. (201 8a)), PIQA (Bisk al., 2020), Wino- grade (Sakaguchi et al., 2020), ARC-Challenge (ARC-C; Clark et al. (2018)), and MMLU (Hendrycks al., 2020), which covers 7 knowledge and common sense based bench- marks. We also use Ifeval (Zhou et al., 2023) and Alpaca Eval 2.0 (Dubois et al., 2023; 2024) for evaluating general instruction following ability. We report the average scores of the seven tasks + Ifeval prompt level loose accuracy as Avg. Score and Alpaca Eval 2.0 length controlled win rate (L.C). Detailed descriptions of benchmarks can be found in Appendix D.
Baselines We compare the following baselines with SIM- PLEMIX. A more detailed comparison of our baselines with SIMPLEMIXis in Table 1.
On-policy DPO: Winning and losing response sampled from the SFT model: Yie Y/ ~ T- x).
Off-policy DPO: Winning and losing response sampled from off-policy dataset: You y1
Don-
HyPO (Song et al., 2024b): Off-policy DPO with KL Regularization using on-policy data: Yw. ~ Dat, y T+(-] | æ). Compared to HyPO (Song et al., 2024b), SIM-
PLEMIX changes the online KL regularization objective
4.2. Main Results
a DPO loss.
Table 2 shows the experiment results Alpaca Eval 2.0 and averaged scores over 8 benchmarks (Avg. Score) for all of our baselines. We report the detailed breakdown of individual benchmarks in Appendix E.
DPO-Mix-P (Shi al., 2024): On-policy data generated with interpolation between the current model To and TSFT- Yw 3t 3/2/ x=1/² 1122 1/2 SST Rg. TSFT Compared to DPO-Mix-P (Shi al., 2024). SIMPLEMIX can be seen "hard" version of sampling from interpolation between off-policy LM and the SFT policy TsFt with equal weights.
We find that SIMPLEMIX consistently yields the best per- formance across both models and evaluation metrics. This indicates that the widely available source of off-policy preference data still provide valuable signals to pref- erence optimization. We plot the per-task performance in Alpaca Eval 2.0 of models trained with SIMPLEMIX in Figure 8 in Appendix G. SIMPIFEMIX offers a balance be- tween the performance on reasoning (math and coding) and open-ended tasks (creative writing and recommendation)
Models and Datasets We experiment with two RSPt mod- els meeta-llama/Lanme 1.w-mma----------------- allenai/ /asenai/Llannn /al/aammma212 1----ma-------t1------------
and with DPO (Rafailov et al., 2024) and experiment on two pairwise pref- erence datasets: Ultrafeedback (Cui al., 2024) and Help- Steer2 (Wang et al., 2024). A detailed description of the collection of prompts and responses in both datasets can be found in Appendix C. We train our models for one epoch on the same amount of (Nw, 3R) pairs for every baseline. This controls that every method has seen the same amount of data, although on on-policy DPO requires more computing since it requires sampling from either TSrT or mo.
5. Ablations in Preference Data Curation
In this section, we investigate curation strategies in prefer- ence data curation. Specifically, we look into the impact of response diversity (5.1), the mixture ratio between on- and off-policy data (5.2), and the effect of filtering off-policy data (5.3).
5

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
Tulu-3-8B-SFT
Llama-311-8B-Instrre
Alpaca Eval 2. (%)
Avg. Score f
Alpaca Eval 2.0 LC (%)
Avg. Score
SFT
8.52
59.72
24.36
60.88
Experiments on UtraFeedback (Cui et al., 2024)
Off-policy DPO
14.23
60.02
27.87
61.67
On-policy DPO
15.17
59.70
27.28
62.09
HyPO (Song et al., 2024b)
18.11
60.03
27.91
62.01
DPO-Mix-P (Shi et al., 2024)
17.25
60.70
27.79
61.61
SIMPLEMIX
20.64
61.14
29.41
63.06
Experiments on HelpSteer2 (Wang et al., 2024)
Off-policy DPO
14.74
59.94
24.81
61.54
On-policy DPO
15.26
60.21
25.09
61.77
HyPO (Song al., 2024b) DPO-Mix-P (Shi et al., 2024)
18.19
59.96
27.15
61.69
17.75
60.04
27.11
61.49
SIMPLEMIX
21.11
61.22
29.12
62.12
Table Performance comparison among various on and off-policy preference optimization methods on UltraFeedback (Cui et al., 2024) and HelpSteer2 (Wang et al., 2024). Avg. Score refers to the average score across 8 benchmarks listed in Appendix D. We found that the performance gap between on- and off-policy DPO minimal. Incorporating off-policy data into on-policy DPO outperforms using on- or off-policy data only. Furthermore, curating balanced mixture of on- and off-police data in DPO consistently yields the strongest performance in both LM-as-a-judge evaluation (Alpaca Eval 2.0) and reference-based evaluation (Avg. Score).
5.1. The Effeet of Preference Data Diversity
LC (%) 1
Avg. Score 1
Avg. Reward
Some works theoretically show that on-policy preference optimization benefits from more exploration (Rashidine- jad Tian, 2024; Song et al., 2024b; Anonymous, 2024a; Xiong et al., 2024). This is to say that eliciting more diverse responses would be beneficial. A possible explanation of the effect of off-policy data is that it increases the diversity of generations. Works have reported that on-policy genera- tions usually share the same prefix (Knight et al., 2017) leading to reduced diversity among responses. On the other hand, off-policy data, esp. Ultrafeedback, the responses are collected from a collection of models rather than single model, thus improving diversity.
Prompting
21.41
58.27
-10.75
1.0
26.94
60.21
-9.44
T = 2.0
26.77
59.44
-10.56
T 3.0
26.18
58.83
-10.77
SIMPLEMIX 29.41
63.06
-3.39
Table 3. Performance comparison between different sampling methods for eliciting diverse responses. Explicitly prompting for diversity increasing the sampling temperature results in generations with lower quality, as measured hy our reward model. Low-quality generations, albeit diverse, lead to worse perfor- mance on our evaluation when performing DPO on them.
To validate this claim, compare on+off policy mix- ture with directly increasing the diversity of responses with:
trafeedback, again we the oracle reward model anno- tate the best and worst response as y/ and yn. We report the results of training TSFT on this diverse response in Table 3.
1. Prompting: We iteratively prompt the language model
to generate N responses for diversity (Lu et al., 2024): we first prompt the language model with x and obtain the first response Y1 x(- x), then condition a written prompt that asks for diverse response for the same in- put 1/2 ~ T(- x, 2). We repeat this process to obtain N responses. We report the prompt z in Appendix H.
We found that eliciting more diverse responses often comes the cost decreasing generation quality, thus hurting performance. We manually inspected the gener- ations of prompting and larger generation temperature т. and found that most of the generation, albeit diverse, often falls short in quality compared to direct sampling. The explanation is that diversity itself is not helpful because the diversity might fall into low-reward regions, increasing diversity while reducing quality. Our findings echo (Anony- mous, 2025b) who found that sampling with a larger tem-
2. Large temperature: Sampling yw: d with a larger sam- pling temperature r. We experiment with 7 = (1,2,3)
We sample N 4 responses from TSFT I.1 3. 1 ructnnstucct the 60k prompts in UI-
6

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
perature leads to degraded performance on various question-
sentence embedding between the chosen and rejected re- sponse proxy for "similarity" and select top-p exam- ples that are most dissimilar.
answering benchmarks.
5.2. The Effect of On- vs. Off-policy Data Mixtures
We train L1 ama -3. --8--Inntrrtt on filtered Ultra- feedback with results on Alpaca Eval 2.0 in Figure 5. We found that select- ing data based on quality (orange) outperforms other criteria except when p = (0.5).
We further investigate different data mixtures of on- and off-policy data by keeping the total amount of data the same but varying the sampling ratio between on- and off-policy data. Figure 4 shows the re- sults of training Llama-3. B.ntaa3.1-uuuuz -u--uuu---sSS on Ultrafcedback with dif- ferent on- to off-policy data ratio. We observe that sampling with equal probability from both data sources (0.5 on cn-policy + 0.5 off- off-policy) outperforms other mixtures, which echos the results in Ball et al. (2023) who finds that a mixture of 0.5 off-policy data + 0.5 on-policy data generally performs the best for traditional RL tasks. Ball et al. (2023) also finds that in traditional RL tasks, a balanced 0.5 - 0.5 mixture is also the most stable. Unfortunately, based on the error bars in 4, we found no clear pattern about the variance of different mixtures.
(0.1,0.2, 0.4, 0.5} and report the
L- 3. 1 1-1--88 rnst and
Since we know that the quality of off- off--polcy data has the most impact on performance, we filter the off- off-policy data according to its reward evaluated by our oracle reward model and mix it with the on-policy data. Specifically, we keep p = 0.4 fraction of Ultrafeedback and mix it with the same amount of on-policy data. We report the results in Figure 6. We found that only keeping the high-quality off- off-policy data, when paired with on-policy data, can further improve the performance.
6. Related Works
Comparing On- and Off-Policy Alignment The liter- ature (Xu et al., 2024c; Ivison et al., 2024) usually com- pares PPO (Schulman et al., 2017) with DPO (Rafailov et al., 2024), arriving at mismatching conclusions with some works (Xu et al., 2024c) showing that PPO outper- forms DPO, some works show the gap between PPO and DPO is minimal (Lambert et al., 2024a; Ivison et al., 2024). Two concurrent efforts are close to our work (Tajwar et al., 2024; Tang et al., 2024). Both studies investigate the differ- ences between on and off-policy alignment, but Tajwar et al. (2024) conducted experiments on a controlled setting that is different from real-world LM alignment, and although Tang et al. (2024) have arrived at conclusion that on-policy outperforms off-policy. the performance gap between on- and off -oficy data in the most general task in their setting (chat arena side by side) is the smallest, which motivates us to investigate deeply. Additional related works on LM alignment is at Appendix F.
5.3. The Effect of Off-poolcy Data Filtering
Off-policy responses are often collected with various open- sourced models for maximizing diversity, which often in- cludes smaller models that are less capable. For example, re- sponses in Ultrafeedback (Cui et al., 2024) are sampled from models ranging from the most capable GPT-4 to smaller open-sourced 7B models, often including low-quality re- sponses. We investigate the effect of filtering out these low-quality responses when combined with on-policy data:
Specifically, select fraction p of the entire Ultrafeed- back dataset (Cui al., 2024) according to the following heuristics:
Quality: We annotate the pairs of generation with Bewrrrk--ewarr--eeeee
2--222-200 2. We se- lect the top-p pairs with the highest total reward.
Existing work shows that LM learns better from on-policy data (Tajwar et al., 2024: Tang et al., 2024). We select top-p of the most on-policy examples. We mea- sure "on-policiness" by the sum of the log- bog-brbbabilities of chosen and rejected responses using TSF.-
On-policiness:
Hybrid RL While there has been many works that bridges on- and off-policy RL (Song et al., 2023; Wagenmaker & Pacchiano, 2023; Gu et al., 2017: Nakamoto et al., 2023; Ball et al., 2023; Lee et al., 2021; Tan et al., 2024; Song et al., 2024a), few work studies combining off-policy and on-policy data in language model alignment (Song et al., 2024b; Shi et al., 2024: Anonymous, 2024b; Bose et al., 2024), whose contributions are mostly theoretical in terms of coverage (Song et al., 2024b), convergence rates (Shi et al., 2024; Bose et al., 2024), and optimality (Xiong et al., 2024). However, these works often only conduct experiments on LL--ass---udde judjue benchmarks, which can be easily hacked (Singhal et al., 2024; Wei et al., 2024; Park et al., 2024), on models that are smaller and weaker (e.g.. the Pythia suite).
Existing work (Kim et al., 2024) re- ports that the policy learns better from highly contrastive pairs, where the quality gap between the chosen and re- jected response is large. We select top-p pairs where the difference between the chosen reward and the rejected reward is largest.
Contrastiveness:
Existing work (Pal et al., 2024; Razin et al., 2024) points out that similar examples harms DPO per- formance. We take the cosine similarity of the last layer
Similarity:
7

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
Meta-Liama- 3-88-Instruct
Llama-3.11 3.1-Tulu-3-88--35
22
Win Rate (againstt GPT-4-turbo)
Rate GPT-4-urbo
30
20
29
18
28
16
27
14
Lergth Controled Wnrraee [LC)
Length Contrelee Winnate (LC)
26
Win Rate (WR)
Win Racc (WR)
0..1.
0.30.7
044,6
0.5.0.5
0460.44
0.7.03 10,00
0.0.1.0
0.3.0.7
0.4..6
0.5.05
0.6.4.
0,7.0.3 1.0,000
On-Policy to Off-Policy Sampling Ratio
On-Policy Off-Policy Sampling Ratio
Figure 4, Perfomance on Alpaca Eval 2.0 for different on- to off-policy data ratio for performing DPO on top of TSFT Met detta-lll.aa 1-88-11 nst rucruct (left) and ama-3. 1 uu- 8-8-88T (right) the Ultrfcedback (Cui al., 2024) dataset. A balanced mixture (0.5 on-policy + 0,5 off-policy) outperforms other mixtures.
meta-llama/Liama-
Selecting High Quality Off-policy
Contral Win Rate (against GPT44 Tuubb))
31.62
29.41
Win Rate (against GPT-4-turbo)
24.36
15
10
Selection Dn-policiness Quality
Similanty
Contrastiveness
SFT
On Off
On Fiitered
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
Data Selection Ratio
Figure 6. Alpaca Eval 2.0 Length Controlled Win Rate for Rsrr- Mixing On + Off-policy Data and Mixing in high-quality off- policy data with on-policy data DPO. Selecting high-quality off-policy data can improve performance using SIMPLEMIX.
Figure 5. Alpaca Eval 2.0 length controlled win rate (LC) for dif- ferent data selection strategies. No criterion other than selecting high-quality data (measured by reward model) significantly outperforms others for off-policy preference learning.
policy preference optimization is minimal, it could be the case the we did not tune our model with the perfect configu- rations. Moreover, although we also exhausted our resources for evaluation on both reference- and reference benchmarks, a-a--a- benchmarks can be hacked (Anonymous, 2025a), exhibit certain biases (Panickssery et al., 2024; Park et al., 2024), and is sensitive prompt for- matting (Zheng al., 2023). Therefore, the applicability of our conclusions is limited to the hyperparameters configura- tions, models, training datasets and evaluation benchmarks on which we experimented.
To the best of our knowledge, our work conducts the most comprehensive evaluation, including reference-breg based and LMMas-a-judde benchmarks, on top of state-of-the-art open- sourced models: Ilama-3. 1--a-B...--BB-B- I-8B-Instruct (Dubey et al., 2024) and Tulu-3-8B-SFT (Lambert et al., 2024a).
7. Limitations and Implications
I Limiations We acknowledge that although we exhausted our resources to perform hyperparameter searching, the combinatorically large hyperparameter space makes chal- lenging to draw decisive conclusions. Therefore, although we observed that the performance gap between on- and off-
Our work implies that data selection for LM alignment should be task dependent: for harder tasks where
Implications
8

---

Cited SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
few high-quality data is available but the answer be easily verifiable, e.g., math, where you can perform string matching for the answer, using on-policy data, outperforms off-policy data. However, in general, for tasks where abun- dant data is available online, performing on- on-policy sampling does not outperform. Therefore, one can resort to off-policy data, saving on the cost of generating multiple responses per query. We leave the impact of fine-grat anee-gaaied oon-policyness" of data on LM performance for future work.
Anonymous. Sample efficient alignment for LLMs. In Submitted The Thirteenth International Conference Learning Representations, 2024a. URL https:// openreview net / forum Corum?iddee 2id=Pfi17cvccch un- der review. Cited on page 6.
Anonymous. SeRA: Self-reviewing and alignment of LLMs using implicit reward margins. In Submitted to The Thir- teenth International Conference on Learning Represen- tations, 2024b. URL https: /atp:://pppeeeeee /oppnreview, net/ forum? dinmnidd-uu da under review. Cited on page 7.
8. Conclusion
In this paper, we show that in preference learning of LMs, on-policy data and eff-policy data are complementary: on- policy data improves upon reasoning tasks that have a ground-truth answer, whereas off-policy data mostly im- proves upon more general tasks. We propose SIMPLEMIX: mixing up on- and off-policy data consistently improves performance when using data from a single source.
Anonymous. Cheating automatic LLM benchmarks: Null models achieve high win rates. In The Thirteenth In- ternational Conference on Learning Representations, 2025a. URL https://p//./ /ttps///oppeeeeeeeeeeeeeeeeeeeee www.//ee//eeee forum? id=syThiTnhWin Cited page 8.
Anonymous. Turning up the heat: Min-p sampling for creative and coherent LLM outputs. In The Thirteenth International Conference on Learning Representations, 2025b. https:/ /oppn net/ forum? id-FBkpCyujts. Cited on page 6.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Azar, M. G., Daniel Guo, Z., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In Dasgupta, S., Mandt, S., and Li, Y. (eds.), Proceedings of The 27th Interna- tional Conference on Artificial Intelligence and Statis- tics, volume 238 of Proceedings of Machine Learn- ing Research, pp. 4447-4455. PMLR, 02-04 May 2024. URL v238 wwwwhhaaghhaaaaa2aaa.oo. page 17.
Acknowledgments
This work is supported by ONR grant (N00014-24-1-2088)) We sincerely thank Jingyu Zhang, Weiting Tan, Mare Marone, Jeffery Cheng, and Orion Weller for fruitful dis- cussions. We also thank the anonymous reviewers for their helpful suggestions.
mlr .press// Cited on
References
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.00 05862, 2022a. URL https:// arxiv axxiv.orgaaie httgg//bb///e 220 05862. Cited on page 3.
Ahmadian, A., Cremer, C., Gallé, M., Fadace, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to basics: Revisiting REINFORCE-styL optimization for learning from human feedback in L.LMs. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 12248-12267, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10. 10.116653/1122224accooo 662. URL nattps:// ac ant aeelannhologyy wwg.:/oooooyy../222../..// acl - long. b.ogg.6. 662. Cited on page 1.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J.,
Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv artp: 2022b URL https: htt//aaxiivvv.///////eee/eeea wtwpxiiw.orrgbbb///2222.2. on page 17.
2022b.
Ball, P. J., Smith, L., Kostrikov, I., and Levine, S. Efficient online reinforcement learning with offline data. In Krause, A., Brunskill, E., Cho, K., En- gelhardt, B., Sabato, S., and Scarlett, J. (eds.), Pro- ceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine
Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Goffinet, E.,, Heslow, D., Lau- nay, J.. Malartic, Q., Noune, B., Pannier, B., and Penedo, G. Falcon-4055 40B:: an open large language model with state- th- performance. 2023. Cited on page 16.

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
Learning Research, pp. 1577-1594, PMLR, 23-29 Jul prrsss////
Cui, G., Yuan, L.,, Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z.., and Sun, M. ULTRA- FEEDBACK: Boosting language models with scaled Al feedback. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 9722-9744. PMLR, 21- 27 Jul 2024. URL https: / http///oceediiee press /
2023. URL.ttps: / /pwpceeedeeeeeeeooooocooccoccdde ngs mol.mllr.... mllr x202/ball1 html. Cited on page 7.
Biderman, S., Schoelkopf, H., Anthony, Q. Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and Van Der Wal, O. Pythia: a suite for analyzing large language models across training and scaling. In International Conference on Machine Learning (ICML). JMLR.org, 2023. Cited on page 16.
.mlr. html. Cited on pages 3, 5, 6.
7, 8, 16, and 18.
Bisk, Y., Zellers, R., Bras, R. L.,, Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural lan- guage. In Conference on Artificial Intelligence (AAAI), 2020. Cited on pages 5 and 17.
Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z.. Hu, S., Liu, Z., Sun, M.. and Zhou, B. Enhancing chat lan- guage models by scaling high-quality instructional con- versations. arXiv preprint 2305.14233 2023. URL pettps//arxiv..oo htt.://rggbbbb//..3ee 14233. Cited on page 16.
Bose, A., Xiong, Z., Saha, A., Du, S. S., and Fazel, M. Hybrid preference optimization for alignment: Provably faster convergence rates by combining offline preferences with online exploration, 2024. URL https:/ / ////axxe arxiv. org/ arg/abs// abs 2412 10616 6. Cited on page 7.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C.,, Ferrer, C. C., Nikolaidis, C., Al- lonsius, D., Song. D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D., Starcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Zhang, F., Syn- naeve, G., Lee, G., Anderson, G. L., Nail, G., Mialon, G., Pang. G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, A., Kloumann, I., Misra, I., Evtimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J.. Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu. K., Bhalla, K. Rantala-Yeary I.., der Maaten, I.., Chen, I.., Tan, L.., Jenkins, L., Martin, L., Madaan, L., Malo, L., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pasupuleti, M., Singh, M. Paluri, M., Kardas, M.. Oldham, M.. Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Duchenne, O., Çelebi. O., Alrassy P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q., Dong, Q. Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang. R., Hosseini, S., Chennabas- appa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie,
Brown, T., Mann, B., Ryder, N., Subbiah, M.. Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learn- ers. Advances in Neural Information Processing Systems (NeurIPS), 2020. URL https: / /tt/aaxivvv/////////////////aata org/ abs/ 2005. 141111664 65. Cited on page 1.
Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating Ilms by human preference, 2024a. URL https:// htpp:///rxiiv wwtg/rgg/////oooeee/o.//eeeeeeee on page 3.
04132. Cited
Chiang, W.-L., Zheng, I.., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonza- lez, E., et al. Chatbot arena: An open platform for evaluating Ilms by human preference. In International Conference on Machine Learning (ICML), 2024b. URL https: /atxxxxxvv///////////a////aaaa org/pdg/////ooooooooo 2403. 04132. Cited on page 16.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Chal- lenge. arXiv preprint arXiv:1803.05457 2018. URL https:/ pumps://arxtee /arxivv.... wttg/bs///aaa /21/22222222 03315. Cited on pages 5 and 17.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, I.., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training ver- ifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL attps://arxiv. org//ddf/210..14 141 68. Cited on page 3.
10

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang. S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Her- man, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov, T., Xiao, T., Karn, U. Goswami, V. Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Petrovic, V., Chu, W. Xiong, W., Fu. W., Meers, W., Martinet, X., Wang. X., Tan, X. E., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y. Zhang, Y., Li, Y., Mao, Y., Coudert, 7. D., Yan, Z., Chen, 7.,, Papakipos, Z Singh, A., Grattafiori, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Vaughan, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Franco, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Wang. C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtenhofer, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Wyatt, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, F., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E.,, Sun, F., Kreuk, F., Tian, F., Ozgenel, F., Caggioni, F., Guzmán, E. Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Thattai, G., Herman, G. Sizov, G., Guangyi, Zhang. Lakshminarayanan, G., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H. Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Damlaj, I., Molybog, I., Tufanov, I., Veliche, I.-E., Gat, I. Weissman, J., Geboski, J.. Kohli, J., Asher, J.. Gaya, J.B.,, Marcus, J., Tang, J., Chan, J., Zhen, J., Reizenstein, J. Teboul, J., Zhong, J., Jin, J., Yang. J.. Cummings. J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J. Wang, J., Wu, K., U, K. H., Saxena, K., Prasad, K., Khandelwal, K., Zand, K., Matosich, K., Veeraragha- van, K., Michelena, K., Li, K., Huang, K., Chawla, K., Lakhotia, K. Huang, K., Chen, L., Garg, L., A, L., Silva, L., Bell, L., Zhang, L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Tsimpoukelli, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Ke- neally, M., Seltzer, L., Valko, M., Restrepo, M., Patel, M.. Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Laptev, N. P., Dong, N., Zhang, N., Cheng, N., Chernoguz, O.,
Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P.,
Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj. P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Li, R., Hogan, R., Battey, R., Wang. R., Maheswari, R., Howes, R., Rinott, R., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C.,, Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyighe, S., Chintala, S., Max, S., Chen, S., Kehoe, S. Satterfield, S., Govindaprasad, S., Gupta, S., Cho, S.. Virk, S., Subramanian, S., Choudhury, S.,, Goldman, S., Remez, T., Glaser, T., Best, T., Kohler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. Mangla, V., Albiero, V., Ionescu, V., Poe- naru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wang, X., Wu, X., Wang. X., Xia, X., Wu, X., Gao, X., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Hao, Y., Qian, Y., He, Y., Rait, Z., Vito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., and Zhao, Z. The Ilama 3 herd of models, 2024. URL https: //ta///axiii orgo abs 2407. 21783. Cited on pages 3, 8, and 18.
Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., and Hashimoto, T. Al- pacafarm: A simulation framework for methods that learn from human feedback. In Thirty-sevntth Con- ference on Neural Information Processing Systems, 2023. URL https: apenreview www. forum?? aa-4hturzLeKK, Cited on pages 5 and 16.
Dubois, Y., Liang, P., and Hashimoto, T. Length alpacaeval: A simple debiasing of automatic evalu- ators. In First Conference on Language Modeling, 2024. URL https:/ /pwp//eeeeeeee.ee////e/ee net forum? 1d=CybBmZWBXC Cited on pages 3, 5, and 16.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Model alignment as prospect theoretic optimiza- tion. In International Conference on Machine Learning (ICML), 2024. Cited on page 18.
Gao, L.,, Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C.,, Phang,
J., Reynolds, I., Schoelkopf, H., Skowron, A., Sutawika,
L., Tang. E., Thite, A., Wang, B., Wang. K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https: / zenodo. org/ records/ 12 16608602. Cited on pages 16 and 17.
Gu, S., Lillicrap, T., Ghahramani, Z.,, Turner, R. E.,

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
Schölkopf. B., and Levine, S. Interpolated policy gradi- ent: merging on-policy and off-policy gradient estimation for deep reinforcement learning. In Advances in Neural Information Processing Systems (NeurlPS), NIPS' 17, PP. 3849-3858, Red Hook, NY, USA, 2017. Curran Asso- ciates Inc. ISBN 9781510860966 Cited on page 7.
Knight, K., Badarau, Bianca, Baranescu, Laura, Bonial, Claire, Bardocz, Madalina, Griffitt, Kira, Hermjakob, Ulf, Marcu, Daniel, Palmer, Martha, O' O'Gorman, Tim, and Schneider, Nathan. Abstract meaning representation (amr) annotation release 2.0, 2017. URL https:// catalog. wwwoggldc....o www. wew.nnnnddd/uDD//// 7T10. Cited on page 6.
Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., Ferret, J., and Blondel, M. Direct language model alignment from online ai feedback, 2024. URL ht https:: happ:///rre: //arxiv. org/ /ass//b//// / 2402. 04792. Cited on pages 3, 17, and 18.
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F,, Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., Gu, Y., Malik, S., Graf, V., Hwang, J. D., Yang, J., Bras, R. L., Tafjord, O., Wilhelm, C., Soldaini, L., Smith, N. A., Wang, Y., Dasigi, P., and Hajishirzi, H. Tülu 3: Pushing frontiers in open language model post- training, 2024a. URL https:/ /arxivv... w.w.g//og/... /bbs/ Cited on pages 1, 3, 7, 8, and 18.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Learning Representations (ICLR), 2020. Cited on pages 3, and 17.
International Conference on
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. Rewardhench: Evaluat- ing reward models for language modeling, 2024b. Cited on page 3.
Hu, J., Wu, X., Zhu, Z., Xianyu, Wang, W., Zhang, D., and Cao, Y. Openrlhf: An easy-to-use, scalable rlhf framework. arXiv preprint arXiv:2405.1 143, 2024. Cited on page 16.
and
Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline- to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In 5th Annual Conference Robot Learning, 2021. URL https: /tle:://oppoe /ppnreeiee.. net / forum? di d= 6J5W. Cited on page 7.
Hu, S., Luo, Y., Wang, H., Cheng, X., Liu, Z., and Sun, M. Won't get fooled again: Answering ques- tions with false premises. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5626-5643, Toronto, Canada, July 2023, Association for Compu- tational Linguistics. doi:
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacae- val: An automatic evaluator of cstruction-followinn models. https: //gpttub.b./////////// com/ tommtatsu-laab a paca_eval, 1, 2023. Cited on page 3.
309. URL https: a/acl anthol ooyy.ooooo.....ooo org/2023. sel-long: 309/. Cited on page 16.
Lin, S., Hilton, J., and Evans, O. TruthfulQA: mea- suring how models mimic human falsehoods. arXiv preprint arXiv:2109.07995 2021. URL https://
Ivison, H., Wang, Y., Liu, Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., and Hajishirzi, H. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2024. Cited pages 1, 3, and 7.
wwxiv.org/abs/// 07958. Cited on page 16.
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. The flan collection: Designing data and meth- ods for effective instruction tuning. arXiv preprint 1xXiv:2301.13688 2023. URL https://aan /arxivv. org/ 2301. 13688. Cited on page 16.
Jang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de Las Casas, D., Bressand, F,, Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.- A., Stock, P., Scao, T. L., Lavril, T., Wang. T., Lacroix, T., and Sayed, W. E. Mistral 7b. ArXiv, abs/2310.06825, 2023. URL https: http://aapp..aaaaaee.eeee semant cscholar http//CopuII:::333388/
Lu, Y., Wang, D., Li, T., Jiang, D., and Khashabi, D. Bench- marking language model creativity: A case study on code generation. arXiv preprint 20X1::2220:: 2024. URL https:/ / / aaxiv www.vorgg/././/eeooooooo 09000 Cited on page 6.
Cited on page 17.
Kim, J., Goyal, A., Zhang, A., Xiong, B., Hou, R., Kam- badur, M., Mahajan, D., Hajishirzi, H., and Tan, L. A systematic examination of preference learning through the lens of instruction instruction-followinng 2024. URL ht tps: //a rxiv. org/ abs 2242. ......200 15282. Cited on page
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference- free reward. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Cited on page 17.
12

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018a. Cited pages 5 and 17.
erations. In The Thirty-eighth Annual Confer- Neural Information Processing Systems, 2024. URL https: /atpss///pppeeeeeweeee wwt/ 1d=4NJBV6WpOU Cited on page 8.
ence
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Process- ing (EMNLP), 2018b. URL https: /etps://arxivy htraaaaivvvooo../....... abs/ 809 02789. Cited on page 17.
Park, J., Jwa, S., Meiying, R., Kim, D., and Choi, OffsetBias: Leveraging debiased data for tuning eval- uators. In Al-Onaizan, Y., Bansal, M., and Chen, Y.- N. (eds.), Findings of the Association for Computa- tional Linguistics: EMNLP 2024, pp. 1043-1067, Mi- ami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10. 1061881111/222. findings-emnlp.577 URL https //t/a/aaaaloooyy.ooaooaao org 3rg/2024.02 findings emn 1.mmmp.1. p. 57/. Cited on pages 7 and 8.
Nakamoto, M., Zhai, Y., Singh, A., Mark, M. S., Ma, Y., Finn, C., Kumar, A., and Levine, S. Cal-QL: Calibrated offline RL pre-training for efficient online fine-tuning. In Advances in Neural Information Processing Sys- tems (NeurIPS), 2023. URL htLLttts:// /ttttp::///ppeeeevei net feet/fowe ftrrm/ormm?em/ (edddd-GEEEViSSse. Cited on page 7.
Penedo, G., Kydlíček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https:/ http://axxvvvo////ee/e/ Cited on page 5.
Nvidia, :, Adler, B., Agarwal, N., Aithal, A., Anh, D. H., Bhattacharya, P., Brundyn, A., Casper, J., Catanzaro, B., Clay, S., Cohen, J., Das, S., Dattagupta, A., Delalleau, O., Derczynski, L.., Dong, Y., Egert, D., Evans, E.,, Ficek, A., Fridman, D., Ghosh, S., Ginsburg, B., Gitman, I, Grzegorzek, T., Hero, R., Huang, J., Jawa, V., Jennings, J. Jhunjhunwala, A., Kamalu, J., Khan, S., Kuchaiev, O., LeGresley, P., Li, H., Liu, J., Liu, Z., Long, E., Maha- baleshwarkar, A. S., Majumdar, S., Maki, J., Martinez, M., de Melo, M. R., Moshkov, I., Narayanan, D., Naren- thiran, S., Navarro, J., Nguyen, Nitski, O., Noroozi, V., Nutheti, G., Parisien, C., Parmar, J., Patwary, M., Pawelec, K., Ping. W., Prabhumoye, S., Roy, R., Saar, T., Sabavat, V. R. N., Satheesh, S., Scowcroft, J. P., Sewall, J., Shamis, P., Shen, G., Shoeybi, M., Sizer, D.,, Smelyanskiy, M., Soares, F., Sreedhar, M. N., Su, D., Subramanian, S., Sun, S., Toshniwal, S., Wang, H., Wang. Z., You, J., Zeng, J., Zhang, J., Zhang, J., Zhang, V., Zhang, Y., and Zhu, C. Nemotron-4 340b technical report, 2024. URL ht https: /tt//aaxiivv.////// org/ abs /202 11704. Cited on page 16.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimiza- tion: Your language model is secretly reward model. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024. URL https: /0t::/arxa /tt//axivvv////////////////aaavv vv. org/ abs/2305. 18290. Cited on pages 2, 3, 5, 7, and 17.
Rashidinejad, P. and Tian, Y. Sail into the headwind: Align- ment via robust rewards and dynamic labels against re- ward hacking, 2024. URL https: htttss///rxxv..or //axxivvorrg///// abs/2412. 09544. Cited on page 6.
Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., and Hanin, B. Unintentional unalignment: Likelihood displacement in direct preference optimization. arXiv preprint 1rXiv:2410.08847, 2024. Cited on page
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. WINOGRANDE: adversarial winograd schema chal- lenge at scale. In Conference on Artificial Intelligence (AAAI), 2020. URL https: /aaa/vivvv..// org/ abs/ 1907. 1907.10641 10641. Cited on pages 5 and 17.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training Language Models to Follow In- structions with Human Feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https:/ a arxiv. org/abss 220322223.22 55. Cited pages and 17.
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint ar7iv: 2017. URL https: 2017.
Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024. URL httpt:/// arxiv. v. 220. Cited on pages 7 and 17.
/ /arxivv. www./oogg/////////// 06347. Cited on pages 3 and 7.
Shi, R., Zhou, R., and Du, S. S. The crucial role of samplers in online direct preference optimization, 2024. URL https: /////xaxvv/ wwgg/bs//409...... 196055. Cited on pages 5, 6, 7, and 18.
Panickssery, A., Bowman, S. R. and Feng, S.
LLM evaluators recognize and favor their own gen-
13

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
Singhal, P. Goyal, T., Xu, J., and Durrett, G. A long way to go: Investigating length correlations in Language Modeling, http:///pppeeiiieeee wwwpeeeeeeeeeeeeeeeeeeeeeeeeeeee net/forum?
// /aaclanthologyy.oooooo wwg./ii...g..111 111NN199110 1421. Cited on pages 5 and 17.
RLHF. In First Conference 2024. URL httph:tt///. id=G81a01PGx) Cited on page 7.
Tan, K., Fan, W., and Wei, Y. Hybrid reinforcement learning breaks sample size barriers linear MDPs. In Advances in Neural Information Processing Systems (NeurlPS), 2024. URL https: htapps://pperree oppeneeviewe...e net/ forum? id Ad-bPuxxFBHyL Cited on page 7.
Song, Y., Zhou, Y., Sekhari, A., Bagnell, D., Krishna- murthy, A., and Sun, W. Hybrid RL: Using both of- fline and online data make RL efficient. In Interna- tional Conference on Learning Representations (ICLR), 2023. URL https::///. /ppopreeeeeeeeeeee net/forume (d-yyBis80ii0 Cited on page 7.
Tang, Y., Guo, D. Z., Zheng, Z.. Calandriello, D., Cao, Y., Tarassov, E.,, Munos, R., Avila Pires, B., Valko, M., Cheng, Y., and Dabney, W. Understanding the per- formance gap between online and offline alignment al- gorithms, 2024. URL https: /arxivv..vv..///////ea///aaaavv/ org/ /aba// 2405. 08448. Cited on pages 3, and
Song. Y., Bagnell, D., and Singh, A. Hybrid re- inforcement learning from offline observation alone. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J. and Berkenkamp, F. (eds.). International Conference on Machine Learn- ing (ICML), volume 235 of Proceedings of Machine Learning Research, pp. 46019-46049, PMLR, 21-27 Jul 2024a. URL https: http///oooedinggee wereediingsmm..... miggs.m... mlr. Sress/v235/songge
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-follooingg Ilama model. https: //ithubb wowocto//aaaaoa. conn/atsu-lab) stanford_alpacee 2023. Cited page 16.
html. Cited on page 7.
Team, M. N. Introducing mpt-30b: Raising the bar for open- source foundation models, 2023, URL www mww. cmml com/b h og ht/p/mtt3000 30b. Accessed: 2023-06-22. Cited on page 16.
Song, Y. Swamy, G., Singh, A., Bagnell, J., and Sun, W. The importance of online data: Understanding prefer- ence fine-tuning via coverage. In Advances in Neural Information Processing Systems (NcurlPS), 2024b. URL https: /awpiv..vvvvv/////////./////aavv org/ abs/ 2406 01462. Cited on pages 1, 5, 6, 7, and 18.
Touvron, H.. Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizen- stein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M.. Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. LLAMA 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Chris- tiano, P. F. Learning to summarize with human Larochelle, H., Ranzato, M., Had- sell, R.. Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, vol- ume 33, pp. 3008-3021. Curran Associates, Inc., 2020. URL htttps://proceeddnng neurips. co co/paper
feedback.
1189685d55692 118955566999d 0a0.2222222200000884444444455111
pdf. Cited on page 17.
Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schnei- der, J., Xie, T.. Ermon, S., Finn, C., and Ku- mar, A. Preference fine-tuning of LLMs should leverage suboptimal, on-policy data. In Forty- first International Conference on Machine Learning, 2024. URL https::///// /ttp/ppprrevieeeee net forum? 1d=bWNPx6tOsE Cited on pages and
2023. URL abs wwtpaa////222/ 09288. Cited hhttps: : /tt///axiiv/ www.vvorr//gooo org/ wwwp////////////..oe on page 16.
Tunstall, L., Lambert, N., Rajani. N., Beeching, E.,, Le Scao, T., von Werra, L., Han, S., Schmid, P., and Rush, A. Creating coding assistant with starcoder. Hugging Face Blog, 2023. on page 16.
Cited
Talmor, A., Herzig, J., Lourie, N., and Berant, J. Com- monsenseQA: A question answering challenge tar- geting commonsense knowledge. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 4149-4158, 2019. doi: 10.18653/v1/N19 1421. URL https:
Wagenmaker, A. and Pacchiano, A. Leveraging offline data in online reinforcement learning. In International Confer- ence on Machine Learning (ICML), CC..' JMLR.org, 2023. Cited on page
14

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
Wang, Z., Dong. Y., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J.,, Sreedhar, M. N., and Kuchaiev, O. Help- steer2: Open-source dataset for training top-performing reward models, 2024. Cited on pages 3, 5, 6, 16, and 18.
Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Pro- ceedings of the 41st International Conference on Ma- chine Learning. volume 235 of Proceedings of Ma- chine Learning Research, pp. 34983-54998 PMLR, 21- 27 Jul 2024c. URL https: http//ooceeeeieee ngs .mlr. press/ htm4h. Cited on pages and
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurIPS), 35: 24824-24837, 2022. URL https:/ http://axxivvoooooe abs/2201. 11903. Cited on page 1.
Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language mod- els, 2024. URL ht https: /ttxxaxxvv.../////////////aaaav/ www.v.orgg//oooooo.oo...oooooo //ggaabl abs//4b00 10020. Cited on page 18.
Zellers, R., Holtzman, A., Bisk. Y., Farhadi, A., and Choi, Y. HellaSwag: Can machine really finish your sentence? In ACL, 2019. Cited on pages 5 and 17.
Wei, Z., Liu, Y., and Erichson, N. B. Emoji attack: A method for misleading judge lims in safety risk detec- tion, 2024. URL https:/ / www.ii.org/bb//24//// 01077. Cited on page? 7.
Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. Wildchat: 1m chatGPT interaction logs in the wild. In The Twelfth International Conference Representations, 2024. URL https: /ttp:///ppeeeeieeeeeeeeee net / forum? wtccummiiiiiie 1d= Bu72R1bM. Cited on page 16.
Xie, T., Foster, D. Krishnamurthy, A., Rosset, C., Awadal- lah, A., and Rakhlin, A. Exploratory preference op- timization: Harnessing implicit appppoooinitio for sample-efficient rlhf, 2024. URL https: arxiv.. org/abs/ 2405. 21 21046. 6. Cited on page 3.
Learning
Zheng, L., Chiang, W.-L., Sheng. Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z.,, Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https: htt////xxvv.orr/ aba/ 2306. 05685. Cited on page 8
Xiong, W., Dong, H., Ye, C., Wang, Z.,, Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In International Conference on Machine Learning (ICML), 2024. Cited on pages 3, 6, 7, and 18.
Zhou, J. Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-folloving evaluation for large language models. arXiv preprint arXi::2311.0791 2023. Cited on pages and 17.
Xu, C., Sun, Q. Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. WizardLM: empowering large language models to follow complex instructions. arXiv preprint 2rXiv:2304.12222 arxiv. abs abs/ /2004 12244. Cited page 16.
2023. URL https: //
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., Murray, K., and Kim, Y. Contrastive preference op- timization: Pushing the boundaries of LLM performance in machine translation. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41 Interna- tional Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 55204- 55224. PMLR, 21-27 Jul 2024a. Cited on page 17.
Xu, J., Lee, A., Sukhbaatar, S.. and Weston, J. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. In International Conference on Machine Learning (ICML), 2024b. Cited on page 18.
Xu, S., Fu, W., Gao, J., Ye, W. Liu, W. Mei, Z., Wang, G., Yu, C., and Wu, Y. Is DPO superior to PPO for LLM alignment? A comprehensive study. In Salakhutdinov, R., Kolter, Z.,, Heller, K., Weller, A.,
15

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
Supplemental Material
A. plperparraeeees
For all of our experiments, we use OpenRLHF (Hu et al., 2024) for training and dn-evaluation-hamees (Gao et al., 2024) for evaluation. Initially, we performed hyperparameter sweeps for {0.01, 0.5, 0.1, 1, 1,5 and max learning rate {5e 7, le 6, 5e 6] for initial exploration for DPO, We report result on 0.1 and max learning rate = for all of our experiments.
B. Alpaca Eval 2.0 Prompt Categories
We define the category of prompts as the same as WildChat (Zhao et al., 2024), which contains in total 16 categories. We prompt gpt pppt-4omm 1oot-44omin for classifying prompts into one of the 16 categories. We report the distribution of 805 prompts in Alpaca Eval 2.0 (Dubois et al., 2023; 2024) in Figure 7.
factual information (general professional), history practices
26. 099
recommendationn
18.14%
assisting or creative writing
7.08%
analysis decision explanation
7.08%
tips, opinions or advice
5.71%
informaionn summarization
5.59%
Topics
matherraiica reasoning calculation
4.35%
awareness ethics and other risks
3.23%
editing rewritingg
2.98%
releplay
1.125
logical measoning
3.87%
dassification
3.87%
other please continue them lother: [category nama))
6.6%
linguistics
0. 62%
multilinuaaiity tranclation
0.50%
25
100
150
175
Figure Distribution different categories of prompts in Alpaca Eval (Dubois et al., 2023; 2024).
C. Dataset Details
Here describe the datasets we used in $4: Ultrafeedback (Cui al., 2024) and HelpSteer2 (Wang al., 2024) in detail:
Ultrafeedback (Cui et al., 2024): Prompts included questions in TruthfulQA (Lin al., 2021), FalseQA (Hu et al., 2023), Evol-Instruct (Xu et al., 2023), UltraChat (Ding al., 2023), ShareGPT (Chiang al., 2024b), and the FLAN (Longpre et al., 2023) collection; Responses are sampled from model pool with 17 models: GPT-4, gpt-3,5-turbo, Bard, UltraL 1EELlMM1BBAA 13B/65B (Ding et al., 2023), Vicuna-33B-vI. 1.3 (Ding et al., 2023), LLaMA2-7B/13BB700B -aattt (Touvron et al., 2023), Alpaca-7B (Taori et al., 2023), MPT-30B-CCCLEC Chat (Team, 2023), Bercon-4OB-Instteee (Almazrouei et al., 2023), StarChat (Tunstall et al., 2023), and Pythia- 12B (Biderman al., 2023).
(Xu et al., 2023),
HelpSteer2 (Wang et al., 2024): Prompts mainly (over 95%) are sampled from ShareGPT (Chiang et al., 2024b), with a small portion from proprietary user prompts; Responses are sampled from Nemotron-2/1 2/3/4 (Nvidia et al., 2024).
16

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
Mixtral-8x7B-Insstrutt--) (Jiang et al., 2023), and human annotators.
D. Evaluation Details
We use the In-eval-harnen (Gao et al., 2024)* library for evaluation on all 8 benchmarks (ARC-Challenge, Com- monsense QA, Hellaswag, MMLU, OpenQA, PiQA, Winograde, and Ifeval).
The details of the benchmarks are listed below:
ARC-Challenge (Clark et al., 2018): A set of multiple-choice questions focusing on reasoning and scientific knowledge across range of domains.
Commonsense QA (Talmor et al., 2019): A benchmark designed to evaluate a model's ability to answer questions based on commonsense knowledge, requiring reasoning beyond factual recall.
Hellaswag (Zellers et al., 2019): A benchmark focused on commonsense reasoning and narrative understanding, where models predict the most plausible continuation of an incomplete story.
MMLU (Hendrycks et al., 2020): The Massive Multitask Language Understanding benchmark evaluates models diverse set of tasks, including STEM, humanities, and social sciences, measuring performance human-level tasks.
OpenQA (Mihaylov et al., 2018b): A benchmark for evaluating models on open-domain question answering, testing how well they retrieve and reason over information to answer diverse queries.
PiQA (Bisk et al., 2020): A benchmark focused on physical commonsense reasoning, requiring models to predict the most plausible solution to problems involving physical world interactions.
Winograde (Sakaguchi et al., 2020): A benchmark that assesses models' ability to perform common-sense reasoning in co-reference tasks, where ambiguous pronouns resolved using contextual clues.
Ifeval (Zhou et al., 2023): benchmark designed to assess models on their ability to follow precise instructions that are verifiable.
In Alpaca Eval 2.0, for reproducibility, we use greedy decoding with max length of 2048 tokens.
E. Full Results
Table 4 and 5 shows the full results on all 8 benchmarks: Commonsense QA (CQA; (Talmor et al., 2019), Hellaswag (HS; (Zellers et al., 2019), Openbook QA (OQA; (Mihaylov et al., 2018a), PIQA (Bisk et al., 2020), Winograde (Sakaguchi al., 2020), ARC- -haalhelee (ARC-C; (Clark al., 2018)), MMLU (Hendrycks al., 2020), and Ifeval (Zhou et al., 2023). respectively.
F. Additional Related Works
Language Model Alignment Alignment of language models is typically done at the post-training stage, in order to make the language model prefer certain types of responses. In our paper, we specifically refer the stage after supervised fine-tuning (SFT) as "alignment". In this stage, two paradigms are typically employed: In Reinforcement Learning with Human Feedback (RLHF; (Stiennon et al., 2020; Ouyang et al., 2022)), an on-policy alignment method assigns a reward to on-policy rollouts. Such reward can be of separate reward model trained on off- off- policy generations (Ouyang et al., 2022), hand-crafted heuristics (Bai et al., 2022b), or even Al feedback (Guo et al., 2024). The other paradigm does not require on-policy rollouts and directly manipulates the log probabilities of the offline chosen and rejected responses, hoping that the model generalizes from these offline responses to learn what principle it should follow. Direct Preference Optimization (DPO; (Rafailov et al., 2024)) is the most canonical one that aims to push the log-probs of chosen responses higher and the log-probs of rejected responses lower. Many variants of DPO have been proposed (Pal al., 2024; Meng et al., 2024; Azar al., 2024; Xu al., 2024a) but they mostly used off-policy examples. However, the contrastive nature of the DPO loss
17

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
ARC-C
CQA
HSwag
MMLU
OQA
PIQA
WG
Ifeval
Avg. Score
-lama-3.1-8B-
51.90
77.14
59.14
67.93
33.40
79.97
73.55
44.00
60.88
Experiments on UltraFeedback (Cui et al., 2024)
Off-policy DPO
53.07
77.40
59.33
68.15
34.40
80.47
74.11
46.40
61.67
On-policy DPO
54.18
77.81
59.88
68.60
34.80
80.41
74.98
46.03
62.09
HyPO (Song et al., 2024b)
54.86
77.64
60.09
68.20
35.40
80.63
74.59
44.60
62.00
DPO-Mix-P (Shi et al., 2024)
53.07
77.89
59.67
68.37
34.00
80.20
74.03
45.66
61.61
SIMPLEMIX
55.03
77.81
61.08
69.47
35.40
81.69
75.00
48.98
63.06
Experiments on HelpSteer2
(Wang et al., 2024)
Off-policy DPO
51.79
76.99
59.22
68.11
33.60
80.09
73.88
48.61
61.54
On-policy DPO
53.58
77.48
59.63
68.45
35.00
80.47
74.11
45.47
61.77
HyPO
53.50
76.82
59.37
68.32
34.00
80.30
73.88
47.32
61.69
DPO-Mix-P SIMPLEMIX
53.49
77.31
59.55
68.38
34.60
79.86
74.34
44.36
61.49
54.27
78.05
59.76
68.39
34.60
80.20
74.51
47.21
62.12
Table 4. Detailed Benchmark results (ARC Challenge, Common QA, Hellaswag, MMLU, Openbook QA, PIQA, Winograde, Ifeval) on
training L 1 a0laamm....m --str-nnttuuct (Dubey al., 2024) on the Ultrafcedback (Cui al., 2024) and HelpStoer2 (Wang et al., 2024)
dataset.
Model
ARC-C
CQA
HSwag
MMLU
OQA
PiQA
WG
Ifeval
Avg. Score
Llama - 3-1--ulu------- 8B-8B--- 88-88BTTTTTT
52.73
75.76
61.87
63.67
36.80
80.79
74.35
31.79
59.72
Experiments on UltraFeddback (Cui et al., 2024)
Off-policy DPO
55.20
76.49
62.94
58.77
37.60
80.96
74.19
33.97
60.02
On-policy DPO
54.18
76.17
62.41
58.70
37.60
80.85
74.27
33.40
59.70
HyPO (Song et al., 2024b)
55.15
76.33
61.91
62.17
37.40
80.19
73.95
33.10
60.03
DPO-Mix-P (Shi et al., 2024)
56.32
76.02
63.01
64.21
39.50
80.36
73.11
33.10
60.70
SIMPLEMIX
56.06
76.82
63.13
63.60
38.60
81.23
74.03
35.62
61.14
Experiments on HelpSteer2 (Wang et al., 2024)
Off-policy DPO
52.90
76.09
61.83
63.68
37.00
80.79
73.80
33.46
59.94
On-policy DPO
51.90
75.92
61.88
63.77
37.00
80.74
74.03
36.41
60.21
HyPO (Song et al., 2024b)
52.65
76.33
61.85
63.90
36.40
80.63
74.43
33.46
59.96
DPO-Mix-P (Shi et al., 2024)
52.55
76.26
62.11
63.71
36.20
80.19
74.17
35.10
60.04
SIMPLEMIX
52.90
76.09
63.84
63.84
37.40
80.74
74.11
40.85
61.22
Table Detailed Benchmark results (ARC Challenge, Common QA. Hellaswag. MMLU, Openbook QA. PIQA. Winograde, Ifeval) on training Ll Llama-3 T-lu--uu------- (Lambert et al., 2024a) on the Ultrafeedback (Cui et al., 2024) and HelpSteer2 (Wang et al., 2024).
can also take on-policy examples. Therefore, many on-policy variants of DPO have also been proposed (Xu et al., 2024b; Yuan et al., 2024; Xiong et al., 2024; Guo et al., 2024). Furthermore, there are also off off-policy works that are non-contrastive (Ethayarajh et al., 2024). Our paper directly studies the difference ($3) and interaction ($4) of vs. off-policy data, disentangling the effect of the contrastive nature of DPO and the non-contrastive nature of standard RLHE.
G. Per-task Performance of SIMPLEMIX
18

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
meta-llama/Laamas 10mm-3.1-88-Instruce
allenavillaa-3.111
22.81
oft-palicy DPC
policy
en-policy DPO
On-policy DPO
35,09
mpleeMx
turbo)
20
30
27:14
27.13
15-
14:11
lagainst
15-
22.86
17.14
15-
Vitn Rote:
10-
5:26
5.71
2.86 3:29
2.63
math
creative
mathh
cocing Categories Alpaca nommmendation
Categortes
Figure 8. Comparison of win rates (against GPT-4-trbo) across different prompt categories in Alpaca Eval 2.0 by training LLama-3.2 1aama-3.1--8 -B-IIBntruueeeeeee and TuLu-3-B8 3-BB-SFI with SIMPLEMIX. SIMPLEMIX offers a good balance between reasoning tasks and open-ended tasks.
H. Diverse Prompt
Figure shows the prompt we used to elicit the language model to generate a response different from previous responses.
Please generate another response that different from all the previous ones. can differ in aspects such as but not limited to tone, formality, strategy, judgment, conciseness, structure. It can also involve asking for clarity the input prompt unclear.
Figure 9, The prompt used to elicit diverse responses from language model.
Full Reward Distributions
Figure 10 shows the aggregated histogram of reward of the responses sampled from Llama-3. 1-88--nntttrre ruct with sampling temperature = {0.7, 3} and prompting it to generate diverse responses (Prompting). Figure 11 shows the individual histograms.
19

---

SIMPLEMIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning
metd:llamm//lam--11118-----n
temp 2.0 -10.511
5000
temp 3.0 (Avg: 10,777
Prompting (Aug:
temp 0.7 (baselina) (Avg: 3339)
4000
3000
2000
Reward
Figure 10. Histogram of reward distribution for different sampling temperatures. Increasing the temperature results in generations of lower quality, as measured by our reward model.
Iamm Reward butions
Hemp = 2.0
sempp 3.0
Frompe Pn Juernge
triro 0.7 [bacelin)
5300
4331
4000
12.0
3000
3000
1259
122
Figure 11. Histograms of the reward for sampling from Llama 1...-..1111.1-- -B-Insuuu with different sampling schemes (prompting and increasing the lemperature).
20


Reviews:
[{"review_id": "6NB3t8yMKr", "reviewer": "Reviewer_9f8w", "summary": "The paper investigates the interplay between on-policy and off-policy preference data in aligning large language models (LLMs) with human preferences. It presents the key finding that on-policy data is more effective for reasoning tasks (e.g., math, coding), whereas off-policy data performs better in open-ended tasks (e.g., creative writing, recommendations). Based on this observation, the authors propose SIMPLEMIX, a method that combines both data sources in a straightforward manner.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful comments and suggestions. We are thankful that the reviewer finds our method “straightforward but effective” and our results “demonstrate clear trends”. We hope to resolve the concerns below:\n\n> Benchmarks like Alpaca Eval 2.0 and Ifeval involve LLM-based evaluation, which can be biased or manipulated. Including more human evaluation results (e.g., actual user studies) would add credibility to the findings.\n\nIt is possible there may be some misunderstanding of the scope of our evaluation. In our work, we have chosen to report scores of **nine** benchmarks, only one of them is LLM-based evaluation (Alpaca Eval 2.0). The other 8 benchmarks include but are not limited to world knowledge acquisition (MMLU), commonsense reasoning (Hellaswag), precise instruction following (IFEval), open-domain question answering (OpenQA). We believe that this is a practical and comprehensive setting since the same benchmarks are also adopted by FineWeb [6] in pre-training data curation. Detailed descriptions of our evaluation benchmarks can be found at Appendix D. \nWe acknowledge that adding real user studies would improve the credibility of our work. Since the cost of high quality human annotators is prohibitively large, we opted to use Alpaca Eval 2.0 due to it having a 0.95 correlation with real human annotators while only having 0.8% of the cost according to [7]. \n\n> The paper lacks a theoretical foundation to explain why SIMPLEMIX works better than other hybrid approaches.\n\nThere exists a mismatch between theory of existing works and practice in LM alignment, and we conjecture that might be the reason why SIMPLEMIX works better:\n\n- HyPO [1] proposes to perform an off-policy DPO while using on-policy data to minimize the KL divergence between the current policy ($\\pi_\\theta$) and the reference policy ($\\pi_\\text{SFT}$). The motivating assumption for adding the additional KL regularization is that the “DPO implicit reward” is unbounded and can lead to infinite KL. In practice, the literature has witnessed the *opposite* trend [2, 3, 4] where KL regularization might not be necessary for LM alignment (because of the reference model’s unreliability, leading to unreliable KL values). Our work combines on- and off-policy data *without explicit regularization* of the KL compared to HyPO, removing the strong KL regularization in HyPO might be a reason that SimpleMix works better.\n\n- DPO-Mix-P [5] samples from an interpolation between $\\pi_\\theta$ (the current policy) and $\\pi_\\text{SFT}$ (reference policy). The theoretical foundation of DPO-Mix-P is designed for faster convergence in terms of DPO loss (achieving a lower DPO loss for less iterations). In practice, the recent works have shown that lower DPO loss doesn’t always go hand-in-hand with better alignment (“Alignment Gap” [6]). Therefore, achieving a lower DPO loss, or equivalently, a higher ranking accuracy, might not contribute to a better aligned model.\n\n> While the study identifies task-dependent benefits of on- and off-policy data, it does not explore task-specific weighting or adaptation. A potential extension could involve dynamically adjusting the data mixture ratio based on task type.\n\nIn section 3, we have shown the complementariness of on- and off-policy data on carefully selected subtopics in Alpaca Eval 2.0. However, in reality, user prompts rarely fall into a single category as many queries require both understanding the user’s personal preference, but also following verifiable constraints. For example, the query from UltraFeedback “explain cache keys to me like im 5” requires both objective knowledge about cache keys and adjustments of tone and style. Therefore, we decided to be safe and only combine two sources of data to achieve a balanced performance across all types of tasks. We agree with the reviewer that algorithmic adjustment of the sampling method (on- or off-policy) based on the prompt category would be a promising future direction.\n\nReferences:\n\n[1] The Importance of Online Data: Understanding Preference Fine-tuning via Coverage (Song et al., NeurIPS 2024)\n\n[2] Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation (Xu et al., ICML 2024)\n\n[3] SimPO: Simple Preference Optimization with a Reference-Free Reward (Meng et al. NeurIPS 2024)\n\n[4] DAPO: An Open-Source LLM Reinforcement Learning System at Scale (Yu et al., 2025)\n\n[5] The Crucial Role of Samplers in Online Direct Preference Optimization (Shi et al., ICLR 2025)\n\n[6] Preference Learning Algorithms Do Not Learn Preference Rankings (Chen et al., NeurIPS 2024)\n\n[7] FineWeb: decanting the web for the finest text data at scale (Penedo et al., 2024)\n\n[8] MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (Ni et al., NeurIPS 2024)", "claims_and_evidence": "Most claims are well-supported.", "methods_and_evaluation_criteria": "Yes, but relies heavily on LLM-based evaluation; lacks human validation.", "theoretical_claims": "No formal proofs provided; lacks theoretical justification for SIMPLEMIX effectiveness.", "experimental_designs_or_analyses": "Yes, but it lacks human evaluation.", "relation_to_broader_scientific_literature": "Builds on preference learning, hybrid RL, and DPO.", "essential_references_not_discussed": "Lacks discussion on recent hybrid RL preference optimization and adaptive weighting.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. The proposed SIMPLEMIX method is a straightforward but effective way to improve language model alignment without additional computational overhead.\n\n2. The results are statistically significant and demonstrate clear trends in performance across different tasks.\n\n**Weaknesses:**\n\n1. The paper lacks a theoretical foundation to explain why SIMPLEMIX works better than other hybrid approaches.\n\n2. Benchmarks like Alpaca Eval 2.0 and Ifeval involve LLM-based evaluation, which can be biased or manipulated. Including more human evaluation results (e.g., actual user studies) would add credibility to the findings.\n\n\n3. While the study identifies task-dependent benefits of on- and off-policy data, it does not explore task-specific weighting or adaptation. A potential extension could involve dynamically adjusting the data mixture ratio based on task type.", "other_comments_or_suggestions": "See Strengths And Weaknesses Part.", "questions_for_authors": "See Strengths And Weaknesses Part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LoBfY1yT84", "reviewer": "Reviewer_KSRd", "summary": "This paper studies the effect of mixing on-policy and off-policy data when fine-tuning language models using direct preference optimization (DPO). They observe that on-policy data (data generated by the current policy) tends to work better for tasks with clear correct answers, like math and coding, whereas off-policy data (generated by other pre-trained models) is better for open-ended tasks, such as creative writing or recommendations. They propose a simple method called SIMPLEMIX, where on-policy and off-policy data are mixed in equal proportions. Their main findings show SIMPLEMIX consistently improves performance compared to using either data source alone, and it also beats more complicated methods like HyPO and DPO-Mix-P, according to evaluations on AlpacaEval 2.0 and other benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review and we would like to extend our gratitude for the reviewer finding our method straightforward and our experiment results clear. We hope to answer the questions and resolve the concerns of the reviewers below:\n\n> The idea itself isn’t very novel—just mixing two data sources is fairly standard.\n\n - **On its novelty:** Our contribution is not limited to mixing of the two data sources, our work also 1) shows that the two data sources are complimentary (Section 3) - which provides a possible explanation for the performance improvement; 2) conduct detailed ablations (Section 5) on data mixtures and other data curation strategies. **To the best of our knowledge, our work is the first to carefully study the interplay between on- and off-policy data in preference optimization.**\n\n - **On simplicity:** While we acknowledge that the method is simple, we believe this is our strength. SimpleMix does not introduce additional hyper-parameters. To put it in reviewer `9f8w`’s words, “SIMPLEMIX method is a straightforward but effective way to improve language model alignment without additional computational overhead.”\n\n> The reported performance improvements are modest, raising doubts about real-world significance.\n\n - **Improvements**: While the reported performance improvements may appear modest, they are averaged across **eight** benchmarks. As reviewer `fktC` denotes “The improvement is modest, but the consistency makes a convincing case.” \n - **Real-world significance** Tulu 3 [1] is a concurrent effort that is similar to our setting and demonstrates a 3-point average improvement across benchmarks. This is a meaningful gain, especially considering that mixing the two data sources incurs minimal cost. As Reviewer `fktC` denotes, our method “has a strong potential to be adopted in practice.”\n\n> Can you explain why Tulu specifically benefits more from SIMPLEMIX? Is there something specific about Tulu’s training or data that makes mixing data sources especially useful here?\n> In Figure 8, SIMPLEMIX shows very little improvement or sometimes no improvement over the baseline (pure on-policy or off-policy data) when applied to the LLaMA instruct-tuned model across all four task categories (math, coding, creative writing, recommendation). This again relates the Q1 above: it seems that mixing is not useful when the model is already aligned?\n\nWe aggregated the two questions here: we conjecture that Tulu-8B-SFT have not gone through a preference optimization process while Llama-3.1-8B-Instruct have already gone through extensive SFT and DPO training, thus making it easier to improve Tulu’s performance compared to improving Llama-3.1-8B-Instruct’s performance. \n\nReferences: \n[1] Tulu 3: Pushing Frontiers in Open Language Model Post-Training (Lambert et al., 2024)", "claims_and_evidence": "The authors claim that mixing on-policy and off-policy data improves alignment performance compared to just using one data source alone. Their main results (Table 2) support this claim, showing SIMPLEMIX outperforms methods like DPO-Mix-P and HyPO. The actual improvement in the length-controlled win rate on AlpacaEval 2.0 is modest (e.g., from around 28 to 30 in Figure 4). Moreover, the authors show that, if off-policy data is curated, the performance of SIMPLEMIX is further improved. They show this on Alpaca Eval 2.0 as well (Figure 6).", "methods_and_evaluation_criteria": "The method itself (mixing responses from on-policy and off-policy sources equally) is very straightforward. They evaluate using standard benchmarks like AlpacaEval 2.0 and FineWeb, which makes sense for the alignment task. These benchmarks clearly separate reasoning tasks (math, coding) from subjective tasks (creative writing, recommendations). Overall, their choice of methods and evaluation criteria seems reasonable and appropriate for the problem they’re studying.", "theoretical_claims": "The paper does not include major new theoretical claims. They mostly rely on intuition and experimental results to justify the SIMPLEMIX idea. There aren't complicated theoretical proofs to verify.", "experimental_designs_or_analyses": "I checked their experimental designs briefly, especially Figure 4, where they compare various mixing ratios of on-policy and off-policy data. This analysis seems reasonable, clearly showing that a balanced mixture (around 50-50) is slightly better than using either purely on-policy or off-policy data. However, from the results in Figure 4, the performance difference between SIMPLEMIX and the two extremes (purely on/off-policy) is modest. The win-rate accuracy curves are close together, suggesting limited practical impact. Thus, while the analysis itself is sound, the paper doesn’t convincingly demonstrate that the added complexity of mixing datasets is justified by these marginal improvements.", "relation_to_broader_scientific_literature": "They relate their paper clearly to recent debates about using on-policy versus off-policy data in preference alignment, and they discuss recent similar works like HyPO and DPO-Mix-P. The idea of mixing on- and off-policy data isn't new, but clearly showing which tasks benefit from each type of data is a useful contribution. However, their contribution is mostly empirical, with no new theoretical advances.", "essential_references_not_discussed": "Nothing noteworthy to me.", "other_strengths_and_weaknesses": "Strength:\n\n1. SIMPLEMIX is straightforward and easy to implement.\n2. They provide clear experimental results that consistently support their claims.\n3. They clarify the conditions (objective vs. subjective tasks) under which on- or off-policy data might be more beneficial.\n\nWeaknesses:\n\n1. The idea itself isn’t very novel—just mixing two data sources is fairly standard.\n2.  The reported performance improvements are modest, raising doubts about real-world significance.", "other_comments_or_suggestions": "The paper presents a very simple method with clearly explained and straightforward experiments. Overall, the experiments are sound, clearly show modest but consistent improvements, and cover relevant datasets. The main limitation is that the theoretical contribution is minimal—it's mostly applying a known idea (mixing data sources) without significant new insights. The modest size of the observed performance gains also limits its practical significance.", "questions_for_authors": "1. In Figure 6 you show that SIMPLEMIX improves performance much more clearly for Tulu-3 SFT compared to LLaMA-3 Instruct-tuned models. Can you explain why Tulu specifically benefits more from SIMPLEMIX? Is there something specific about Tulu’s training or data that makes mixing data sources especially useful here?\n\n2. In Figure 8, SIMPLEMIX shows very little improvement or sometimes no improvement over the baseline (pure on-policy or off-policy data) when applied to the LLaMA instruct-tuned model across all four task categories (math, coding, creative writing, recommendation). This again relates the Q1 above: it seems that mixing is not useful when the model is already aligned? \n\nCould the authors share some insights on this? Thank you.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c9qXWoTIO1", "reviewer": "Reviewer_fktC", "summary": "Authors' first claim that on-policy data are most effective on tasks with objective answers, while off-policy data are most effective on open-ended tasks like creative writing. Analysis on AlpacaEval supports the claim. Then, authors propose a simple method of mixing on-policy and off-policy: just sample from two policies with equal probability: SimpleMix. SimpleMix method makes modest but consistent improvement over strong baselines. Authors also provide good ablations with different mixtures, filtering mechanisms, response diversity.\n\n## update after rebuttal\n\nThe new result with DNO further strengthens the benefit of simplicity. I increased my score accordingly.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful comments and suggestions. We appreciate that the reviewer finds our work simple and has a strong potential to be adopted in practice. We are also encouraged that the reviewer finds our paper “clearly written”.\n\n> It could've been nicer if it was discussed more prominently in earlier literature review sections. Currently it is only discussed in Section 5.1.\n\nWe thank the reviewer for relating our work to the broader RL literature. We will discuss our relation with existing work in RL that is not about language model alignment earlier in our updated manuscript, and extend our discussion in the appendix.\n\n> Methods like Direct Nash Optimization (Rosset et al 2024 https://arxiv.org/abs/2404.03715) do mix samples from online policy and offline policy together, hence SimpleMix is not the only paper which mixes online and offline policy samples. DNO could've been an interesting baseline, at least a non-iterative version.\n\nWe thank the reviewer for bringing up the interesting discussion on DNO. Although the authors of DNO defined it as “a batched on-policy algorithm” (page 9 in [1]), DNO selects the best and worst response from N on-policy generations and one off-policy generation from gpt-4-turbo to perform DPO, therefore it can be seen as an hybrid method. As suggested by the reviewer, we conduct an experiment comparing SimpleMix with one iteration of DNO on Tulu-3.1-8B-SFT, where we sample N = 4 generations and mix it with the off-policy “chosen” generation from Ultrafeedback, and select the best and worst response according to our oracle reward model. We report the results below:\n\n\n### Alpaca Eval 2.0\n\n| Model                    | LC    | WR    | Std Error | Length |\n|--------------------------|-------|-------|-----------|--------|\n| Tulu 3 SFT + DNO         | 16.22 | 14.15 | 1.29      | 1521   |\n| Tulu 3 SFT + SimpleMix   | 20.64 | 18.02 | 1.36      | 1474   |\n\nCompared to the setting in [1], we changed the number of on-policy generations (5 -> 4), the off-policy model (gpt-4-turbo -> the chosen response in ultrafeedback), and the oracle reward (gpt-4-turbo -> Skywork/Skywork-Reward-Gemma-2-27B)  to make the results comparable to our work. We will make sure to add discussions on DNO in our latest manuscript.\n\n[1] Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences (Rosset et al., 2024)", "claims_and_evidence": "Authors' first claim is that on-policy data are most effective on tasks with objective answers. This is backed by AlpacaEval 2.0, but it is a bit weak because AlpacaEval has very small samples on Math and Coding. Also, on-policy algorithms are only trained for a single epoch, which could've underestimated their performances. SimpleMix shows consistent benefit over reasonable baseline methods. The improvement is modest, but the consistency makes a convincing case.", "methods_and_evaluation_criteria": "I checked the experimental setup in Section 3.1. In terms of base model choices, two Llama 3.1-8B based SFT models are good choices. They are performant and represent best practices in the literature. The setup could've been improved with more diverse models, in particular more diverse sizes, although the computational constraint is understandable. AlpacaEval 2.0 is appropriate for this initial experimentation, but its math subset is pretty small (<50) hence I am not sure how much 2% ~ 5% difference in Figure 2 on math is meaningful. Not a major concern because the observation is consistent on Coding, but it could've been more convincing if bigger benchmarks were used for this analysis.\n\nI also checked the experimental setup in Section 4.1. Again base model choices are standard. UltraFeedback is also a good choice as it is well-established and covers various capabilities (for ex per Ivision et al https://arxiv.org/abs/2406.09279). Evaluation on AlpacaEval is good for overall conversation quality, but the rest of benchmarks is a bit too much focused on knowledge tasks, which don't move much from DPO training. I suggest a setup like https://arxiv.org/abs/2406.09279 which more broadly covers coding, safety, truthfulness, etc. Evaluating on coding and reasoning could've helped better validating authors' claim that on-policy performs better on reasoning and off-policy performs better on creative writing. Authors compare against a good set of baselines (HyPO and DPO-Mix-P).", "theoretical_claims": "The paper doesn't make theoretical claims, but make a good connection to theoretical works and compare against them Shi et al 2024, and Song et al 2025.", "experimental_designs_or_analyses": "As discussed in Methods And Evaluation Criteria, I checked experimental setups in Section 3.1 and 4.1. Issues are discussed in the section.\n\nIn addition, I have a concern that all algorithms are only run for a single epoch. I hypothesize that online policy algorithms benefit more from more epochs since new responses are generated for every epoch. Hence, authors' experiments can be underestimating the benefit of on-policy methods.", "relation_to_broader_scientific_literature": "The connection to symmetric sampling from Ball et al (2023) is interesting because the method is highly similar, hence establishes a connection to the broader reinforcement learning literature. It could've been nicer if it was discussed more prominently in earlier literature review sections. Currently it is only discussed in Section 5.1.", "essential_references_not_discussed": "Methods like Direct Nash Optimization (Rosset et al 2024 https://arxiv.org/abs/2404.03715) do mix samples from online policy and offline policy together, hence SimpleMix is not the only paper which mixes online and offline policy samples. DNO could've been an interesting baseline, at least a non-iterative version.", "other_strengths_and_weaknesses": "Proposed method is very simple, and hence has a strong potential to be adopted in practice.", "other_comments_or_suggestions": "Summary in line 080-094 is too much of a repeat from only a few paragraphs above. I suggest making the summary more concise so that it becomes less of a repeat.", "questions_for_authors": "Paper is clearly written, hence I don't have major questions which would change my evaluation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: My recommendation is to accept the paper.

The paper proposes a simple preference alignment approach that mixes on-policy and off-policy in equal parts in a DPO objective. The method follows from existing discussion in the literature about the relative strengths of on- and off-policy data, and significantly simplifies existing approaches that attempt to mix data from these distributions. The method shows significant improvement over standard baselines, and consistent (if small) improvement over more complex but "mixed" baselines.

There was strong disagreement among reviewers about this paper. The key disagreements came down to (1) the necessity of human evaluation, and (2) necessity for some theoretical justification.

One reviewer made a strong case that human evaluation is important for the class of tasks that are more subjective, while another noted that much progress has been made on preference alignment (even on subjective criteria) using automatic evaluation. While I am sympathetic to the idea that human evaluation is ultimately necessary, I think the primary object of study here is the optimization algorithm itself, so I think it is valid to perform experiments with a proxy, although human evaluation or replication on multiple diverse proxy reward models could make the case more strongly.

Regarding theoretical justification, I agree that it would be useful for the literature to have more theoretical threads to pull on here, but I think that the main contribution of this work is to show that something simple and easily implemented does work. The algorithm itself is immediately useful, and there is room for subsequent work to take this observation and work out the science.

I would encourage the authors to address these issues directly in the text of the paper.
Decision: Accept (poster)