DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learning
Authors: Yifan Wang
Bolian Li
Junlin Wu
Zhaoxuan Tan
Zheli Liu
Ruqi Zhang
Ananth Grama
Qingkai Zeng
Venue: ICLR 2026 Poster (2026)
Keywords: large language models
preference learning
user feedback
post-training
self-improvement
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Real-world large language model deployments (e.g., conversational AI systems, code generation assistants) naturally generate abundant implicit user dissatisfaction (DSAT) signals, as users iterate toward better answers through refinements, corrections, and expressed preferences, while explicit satisfaction (SAT) feedback is scarce. Existing preference learning approaches are poorly aligned with this data profile, as they rely on costly human annotations or assume plentiful positive responses. In this paper, we introduce \textbf{DRIFT} (\textbf{D}issatisfaction-\textbf{R}efined \textbf{I}terative pre\textbf{F}erence \textbf{T}raining), which anchors training on real-world DSAT signals and samples positives dynamically from the evolving policy. Empirically, DRIFT models trained on real-world \textit{WildFeedback} datasets and synthetic \textit{UltraFeedback} datasets achieve up to +6.23\% (7B) / +7.61\% (14B) on WildBench Task Score and up to +8.95\% (7B) / +12.29\% (14B) on AlpacaEval2 win rate over base models, outperforming strong baseline methods such as iterative DPO and SPIN. At larger scales, the improvements are particularly pronounced: 14B models trained with DRIFT surpass GPT-4o-mini on WildBench. Further analysis shows that DRIFT also preserves exploratory capacity, yielding more diverse high-reward solutions rather than collapsing to narrow subsets. Theoretically, we demonstrate that this design preserves preference margins and avoids the gradient degeneration. These results show that DRIFT is an effective and scalable recipe for real-world post-training that leverages the most abundant and informative signal.

Published as a conference paper at ICLR 2026
DRIFT: LEARNING FROM ABUNDANT USER DISSAT- ISFACTION IN REAL-WORDD PREFERENCE LEARNING
Yifan Wang', Bolian Li', Junlin Wu², Zhaoxuan Tan Zheli Liu", Ruqi Zhang',
Ananth Grama', Qingkai Zeng**
"Department of Computer Science, Purdue University
2 Department of Computer Science and Engineering, Washington University in St. Louis
3 DDpartment of Computer Science and Engineering, University of Notre Dame
*College of Computer Science, Nankai University
(wang5617, 114468, ruqiz, 2 a03dnd. edu, liuzhell iennliiinnnkiii edu, en, qzengnkca com
@epprduueedee junlin, wu@wustl edu
ABSTRACT
Real-world large language model deployments (e-g., conversational Al systems, code generation assistants) naturally generate abundant implicit user dissatisfac- tion (DSAT) signals, users iterate toward better answers through refinements, corrections, and expressed preferences, while explicit satisfaction (SAT) feedback is scarce. Existing preference learning approaches are poorly aligned with this data profile, as they rely on costly human annotations or assume plentiful positive Iterative
responses. In this paper, we introduce DRIFT (Dissatisfaction-Reeif preFerence Training), which anchors training on real-world DSAT signals and samples positives dynamically from the evolving policy. Empirically, DRIFT models trained on real-world WildFeedback datasets and synthetic U'traaeedbbcc datasets achieve up to +6.23% (7B) 17.611 % (14B) on WildBench Task Score and up to +8.95% (7B) / +12.99% (14B) on AlpacaEval2 win rate over base models, outperforming strong baseline methods such as iterative DPO and SPIN. At larger scales, the improvements are partieularly pronounced: 14B models trained with DRIFT surpass GPT--0-mini on WildBench. Further analysis shows that DRIFT also preserves exploratory capacity, yielding more diverse high-reward solutions rather than collapsing to narrow subsets. Theoretically, we demonstrate that this design preserves preference margins and avoids the gradient degeneration. These results show that DRIFT is an effective and scalable recipe for real-world post- training that leverages the most abundant and informative signal. Code and data are available at httpp::: / thub. htt.pp////ggtth.b..cooocccaaaa// wwwp:/caaaaaa//eeaaccccc.
INTRODUCTION
Large language models (LLMs) now power wide range of real-world applications, including con- versational assistants (e.g., GPT, Claude, Gemini), customer support, search and recommendation, productivity and education tools, and code generation. A key driver of this success is preference learning, a critical component post-training that aligns model behavior with human judgments and values. Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al. 2022) pioneered this approach by training a reward model on human preference data and subsequently optimizing the policy using reinforcement learning algorithms (Schulman al. 2017). Direct Preference Opti- mization (DPO) (Rafailov et al.) 2023) simplified this process by directly optimizing on preference pairs without requiring an explicit reward model, making the training procedure more stable and computationally efficient, while achieving comparable alignment performance.
However, these approaches depend costly, carefully curated human preference annotations that are difficult to scale across domains and evolving user needs. In contrast, deployed LLM systems continuously generate vast amounts of real-world interaction data. Beyond offering scalability, such
"Corresponding author

---

Published as a conference paper at ICLR 2026
Explicit User Feedback
Implicit User Feedback
DRIFT
Compare responses
[s] Diseatiefied Example Wildfeddback dataset)
Preference Training chosen
Which response you prefer?
currency
rejected
DSAT response
posiove resporse
resporse A
resporse
The current currency Croatia the Creatian kuna (HRK).
Wrong
happoned 2023
with user feedback
DSAT Chat history
sample
Thumbs
Satisfied Example (5% WildFoedback dataset)
148 Beats Cutting Edge Models
currancy
The of Croatia Euro.
1-3% provide feedback
Thanks!
SAT User Feedbook
Sparse, Biased
Abundant, Informative
Learning from DSAT
Figure I: Overview of user feedback signals and the DRIFT framework. Explicit feedback (left) sparse and biased, as most users are passive consumers. In contrast, implicit feedback (middle) provides abundant and informative signals, where dissatisfaction (DSAT) is far more prevalent than satisfaction (SAT) (e.g., 12% vs 5% in the WildFeedback dataset). DRIFT (right) leverages these DSAT signals for preference learning, enabling our 14B model to surpass commercial models.
real data often captures richer and more nuanced spectrum of human preferences than small, cu- rated annotation datasets, as users naturally convey satisfaction, dissatisfaction, and refinement in- tents during conversation. This motivates a key question:
How can we transform abundant but implicit user feedback from real-world inter- actions into scalable and effective preference learning signals for LLMs?
From a data collection perspective, existing chatbot platform as shown in Figure I (left) attempts to gather user feedback through explicit mechanisms: (I) asking users to compare and rank multiple model responses, or (2) providing simple feedback buttons (e.g. thumbs up/down) at the end of chat interfaces. However, these collection methods are inefficient, as most are passive consumers (Louna- maa 2024). with only 1-3% users willing to provide explicit feedback. Moreover, those who do provide feedback often express extreme opinions (strongly positive or negative) that may not reflect the broader distribution of user preferences. However, as illustrated by the example in Figure D (middle) users naturally express their preferences through the conversation itself through follow-up questions, correction requests, and iterative refinements, creating a rich source of implicit feedback. Beyond scalability, such real interaction data can contain richer and more representative preference information than curated annotation datasets, capturing fine- tne-grained user intents that explicit labels often miss. Recent datasets such as WildChat- 1M (Zhao al., (2244) and L.MSYS--Chat IM (Zheng et al. 2024) have collected over one million real-world conversations, creating rich foundation for studying naturally occurring user feedback. Building on these resources, several studies have ex- plored ways to extract preference signals directly from user interactions. For example. Don- Ychiya ct al. (2025) demonstrate that naturally occurring user feedback appears in approximately 30% of conversations and propose mining preferences by detecting explicit evaluative user responses. Simi- larly, WildFeedback (Shi et al. (025) applies user satisfaction estimation (Lin et al. (2024)) to auto- matically extract satisfaction and dissatisfaction labels construct large-scale preference datasets.
From a preference learning method perspective, recent works explored self-generated strategies to reduce reliance on human annotation. Self-Rewarding Language Models (Yuan et al. 2024) prompt the training model itself to score its own rollouts, but face a key limitation: the synchronized im- provement of chosen and rejected responses progressively reduces their contrast, which in turn un- dermines effective preference learning (Wang et al. (202a). An alternative approach, SPIN (Chen) al. 2024c) treats ground-truth responses from the SFT dataset as chosen and self-geenrated ones as rejected, yet is difficult to apply in practice where gold-standard responses are often rare, limiting its ability to generalize to broader scenarios. In contrast to positive feedback, dissatisfaction signals are naturally abundant as users refine suboptimal outputs through interaction. To leverage this un- derutilized signal, we introduce DRIFT (Dissatisfaction -essatiiiationnRReiined Iterative preFerence Training), a simple yet scalable method that directly leverages user dissatisfaction (DSAT) signals from authen- tic conversations to iteratively enhance model performance. Unlike SPIN, which fixes supervised responses positives and treats self-generated ones negatives, DRIFT anchors each training
2

---

Published as a conference paper at ICLR 2026
pair with real DSAT negative and samples the chosen responses from the current policy, enabling dynamic and policy-aligned adaptation. Our contributions are:
Empirical Validation: DRIFT consistently surpasses other iterative self-improving methods, SPIN and IterDPO, yielding gains of up to +6.23% (7B) / +7.61% 61% (14B) in WildBench Task Score and up to +8.95% (7B) +12.29% (14B) AlpacaEval2 win rate over base models.
Enhanced Exploration: DRIFT preserves a larger exploration space and generates more diverse responses with substantially broader coverage of the global high-reward region.
Theoretical A Analysis: We show that DRIFT maintains non-vanishing expected preference mar- gin and prevents gradient collapse, which is a critical limitation in existing self-improving models.
2
RELATED WORK
2.1 LEARNING FROM REAL-WORLD USER FEEDBACK
Since relying human labeling is not only expensive and time-consuming, but also highly sub- jective to a small set of annotators, recent work has shifted toward leveraging naturally occurring signals "in the wild". A natural starting point is to incorporate the most explicit feedback of user input, including edits and demonstrations. [ao et al. (2024) use edits in writing assistant settings to infer latent preferences, keeping the base LLM frozen and training separate preference mod- ule that conditions future outputs. Similarly, Shaikh et al. (025); Tuucker et al. (2024) rely on a handful of user- eser-provided demonstrations to bootstrap alignment, iteratively generating comparison pairs by treating user examples as preferred over LLM outputs and their intermediate revisions. Another stream of work draws on implicit feedback that emerges naturally during conversations. Hancock et al. (2019) introduced self-feeding chatbot that monitors user satisfaction during de- ployment: satishidd turns are added as new training data, while explicit feedback requested when dissatisfaction is detected. Liu al. (2025) extended this idea, regenerating improved responses for dissatisfaction and applying them in supervised fine-tuning (SFT). While this provides some bene- fit short tasks like MT-Bench (Bai et al. 2024), gains are limited more complex real-world task benchmarks such as WildBench (Lin et al. 2024a). Building further on implicit signals, re- cent approaches transform them into pairwise preferences for direct optimization. [Shi et al. (2025) identify dissatisfaction with GPT-4, summarize user preferences, and generate improved responses as chosen answers, contrasting them with the original unsatisfactory replies rejected. (Tan et al. (2025) follow a similar philosophy by extracting ceader-centrie questions from user-generated con- tent, sampling multiple candidate answers with an LLM, and ranking them with reward model to construct mhosen-rejecctrr pairs. In contrast, our approach requires no positive responses from stronger models, no reward model, and certainly no human- provided golden truth, relying solely on abundant real-world dissatisfaction (DSAT) signals and dynamic positives from the evolving policy.
2.2 SHL.F-MMPOVEMENN AND ITERATIVE DIRECT PREFERENCE OPTIMIZATION
Self-improvement strategies have emerged as an important avenue for iteratively enhancing model performance. SPIN (Chen et al. 2024ch formulates this framework by treating the previous iteration model as the opponent and the current iteration model as the main player, constructing preference data with the SFT response the chosen response and the prior iteration's 's response the rejected response, thereby fully utilizing the SFT data without requiring additional human annotation. How- ever, pairing fixed ground-truth positives with self-generated negatives has been shown to risk reward hacking and model collapse (Wang et al. 2025b). Beyond this, Iterative DPO (Xiong et al. 2024 Xu
al. 2024) and Relf-Rewarding Language Models Yuaan al. 2024) and variants (Pang et al.,
Chen et al., 2024a Zeng et al. 2025 Tu et al. 22025 Chen et al.) 2024b) explore generating on policy preference data via ranking responses by the model itself or reward model verifier and then conducting iterative DPO training. However, subsequent studies reveal that self-improving models face a critical limitation: the chosen and rejected responses can become too similar, leading to weak preference signals. To address this, Temporal Self- Belf-Rewarding LMs (Wang et al.) (025a)) decouple chosen and rejected responses through past-future anchoring, while CREAM (Wang et al. 2025c) introduces consistency regularization to stabilize the preference signal. Our method naturally avoids this issue by anchoring on genuine DSAT negatives and sampling fresh positives from the evolving policy, thereby maintaining non-vanishing preference margin and preventing gradient collapse.
2024
3

---

Published as a conference paper at ICLR 2026
3 DRIFT: DISSATISFACTION-REEII ITERATIVE PRE- FRE-FFRENCE TRAINING
User feedback in real-world systems is inherently asymmetric, while satisfied rarely provided explicit positive responses, dissatisfied users are more likely to offer abundant and detailed feedback in the form of complaints, corrections, and stated preferences. As result, dissatisfaction (DSAT) signals are not only more frequent but also richer in information than satisfaction (SAT) signals. Instead of viewing this imbalance as a limitation, DRIFT exploits it by treating authentic dissatis- faction high-quality negative supervision, while generating positive feedback dynamically from evolving model itself.
Our approach is motivated by two key insights:
Genuine dissatisfaction reflects real deployment failure modes, offering more informative and reliable supervision than synthetically constructed negatives.
Iteratively sampling fresh positives from the current policy maintains the margin between chosen and rejected responses, thus mitigating the gradient collapse that plagues most self- improvement methods as these responses become increasingly similar over time.
Formally, let X denote the prompt space and y the response space. The current model is X' y - (0, 1), and Tee is a frozen reference model. Let XDSAT S X' denote prompts with observed dissatisfaction signals. For each XDSAT, observe set of negative responses:
DSAT(x) {v
expressed dissatisfaction).
DRIFT proceeds in iterative refinement cycles. where each round builds upon the improved policy from the previous iteration (Algorithm (T). We begin by filtering the wild dataset to extract dissatis- faction (DSAT) cases, producing prompi-response pairs (z, y that reflect concrete failure modes encountered in real-world scenarios. At each iteration, itive response yt for the same prompt x, allowing the positive response to evolve alongside the model's capacities. The model is then updated by minimizing the DPO loss:
current model To, generates a fresh pos-
Lovo -
-B (a.tt .))) log log Sely log 5e x) (
, (x))
where 3 controls the preference margin and denotes the logistic function.
Algorithm DRIFT: Dissatisfaction-Reiine Iterative Preference Training
1: Input: Wild implicit feedback dataset, current model Ty. reference model Teet, number of iterations K
2: Output: Updated model parameters 0x
3: Filter: Extract DSAT signals form D {(x,y-] DSAT(=))
4: for k == I.....K K do
5: Positive Sampling: For each (æ, ) E D, sample fresh positive response y Tep (: xx)
6: Loss Update: Update 0₂ by minimizing Lero (Eq.
7: end for
EXPERIMENT
In this section, evaluate DRIFT against strong self-improvement baselines, focusing real world task performance. Sec. 4.1 outlines datasets, training recipe, and evaluation benchmarks. Sec. 4.2 presents the task performance on WildBench and AlpacaEval2. Then, in Sec. 4.3 we analyze exploration cability on response space of each method through global high-reward coverage.
4.1 SETUP
Datasets. WildFeedback (real-world, user-feedback). The WildFeedback dataset derived from conversations, by assigning per-turn
WildChat-1M, corpus of over million
4

---

Published as a conference paper at ICLR 2026
labels Satisfaction (SAT), Dissatisfaction (DSAT), Neutral (Non-DSAT/Non-SAAT Labels de- rived using SPUR (L al. 2024b), which recursively prompts GPT-4 to learn SAT/DSAT rubrics from thumb-annotated conversations and applies them to score satisfaction/ dissatisfaction. Among all 88,920 unique conversations, only 4,478 (5.04%) conversations were labeled SAT, while 10,632 (11.96%) were labeled DSAT, which is more than twice the SAT count. We also curate 491 seed data items (0.55%) in which LLM responses transition from DSAT to SAT after revision, naturally yielding preference pairs.
UltraFeedback (synthetic, LLM-labeled). For completeness and comparability with prior self- improvement work, we also evaluate on Utrraeedbbcck in which each prompt has four completions from different models that are scored by GPT-4. This synthetic setting provides a complementary evaluation to the real-world data setting and ensures fair comparison with SPIN/ IterDPO on com- monly used LLM--lbelldd preference data.
Table I: Comparison of preference data construction strategies across different methods. "Self-Gen"
means responses are generated by the current policy. "Real" is from user feedback.
Chosen Response
Rejected Response
No Positive Examples
Leverage Real
Method
User Feedback
Self-Gen
Real
Self-Gen
Real
SPIN
x
(SAT) X
X
X /
IterDPO
X
X
DRIFT (Ours)
X
/ (DSAT)
Training Recipe. Our experiments are conducted Qwen2.5-7B-Instruuc and Qwen2.5-1444
Instruct. We adopt a two-stage training:
(1) Warm start: train on the 491 seed DSAT-+SAT pairs, which provides an initial aligned policy, (2) Iterative preference training: After warm start, each method constructs fresh preference pairs.
Per-iteration preference data construction (Table 77:
DRIFT: In WildFeedback, we keep the DSAT reply as the rejected response and, at each iteration, sample a fresh response from the current policy using prompt which contains the full conversation including the DSAT user turn and an explicit improvement instruction. In UltraFeedback, we re- place the original chosen with a fresh policy sample.
use the SAT reply as the chosen response and a policy sample as the rejected response using the prompt which is the conversation before the SAT user turn. In Ultra- Feedback, we replace the original rejected with fresh policy sample.
SPIN: In WildFeedback,
IterDPO: In WildFeedback, we generate two responses using different prompts: the chosen context includes the full conversation including the DSAT turn and explicit improvement instruction, while the rejected context is the conversation before the DSAT user turn which does not reveal the user preference and instruction information. In UltraFeedback, both responses are generated from the same prompt and ranked by the reward model the higher-scored response is chosen and the other is rejected.
We then perform one epoch of DPO training after data generation, which prevents overfitting during iterative training. Full training details are presented Appendix D
Evaluation. We evaluate on WildBench (Elo, Task Score) and AlpacaEval2 (win rate, length- controlled; LCC. WildBench is built from challenging real-world user queries in WildChat-IM and spans five diverse categories: Creative, Reasoning, Math, Info Seek, and Coding, making it well suited for assessing our method for real-world performance.
4.2
PERFORMANCE EVALUATION
4.2.1
RESULTS ON WILDFEEDBACK
We first examine performance on the real-world WildFeedback dataset, which contains authentic user satisfaction/ dissatisfaction labels and exhibits strong imbalance: dissatisfied responses (DSAT) outnumber satisfied ones (SAT) by more than 2:1. Hence, we consider two configurations: a Con- trolled setting with around 4k samples (matching SPIN for fair comparison), and Full setting using all 1 1k DSAT samples to demonstrate DRIFT's ability to exploit abundant negative feedback.
5

---

Published as a conference paper at ICLR 2026
Table Results of training on WildFeedback. Appendix for detailed per-task results.
WildBench
AlpacaEval2
Method
7B
14B
7B
14B
Elo
Score
Elo
Score
Win
LC
Win
LC
Base Seed
1194.67
48.66
1213.17
55.08
37.69
39.73
36.65
43.58
1193.66
49.11
1213.50
54.93
42.67
42.06
40.25
45.78
Controlled Setting
SPIN
iterl
1180.75
42.86
1200.63
47.16
26.21
34.09
25.53
37.28
iter2
1173.45
37.86
1 1192.56
44.04
20.56
29.00
18.57
30.91
IterDPO
iterl
1189.43
47.07
1206.65
51.79
41.55
41.35
37.14
43.14
iter2
1192.46
48.94
1211.69
56.63
41.18
40.14
48.32
47.28
DRIFT (Ours)
iter1
1197.13
51.06
1215.73
58.37
42.73
41.41
48.76
45.42
iter2
1195.33
51.06
1214.03
57.59
43.79
41.49
46.83
43.48
Full Setting
IterDPO
iterl
1185.11
46.31
1205.48
52.34
40.36
39.85
38.07
43.99
iter2
1182.33
46.17
1206.63
51.38
35.76
37.06
32.88
37.92
DRIFT (Ours)
iterl 1194.81
50.61
1212.83
57.27
43.90
40.32
48.63
47.46
iter2
1199.09
51.69
1217.61
58.30
46.64
42.72
45.33
44.93
As shown in Table 2 DRIFT raises the WildBench Task Score by 6.23% (+3.03) for 7B and 5.97% (+3.29) for 14B, and boosts A AlpacaFval2 win rate by 8.95% for 7B and 12. 12.11% for 14B compared to the base models. And method consistently outperforms both SPIN and IterDPO across all metrics in both controlled and full data settings.
While SPIN shows degraded performance with iterations likely due its reliance on fixed set of satisfied responses becoming stale, DRIFT maintains steady improvements, suggesting that its strat- egy prevents distribution shift. IterDPO performs better than SPIN but still lags behind DRIFT in both settings, indicating that while reward model guidance helps. the real world informative DSAT examples provides superior training signal. Notably, DRIFT's controlled setting (using only 4k sam- ples) already matches or exceeds IterDPO's full setting performance, demonstrating the efficiency of wwwpsssfaacttonaannnnroo
learning. The stronger gains the 14B scale suggest that DRIFT benefits larger models more, likely because their greater capacity makes it casier to discover better positives while being anchored by real negatives. This making DRIFT well suited for scaling up.
Long- Long-horizon stability beyond 2 iterations.
To further investigate the stability and perfor- mance trends for longer iterations, we extended all methods to five iterations on Qwen2.5-7B,
WildBench Task
Figure
2 visualizes the performance trajectories across iferations.
42.86
62.40
50
Both SPIN and IterDPO peak early at iterl and then exhibit performance degradation, with SPIN showing the most pronounced decline. In contrast, DRIFT demonstrates sustained im- provement up to iter4 (52.47), and then forms a stable plateau with minimal variation (51.2 at iter5). This stability suggests that DRIFT's
33.55
10
Ea22
Ber3
Hlac
Iteration
Figure Performance across five iterations on Qwen2.5-7B. Bold values indicate the maximum score achieved by each method across iterations.

---

Published as a conference paper at ICLR 2026
strategy of anchoring on real dissatisfied responses while continuously sampling fresh positives pre- vents mode collapse that plagues other self-play or self-im improve methods during extended iterative training. The limited performance collapse observed even at iter5 further validates DRIFT's robust- ness for long-horizon self-improvement.
Unguided ablation study and the source of DRIFT's advantage. An important consideration is whether DRIFT's observed gains are attributable to the guidance prompts provided during positive generation. To examine this, we evaluated an unguided variant which DRIFT and IterDPO gener- ate responses solely from the original user prompt, matching SPIN's configuration. The results (see Appendix C Table 5) show that DRIFT remains consistently stronger than both baselines in this setting, and the gap between the unguided and full versions is small. This suggests that DRIFT's gains are not attributable to instruction prompt and reference point, but instead to design principle of anchoring off-policy DSAT negatives while drawing positives from the evolving policy.
4.2.2 RESULTS ON ULTRAFEEDBACK
To ensure comprehensive evaluation and fair comparison with prior work, we also evaluate on the synthetic UltraFeedback dataset, where preferences are scored by GPT-4 rather than derived from real user interactions. This complementary evaluation helps assess whether DRIFT's advantages generalize beyond the specific characteristics of real-world dissatisfaction signals more conven- tional preference learning settings. As shown in Table DRIFT outperforms the base model with gains of 4.62% (+2.25) for 7B and 7.61% 61% (+4.19) for 14B on WildBench Task Score, and im- provements of +3.35% (7B) and + +12.29% (14B) on AlpacaEval2 win rate. Compared to the best SPIN/IterDPO results, DRIFT achieves additional gains of +2.14 (7B) and +4.49 (14B) on Task Score, well +6.51% (7B) and +6.10% (14B) win rate.
Table 3: Results of training on Ultraeeddack. Appendix C for detailed per-task results.
WildBench
AlpacaEval2
Method
7B
14B
7B
14B
Elo
Score
Elo
Score
Win
L.C
Win
LCC
Base
1194.67
48.66
1213.17
55.08
37.69
39.73
36.65
43.58
SPIN
iterl
1163.16
35.10
1178.88
36.99
18.39
26.62
16.09
28.20
iter2
1139.66
25.93
1155.07
28.01
13.23
19.91
13.66
24.29
IterDPO
iterl
1194.45
48.77
1214.14
54.21
34.53
40.55
33.29
42.84
iter2
1192.01
48.49
1215.12
54.78
32.15
39.92
28.51
40.47
DRIFT (Ours)
iterl
1197.04
50.91
1215.67
58.52
41.04
40.37
47.89
48.46
iter2
1197.94
50.32
1218.75
59.27
40.47
37.09
48.94
47.43
4.3 EXPLORATORY CAPACITY ANALYSIS: DRIFT EXPLORES MORE DIVERSE HIGH REWARD SOLUTIONS
A central question in preference learning is whether push- ing rewards upward narrows the response distribution and erodes exploration capacity and diversity. Methods that op- timize aggressively for top scores or fixed chosen responses can shift toward mode seeking: peak metrics improve while alternative high-quality modes are under explored. We fur- ther investigate whether DRIFT's strategy of sampling fresh positives while anchoring on real dissatisfied responses en- hances the model's ability explore the space of high- quality solutions than SPIN or iterDPO, which may pro- gressively constrain the solution space.
Devarnt Po Region Share (9))
70
32,6
20.9
IterDPO
Figure 3: Comparison of high reward region coverage.

---

Published as a conference paper at ICLR 2026
Semantic reward topography construction. For each prompt, first sample 128 responses from each method and embed all collected response We obtain 2D semantic projection via UMAP and estimate weward-weighted density surface Zan(g) [0. over regular grid g using Gaussian KDE. The global high-reward region is defined as
H = (g)(g) Z Zhigh }
Thigh Quantile Zan 0.8).
For each method m, we construct a corresponding surface Dm (m(g) on the same grid and bandwidth, and measure its coverage inside the global high-reward region by
|Sml [
Sm = (g H: (g) Thigh }.
Share(m)
We render Zan the background terrain, overlay the boundary of H (dashed), and shade S. for each method. We report the average high-reward coverage across 50 prompts in Figure [ DRIFT consistently attains the largest share both 7B and 14B scales, with a larger margin 14B, indicat- ing greater high reward solutions diversity and better scalability.
Case Study: High-Reward Reward Coverage and Response Diversity. Figure shows an example that DRIFT distributes responses across broader set of high-reward semantic regions, whereas SPIN and IterDPO concentrate their outputs within a much smaller subset of the space. Notably, DRIFT also discovered distinct region (circled in the reward topography) where it uniquely employed markdown formatting to structure research papers, demonstrating alternative presentation styles for the same prompt.
Prompti Generate research poper that unalytes and compares the farm and Fanction diffarr diffarent types of cernivorovs plantss.
high
herDPO
IterDPO points high reward area DRIFT points high-reward Global high reward area (O80)
DRIFT
0300
0720
0.85
Share of Global High- Rewerd Reçion SPIN 51, 3%
DRIFT uniquely
explored region
torrPO 51.0%
87.9%
0
SPIN Representative Response
IterDPO Representative Response
DRIFT Representative Response
Title: Carnivorous Plants: Form, Function, Evelutionary History, Applications
Titie Camivorous Plants: Function, Evolutionary Significance
"Title: Comparative Analysis Carnivorous
Adaptations: Flytraps.
Plants, and Sundews**
Introduction:
Introduction: Carnivorous
"'Aastrat"
Camivorous plants fascinating
fascinating group
Carniverous plants evolved remarkable adaptations...
have unique capture and digest prey...
organisms...
"Introduction"
Markdown format!
These plants typically grow in nutrient- poor environments supplement their through carnivory...
These plants are found in various habitats. primarlly nutrient-poo solls
Physical Characteristics and Adeptations: flytraps
"Dionass musciputa!
Figure 4: Example of response diversity and quality comparison via semantic reward topography. Two central plots: Left is the UMAP scatter all responses; Right is the semantic reward topogra- phy showing the global high-reward region and the coverage of the three methods. Full prompt and responses are in Appendix
Anchoring authentic DSAT negatives while sampling fresh positives enables DRIFT to maintain and amplify exploratory capacity. DRIFT not only reaches high reward but also occupies a far broader set of high-reward areas, explaining why it continues improve over iterations without collapsing to small family of solutions.
Embeddings computed with
kmbedd nereeddiig---00 6H
8

---

Published as a conference paper at ICLR 2026
5 THEORETICAL ANALYSIS
DRIFT shows superior performance by leveraging real-world user dissatisfaction as high-quality negatives, which leads further investigate and analyze how real-world data shapes the success DRIFT and why some other strong baselines like SPIN and IterDPO fall short. this section, we prove that DRIFT maintains a usable gradient signal through fresh positives anchored by genuine DSAT negatives; in contrast, updates fitted to a fixed SAT set like SPIN can easily overfit to a small sub-optimal subset with gradient collapse.
Notation.
Let A' be the prompt set. For a particular prompt ~ X, denote yt as the chosen response and y the rejected response. Let the generation likelihoods of y* and y to be x To(y* x) and æ = T z) respectively, where To is the language model we aim to train.
For formula simplicity, we denote the implicit reward margin (Rafailov et al. 2023) as:
s = (log /TTt///al'" (1 (x)) - log (i /Teely
(3)
and the loss function:
i log (s),
= - Vo log my 1x) Vo log To
x))
(4)
where (ss) (1 + - is the logistic function. We also denote dg V
In To (y+
x)
In (y x) and (0) = v(()] = B (-s) dol. Finally, let X' x y + (0, 1
be the
unknown gold reward, and (6) B Ey Ex ( a) (r* (z, Y)] be the overall objective.
Reward Margin Hypothesis. With a probability of at least Pamp. both the implicit reward margin and the gold reward margin have positive lower bounds. Specifically, there exists some T E (0, 2 and m, > 0 such that
Emp = o(-s) 2 and (zxxx²) (2, ) m.
P(Eimp) Pimp 0. (5)
This hypothesis ensures that chosen responses are mostly ranked higher than rejected responses.
Non-vanishing expected training signal. We first certify that DRIFT maintains a uniform pected gradient magnitude under positive-mass "quality" event.
Lemma 1 (Expected gradient lower bound under local quality). Let E fo(-s) 2 r) with P(E) pa > 0 for some E (0, 1 I [[dee]] E] Acond > 0, then
E [
2 P Aoond.
(6)
Proof. From Eq. 4 and (-- (-s) 0,
E V = E E[Bo(-s)
do 2 8 EE [| E(E) E].
This bound shows that long non-negligible fraction of pairs satisfy a(-s) 2 and have nonzero conditional gradient gap [E[lldel a(-s) 2 r| > 0), the expected training signal stays away from zero.
Expected improvement of actual utility. We now state general improvement guarantee: the expected DPO step increases the true utility J, with the gain quantified based on the key data condi-
tion.
Theorem (Expected improvement of J), Assume the improvement event Eq. 5 holds with proba- bility at least Pimp and there exists such that
E ((),,, de)
Emp 2 A.
If J is Lm-ohhoth in neighborhood of 0. then for any 0.
L,J n² [[1
E [[ + (9)))] 2 J(0) + 3 Pump X
(8)
2
In particular, for sufficiently small n,, the right-hand side exceeds J(0) by a linear-in-n margin @Tpimpp up to O1r)).

---

Published as a conference paper at ICLR 2026
Proof sketch. g(0) 3Elo(-s)dd)) gives
g)) = =B do)] 2 B [ Eamp] 2 BBT Pimp A.
smoothhhssss yields J(0 -[0+-g]) 2 ((0) + (VJ.)) 11 then take expectations obtain
Eq Full proof in Appendix B.2
Why DRIFT outperforms SPIN? When SPIN concentrates probability on a finite SAT catalogue and reaches a fixed point, the magnitude of the pairwise DPO signal on SPIN pairs is controlled by catalogue's log-density-ratin variance:
B[ db1
8
Var(s) VENd,I.. Var(s) = 2,3 Vary (pssa(((())))) (9)
4
where (ty) == (y x)) In Trer æ)) (Proposition I in Appendix). Thus, a small Varysar (h) implies weak training signal that can quantitatively degenerate. By contrast, DRIFT maintains non-vanishing signal and practical gains. Full discussions are in Appendix B.3
Summary
Signal: If improvement events occur with nonzero probability, DRIFT's expected gradient stays non-vanishing (Lemma
Utility: Under the local correlation, a small step along the expected DPO direction improves up to (17²² (Theorem 1.
Contrast: At SPIN fixed points finite SAT catalogue, the signal is controlled by catalogue
log goy-density-ratio variance and can degenerate (Proposition
6 DISCUSSION
Generalization. To test our method's generality, we applied the same training procedure to Gemma-3-12B-it, a multimodal and structurally distinct architecture. The results (see Appendix C Table show that DRIFT again outperformed SPIN and IterDPO across all WildBench categories, exhibiting similar stability and improvement patterns. These results confirm that DRIFT generalizes beyond single model family and performs well even on different model architecture.
Safety implications of training with DSAT signals. Since DRIFT explicitly anchors on real user dissatisfaction, it is important vulnerabilities or demographic biases. Evaluations on AdvBench and ToxiGen (see Appendix C Table show that DRIFT does not increase jailbreak success rates, toxicity, group-specific harms relative to baseline models across iterations.
whether such supervision inadvertently amplifies adversarial
Taken together, these results show that DRIFT's asymmetric pairing of off-policy DSAT negatives with on-policy positives provides a stable and informative learning signal, generalizes across model families, and preserves baseline safety characteristics. Grounding preference optimization in real dissatisfaction thus offers a reliable and scalable direction for real-world post- uest-training.
7 CONCLUSION
Real-world post-training rarely comes with abundant golden positives; comes with abundant dis- satisfaction and iterative user edits. In this paper, we introduced DRIFT, simple. scalable recipe that pairs authentic DSAT negatives with policy sampled positives, turning in-situ feedback into sta- ble, exploration preserving updates. Empirically, on real-world user feedback dataset WildFeedback, DRIFT outperforms SPIN and IterDPO on WildBench and AlpacaEval2 (with the stronger margins at larger base models); on synthetic LLM-labeled dataset UltraFeedback, it retains its superiority. Exploratory capacity analysis indicates that DRIFT explores more diverse high-reward solutions rather than overfitting to a narrow region. Theoretically, we show that DRIFT's admits uniform, non-vanishing gradient lower bound, avoiding the collapse that arises when training concentrates probability on a finite fixed chosen (or SFT) set as in SPIN. Together, these results suggest DRIFT promising practical recipe for preference learning with real-world user feedback.
10

---

Published as a conference paper at ICLR 2026
This work relies publicly datasets WildFeedback that contain anonymized human-LLM conversations. No personally identifiable information was collected or used. All ex- periments comply with dataset licenses and terms of use. The research has foreseeable negative social or ethical impacts.
Ethics Statement
Reproducibility Statement We have made extensive efforts to ensure the reproducibility of our work. The main paper details our training setup, datasets, and evaluation metrics (Secs. 4.3 Dataset construction and filtering steps are described in Sec. 4.1 with references to the source corpora. Training hyperparameters. iteration procedures and training dynamics are presented in Appendix D To further facilitate verification and reuse, we will open-source our code, including data-processing pipelines, training scripts and analysis upon paper acceptance.
REFERENCES
Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and Wanli Ouyang. Mt-bench- 101: A fine-grained benchmark for evalu- ating large language models in multi-turn dialogues. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7421-7454, As- sociation for Computational Linguistics, 2024. doi: 10. 10.188655 12211166///12222.oo http: dx. doi. ott/ 1tt.//10.8163338 1101////..113/////222.... acl -cl-oon.................o.. .oogooo...0..
URL
Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang, Qian Liu, Arunesh Sinha, Pradeep Varakan- tham, and Min Lin. Bootstrapping language models with dpo implicit rewards. arXiv preprint (rXiv:2406.09760 2024a.
Kaiyuan Chen, Jin Wang, and Xuejie Zhang. Learning to reason via self-iterative process feedback for small language models, 2024b. URL
08393
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models, 2024c. URL Utt/httss:://axii
g/abs/2401. 01335
Shachar Don- Yehiya, Leshem Choshen, and Omri Abend. Naturally occurring feedback is common, extractable and useful, 2025.
Ge Gao, Alexey Taymanov, Eduardo Salinas, Paul Mineiro, and Dipendra Misra. Aligning Ilm agents by learning latent preference from user edits, 2024. URL https://aaxxii wwtpzxivv.rrr///o /bb
112404.12269
Braden Hancock, Antoine Bordes, Pierre-Emmaauel Mazare, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! In Anna Korhonen, David Traum, and Lluís Märquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3667 -384,, Florence, Italy, July 2019. Association for Computational Linguis- tics. doi: 10. 10.18653/v11/ 1/139////1P1111///1 1358. http://tss..eclaaaaooooo.../////
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild, 2024a. htt.httss///axiv....oo
006. 04770
Ying-Chun Lin, Jennifer Neville, Jack W Stokes, Longqi Yang. Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri. Reid Andersen, Xiaofeng Xu, et al. Interpretable user satisfaction esti- mation for conversational systems with large language models. arXiv preprint 1xX.::224433.388 2024b.
Yuhan Liu, Michael J. Q. Zhang, and Eunsol Choi. User feedback in human- Iim dialogues: A lens to understand users but noisy as a learning signal, 2025. URL https:://///. http:httpss/aaatee http::///axxvvvvxxa/
2507 23158
Tuomas Lounamaa. Explicit and implicit IIm user feedback: A quick guide, May 2024. URL com/ h http///oo/eppliittimmmpppmmmpppp
user- - feedbac
k lo-quick-uuidd Accessed: 2025-09-10.

---

Published as a conference paper at ICLR 2026
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston, Iterative reasoning preference optimization, 2024. URL /ttps:://axxxv..x///////////
S 8/2404.197 14404.19733
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 36:53728-53777 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Omar Shaikh, Michelle S. Lam, Joey Hejna, Yijia Shao, Hyundong Cho, Michael S. Bernstein, and Diyi Yang. Aligning language models with demonstrated feedback, 2025. URL. https:
//arxiv.crg/abbe 22/5///aa/2244.... 2406 00888
Taiwei Shi, Zhuoer Wang, Longqi Yang. Ying-Chun Lin, Zexue He. Mengting Wan, Pei Zhou, Sujay Jauhar, Sihao Chen, Shan Xia, Hongfei Zhang, Jieyu Zhao, Xiaofeng Xu, Xia Song, and Jennifer Neville. Wildfeedback: Aligning ilms with in-situ user interactions and feedback, 2025. URL
https::/ www.//ggg/.... 2408 15549
Zhaoxuan Tan, Zheng Li, Tianyi Liu, Haodong Wang, Hyokun Yun, Ming Zeng, Pei Chen, Zhihan Zhang, Yifan Gao, Ruijie Wang, Priyanka Nigam, Bing Yin, and Meng Jiang. Aligning large language models with implicit preferences from user-generated content, 2025. URL https:
//arxiv.org/aaaaaaai. 2506. 04463
Songjun Tu, Jiahao Lin, Xiangyu Tian, Qichao Zhang, Linjing Li, Yuqian Fu, Nan Xu, Wei He, Xiangyuan Lan, Dongmei Jiang, and Dongbin Zhao. Enhancing Ilm reasoning with iterative dpo: A comprehensive empirical investigation, 2025. URL 2
2854
Aaron David Tucker, Kianté Brantley, Adam Cahall, and Thorsten Joachims. Coactive learning for large language models using implicit user feedback. In Forty-first International Conference on Machine Learning, 2024.
Yidong Wang, Xin Wang, Cunxiang Wang, Junfeng Fang, Qiufeng Wang, Jianing Chu, Xu- ran Meng, Shuxun Yang, Libo Qin, Yue Zhang, Wei Ye, and Shikun Zhang. Temporal self- rewarding language models: Decoupling chosen-rejected via past-future, 2025a. URL htLhtt:::
06026
Yifan Wang, Runjin Chen, Bolian Li, David Cho, Yihe Deng, Ruqi Zhang, Tianlong Chen, Zhangyang Wang. Ananth Grama, and Junyuan Hong. More is less: The pitfalls of molti-model synthetic preference data in dpo safety alignment. arXiv preprint arXiv:2504.02193 2025b.
Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, and Huaxiu Yao. Cream: Consistency regularized self-rewarding language models, 2025c.
URL /ettts::/arxxt
org/ wwtp///....... 12735
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang- Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, 2024. URL /tattss///rxiv... www.riiiv.......eooo abs 2abs//232..1e..
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss, 2024. URI. https:
16682
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho. Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint 240.:2200.0002000 3, 2024.
12

---

Published as a conference paper at ICLR 2026
Yongcheng Zeng, Xuanfa Jin, Guoqing Liu, Quan He, Dong Li, Jianye Hao, Haifeng Zhang, and Jun Wang. Aries: Stimulating self-refinement of large language models with and for iterative preference optimization. In Workshop Reasoning and Planning for Large Language Models, 2025.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatgpt interaction logs in the wild, 2024. URL https:////ax org/ absaabs//2// abs/2405. 01470
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat-Im: A large-scale real-world Ilm conversation dataset, 2024. URL http:::///// xiv. xiv.org/ abs 1399.1998
13

---

Published as a conference paper at ICLR 2026
A DISCLOSURE OF LLM USE IN PAPER PREPARATION
We acknowledge the use of LLMS for assistance with writing and polishing text. All the content suggested by LLMs in writing proofread and manually adjusted before being integrated into the final manuscript. The authors take full responsibility for the accuracy and factuality all content presented.
B THEORETICAL PROOFS
B.1 ASSUMPTIONS
(A1) Improvement event (data-level). There exist (0, my 0, and Pimp > such that, with
Eimp {((-s) and r"(x,"") (a, y z m, }
one has P (Fimp) 2 Pimp-
(A2) Local smoothness of J. There exists LJ < such that, for all sufficiently small v, J(0 u) 2 J(0) + (VJ(0), v) L |||||
(A3) Finite second moment for the score difference. ||do] I Ca < co.
B.2 PROOF OF THEOREM T (EXPECTED IMPROVEMENT J))
Theorem (Restatement of Theorem D. Under Assumptions [Al) (A2) (A3)
and the local advan-
tage correlation condition 7 for any r > and g(0) Elo 8)dg do).
LJ
E [ + + 2 ((0) + n Phmp A
n² 82 Ca.
2
Proof. By definition, g(0) -3 [[(((-s) do}. Taking inner product with VJ(0) and then expecta- tion,
E ((V(((), (0))
BE
a () (VJ()),
On the improvement event from Assumption 7
(AI)
((-8) 2 т; conditioning on Eimp and using
E (V)( .(0) 2 P E [VJ de) Empp] 2
B Pimp A.
By the Lssmmothhesss in Assumption (A2)
(( 7g) 2 (() + 7 (V) - بي x² ||g1|2
Taking expectations and combining the previous bound gives
E ( 7g) 2 ((0) + n. Pimp A
I, 77 E 1
It remains to bound E (l)(1) Since a E (0. 1), [|g(0) = || ([3 dolll < 8 BBlldoll 3 / 8 CC., where used Assumption (A3) and Jensen. Hence E. []((0)]1 B²Ca, which yields the stated inequality.
B.3 SPIN REAL-WORLD PERFORMANCE DISCUSSION
SPIN updates concentrate probability on the finite SAT catalogue SAT(x) and, under trust-region style update, admit the closed-form iteration (see, e.g., (Chen et al. 2024c)):
Pe. (y x:) Pa, (y x)) PSAT(U x)) 1/A
(5.3)
Po, (y x)
which drives po, (: a:) toward P-AT(: a:) supported on SAT(w). Consequently, at any fixed point i one has Tp(- æ) P-AT(- æ); under the SPIN data rule (positive sampled from PSAT, negative sampled independently from Ta given æ). the positive y* and negative y conditionally i.i.d. on (ZT(2z) with the common marginal PSAT- The DPO signal then depends on the variance of the log-density ratio on that finite set and quantitatively degenerate.
14

---

Published as a conference paper at ICLR 2026
Proposition (Quantitative degeneration at SPIN fixed point). At SPIN fixed point 0 with T = (-Ar(-12), if y², y are conditionally i.i.d. from TT([12) and E Egll < co. then for h(y) = In (3 2) In (3) xx),
[[Bo(-s) del
B Var(s) V B Var(s) = -23² Vary~paAT
(10)
Proof. By (A5), for each we have Eldg x since yt, . are i.i.d. under Ta(-1xr). Therefore
- -oo(s)) do) B[Bo(-s) dal = 3 -
Fix any unit vector u. Scalar Bauchy-Schward yields
u [[[Bo((-s) 3) d6} | (o(-s) B)((-)) u da) S B (Var(o(-s)) VB((uT d
Taking the supremum over all unit u,
[[ ddl del VVar(o(-s)) v /Elldg1e.
Since a is 1/4-Lipschitz. Var(o( I 1o Var(s). With = 3[h(z)+) h1y }] and y y iid.
Var(s) 8² Var Var(h(y*) h(y)) 2,3² Vary ~PSA (n(Y)),
which gives the stated inequality.
15

---

Published as a conference paper at ICLR 2026
C ADDITIONAL RESULTS
C.1 WILDBENCH LEADERBOARD FULL TABLE
Table Complete WildBench leaderboard showing all evaluated models with comprehensive task-
specific scores across multiple evaluation dimensions. Our models highlighted in gray.
Model
Elo
Task
Creative
Reasoning
Math
Info Seek
Coding
Length
GPT-4o (2024-05-13)
1256.86 86
59.30
59.12
60.47
3723.52
Claude335-Sonnet (20240620)
1238.93
54.70
55.61
55.64
50.16
55.54
56.51
2911.85
GPT-4-Turbo (2024-04-09)
1233.99
55.22
58.66
56.20
51.00
57.18
55.07
3093.
Remini-1.5-PP
1228.55
52.95
55.12
53.73
48.59
52.23
55.22
3247.97
DeepSeek-v2 w2-Char (0628)
1221.66
53.99
56.43
54.83
51.43
52.72
55,00
3252.38
GPT-4 (0125-preview)
1221.30
52.28
57.57
53.45
45.79
54.36
52.92
3335.64
Claude-3-OOus (20240229)
1219.27
51.71
53.02
52.53
46.75
53.47
53,30
2685. 98
Qwen2.5-14B UltraFeedback DRIFT iter2
1218.75
39.27
59.48
61.05
59.44
58.66
57.64
4275.27
Mistral-Large-2
1217.99
55.57
58.86
57,22
52.67
57.38
53.84
3503.65
1217.61
58,30
58.
60.60
59.37
58.37
55.19
4492.77
GPT-4o-mini (2024-07-18)
1217.35
57.14
60.05
58.24
$4.05
57.43
57.17
3648.
1215.73
58.37
58.55
55,57
1ron2.5-14B-Uttraossddeee
1215.67
58.52
58.65
60.30
58,02
57.57
57.64
4288.25
Qwen2.5-14BB-u htrr4///-Urr///eeuhceeeeeeee Ite--iiie
1215.12
54.78
55.35
53.08
3152.61
Qwen2.5-14B UltraFoedback- Itsrbaakkkk--------iii
1214.14
54.21
55.13
$6.27
$2.02
55.74
52.64
3424.60
Qwen2.5-1.B. 1-122-5-------- WildFeebback-DRIi--eeeee----
1214.03
57.59
57.98
59.73
57.45
57.62
55,38
5333.00
Qwen2 WildFeadbackk SSedd
1213.50
54.93
54.94
57.30
53.60
55.45
53.30
3878.89
Qwen2 1-B--.5--4BInsttruee
1213.17
55.08
55.71
57.84
54.98
54.95
52.23
3682.95
Qwen2.5-114B. 100.5-114--WiilFeehhhaae WilldFeeeback-DRIIIFI
1212.83
57.27
58.09
59.31
55.78
57.22
56.13
4485.35
Qwen2.5- 14B 14B
WildFeedback IterDPO-ite -terttr224 4k
1211.69
56.63
56. 18
59.19
56.51
56.78
54,25
4491.00 1.91.05
DeepSeck-42-oodr (0628)
1206.89
45.66
40.78
47.17
46.43
40.05
48.87
2580. 18
Scmini-1.5--laat
1206.77
48.85
51.66
50.79
45.32
48.67
48.73
3654.40
1206.65
51.79
50.75
54.49
51.24
52.67
19.43
3925.74
Qwen2.5 -40 WildFeedback IteDDPO iter2
1206.63
$1.38
50.23
54.17
$1.03
52.48
48.68
Qwen2.5-14BB (enm::5--4BBWillleeddddee----//
1205.48
52.34
52.87
54,97
51.71
54.06
48.96
4054.28
DeepSeek.v2-Chul
1205.02
48.21
53.59
50.63
44.52
51.81
44.43
2896.97 .7
Qwen2.5-11 1-1-
WildFeedbackSSIINNiic
1200.63
47.16
49.90
47,25
47.67
43.11
2707.
Swen2.5-7B-WWinn S
1199.09
51.69
52.45
53.21
50.63
52.38
50.28
4707 -407.48
Qwen2.5.77 1177B-UUtrFFeedhaass Tatraarccddaakk-DRiiiiieeeee i-tert
1197.94
50.32
50.39
52.50
48.41
52.08
48.58
$121 20
Qwen2.5-P7 WildFeeeDbackDDRIFT--8
1197.13
51.06
53.23
53.44
48.32
52.28
49.29
4517.02
Qwen2.5-1 wBB.BBBaaeeeeaackkDDDa DRIFFIttee
1197.04
50.91
50.08
53.77
48.92
52.72
48.87
4856.83
Qwen2.5-78 w1B.-WiiFFeecbbu-DDRFF---44
1195.33
51.06
51.42
53.45
49.16 16
52.13
49.38
4686.39
Qwen2.5. wtt.:/iideeeaahcckeneo Miidieeadhaakk-DRiiiieeeee
1194.81
50.61
$2.09
53.03
48.56
$2.13
48.34
4652. 38
Qwen2.5-78 -B--nnutttt
1194.67
48.66
50.08
51.80
47.09
50.69
45.00
4275.08
Qwen2. 1.-BB--UtaFFecduuck-eeeeeet
1194.45
48.77
49.35
51.39
46.03
50.89
46.82
3888, 79
Qwen2 7B WildFeedhack DRIFT-iter2
1194.24
50.42
51.89
53.31
48.88
$1.49
47.52
4711.01
Qwen2.5-78 WildFeedbac-SSed
1193.66
49.11
49.35
51,57
47.54
50.15
47.17
4624.30
Nemotron 4-00004444400--1-----
1193.60
47.67
53.32
46.26
2754.01
Qwen2.5- SilFFecdaac-SPII-iieee
1192.56
44.04
45.54
45.95
40.64
45.89
43.13
2820.38
Qwen2.5-7B, WildFeedbbck Seeedblaakk-eeePOO.O. iter2
1192.46
48.94
50.39
46.37
46.79
4731.32
Claude-3-Sonne (20240229)
1192.03
45.48
46.30
47.43
40.64
47.13
46.10
2670.24
Qwen2 [lltra/eedbbuck-eeeDPOOO
1192.01
48.49
50. 18
50.73
64.68
50.22
47.58
3708 61
12-nn--eB Qwen2 72B Instruct
1189.43
44.50
49.92
46.85
40.95
49.50
39.81
2856.45
Qwen2.5-7B.
1189.43
47.07
48.6 63
49.13
44,50
49.26
45.12
4602.06
Mistral Nemo--mooo--tttt (2407)
1187.35
44.38
54.57
47.41
35.63
$1.93
39.72
3318.21
Qwen2.5-7B 7anu2_5-77BWiiFecbbaaa www.eeebbbck-eeeeeeeee IteDPO-iterl
1185.11
46.31
46.77
48.61
44.46
48.02
44.27
4662.34
Qwen2.5-775 httpB//ilddeeebuuc--eeeeeeeeeeee
1182,33
46.17
45.74
49.22
45.60
46.24
43.70
4520.55
Qwen2,5-7B-Wel htt.://WWlddFecduu--SPPi......
1180.75
42.86
43.26
45.45
41.59
46.29
39.06
361 3611.19
1178.88
36.99
38.86
40.09
39.01
33.40
26422.6
Claude 3-Hik- (20240307)
1175.97
38.89
42.95
41.29
31.43
45.35
36.98
2601.05
2imm2,5-7BBWildFeeeddbdd
1173.45
37.86
37.36
40,72
37.06
42.13
33.27
2839900
Mistral-I Lrge (2402)
1172.18
38.89
49.66
46.14
33.74
2514.98
Q hwt.s778--Uuuuaeedddu--ssSPS
1163.16
35.10
34.78
37.61
30.87
39.90
33.21
3475. 16
QwenI S-reee-haa
1160.02
39.93
50.36
43.45
29,80
48.22
35,36
2392.33
Qwen2 14B- UltraFesdback
1155.07
28.01
29.77
31.69
25.08
32.72
23.13
2504.57
Hixxtrll-xx7B-nnstruceee
1145.51
31.47
42.75
34.59
22.14
41.94
25.02
2653.55
Qwen2
1139.66
25.93
25.79
28.88
19.60
32.03
24.43
3293.30
GPr- (01 25)
1135.69
30.02
37.42
33.39
21.59
36.49
26.54
1844. 14
16

---

Published as a conference paper at ICLR 2026
C.2 UNGUIDED ABLATION STUDY
Table 5: Unguided ablation study results on Qwen2.5-7B (full WildFeedback setting).
Unguided
variants remove the reference DSAT response and explicit improvement instruction, matching SPIN's setting.
Method
WildBench Score
AlpacaEval2 WinRate
SPIN
iterl
42.86
26.21
iter2
37.86
20.56
IterDPO
iterl (Unguided)
49.44
40.30
iter2 (Unguided)
49.66
41.24
iterl (Full)
46.31
40.36
iter2 (Full)
46.17
35.76
DRIFT (Ours)
iterl (Unguided)
50.77
46.77
iter2 (Unguided)
51.04
45.47
iterl (Full)
50.61
43.90
iter2 (Full)
51.69
46.64
C.3 GENERALIZATION ACROSS MODEL FAMILIES
Table 6: Experiment results on Gemma-3-12B- 12B-it (full WildFeedback setting). All methods were trained using the same recipe and evaluated on WildBench. Bold indicates the highest score for each metric.
Method
Task
Creative
Reasoning
Math
Info Seek
Coding
Base
60.77
66.10
62.46
54.10
65.21
59.62
SPIN
iterl 59.06
65.79
61.14
50.40
63.96
58.30
iter2
51.86
58.76
53.81
42.06
59.60
50.33
IterDPO
iterl 60.10
62.74
60.48
54.92
61.29
62.10
iter2
48.98
52.40
49.27
43.89
47.61
52.23
DRIFT (Ours)
iterl
61.73
66.37 66.10
63.02
56.19
64.71
61.23
iter2
60.88
62.07
54.76
62.82
61.33
C.4 SAFETY AND ETHICAL CONSIDERATIONS
17

---

Published as a conference paper at ICLR 2026
Table 7: Safety and ethical norms evaluation results. AdvBench measures adversarial jailbreak
attack success rate (%). ToxiGen measures toxicity scores (I-5 scale). Lower values indicate better safety.
Method
Attack Success Rate (%)
Toxic Score (1-5)
Base
0.01
1.56
SPIN
iterl
0.01
1.65
iter2
0.00
1.70
IterDPO
iterl
0.02
1.50
iter2
0.00
1.66
DRIFT (Ours)
iterl
0.02
1.54
iter2
0.01
1.58
18

---

Published as a conference paper at ICLR 2026
D IMPLEMENTATION DETAILS
D.1 WARM START TRAINING DETAILS
We curate DSAT- seed set (491 pairs) from WildFeedback, where dissatisfied user turn (DSAT) is followed by revised model response that satisfies the user (SAT). Each pair provides natural preference: the DSAT response fails to meet expectations, while the subsequent SAT re- sponse is preferred.
For our warm start phase, we initialize training using pre-trained instruction-tuned models as the base models. The warm start training utilizes seed preference data to establish initial alignment before iterative refinement. We did DPO training with carefully tuned hyperparameters to ensure stable convergence. All experiments were conducted on H100 GPUs with the same hardware configuration maintained across all training phases. The detailed hyperparameters for warm start training are presented Table g
Table 8: Warm Start Training Hyperparameters
Learning rate
Batch size
8
Optimizer
LR scheduler
Seg length
Epochs
Precision
5.0e-7
4
0.1
RMSprop
Linear
2048
3
bfloat16
D.2 ITERATIVE TRAINING DETAILS
After the warm start phase, we conducted iterative training to progressively refine model alignment using dynamically generated preference data. Data generation details are in Sec. [.1] Each iteration builds upon the previous model checkpoint, incorporating newly created preference data. The iter- ative training process maintains consistent hyperparameters across iterations, with only the training data and base model checkpoint changing between iterations. We trained each iteration for a single epoch prevent overfitting on the iteratively generated data. Table details the hyperparameters used for iterative training phases.
Table 9: Iterative Training Hyperparameters
Learning rate
Batch size
8
Optimizer
LR scheduler
Seq length
Epochs
Precision
5.0e-7 4
0.1
RMSprop
Linear
2048
b
D.3 TRAINING DYNAMICS
For better training illustration, report the Owen2.5-114B-Insrre DRIFT iterl & iter2 training dynamics in Figure 5 which shows dpo training loss, chosen reward, and rejected reward. The loss curves exhibit stable convergence across both iterations. The reward signals show the expected separation pattern: chosen rewards consistently increase while rejected rewards decrease. This trend observed in both iterations, confirming the effectiveness of the training.
19

---

Published as a conference paper at ICLR 2026
trainyless
trakerewwrfSichhsen
tralnieewaddsnneeeteed
Nu /// A//l///wwaa A//
-5
0/4
10
ww/ Anhy
V1
2000
Step
20
350
320
393
(a) Qwen-2.5- 111-:22.-11B--iiiFoddaaaee SiaddFoodbackDDRRFFFES
traierewwarta/chawee
train/ maiii/ee
24
Me Av M Mow Ww Wmnag
200
Step
30
200
300
300
(b) Qwen-2.5-1ee 105-11--WildFeedfeeeee Serereeedbac--DRIITT-MM DRIIFTitte2
Figure 5: The top row shows DRIFT training dynamics for iteration on Qwen2.5-14B In--trrrctt The bottom row shows the training dynamics for iteration
20

---

Published as a conference paper at ICLR 2026
E
MODEL RESPONSES EXAMPLE
Using at least two different scientific sources, generate a research paper that analyzes and compares the form and function of three different types of carnivorous plants, including but not limited to Venus flytraps, pitcher plants, and sundews. The paper should include detailed descriptions of the physical characteristics of each plant, as well as analysis of their unique adaptations for capturing and digesting prey. In addition, the paper should explore the evolutionary history and ecological significance of carnivorous plants, and discuss current research on their use in medicine and biotechnology. The paper should be written in APA format and include least 10 peer-reviewed references.
Prompt:
SPIN Representative Response:
Title: Carnivorous Plants: Form, Function, Evolutionary History, and Applications
Introduction:
Carnivorous plants are fascinating organisms that have evolved unique adaptations to capture and digest prey. These plants are found in nutrient-poor environments where soil lacks sufficient nitrogen and phosphorus. The three most common types of carnivorous plants are Venus flytraps (Dionaea muscipula), pitcher plants (Nepenthes spp.), and sundews (Drosera spp.). This research paper will analyze and compare the form and function of these three types of carnivorous plants, as well as their evolutionary history and ecological significance.
Physical Characteristics and Adaptations;
Venus flytraps are known for their trap-like leaves that can close rapidly when triggered by insects. Each leaf has six trigger hairs that, when touched twice within 20 seconds, cause the trap to close. Once closed, the trap secretes digestive enzymes break down the insect. Venus flytraps are native to North Carolina and South Carolina and require specific environmental conditions, such as high humidity and acidic soil, to thrive.
Pitcher plants are named for their modified leaves that resemble pitchers or cups. These leaves covered in nectar and have slippery surface that makes it difficult for insects to escape once they enter. Once inside, the insect is digested by enzymes secreted by the plant. Pitcher plants are found throughout the world, from tropical rainforests temperate regions, and come in various shapes and sizes. Nepenthes rafflesiana, also known as the Rafflesia pitcher plant, is one of the largest pitcher plants, with pitchers up 30 centimeters tall.
Sundews are named for their sticky glandular hairs that resemble dewdrops. When an insect lands on the plant, the sticky hairs trap it, and the plant secretes digestive enzymes to break down the insect. Sundews are found in many parts of the world, including North America, Europe, and Asia. Drosera capensis, also known the Cape sundew, is a popular species used in horticulture due to its attractive appearance and case care.
Evolutionary History and Ecological Significance:
The evolutionary history of carnivorous plants is still not fully understood, but it is believed that they evolved independently multiple times across different plant lineages. Carnivorous plants likely evolved in nutrient-poor environments a way supplement their nutrition through the capture and digestion of insects.
GPT-5 Score: 5/10
Explanation: Covers the basic features of the plants lacks depth; references are not standardized (limited to general descriptions rather than APA format); missing abstract and overall not well structured as a research paper.
Figure 6: SPIN model response example.
21

---

Published as a conference paper at ICLR 2026
Using least different scientific sources, generate a research paper that analyzes and compares the form and function of three different types of carnivorous plants, including but not limited to Venus flytraps, pitcher plants, and sundews. The paper should include detailed descriptions of the physical characteristics of cach plant, as well as an analysis of their unique adaptations for capturing and digesting prey. In addition, the paper should explore the evolutionary history and ecological significance of carnivorous plants, and discuss current research on their use in medicine and biotechnology. The paper should be written in APA format and include at least 10 peer-reviewed references.
Prompt:
IterDPO Representative Response:
Title: Carnivorous Plants: Form, Function, and Evolutionary Significance
Abstract:
Carnivorous plants are fascinating organisms that have evolved unique mechanisms to cap- and digest prey. This paper compares the form and function of three different types of carnivorous plants: Venus flytraps (Dionaea muscipula), pitcher plants (Nepenthes spp.), and sundews (Drosera spp.). We also discuss the evolutionary history and ecological significance of these plants, well as their potential applications in medicine and biotechnology.
Introduction:
Carnivorous plants are found in nutrient-poor environments where other sources of nitrogen and phosphorus scarce. They have evolved specialized trapping mechanisms obtain these nutrients from insects and small animals. Venus flytraps, pitcher plants, and sundews are three examples of carnivorous plants with distinct trapping strategies.
Venus Flytraps:
Venus flytraps are native to North Carolina and South Carolina in the United States. Their leaves contain trigger hairs that, when touched twice, cause the leaf snap shut, trapping the insect inside (Gurevitch et al., 2017). Once trapped, the insect is digested by enzymes secreted by plant. Venus flytraps have evolved this mechanism to supplement their nutrient intake in nutrient-poor soil (Hunt & Raker, 2018).
Pitcher Plants:
Pitcher plants are found in tropical and subtropical regions worldwide. They have evolved a modified leaf that forms deep, liquid-filled pitfall trap (Kress & Wurdack, 2006). Insects are attracted to the nectar produced by the pitcher and fall into the liquid, where they are digested by enzymes and bacteria. Pitcher plants have evolved this mechanism obtain nutrients from a wide range of prey, including ants, flies, and spiders (Rice, 2014).
Sundews:
Sundews are found in temperate and tropical regions worldwide. They have evolved sticky glands on their leaves that trap insects (Onozco-terWengell & Brodersen, 2015). Once trapped, the insect digested by enzymes secreted by the plant. Sundews have evolved this mech- anism to obtain nutrients from wide range of prey, including flies, mosquitoes, and ants (Rice, 2014).
GPT-5 Score: 6/10
Explanation: Presents a clearer structure with some academic tone, but the discussion is still surface-level and misses key elements like applications and conclusion.
Figure 7: IterDPO model response example.
22

---

Published as a conference paper at ICLR 2026
Using least different scientific sources, generate a research paper that analyzes and compares the form and function of three different types of carnivorous plants, including but not limited to Venus flytraps, pitcher plants, and sundews. The paper should include detailed descriptions of the physical characteristics of cach plant, as well as an analysis of their unique adaptations for capturing and digesting prey. In addition, the paper should explore the evolutionary history and ecological significance of carnivorous plants, and discuss current research on their use in medicine and biotechnology. The paper should be written in APA format and include at least 10 peer-reviewed references.
Prompt:
DRIFT Representative Response:
**Title: Comparative Analysis of Carnivorous Plant Adaptations: Venus Flytraps, Pitcher Plants, and Sundews**
**Abstrac***
Carnivorous plants have evolved remarkable adaptations survive nutrient- poor envi- ronments by capturing and digesting insects. This paper compares the form and function of three prominent carnivorous plants: *Dionaea muscipula* (Venus flytrap), *Nepenthes spp. (pitcher plants), and *Drosea spp.* (sundews). We analyze their physical characteristics. trapping mechanisms, and digestive processes. Additionally, we discuss their evolutionary history, ecological roles, and potential applications in medicine and biotechnology. Our review synthesizes data from multiple studies to provide a comprehensive understanding these fascinating organisms.
** #Introdduttion***
Carnivorous plants are specialized angiosperms that have developed unique strategies to obtain nutrients from animal prey (Lloyd, 1983). They thrive in nutrient-poor soils, where traditional photosynthetic pathways alone cannot sustain growth. Threc notable examples of carnivorous plants are the Venus flytrap (*Dionaea muscipula*), pitcher plants (*Nepenthes *pp.*), and sundews (*Drosera spp. "). Each species has distinct morphological features and physiological mechanisms adapted for prey capture and digestion.
#* Venus Flytrap (*Dionaea muscipula*)e*
The Venus flytrap is characterized by its hinged leaves with sensitive trigger hairs. When insect touches these hairs, the leaf rapidly closes within seconds (Braun & Speck, 2006). The internal structure includes complex system of mechanical sensors and hydraulic pressures that facilitate rapid closure (Lischka et al., 2014).
** **Pitcher Plants (*Nepenthes spp. *)**
Pitcher plants possess modified leaves forming deep, fluid-filled trap. Insects are attracted to the nectar produced the pitcher's entrance and fall into the digestive fluid below (Givnish, 2000). The pitchers contain enzymes and acids that break down the prey (Bramwell & Chaloner, 1975).
**Sundews (*Drosera spp.*) *p*
Sundews are covered sticky glandular hairs that ensnare insects. Once an insect becomes stuck, the plant secretes digestive enzymes to dissolve the prey (O'Leary al., 2010).
GPT-5 Score: 8/10
Explanation: The format is closer to an academic paper (title, abstract. introduction, sub- sections) and using markdown format; covers morphology, function, evolution, ecology, and applications; relatively comprehensive and consistent with review style.
Figure 8: DRIFT model response example.
23


Reviews:
[{"review_id": "Dm5FW5k0gI", "reviewer": "Reviewer_i1Ue", "summary": "This paper proposes DRIFT, a preference-optimization framework that leverages abundant real-world dissatisfaction (DSAT) signals rather than relying primarily on scarce explicit satisfaction (SAT) labels. DRIFT anchors learning on negative examples and dynamically samples positives from the model itself, promoting continued policy improvement. The authors provide theoretical guarantees showing non-vanishing gradients and expected utility improvements, and empirically demonstrate strong performance over IterDPO and SPIN across WildFeedback, WildBench, and AlpacaEval2 benchmarks.", "questions": "- Theoretically, DRIFT maintains a gradient with a lower bound, but could this persistent gradient signal introduce instability?\n\n- Have you evaluated more than two iterations? What happens when DRIFT is run for longer iterative cycles?\n\n- I do not fully understand how Z_all (g) is computed in line 323. What exactly does \"all responses\" refer to in this context?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper provides a clear and practical motivation for leveraging DSAT signals, which are abundant in real-world user logs and therefore highly scalable compared to SAT-based approaches.\n\n- The empirical results are strong, showing consistent improvements over iterative baselines across multiple benchmarks.\n\n- The theoretical analysis is meaningful, demonstrating that DRIFT maintains a non-vanishing learning signal and guarantees improvement under local correlation assumptions with small update steps.\n\n- The method is simple and scalable, integrating easily with existing preference-optimization pipelines without significant architectural changes.", "weaknesses": "- Baseline behavior appears inconsistent with prior literature: IterDPO and SPIN degrade after iteration here, whereas their original works report improvement. This raises questions about reproduction fidelity and tuning fairness.\n\n- The method's reliance on self-generated positives may amplify noise or bias, potentially causing training instability, especially in iterative settings.\n\n- Limited discussion on long-horizon stability; only up to iter-2 results are shown, making it unclear whether performance remains stable or oscillates over extended iterative training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KescU8lXgx", "reviewer": "Reviewer_ukXJ", "summary": "My understanding of the key contribution of the paper, is that performing iterative DPO while using an off-policy sample with negative feedback as the rejected response, and an on-policy generation from the previous model iteration as the chosen response (called DRIFT hereafter), produces better results than: \n- The inverse i.e. iterative optimization where an off-policy sample with positive feedback is used as the chosen response, and an on-policy generation from the previous model iteration is used as the rejected response (called SPIN hereafter) \n- On-policy self-play, where in each iteration multiple on-policy responses are generated and the model itself is used to choose the best and worst response as the chosen resp. rejected response (called IterDPO hereafter) \n\nThe “goodness” is measured via datasets UltraFeedback and WildFeedback, using WildBench (Elo) and AlpacaEval2 (Win rate). \n\nThis finding is relevant because off-policy samples with negative feedback are more ubiquitous than samples with positive feedback, in a context of real-world preference learning, i.e. beyond learning from human-annotated datasets, which are costly to produce.", "questions": "I have one main question, related to the weakness put forward above:  would it be feasible to rerun (part of) the experiment without giving InterDPO and DRIFT the reference point and an improvement instruction, so that we get an “apples to apples” comparison with SPIN, where the on-policy generation is not guided?   \nPlease provide further motivation for the experimental choices, and if my question is deemed justified, additional insights from a more direct comparison may convince me to upgrade my score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The rationale for the proposed approach and experiments is well explained, i.e., the need to move beyond annotation, followed by the observation that in a real-world setting negative feedback is more abundant than positive feedback. \n\n- The experiments are well set up, including reruns based on the papers this paper builds on, i.e. a setup for SPIN, and a setup for IterDPO. \n\n- The paper features valuable analyses, both empirical and theoretical.", "weaknesses": "- In the results we see that SPIN lags behind IterDPO, and IterDPO lags behind DRIFT. However, when looking at how the training samples are created in section 4.1 (Training Recipe), we see that for both IterDPO and DRIFT, in order to create the positive response, the model is guided, i.e. it is given the negative response and an instruction to improve. There does not seem to be an equivalent for SPIN: there is no instruction to create a “worse” negative relative to a reference point. I’m wondering to what extent this has an impact on the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2SdvOlzExR", "reviewer": "Reviewer_aMHA", "summary": "In this paper, the authors introduce DRIFT, a direct (preference) alignment approach that leverages abundant signals from dissatisfactory user-chatbot interactions. The method samples preferred responses from the policy and employs real-world DSAT responses as non-preferred responses, in an iterative, on-policy pipeline.\nThe authors provide experimental validation of their method, theoretical guarantees, as well as insightful analysis of the semantic and reward coverage of the induced policy.", "questions": "# Questions and comments\n\n- Eq2, L184: it is unclear what \"l(*)\" refers to since it does not appear in the equation\n- Algorithm 1 (L186) is never referenced in the text\n- Section 4.1, L211: Could you please provide more details about the curation process of this subset? Were annotators employed? if so, what were they expertise? How many annotations were gathered per prompt or interaction? what was the inter-annotator agreement? what were the rubrics for such annotation? This kind of information would help the community replicate or expand this subset, since it seems it is an important component of the method recipe.\n- L243. The actual HF ID name would be better mentioned as a footnote or in the Appendix. In the main text, a short explanation of what this reward model is would be better for the story.\n- L263. There seems to be different reported percentages for 14B on AlpacaEval2, wrt to numbers in the abstract and the introduction (e.g. 12.11 vs 12.29 %)\n- L267, Table2. It would be beneficial to have a more detailed, realistic discussion of the results, pointing our also the limitations of the method. For instance, on AlpacaEval2, only 7B shows slight improvement, while 14B shows degradation; on both setups (full, controlled)\n- L297, Table2. The statement \"DRIFT maintains steady improvements\" only holds for the full setup, and doesn't for AlpacaEval2 14b, where it shows degradation after the first iteration\n- Table 3. it seems that the wrong cell was bolded in the AlpacaEval2--14B-LC column and AlpacaEval2--7B-LC column. Please add an explanation in the caption.\n- Section 4.3, L323. Could you please elaborate on how the embeddings were calculated? and how is the response reward calculated?\n- Figure 3. Could you please elaborate on what the axes are, and their scales, of the UMAP plot on the left? Example responses from across the region would help us understand the semantic coverage of the represented region", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "S1. The method aims to leverage real-world interactions, allowing the modelling of more nuanced aspects of human preference found in satisfaction / dissatisfaction data.\n\nS2. The authors provide sound theoretical guarantees of vital properties of the method such as avoiding gradient degeneration and maintaining preference margins.", "weaknesses": "W1. The experiments section could benefit from more clarity and details, especially in the diversity-reward coverage analysis\n\nW2. Rather inconclusive results on wildbench and alpaca-eval2, specifically at lower scales or later iterations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NUb9Beb3Qz", "reviewer": "Reviewer_zGYz", "summary": "The paper introduces DRIFT (Dissatisfaction-Refined Iterative Preference Training), a method for aligning large language models using abundant real-world user dissatisfaction (DSAT) signals instead of scarce human-labeled or positive feedback. DRIFT anchors on real DSAT negatives and dynamically samples positives from the evolving model, maintaining effective learning signals.\n\nExperiments on WildFeedback and UltraFeedback show DRIFT outperforms SPIN and IterDPO, achieving up to +12% win-rate gains and even surpassing GPT-4o-mini at 14B scale. The approach preserves response diversity and avoids gradient collapse. Theoretically, it guarantees non-vanishing gradients and stable preference margins.", "questions": "1. How robust is DRIFT to noisy or ambiguous DSAT signals, especially when user dissatisfaction stems from subjective or stylistic preferences rather than factual errors?\n\n2. Could the authors provide insight into how DRIFT scales or adapts when applied to different model families or domains beyond conversational tasks (e.g., coding, reasoning, or multimodal settings)?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Novel Supervision Source: Effectively leverages abundant real-world user dissatisfaction (DSAT) signals—an underutilized yet rich feedback channel—for scalable preference learning.\n\n* Strong Empirical Performance: Demonstrates consistent and significant gains over SPIN and IterDPO on WildBench and AlpacaEval2, even surpassing GPT-4o-mini at 14B scale.\n\n* Theoretical and Practical Robustness: Provides clear theoretical guarantees (non-vanishing gradients, stable preference margins) and preserves exploration diversity, avoiding mode collapse common in self-improving methods.", "weaknesses": "* Label Noise Risk: The approach depends on automatically inferred DSAT labels (e.g., via SPUR), which may introduce noise or misclassifications that affect training quality.\n\n* Limited Generalization Evidence: Experiments are confined to Qwen2.5 models; results on other architectures or tasks would strengthen claims of broad applicability.\n\n* Insufficient Bias and Ethics Discussion: The paper minimally addresses potential biases in user dissatisfaction data or ethical implications of learning from negative, possibly adversarial, feedback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)