DRIFT: Learning from Abundant User Dissatisfaction in Real-World
  Preference Learning
Authors: name: Yifan Wang; name: Bolian Li; name: Junlin Wu; name: Zhaoxuan Tan; name: Zheli Liu; name: Ruqi Zhang; name: Ananth Grama; name: Qingkai Zeng
Published: 2025-09-27 03:06:27

Abstract: Real-world large language model deployments (e.g., conversational AI systems,
code generation assistants) naturally generate abundant implicit user
dissatisfaction (DSAT) signals, as users iterate toward better answers through
refinements, corrections, and expressed preferences, while explicit
satisfaction (SAT) feedback is scarce. Existing preference learning approaches
are poorly aligned with this data profile, as they rely on costly human
annotations or assume plentiful positive responses. In this paper, we introduce
DRIFT (Dissatisfaction-Refined Iterative
preFerence Training), which anchors training on real-world
DSAT signals and samples positives dynamically from the evolving policy.
Empirically, DRIFT models trained on real-world WildFeedback datasets
and synthetic UltraFeedback datasets achieve up to +6.23\% (7B) /
+7.61\% (14B) on WildBench Task Score and up to +8.95\% (7B) / +12.29\% (14B)
on AlpacaEval2 win rate over base models, outperforming strong baseline methods
such as iterative DPO and SPIN. At larger scales, the improvements are
particularly pronounced: 14B models trained with DRIFT surpass GPT-4o-mini on
WildBench. Further analysis shows that DRIFT also preserves exploratory
capacity, yielding more diverse high-reward solutions rather than collapsing to
narrow subsets. Theoretically, we demonstrate that this design preserves
preference margins and avoids the gradient degeneration. These results show
that DRIFT is an effective and scalable recipe for real-world post-training
that leverages the most abundant and informative signal. The code and data are
available at https://github.com/cacayaya/DRIFT.git.