Learning to Reason under Off-Policy Guidance
Authors: name: Jianhao Yan; name: Yafu Li; name: Zican Hu; name: Zhi Wang; name: Ganqu Cui; name: Xiaoye Qu; name: Yu Cheng; name: Yue Zhang
Published: 2025-04-21 08:09:13

Abstract: Recent advances in large reasoning models (LRMs) demonstrate that
sophisticated behaviors such as multi-step reasoning and self-reflection can
emerge via reinforcement learning (RL) with simple rule-based rewards. However,
existing zero-RL approaches are inherently ``on-policy'', limiting learning to
a model's own outputs and failing to acquire reasoning abilities beyond its
initial capabilities. We introduce LUFFY (Learning to reason Under oFF-policY
guidance), a framework that augments zero-RL with off-policy reasoning traces.
LUFFY dynamically balances imitation and exploration by combining off-policy
demonstrations with on-policy rollouts during training. Notably, we propose
policy shaping via regularized importance sampling to avoid superficial and
rigid imitation during mixed-policy training. Remarkably, LUFFY achieves an
over +7.0 average gain across six math benchmarks and an advantage of over +6.2
points in out-of-distribution tasks. It also substantially surpasses
imitation-based supervised fine-tuning (SFT), particularly in generalization.
Analysis shows LUFFY not only imitates effectively but also explores beyond
demonstrations, offering a scalable path to train generalizable reasoning
models with off-policy guidance.