On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised
  Fine-Tuning and Reinforcement Learning via Dynamic Weighting
Authors: name: Wenhao Zhang, name: Yuexiang Xie, name: Yuchang Sun, name: Yanxi Chen, name: Guoyin Wang, name: Yaliang Li, name: Bolin Ding, name: Jingren Zhou

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are two
prominent post-training paradigms for refining the capabilities and aligning
the behavior of Large Language Models (LLMs). Existing approaches that
integrate SFT and RL often face the risk of disrupting established model
patterns and inducing overfitting to expert data. To address this, we present a
novel investigation into the unified view of SFT and RL through an off-policy
versus on-policy lens. We propose CHORD, a framework for the Controllable
Harmonization of On- and Off-Policy Reinforcement Learning via Dynamic
Weighting, which reframes SFT not as a separate stage but as a dynamically
weighted auxiliary objective within the on-policy RL process. Based on an
analysis of off-policy expert data's influence at both holistic and granular
levels, we incorporate a dual-control mechanism in CHORD. Specifically, the
framework first employs a global coefficient to holistically guide the
transition from off-policy imitation to on-policy exploration, and then applies
a token-wise weighting function that enables granular learning from expert
tokens, which preserves on-policy exploration and mitigates disruption from
off-policy data. We conduct extensive experiments on widely used benchmarks,
providing empirical evidence that CHORD achieves a stable and efficient
learning process. By effectively harmonizing off-policy expert data with
on-policy exploration, CHORD demonstrates significant improvements over
baselines. We release the implementation at
https://github.com/modelscope/Trinity-RFT/tree/main/examples/mix_chord to
inspire further research.