SRFT: A Single-Stage Method with Supervised and Reinforcement
  Fine-Tuning for Reasoning
Authors: name: Yuqian Fu; name: Tinghong Chen; name: Jiajun Chai; name: Xihuai Wang; name: Songjun Tu; name: Guojun Yin; name: Wei Lin; name: Qichao Zhang; name: Yuanheng Zhu; name: Dongbin Zhao
Published: 2025-06-24 16:31:37

Abstract: Large language models (LLMs) have achieved remarkable progress in reasoning
tasks, yet the optimal integration of Supervised Fine-Tuning (SFT) and
Reinforcement Learning (RL) remains a fundamental challenge. Through
comprehensive analysis of token distributions, learning dynamics, and
integration mechanisms from entropy-based perspectives, we reveal key
differences between these paradigms: SFT induces coarse-grained global changes
to LLM policy distributions, while RL performs fine-grained selective
optimizations, with entropy serving as a critical indicator of training
effectiveness. Building on these observations, we propose Supervised
Reinforcement Fine-Tuning (SRFT), a single-stage method that unifies both
fine-tuning paradigms through entropy-aware weighting mechanisms. Our approach
simultaneously applies SFT and RL to directly optimize the LLM using
demonstrations and self-exploration rollouts rather than through two-stage
sequential methods. Extensive experiments show that SRFT achieves 59.1% average
accuracy, outperforming zero-RL methods by 9.0% on five mathematical reasoning
benchmarks and 10.9% on three out-of-distribution benchmarks.