SFT or RL? An Early Investigation into Training R1-Like Reasoning Large
  Vision-Language Models
Authors: name: Hardy Chen; name: Haoqin Tu; name: Fali Wang; name: Hui Liu; name: Xianfeng Tang; name: Xinya Du; name: Yuyin Zhou; name: Cihang Xie
Published: 2025-04-10 16:54:05

Abstract: This work revisits the dominant supervised fine-tuning (SFT) then
reinforcement learning (RL) paradigm for training Large Vision-Language Models
(LVLMs), and reveals a key finding: SFT can significantly undermine subsequent
RL by inducing ``pseudo reasoning paths'' imitated from expert models. While
these paths may resemble the native reasoning paths of RL models, they often
involve prolonged, hesitant, less informative steps, and incorrect reasoning.
To systematically study this effect, we introduce VLAA-Thinking, a new
multimodal dataset designed to support reasoning in LVLMs. Constructed via a
six-step pipeline involving captioning, reasoning distillation, answer rewrite
and verification, VLAA-Thinking comprises high-quality, step-by-step visual
reasoning traces for SFT, along with a more challenging RL split from the same
data source. Using this dataset, we conduct extensive experiments comparing
SFT, RL and their combinations. Results show that while SFT helps models learn
reasoning formats, it often locks aligned models into imitative, rigid
reasoning modes that impede further learning. In contrast, building on the
Group Relative Policy Optimization (GRPO) with a novel mixed reward module
integrating both perception and cognition signals, our RL approach fosters more
genuine, adaptive reasoning behavior. Notably, our model VLAA-Thinker, based on
Qwen2.5VL 3B, achieves top-1 performance on Open LMM Reasoning Leaderboard
(https://huggingface.co/spaces/opencompass/Open_LMM_Reasoning_Leaderboard)
among 4B scale LVLMs, surpassing the previous state-of-the-art by 1.8%. We hope
our findings provide valuable insights in developing reasoning-capable LVLMs
and can inform future research in this area.