Title: Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal

Reinforcement Fine-Tuning

Visual Reasoning
Abstract: Visual reasoning abilities play a crucial role in understanding complex multimodal data, advancing both domain-specific applications and artificial general intelligence (AGI). Existing methods enhance Vision-Language Models (VLMs) through Chain-of-Thought (CoT) supervised fine-tuning using meticulously annotated data. However, this approach may lead to overfitting and cognitive rigidity, limiting the model’s generalization ability under domain shifts and reducing real-world applicability. To overcome these limitations, we propose Reason-RFT, a two-stage reinforcement fine-tuning framework for visual reasoning. First, Supervised Fine-Tuning (SFT) with curated CoT data activates the reasoning potential of VLMs. This is followed by reinforcement learning based on Group Relative Policy Optimization (GRPO), which generates multiple reasoning-response pairs to enhance adaptability to domain shifts. To evaluate Reason-RFT, we reconstructed a comprehensive dataset covering visual counting, structural perception, and spatial transformation, serving as a benchmark for systematic assessment across three key dimensions. Experimental results highlight three advantages: (1) performance enhancement, with Reason-RFT achieving state-of-the-art results and outperforming both open-source and proprietary models; (2) generalization superiority, maintaining robust performance under domain shifts across various tasks; and (3) data efficiency, excelling in few-shot learning scenarios and surpassing full-dataset SFT baselines. Reason-RFT introduces a novel training paradigm for visual reasoning and marks a significant step forward in multimodal research.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Reason-RFT, a two-stage post-training pipeline that aims to endow Vision–Language Models (VLMs) with stronger and more transferable visual-reasoning skills.\n\nStage 1 – SFT activation. The authors first perform SFT with CoT–annotated date so that the model learns step-by-step solutions with correct format.\n\nStage 2 – RL. They then continue training with GRPO and two rewards are leveraged: format reward and accuracy reward.\n\nTo evaluate the models, authors reconstructed a benchmark covering visual counting, structural perception, and spatial transformation.\n\nEmpirically, Reason-RFT, built on Qwen2-VL-2B/7B, (i) reaches or surpasses SOTA open and proprietary models on all three tasks, (ii) retains the lead under significant domain shift, and (iii) achieves > 90 % of full-data SFT performance using < 5 % of the data.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n* Empirical results indicate that the proposed Reason-RFT is effective on visual tasks like counting, perception, etc.\n* Evaluations and ablation studies are detailed and comprehensive.\n* The code and data release could be useful for future research.\n\n## Weaknesses\n\n* Two core parts of the Reason-RFT is directly from existing work. The findings are expected and the novelty is limited.\n* The designed tasks are relatively easy, which limits the contribution of this work and the performance of RL. In Fig.5a, the avg response  length is ~70, which is far shorter than modern reasoning models (e.g., 8k, 16k, 32k generation lengths).\n* Lack of evaluations on more challenging benchmarks, such as MMMU, MMMU-Pro, MathVista, MathVision, etc. These are more crucial benchmarks for reflecting the reasoning abilities of VLMs.\n\nQUESTIONS:\n* Have you tried more recent and advanced VLMs, such as Qwen-2.5-VL?\n\n* Have you tried RL without the format reward?\n\n* Do you observe any \"thinking\" words like \"wait\" or \"let me verify that again\"? From the examples you provided, I don’t see a clear difference between the CoT content and what's inside the <think></think> tags. Does this format really make a difference here?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a two-stage framework for improving the visual reasoning capabilities of existing VLMs. Specifically, the authors propose to first fine-tune a model using SFT on a dataset with CoT reasoning traces, and then apply RL (i.e. GRPO) to further improve performance. The paper includes a systematic evaluation and analysis of the framework.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* Comprehensive evaluation across multiple visual reasoning tasks.\n\n* Analysis of different training dynamics .\n\nWeaknesses:\n\n* While I appreciate the thorough evaluation across three visual reasoning categories, the paper overstates its main technical contribution. The use of SFT on CoT data followed by GRPO-based RL has already been explored in several reasoning works after  DeepSeek-R1. As a result, I believe the claim of Reason-RFT being a “novel training paradigm” in line 24 is a bit overstated. \n\n* The separation between in-domain (ID) and domain-shift (DS) data feels somewhat artificial, especially considering that the base model may have encountered similar samples during pretraining. Moreover, in realistic scenarios, practitioners would likely perform SFT + GRPO over a broader, more diverse dataset rather than separate curated subsets.\n\n* The paper’s presentation lacks clarity. I would recommend that the authors restructure it as an analysis paper , rather than presenting it as a  “novel training paradigm”.  For instance, the  paper should focus  more on how the SFT visual CoT traces were generated or how the reasoning quality influences downstream performance.   In fact, as it stands, it remains unclear how the SFT data was generated and most importantly whether  the SFT with CoT data is actually necessary. For instance, Table 1 shows that Reason-RFT-Zero achieves better average performance on 2 out of 3 tasks.\n\n* While I appreciate the introduced benchmark, there are already a plethora of well-established visual reasoning benchmarks with public leaderboards. I would recommend additionally evaluating the proposed method on at least one of these to facilitate comparison with existing approaches and models.\n\nQUESTIONS:\nHow was the CoT data for the SFT stage generated? Was it generated using another VLM (i.e. GPT), was it human generated or a combination of both?\n\nWhat sampling mechanism and resolution was used in Table 1 for the baselines and the proposed methods? Was it greedy decoding or sampling with temperature? Was there any difference?\n\nHow many runs were used to compute the numbers presented in table1? Can the authors also report std?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a two-stage post-training strategy for vision-language models. The post-training framework comprises two stages: 1. SFT with high-quality CoT data, 2. RLVR with GRPO. The paper provides a comprehensive evaluation of various post-training strategies on three visual reasoning tasks including visual counting, structural perception, and spatial transformation. The experimental study analyses the roles of CoT-SFT and GRPO and demonstrate that the proposed two-stage post-training improves the model performance consistently.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. a solid work that studies the roles of SFT and GRPO in the post-training of vision-language models\n2. the designed evaluation tasks/datasets are interesting and valuable for the future research\n3. the experiments provide insightful analysis and findings\n\nWeaknesses:\n\n1. the choices of base models are limited to QwenVL family. The analysis would be more comprehensive and impactful if the authors can provide the analysis with other base models when considering the interesting findings with Qwen-series models and RLVR in [1]\n\n[1] Reinforcement learning with random rewards actually works with Qwen 2.5, https://www.interconnects.ai/p/reinforcement-learning-with-random?utm_campaign=post&utm_medium=web\n\nQUESTIONS:\n1. Why both ANS-SFT and CoT-SFT leads to worse results than the zero-shot performance of Qwen2VL-7b in visual counting (ID and DS-D)?\n2. In spatial transformation, why ANS-SFT shows very strong results and pure RL gives the worst performance? \n3. Related to the above two questions: it seems that some tasks benefit more from SFT and some of them benefit more from RL. Do you have more analysis with other visual reasoning tasks and any insights how are the visual tasks coupled with the post-training strategies?\n4. How much does the ratio of data size for SFT and RL affect the performance?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the reasoning ability of vision-language models (VLMs) by training them to generate chain-of-thought (CoT) reasoning through supervised learning and reinforcement learning. The authors propose a framework called Reason-RFT, which uses a two-stage training process on vision tasks such as counting, structural perception, and spatial transformation.\nIn the first stage (supervised fine-tuning, or SFT), they create <image, question, chain-of-thought, answer> annotations and train the VLM to produce token-level predictions, meaning it generates both the CoT and the final answer given an image and question.\nIn the second stage, they apply reinforcement learning with a GRPO loss. This involves generating multiple CoT traces, assigning each a reward score, and using these rewards to further refine the model’s parameters.\nExperiments on tasks such as counting, perception, and spatial transformation demonstrate that combining supervised learning and reinforcement learning yields improvements and enhanced generalization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper investigates how to enhance the reasoning process (chain-of-thought, or CoT) through a two-stage approach that combines supervised learning and reinforcement learning. In the first stage, supervised learning is achieved by instruction tuning the VLM. In the second stage, reinforcement learning is applied to the VLM using the GRPO loss.\n\n2. Experiments on VQA tasks covering counting, perception, and spatial reasoning show that the two-stage training pipeline improves performance compared to existing models and offers greater transparency.\n\nWeakness:\n\n1. The two-stage training pipeline is quite similar to the training approach used in DeepSeek-R1. It is essentially an extension of training VLMs in the same manner as LLMs. The main differences lie in the chain-of-thought (CoT) generation in the first stage and the new reward function for VQA tasks, which limits its theoretical contribution.\n\n2. The process for generating new CoT annotations used in the first-stage training (Section 3.1) is not explicitly described. Explaining how the initial CoT data for SFT tuning is created is important, as it affects the ability to reproduce the work.\n\n3. Do the open-source VLM models in Table 1 follow the same zero-shot setting? To ensure a fair comparison, these open-source VLMs should be trained using the same dataset.\n\n4. The Reason-RFT does not generalize well to more generic VQA tasks, such as MMMU, RealWorldQA, and AI2D, as shown in Table 6, across different backbones. The improvements are unstable.\n\n5. Relevant works line NMN [1], IPRM [2] are missing in discussing neuro-symoblic methods (line 79)\n\n[1] Neural Module Networks，CVPR2016\n\n[2] Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios, NeurIPS 2024\n\nQUESTIONS:\n1. How is the initial chain-of-thought (CoT) data for the first-stage supervised fine-tuning (SFT) generated? How do you ensure the reliability and correctness of these first-round CoT annotations?\n2. After reinforcement learning, I would like to see more analysis of the quality of the CoT. Is there any quantitative comparison of CoT outputs before and after GRPO tuning?\n\nLIMITATIONS:\nYes, the author discusses limitations in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Reason-RFT, a two-stage reinforcement fine-tuning (RFT) framework aimed at improving the generalization capabilities of visual reasoning models. The approach combines SFT with CoT data in Stage 1, followed by reinforcement learning via GRPO in Stage 2. The authors construct a visual reasoning benchmark covering visual counting, structure perception, and spatial transformation tasks to evaluate the framework. They report that Reason-RFT enhances reasoning ability, improves robustness under distribution shifts, and achieves data-efficient learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The combination of SFT with CoT and GRPO-based RL is well-motivated, aiming to activate and then further refine reasoning capabilities.\n\n2. The differentiated reward structures for discrete, mathematical, and function-based tasks reflect an attempt to tailor the training to task specifics.\n\nWeaknesses:\n\n1. Lack of Novelty in Training Paradigm: The core idea of combining SFT (especially with CoT) and reinforcement learning is not fundamentally new. Similar SFT + RL pipelines have been widely explored in both language and vision-language models. The distinction of Reason-RFT over existing methods is not made sufficiently clear.\n\n2. Insufficient Detail on SFT Data: The paper does not specify which base model generated the SFT data, nor does it elaborate on the diversity of the reasoning trajectories (e.g., variation in trajectory length, problem types, complexity). This raises concerns about the breadth of the reasoning coverage during SFT.\n\n3. Reward Application Ambiguity: It is not clearly explained how the system determines which reward type (discrete, mathematical, or function-based) to apply during training for each question. \n\n4. Model Choice Unjustified: The authors use Qwen2VL for experiments, but do not explain why Qwen2.5VL was not used. Given that Qwen2VL’s zero-shot performance is already competitive, this choice needs clarification.\n\n5. Marginal Gains over RL Baseline: The performance gap between the RL-only and the combined RL+SFT models is surprisingly small in some tasks (except spatial transformation). This raises questions about the actual contribution of the SFT stage, whether it mainly improves the spatial transformation task or if it offers limited value elsewhere.\n\n6. Limited Novelty in Reward Structure: Although the reward designs are carefully specified, they largely follow standard practices (exact match for discrete, tolerance-based for numeric, sequence overlap for functions).\n\nQUESTIONS:\n1. Novelty of the Training Paradigm: How does Reason-RFT fundamentally differ from other SFT + RL frameworks? What specific innovations make this approach unique beyond using GRPO?\n\n2. SFT Data Generation: Which model was used to generate the SFT-CoT data? How diverse are the reasoning chains in terms of trajectory length and problem complexity?\n\n3. Reward Assignment: How does the system know which reward type (discrete, mathematical, function-based) to apply to each question? Is this predefined in the dataset annotations?\n\n4. Model Selection: Why was Qwen2VL used instead of Qwen2.5VL? Would results change significantly with the more recent model?\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}