Title: OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision-language reasoning

iterative improvement

distillation

reinforcement learning
Abstract: We introduce *OpenVLThinker*, one of the first open-source large vision–language models (LVLMs) to exhibit sophisticated chain-of-thought reasoning, achieving notable performance gains on challenging visual reasoning tasks. While text-based reasoning models (e.g., Deepseek R1) show promising results in text-only tasks, distilling their reasoning into LVLMs via supervised fine-tuning (SFT) often results in performance degradation due to imprecise visual grounding. Conversely, purely reinforcement learning (RL)-based methods face a large search space, hindering the emergence of reflective behaviors in smaller models (e.g., 7B LVLMs). Surprisingly, alternating between SFT and RL ultimately results in significant performance improvements after a few iterations. Our analysis reveals that the base model rarely exhibits reasoning behaviors initially, but SFT effectively surfaces these latent actions and narrows the RL search space, accelerating the development of reasoning capabilities. Each subsequent RL stage further refines the model's reasoning skills, producing higher-quality SFT data for continued self-improvement. OpenVLThinker-7B consistently advances performance across six benchmarks demanding mathematical and general reasoning, notably improving MathVista by 3.2\%, EMMA by 1.4\%, and HallusionBench by 2.7\%. Beyond demonstrating the synergy between SFT and RL for complex reasoning tasks, our findings provide early evidence towards achieving R1-style reasoning in multimodal contexts.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces OpenVLThinker-7B, an open-source Large Vision-Language Model (LVLM) based on Qwen2.5-VL-7B, designed to exhibit complex chain-of-thought (CoT) reasoning. The core methodology is an \"iterative SFT-RL cycle\", where Supervised Fine-Tuning (SFT) is alternated with Reinforcement Learning (RL) using Group Relative Policy Optimization (GRPO). The authors propose that SFT helps \"surface latent reasoning actions\", and RL subsequently refines these. In each iteration, the base model is retrained using data generated by the model from the preceding iteration. The paper reports performance gains on six benchmarks, such as +3.2% on MathVista and +2.7% on HallusionBench, claiming these are achieved with high data efficiency (12K samples per iteration) compared to concurrent methods.\n\nSTRENGTHS AND WEAKNESSES:\n$$\\textbf{Strengths}$$\n\n- The paper claims data efficiency using only 12K training samples per iteration to achieve its results, which is considerably less than several concurrent 7B models using single-iteration SFT+RL (e.g., 150K-270K). \n\n- Provides an open-source LVLM (OpenVLThinker-7B) capable of enhanced reasoning is valuable for the research community. \n\n- The process of iteratively generating better training data using the previous iteration's model to train a new model instance from a base checkpoint is systematic. Figure 3 shows progressive improvement on MathVista, supporting this data refinement strategy. \n\n- The attempt to analyze how SFT and RL affect the occurrence of specific keywords (e.g., \"wait,\" \"check\") is a good step towards understanding model behavior changes, forming the basis of the \"action highlighting\" hypothesis. \n\n$$\\textbf{Weaknesses}$$\n\n- The paper states the model is retrained \"from scratch at each iteration\" or from the \"base model Qwen2.5-VL-7B\" using data from the prior iteration. This is more accurately an iterative data refinement and curriculum generation strategy for training new model instances rather than the \"self-improvement\" of a single, continuously evolving model. \n\n- The SFT \"action highlighting\" hypothesis, while interesting, is not clearly demonstrated to be mechanistically distinct from SFT providing a good initialization or behavioral prior for RL. \n\n- Some effort could help address conflicting findings from relevant concurrent research. For instance, the paper's premise that the \"base model seldom exhibits reasoning behaviors initially\" is challenged by Chen et al. (VLAA-Thinker), who report observing reflective reasoning patterns in the same Qwen2.5-VL base model before RL. This discrepancy undermines a key motivation for the proposed SFT role. Similarly, the paper's positive framing of SFT (surfacing latent actions, narrowing RL search space) contrasts with VLAA-Thinker's caution that SFT can induce \"pseudo reasoning paths\" detrimental to subsequent RL. Given VLAA-Thinker is a cited concurrent model using the same base architecture, this lack of discussion is a major omission. \n\n- The paper argues that initial SFT on text-based CoTs (with image captions) can degrade performance due to \"imprecise visual grounding\". It then claims the iterative SFT-RL cycle, where RL \"refines the model's reasoning skills\", rectifies this. Can you provide some clarity on RL's Mechanism for Improving Visual Grounding.\n\n- There is insufficient explanation of how the RL component (GRPO), which primarily optimizes policies based on ranked preferences of textual outputs, directly addresses or improves multimodal visual grounding. This link is critical but underspecified.\n\nQUESTIONS:\n- The paper asserts the Qwen2.5-VL-7B base model \"seldom exhibits reasoning behaviors initially\". This contrasts with findings from Chen et al. (VLAA-Thinker), who note reflective patterns in the same base model family prior to RL. Could you please discuss this discrepancy? \n\n- VLAA-Thinker also suggests SFT can induce \"pseudo reasoning paths,\" contrasting with your view of SFT effectively surfacing actions and narrowing the RL search space. How do you reconcile these differing perspectives on SFT's role? \n\n- Given that SFT on text-based CoTs (from DeepSeek-R1 and captions) is stated to cause a performance drop due to \"a lack of precise visual grounding\", could you provide a detailed explanation of the specific mechanism by which GRPO, an RL algorithm optimizing policy based on textual output preferences, directly rectifies this multimodal visual grounding issue? \n\n- \"Iterative Self-Improvement\" involves retraining from the base Qwen2.5-VL-7B model in each iteration using data from the previous iteration's model. Why is this termed \"iterative self-improvement\" of a single evolving model rather than an iterative data refinement strategy that produces a sequence of distinct, newly trained model instances? \n\n- What are the advantages of this full retraining approach versus continuously fine-tuning the Iter(i-1) checkpoint to create Iter(i), especially if aiming for \"self-improvement\"?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces OpenVLThinker-7B, one of the first open-source large vision-language models (LVLMs) to demonstrate sophisticated chain-of-thought reasoning capabilities similar to proprietary models like GPT-o1. The authors propose an iterative training methodology that alternates between supervised fine-tuning (SFT) and reinforcement learning (RL) stages. The key insight is that SFT acts as an \"action highlighting\" mechanism that surfaces latent reasoning behaviors (triggered by keywords like \"wait\", \"check\"), which RL can then effectively optimize. Starting from Qwen2.5-VL-7B, the model is trained through multiple iterations using only 12K examples per iteration, achieving competitive performance across six challenging benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe iterative SFT-RL approach is innovative and well-motivated. The authors provide compelling analysis (Figure 6) showing how SFT surfaces reasoning keywords that are rarely present in the base model, while RL refines performance without dramatically changing the action distribution.\n\n Despite using only 12K training examples, OpenVLThinker-7B achieves competitive or superior performance compared to concurrent models using 10x more data (150-270K examples). The 2% average improvement over the base model and 2.7% reduction in hallucinations on HallusionBench are notable.\n\nThe paper includes insightful analysis of reasoning behaviors, including keyword frequency analysis and the role of SFT in guiding RL exploration. The curriculum RL approach based on data difficulty is also well-designed.\n\nThe paper is generally well-written with clear figures illustrating the methodology and results. The examples in Figure 4 and Appendix C effectively demonstrate the model's reasoning capabilities.\n\nWeaknesses\n\nThe paper doesn't discuss the computational requirements of the iterative training process. With RL requiring 20 hours per iteration (mentioned in Appendix B), the total training cost could be substantial. How does it compare to baseline methods? I think the improvements seem marginal considering the great cost.\n\nThe paper primarily compares against the base model and concurrent methods. More comparisons with standard vision-language training approaches (e.g., single-stage SFT or RL) would strengthen the evaluation.\n\nThe initial SFT data generation process using image captions and QwQ-32B is somewhat ad-hoc. The impact of caption quality on final performance could be better analyzed beyond the brief discussion in Appendix A.\n\nAll experiments use models from the Qwen family. It's unclear whether the approach generalizes to other model architectures or if the benefits are specific to Qwen's pretraining. See the paper \"ognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs\".\n\nQUESTIONS:\nWhat is the total computational cost of the iterative training process compared to standard approaches? Could you provide FLOPs or GPU-hours comparisons?\n\nCan you provide results for single-stage SFT-only and RL-only baselines trained on the same 12K examples? This would clarify the benefit of iteration versus simply using both techniques.\n\nHave you tested the approach on other base models (e.g., LLaVA, InternVL)? Is the success tied to specific properties of Qwen models?\n\nCan you elaborate on the connection to EM algorithms mentioned in line 173? A more formal analysis of why iteration helps could strengthen the paper.\n\nIn Table 3, \"Vanilla\" SFT shows a large performance drop (68.5→57.5). Could you analyze what specific aspects of the distilled reasoning traces cause this degradation?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an open-source Large Vision-Language Model (LVLM), OpenVLThinker. The model adopts an iterative training strategy that alternates between Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), gradually introducing and enhancing complex Chain-of-Thought (CoT) reasoning capabilities on top of a non-reasoning vision-language base model. Results show that OpenVLThinker achieves certain performance improvements across six benchmark tasks that require mathematical and general reasoning abilities.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths：\n1、\tThe study proposes an iterative SFT-RL loop. The resulting model outperforms open-source vision-language models of comparable scale on multiple reasoning benchmarks and achieves performance comparable to proprietary models such as GPT-4o. Furthermore, compared to models trained with SFT or RL alone, the proposed approach attains superior reasoning capabilities while using only about one-tenth of the training data.\n2、\tThe paper investigates the synergy between SFT and RL in acquiring reasoning capabilities. Specifically, the SFT stage helps narrow the RL search space, guiding the model toward reasonable reasoning traces. RL then further optimizes and enhances the model’s performance.\n3、\tThe study also explores the impact of data difficulty during the RL stage. The results show that a progressive two-stage training strategy—starting with medium-difficulty samples followed by hard ones—more effectively enhances the model’s complex reasoning ability than using simple or uniform difficulty data.\n\n## Weaknesses：\n1、\tHow were the representative keywords for inference-time actions selected? For example, why was \"maybe\" chosen as the representative keyword for \"seeking alternatives,\" and why was \"identify\" selected, given that its usage frequency significantly decreases after supervised fine-tuning?\n2、\tEach training iteration restarts from the base model rather than continuing from the intermediate model of the previous iteration. This design choice might be further explained.\n3、\tThe evaluation benchmarks are relatively limited. Expanding the diversity of benchmarks would better demonstrate the model’s general reasoning capabilities.\n\nQUESTIONS:\nPlease see weaknesses for details.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}