Title: Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large vision-language model

Slow-thinking reasoning
Abstract: Enhancing large vision-language models (LVLMs) with visual slow-thinking reasoning is crucial for solving complex multimodal tasks. However, since LVLMs are mainly trained with vision-language alignment, it is difficult to adopt on-policy reinforcement learning (RL) to develop the slow thinking ability because the rollout space is restricted by its initial abilities. Off-policy RL offers a way to go beyond the current policy, but directly distilling trajectories from external models may cause visual hallucinations due to mismatched visual perception abilities across models. To address these issues, this paper proposes **SOPHIA**, a simple and scalable **S**emi-**O**ff-**P**olicy RL for vision-language slow-t**HI**nking re**A**soning. SOPHIA builds a semi-off-policy behavior model by combining on-policy visual understanding from a trainable LVLM with off-policy slow-thinking reasoning from a language model, assigns outcome-based rewards to reasoning, and propagates visual rewards backward. Then LVLM learns slow-thinking reasoning ability from the obtained reasoning trajectories using propagated rewards via off-policy RL algorithms. Extensive experiments with InternVL2.5 and InternVL3.0 with 8B and 38B sizes show the effectiveness of SOPHIA. Notably, SOPHIA improves InternVL3.0-38B by 8.50\% in average, reaching state-of-the-art performance among open-source LVLMs on multiple multimodal reasoning benchmarks, and even outperforms some closed-source models (e.g., GPT-4.1) on the challenging MathVision and OlympiadBench, achieving 49.08\% and 49.95\% pass@1 accuracy, respectively. Analysis shows SOPHIA outperforms supervised fine-tuning and direct on-policy RL methods, offering a better policy initialization for further on-policy training.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SOPHIA, a novel Semi-Off-Policy Reinforcement Learning framework designed to enhance Large Vision-Language Models (LVLMs) with visual \"slow-thinking\" reasoning. It addresses the limitations of both on-policy RL's restricted exploration and off-policy RL's risk of visual hallucinations. SOPHIA proposes a semi-off-policy behavior model that integrates the LVLM's visual understanding with off-policy reasoning from external Language Models, using outcome-based and propagated visual rewards. Experiments across diverse visual reasoning benchmarks demonstrate SOPHIA's significant performance improvements for LVLMs like InternVL2.5 and InternVL3.0.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n-  The paper clearly identifies a significant problem in LVLMs' slow-thinking reasoning and presents a well-motivated solution. The writing is notably clear, making the proposed methodology easy to understand.\n-  The core design of SOPHIA is innovative and highly sensible, effectively tackling challenges faced by current RL approaches for LVLMs.\n-  The work demonstrates high quality through comprehensive experimental validation and thorough ablation studies, providing robust support for its effectiveness.\n\nWeaknesses\n- The paper could benefit from a more detailed qualitative analysis of typical error modes before and after SOPHIA training, providing deeper insights into how visual hallucinations or logical inconsistencies are specifically mitigated.\n- While SOPHIA is validated using InternVL2.5 and InternVL3.0, demonstrating its effectiveness when applied to other strong base LVLMs, such as Qwen-VL, as the core model for enhancement, would further strengthen the generalizability claims.\n- The paper also lacks a detailed analysis of computational costs, covering both training phase resource consumption and the inference latency associated with multi-step \"slow-thinking\" reasoning.\n\nQUESTIONS:\nPlease refer to weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SOPHIA, a novel reinforcement learning framework designed to enhance visual slow-thinking reasoning capabilities in large vision-language models (LVLMs). The work proposes the semi-off-policy sampling and reward evaluation and propagation to address a critical limitation in current LVLM training approaches for complex multimodal reasoning tasks. The method also achieves competitive performance across several benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n-\tThe semi-off-policy proposed in the paper cleverly combines the advantages of on policy and off policy, which is a very interesting work.\n-\tThe design of the reward mechanism is relatively novel, avoiding human evaluation and establishing effective end-to-end optimization, with good scalability\n-\tThis method has achieved competitive results on different benchmarks\n\n**Weaknesses**\n\n-\tThe paper lacks a complete description of the process, especially regarding the training and inference processes. It is not easy to understand the complete workflow, for example, does the inference stage require a reasoning language model?\n-\tThe design of the reward mechanism is based on the assumption that high-quality visual descriptions are strongly correlated with final results. Is there clear evidence to support this assumption?\n-\tThe paper only validated InternVL as a VLM, and it is recommended to try other VLMs as this method has been proven to be universal for different VLM models.\n-\tDoes SOPHIA have advantages over direct distillation data? For example, first generate capture data using VLM, and reasoning trajectories with reasoning language model, then build a dataset using reject sampling, and train the model using SFT.\n-\tIt seems that the paper does not analyze the impact of different reasoning models on performance.\n-\tSuggest the author to provide a more detailed explanation of the fundamental difference between the proposed method and knowledge distillation.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SOPHIA, a semi-off-policy reinforcement learning framework designed to enhance slow-thinking reasoning abilities in large vision-language models (LVLMs). SOPHIA combines the on-policy visual understanding of a trainable LVLM with off-policy reasoning trajectories generated by a language model, enabling more robust and scalable learning. Through extensive experiments on InternVL2.5 and InternVL3.0 (at both 8B and 38B scales), SOPHIA achieves state-of-the-art performance on multimodal reasoning tasks, outperforming both open-source and some closed-source models.\n\nSTRENGTHS AND WEAKNESSES:\nFirst of all, my research primarily focuses on 3D vision. RL and VLM are out of my expertise. Therefore, I can only provide a general assessment rather than insightful and critical review.\n\n**Strengths:**\n\n- SOPHIA’s semi-off-policy design smartly combines stable on-policy visual understanding with the flexibility of off-policy reasoning from a stronger language model. This setup addresses key issues like limited exploration in on-policy RL and hallucinations in off-policy RL.\n- The method shows clear performance gains on tough multimodal benchmarks like MathVision and OlympiadBench, outperforming other open-source models and even GPT-4.1 in some cases.\n\n**Weaknesses:**\n- Analysis on visual hallucinations: While SOPHIA aims to reduce hallucinations through visual reward propagation, qualitative examples or failure modes would be benficial to strengthen the claim.\n- Binary reward signal: As acknowledged in the limitation section, the reward mechanism is based on a simple binary outcome and a preference for the shortest correct trajectory. This is a coarse signal that may not capture the nuances of a high-quality reasoning process and could prematurely penalize more complex but valid reasoning paths.\n- Limited generalization in some cases: The results show that performance can degrade on certain tasks or data distributions not well-represented in the training data (e.g., MathVista for the 8B model). This suggests that while powerful, the method's robustness may be sensitive to the domain and quality of the reasoning trajectories it learns from.\n\nQUESTIONS:\nIn general, I believe this paper is solid and the results are impressive. I have some questions that needs further clarification\n\n1. Can the authors provide qualitative examples or visualizations of improved visual grounding after SOPHIA training?\n2. What is the inference latency introduced by using SOPHIA compared to the base LVLM?\n3. Could SOPHIA be applied to other modalities or tasks beyond math reasoning?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a semi-off-policy reinforcement learning framework to\nenhance slow-thinking reasoning in large vision-language models. The approach\nworks by combining on-policy visual captioning from a trainable LVLM with\noff-policy slow-thinking trajectories generated by a reasoning LLM. Visual and\nreasoning rewards are computed and used to fine-tune the LVLM using off-policy\nRL. Experiments on different benchmarks demonstrate the effectiveness of the\nproposed method.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n* The studied problem is important and interesting.\n\n* The motivation for improving the on-policy and off-policy LLM RL is clear.\n\n* The proposed method outperforms supervised fine-tuning and direct on-policy RL methods\n\n# Weaknesses\n\n* A central assumption in this work is that image content relevant to reasoning\ntasks can be effectively captured through textual descriptions. While this may\nhold for certain structured scenes, it overlooks a fundamental difference\nbetween modalities: images are continuous and often encode fine-grained spatial\nand visual information that discrete language space may fail to represent fully.\nRelying entirely on language-based captions generated by an MLLM risks omitting\ncritical visual cues, especially in tasks where spatial relations or low-level\nvisual details influence the reasoning outcome. This paper would benefit from a\nmore explicit discussion of this and an analysis of how potential information\nloss in the captioning step might affect downstream performance.\n\n* The current formulation of the proposed method appears particularly\nwell-suited for tasks in which visual inputs already contain structured symbolic\ncontent, such as text-labeled diagrams or schematic illustrations. In these\ncases, the conversion from image to language retains most of the task-relevant\ninformation, enabling the reasoning LLM to perform effectively. However, for\nscenarios that require more perceptual visual understanding, e.g., interpreting\nnatural images, or abstract style and spatial patterns, this kind of assumption\nmay no longer hold. This limitation is also reflected in the choice of\nevaluation benchmarks: most of the datasets used (e.g., MathVision,\nOlympiadBench, DynaMath) focus on math, science, or diagram-based reasoning\ntasks where textual abstraction is relatively reliable. There is a lack of\nevaluation on more diverse, perception-centric benchmarks that stress visual\ngrounding beyond captionable content.\n\n* The performance gains achieved by SOPHIA are largely the result of\nincorporating reasoning trajectories from stronger language-only models such as\nQwQ-32B-Preview and DeepSeek-R1. Since QwQ and R1 has much stronger inherent\nreasoning capabilities than InternVL, part of the improvement may come simply\nfrom distilling a more capable teacher. While this is a valid and useful\nstrategy, it complicates the interpretation of the results: it is difficult to\ndisentangle the contribution of the semi-off-policy framework itself from that\nof the teacher model’s strength. An ablation study using different teacher\nmodels of varying quality or complexity could help clarify this point.\n\n* Another limitation relates to the training data used to support the method’s\nperformance. A significant portion of the results is based on a private dataset,\nwhich is not released or described in sufficient detail. While the authors take\ncare to evaluate SOPHIA on public benchmarks as well, the strongest gains appear\non the internal dataset, whose construction, domain coverage, and difficulty\nlevel remain unspecified. This makes it challenging to fully assess the\ngenerality of the results or to reproduce them independently. More transparency\non dataset characteristics or representative examples would strengthen the\ncontribution.\n\n* Additionally, the entire experimental study is conducted on InternVL models\nonly, across both 8B and 38B scales. While InternVL is a strong baseline, it\nwould be valuable to test the framework on other MLLM families with different\narchitectures or pretraining approaches. \n\n* What is the time and computation resouces overhead from the proposed method?\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}