Title: Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision-Language Models

Reinforcement Learning
Abstract: Reinforcement Learning (RL) has proven to be an effective post-training strategy for enhancing reasoning in vision–language models (VLMs). Group Relative Policy Optimization (GRPO) is a recent prominent method that encourages models to generate complete reasoning traces before answering, leading to increased token usage and computational cost. Inspired by the human-like thinking process—where people skip reasoning for easy questions but think carefully when needed—we explore how to enable VLMs to first decide *when reasoning is necessary*.
To realize this, we propose \ours, a two-stage training strategy:
**(i)** a supervised fine-tuning (SFT) stage with a simple yet effective “**thought dropout**” operation, where reasoning traces are randomly replaced with empty thoughts. This introduces a think-or-not format that serves as a cold start for selective reasoning;  **(ii)** a GRPO stage that enables the model to freely explore when to think or not, while maximizing task-aware outcome rewards.
Experimental results show that \ours can *reduce the completion length by up to **90%** compared to vanilla GRPO, without sacrificing performance or even improving it*. Further evaluations across LLM (GSM8K), VLM (CLEVR, Super-CLEVR, GeoQA), and Agentic (AITZ) tasks—covering a range of reasoning difficulties under both 3B and 7B models—consistently reveal that the \textit{model progressively learns to bypass unnecessary reasoning steps as training advances}.
These findings shed light on the path toward human-like reasoning patterns in RL approaches.
Our code is available at https://github.com/kokolerk/TON.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an adaptive training framework that lets Vision-Language Models flexibly produce either short or long chains of thought (CoT). The method uses a two-stage pipeline: **(1) supervised fine-tuning (SFT)** to establish two basic reasoning patterns followed by **(2) guided reinforcement preference optimization (GRPO)** to adjust CoT length to task demands. Experiments show that the framework simultaneously shortens unnecessary reasoning steps and improves overall performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* Delivers strong performance on all reported benchmarks.\n\n**Weaknesses**\n\n* The motivation for “over-dense” chains of thought is not fully substantiated. The evaluated tasks never exceed ≈ 1 k tokens of reasoning, so it is unclear why such CoTs are considered excessively long.\n\nQUESTIONS:\n**Polished reviewer questions (with citations added)**\n\n1. **Scope of application.**\n   The framework is tested only on VLM reasoning tasks, yet recent length-adaptive methods in *text-only* settings—e.g., **ThinkLess** and **LASER**—report that LLM CoT traces can easily reach thousands of tokens, making compression arguably more urgent there. Could the authors clarify why the vision-language modality was chosen as the primary testbed, and whether the approach generalises to purely textual reasoning? ([arxiv.org][1], [arxiv.org][2])\n\n2. **Source of performance gains.**\n   The stated goal is to remove redundant reasoning steps; in principle this should *preserve* accuracy, not improve it. What specific mechanisms (e.g., better credit assignment in GRPO, reduced exposure bias) lead to the observed accuracy gains? Please provide ablations that disentangle efficiency improvements from accuracy improvements.\n\n3. **Positioning against related work.**\n   Please situate the contribution with respect to existing length-adaptive approaches such as **ThinkLess**, **LASER**, **AdaCoT**, and other “think-less” / difficulty-aware reward-shaping methods. What is the key novelty—reward design, the two-stage SFT + GRPO pipeline, or the adaptation to VLMs? A concise related-work discussion with explicit citations would help readers see the incremental contribution. ([1，2])\n\n[1]: \"ThinkLess: A Training-Free Inference-Efficient Method for Reducing Reasoning Redundancy\"\n[2]: \"Learn to Reason Efficiently with Adaptive Length-based Reward Shaping\"\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies whether the model should conduct chain-of-thought reasoning or not in the multimodal LLM. The idea is quite simple and straightforward: replacing the thinking process with an empty thinking token. The motivation is the author found that in some cases, the thinking process might hurt the performance. Especially there are a high percentage where the model answered the questions incorrectly with the thinking mode on, but it can answer correctly without thinking mode. This motivated the author to study the thinking dropout approach. This is activated in the SFT stage. The GRPO algorithm is also updated with the thinking dropout process. This simple approach reduces the completion length by up to 90% and achieves quite good performance on counting (CLEVR) and Math(GeoQA) tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper is well written. It is very easy to follow the whole paper. It is actually a joy to read this paper.\n2. The proposed approach is very well motivated. Especially the Figure 2, which demonstrates the necessity for enabling the thinking dropout approach.\n3. The approach is quite simple and straightforward. Frankly speaking, it just several line changes in the data processors. I am quite surprising about the simplicity of this approach and eager to try it out by myself.\n4. The proposed approach achieves quite good performance in both efficiency and accuracy.\n\nWeakness:\nFrankly speaking, i didn't observe significant weakness of this work. The only thing that can improve is the clarity for how to update the GRPO. But the rest looks quite good to me. I would love to see this paper being accepted to this conference. It would be great if the author could also release the implementation.\n\nQUESTIONS:\nThis is a good paper. I have several questions:\n\n1. For the GRPO part, I didn't fully grasp how to update the GRPO algorithm to adapt the change of thinking dropout. Specifically, since the format reward is being added, how to handle the format reward for the sequence without thinking?\n2. This approach should also be applicable to the text only LLM. I wonder what is the reason for studying this in the MLLM domain? Why not also studying this in the text only LLM domain and observe the performance improvement?\n3. For the reverse thinking, how to verify the correctness of the thinking process?\n4. An naive question for L141, why can't we just use the third party (closed source APIs) to generate the thinking process as the starting point?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper introduces TON, a two-stage training framework for vision-language models (VLMs) that instills selective reasoning ability by combining supervised fine-tuning with “thought dropout” and reinforcement learning. TON, evaluated on multiple benchmarks, substantially reduces reasoning trace length, while sometimes improving accuracy over GRPO. The approach demonstrates strong out-of-distribution generalization.

STRENGTHS AND WEAKNESSES:
[Strength]
1. The idea to equip VLMs with selective reasoning is practical and well-motivated.
2. TON achieves a significant reduction in average reasoning length without sacrificing performance, surpassing vanilla GRPO.
3. The reward design with continuous matching is straightforward yet effective across benchmarks.


[Weaknesses]
1. Applying random dropout of reasoning traces during SFT may harm the model’s reasoning on complex queries, such as GeoQA, where reasoning steps are critical.
2. This paper lacks a detailed analysis of which query types the model learns to skip reasoning for after training. Providing quantitative insights into these categories would clarify the model’s selectivity and failure modes.

QUESTIONS:
Could the authors clarify the statement in L217 regarding why directly applying GRPO leads to a zero coordinate reward? The explanation is unclear.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConsidering current vision-language model using RL to enhance reasoning ability causing redundant response, the author is motivated to enable models to learn selective reasoning—knowing when to reason and when to skip it—rather than indiscriminately generating reasoning traces, this paper proposes TON, a two-stage method: SFT+GRPO. In the first stage, the author teaches the model to skip reasoning appropriately by prompting model to self-generate reasoning trace given image, question, answer and later randomly replacing reasoning traces with empty “\\n\\n” using to construct dataset. In the second stage, the author designs discrete matching and continuous matching reward to optimize model’s selective reasoning ability. The primary contribution of this work lies in introducing “selective reasoning” as a trainable capability for vision-language models, significantly reducing redundancy in model responses.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper presents a technically sound framework with comprehensive experiments across multiple tasks and model scales. The use of thought dropout and reinforcement learning is well-motivated and evaluated through thorough ablations and clear metrics. However, the construction of dataset by randomly dropout reasoning trace may lead to model skip some complex reasoning after SFT which can’t be optimized effectively by GRPO with task-specific two kinds of reward designs.\nThe paper is well-organized and clearly communicates its motivation and methodology. By implementing ablation experiments of with/without thinking on AITZ task, the paper introduces its motivation of selective reasoning. And the experiment part, especially the ablation of 3 questions well suits the intuitive consideration of the “think or not” mechanism. The only slight shortcoming lies in the implementation of selective reasoning — the rationale for adopting a two-stage approach requires further clarification.\nThe work addresses reducing reasoning overhead in vision-language models by reinforcing selective reasoning. This has practical value for efficient inference in real-world applications.\nThe idea of teaching models when to reason, rather than how, is a novel and insightful shift. Although similar ideas exist in adaptive computation, the integration into multimodal RL training is new and well-executed.\n\nQUESTIONS:\n（1）Clarification of the Two-Stage Design Rationale (SFT + GRPO)\nThe paper proposes a two-stage framework for selective reasoning, beginning with SFT using thought dropout and followed by a GRPO-based reinforcement learning stage. However, the motivation for structuring the method in this specific two-stage manner is not fully explained. In particular, it remains unclear whether random dropout may lead to skip complex reasoning after SFT. Could the authors provide more conceptual reasoning or design analysis supporting this choice? Including additional ablation experiments comparing dropout strategies would help clarify this point. Addressing this question could significantly improve the paper’s Clarity score.\n\n（2）Generalizability of Reward Design in the GRPO Stage\nDoes the reward design satisfy the case when outcomes may not be easily judged by binary or discrete metrics? It may be more convincible to comment on the adaptability of reward design to other situation? For instance, would the current selective reasoning policy be viable in tasks lacking deterministic rewards? If the approach can be extended or generalized to such settings, it would enhance the broader applicability of selective reasoning and justify raising the Significance rating.\n\n（3）Deeper Analysis of “Without Think but Correct” Cases in Ablation\nIn the early-stage ablation where responses with and without “think” are compared, a significant proportion of cases are correctly answered without reasoning while being incorrect with it. This is intriguing but underexplored. Could the authors analyze these cases more deeply—are the errors in the “think” version due to hallucinated reasoning, misaligned chains of thought, or memory interference? Furthermore, after applying TON, how do such cases behave? Are they handled better by skipping reasoning, and if so, what insights can we draw about the model’s internal reasoning reliability? A targeted analysis would shed light on whether selective reasoning improves not just efficiency but also robustness. This could positively impact the Clarity score if addressed with empirical evidence or case studies.\n\n（4）The specificity of vision-language tasks compared to only language task\nThe proposed method does not appear to differentiate significantly between pure-text and vision-language scenarios. While the perspective of exploring reasoning length in vision-language tasks is novel, the method itself does not explicitly leverage the unique characteristics of vision-language reasoning. Could the authors consider incorporating task-specific aspects of vision, or provide a comparison that highlights how this approach differs from existing methods for reducing reasoning length in pure language models? Addressing this could support an increase in both the Significance and Originality scores.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to enable selective reasoning for VLMs. The\nproposed method works by using a simple \"random thought dropout\" approach during\nsupervised fine-tuning and incentivize \"selective thought dropout\" during RL\npost-training (e.g., GRPO). Experiments on models from 3B to 7B demonstrate the\neffectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n* The studied problem is interesting. The motivation is clear.\n\n* The overall writing of this paper is good.\n\n* Both out-of-distribution data and in-distribution data is involved in the evaluation.\n\n# Weaknesses\n\n* The proposed thought dropout mechanism dring SFT is applied randomly without\nconsidering task difficulty or reasoning necessity. This could interfere with\nthe model’s reasoning capability and the ability to calibrate when reasoning is needed. It is suggested to\nadd more discussion about this potential drawback (although the RL stage might be able to fix, it could still introduce potential negative effects). More principled approaches,\nsuch as difficulty-aware or confidence-based dropout might be also helpful.\nThese adaptive approaches are not explored or discussed. In addition, the optimal dropout rate is likely to depend on the difficulty distribution of the SFT dataset. A discussion on how to align dropout rates with dataset characteristics is missing and would improve the soundness of the proposed method.\n\n* Although the scope of this paper is limited to VLMs, the method itself is\ngeneric and could easily be applied to text-only LLMs. However, there are many\nexisting works on adaptive reasoning on LLMs, such as those mentioned in the\nrelated work section, but none are included in the experimental comparison.\nThis limits the clarity of the paper’s contribution comparing to prior works.\n\n* All results are based solely on Qwen2.5-VL 3B and 7B. There is no evaluation\non other model families (e.g., InternVL3 and LLaMA 3.2 vision) or larger models.\nIt is unclear whether the method scales to stronger models or transfers across\ndifferent VLM architecturesa, which weakens the generality of the proposed\nmethod and the contribution. Is the injection of the thought dropout harder for\nlarger models as they have stronger reasoning capability and better inherent\ncalibration of when reasoning is needed?\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}