Title: To Think or Not To Think: A Study of Thinking in Rule-Based Visual Reinforcement Fine-Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Visual Reinforcement Fine-Tuning

explicit thinking

overthinking
Abstract: This paper investigates the role of explicit thinking process in rule-based reinforcement fine-tuning (RFT) for multi-modal large language models (MLLMs). We first extend \textit{Thinking-RFT} to image classification task, using verifiable rewards for fine-tuning~(FT). Experiments show {Thinking-RFT} significantly outperforms supervised FT and yields a cross-dataset generalization effect. We then rethink and question whether explicit thinking in RFT is always necessary and beneficial. Challenging the convention that explicit thinking is crucial for the success of RFT, we introduce \textit{No-Thinking-RFT}, exploring RFT without thinking by introducing a simple equality accuracy reward. We evaluate No-Thinking-RFT on six diverse tasks across different model sizes and types. Experiment results reveal four key findings: \textbf{(1).} Visual perception tasks do not require thinking during RFT, as No-Thinking-RFT consistently outperforms or matches Thinking-RFT across model sizes and types. \textbf{(2).} Models with limited capabilities struggle to generate high-quality CoT for RFT, making Thinking-RFT less effective than No-Thinking-RFT. \textbf{(3).} There are inconsistencies between the answers in the thinking tags and answer tags for some responses of Thinking-RFT, which show lower average accuracy than the overall accuracy. \textbf{(4).} The performance gain of No-Thinking-RFT mainly stems from improved learning during no thinking FT and the avoidance of inference overthinking, as evidenced by the partial gains from appending empty thinking tags at inference time of Thinking-RFT. We hypothesize that explicit thinking before verifiable answers may hinder reward convergence and reduce performance in certain scenarios. To test this, we propose \textit{Think-After-Answer}, which places thinking after the answer to mitigate this effect for experimental verification. Lastly, we conduct a pilot study to explore whether MLLMs can learn when to think during RFT, introducing an \textit{Adaptive-Thinking} method. Experiments show that model converges to either thinking or not depending on model capability, achieving comparable or better performance than both Thinking and No-Thinking-RFT. Our findings suggest MLLMs can adaptively decide to think or not based on their capabilities and task complexity, offering insights into the thinking process in RFT.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author investigates whether the \"think\" process is essential during the Rejection-Fine-Tuning (RFT) stage. Models that undergo Reinforcement Learning from Verifier Feedback (RLVR) typically show significant improvements in reasoning performance, a gain often attributed to the establishment of a stable thinking process. However, the author finds that even when the model is not required to output a \"think\" process during training, it can achieve performance comparable to conventional RLVR. The topic explored in this paper is very interesting, and the experiments are comprehensive.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n- I quite like the motivation of this paper.\n- The paper explores many aspects and challenges the community's belief that the \"think\" process is the primary source of gains from RFT. The experiments are solid. \n\n\n### Weaknesses\n\n1.  The use of a 4-shot training setting (i.e., 4 examples per class), especially when sufficient data is available, seems somewhat \"toy-like.\" This small-data regime is also not the typical scenario where Supervised Fine-Tuning (SFT) is claimed to be most effective.\n\n2.  Although this work largely argues that the \"think\" process is not essential, it can also be seen as fundamentally challenging the RFT process itself. The current characteristics of RFT can be broken down into the following points:\n    - The model samples both \"think\" and \"answer\".\n    - A verifiable reward that is not easily \"hacked.\"\n    - Learning from a set of positive and negative samples.\n    \n    If the \"think\" process is removed from RFT, the model only needs to sample the \"answer.\" For training samples with verifiable rewards (common in domains like math or code), constructing answers is straightforward. For example, given the problem \"1+1 = ?\", we can easily generate answers from the range [1-10], treating \"2\" as the positive sample and the rest as negative samples. By applying a list-wise DPO algorithm, one could achieve a similar effect to \"no-think RFT.\" Since offline DPO does not involve online sampling, its cost is much lower than that of \"no-think RFT.\" This could potentially lead to a scenario where **offline DPO ≥ no-think RFT ≥ RFT**. I believe this comparative analysis would be very valuable, as this transitive relationship might suggest a new RL paradigm.\n\n3.  There is another difference between \"no-think RFT\" and RFT that needs consideration. \"no-think RFT\" can essentially be considered a learning process with one positive example versus multiple negative examples. In contrast, RFT can generate diverse positive examples by combining various \"think\" processes with the correct answer. In \"no-think RFT,\" however, the positive example is unique since it only consists of the final answer. This analytical experiment is also missing from the paper.\n\n4.  Section 4.1 (L229-230) directly assumes that the \"explicit\" think process is harmful. However, based on the experimental data, this conclusion only seems to hold true for the small model (2B). In the experiments with the 7B model, although \"no-think\" scores higher than \"think,\" the \"adaptive think\" mode, which clearly converges to a thinking pattern, achieves the highest score. This results in a relationship of `think < no-think < adaptive-think`. Or, given the small differences, it could be interpreted as `think ≈ no-think ≈ adaptive-think`. In either case, for the 7B model, one cannot conclude that the explicit \"think\" process is harmful.\n\n5.  The significant differences between the 2B and 7B models—in their convergence trends for \"adaptive think\" and in the perceived harm of the explicit \"think\" process—are noteworthy. This raises the question of which of this paper's conclusions can be generalized to larger 32B or 72B models. If the trends for 2B and 7B were similar, we could be more optimistic that the paper's conclusions have good scaling properties.\n\n6.  The \"free lunch phenomenon\" sounds strange. Isn't this just about performance transfer? I don't see this as a particularly novel finding. What is the author trying to convey in this section?\n\n7.  Figure 6: A figure belonging in the main text is placed in the appendix?\n\nQUESTIONS:\nPlease refer to the weakness\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first extends Thinking-RFT to visual tasks and then investigates whether explicit thinking is always necessary and beneficial in RFT. This is motivated by Deepseek R1 shows that explicit thinking can help model improve the performance. The authors propose No-Thinking-RFT on 2B and 7B models, demonstrating that explicit thinking does not consistently improve performance during RFT in visual perception tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper extends the question of explicit thinking to multimodal LMs, providing valuable insight into scenarios where explicit thinking during RFT may not always be beneficial, particularly highlighting that for small models explicit thinking can sometimes negatively impact performance.\n- The authors also introduce an Adaptive-Thinking approach, enabling the model to learn autonomously when explicit thinking should be applied, which is an interesting contribution.\n\nWeaknesses:\n- The authors should include a section in related work about the literature on the negative impacts of explicit thinking (e.g. cot & overthinking) This would provide necessary context and better frame the motivation for their proposed No-Thinking-RFT method. (e.g. https://arxiv.org/abs/2503.16419 (survey paper as a good reference), https://arxiv.org/abs/2410.21333 (task2)).\n- The summary of contributions in the introduction is lengthy. It would be more effective and reader-friendly if presented in a concise and clear short sentences, highlighting the key contributions explicitly.\n- The implementation details section needs more clarity and completeness. Important hyperparameters such as temperature, random seeds, and other settings are important for reproducing the results should be explicitly mentioned.\n- Figures 2 and 3 have readability issues—the font size is too small, and the colors (particularly blue, representing decreases in Figure 2) are not clearly distinguishable. Authors should improve these visualizations to better show their results and enhance readability.\n- Section 4 highlights several critical findings and contains substantial insights. This section might be better placed earlier in the paper (e.g., before the case study on classification tasks). Reorganizing the structure to prioritize main results and findings would enhance readability and clarity in presenting the core contributions.\n\nQUESTIONS:\n1. The authors mention Deepseek-R1 as the motivation; however, I think Deepseek-R1 does not emphasize the benefits of explicit thinking specifically in small-scale models. Since the experiments presented in this paper focus exclusively on models up to 7B, the research question \"Is explicit thinking always necessary and beneficial for RFT?\" feels overly broad. The research question should be refined explicitly to address smaller-scale models (e.g., below 7B) or include additional experiments with larger models to strengthen the generality of the conclusion.\n2. In some tasks, such as EuroSAT, the No-Thinking-RFT method underperforms compared to SFT. Do authors consider these cases as the outliers? Hope to see more insights and explanations regarding these discrepancies.\n3. Do you think the Thinking-RFT has poor performance is because of the low-quality explicit thinking? Also, an open question is whether more fine-grained or nuanced reward designs for the thinking process could potentially improve Thinking-RFT’s performance. An ablation study or additional analysis would great to have.\n\nNote: I'm happy to increase my score if the authors can address my concerns in weakness and questions sections.\n\nLIMITATIONS:\nThe experiments lack error bars, confidence intervals, or multiple runs with different random seeds to ensure robustness and statistical significance of the results.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes No-Thinking-RFT, replacing the reward of RFT with an equality accuracy award. The No-Thinking-RFT is compared against RFT first in an image classification task, followed by a suite of visual reasoning tasks. Further analyses, including Think-After-Answer and Adaptive-Thinking are included to strengthen the paper's comprehensive analysis on thinking in RFT. Overall findings suggest that some tasks benefit from RFT, but often times No-Thinking-RFT is better --- creating a new paradigm to consider in training large multimodal models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- Timely paper on up to date methodologies. \n- Analyses seem thorough. I especially appreciated the think-after-answer and adaptive-thinking experiments, which add validity to the authors' claims. \n- Explanation for why the convergence is faster and why performance increases both make a lot of sense; creating a cohesive and believable set of takeaways.\n\nWeaknesses:\n- Framing could be stronger in the introduction and (especially) the abstract. I think the writing lacks motivation for why the problem is important. \n- Some missing citations: On overthinking harming performance [1], on adaptive selection of thinking/no thinking [2].\n\n[1] Liu, R., Geng, J., Wu, A. J., Sucholutsky, I., Lombrozo, T., & Griffiths, T. L. (2024). Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse. ICML 2025.\n[2] De Sabbata, C. N., Sumers, T. R., & Griffiths, T. L. (2024). Rational metareasoning for large language models. arXiv preprint arXiv:2410.05563.\n\nQUESTIONS:\nIn addition to weaknesses, I have a few questions:\n- Were the datasets reported in the paper the only datasets tested? If so, is there a pre-registration (i.e., on OSF) to indicate this? This would greatly increase the findings. \n- It would be good to discuss your results a bit more via a discussion section. Currently, it is a bit hard to contextualize the results of the work into the existing research landscape. It is also necessary to include a discussion of limitations and future directions to help support future work.\n\nLIMITATIONS:\nNo - based on my reading, I did not find any limitations discussed in the paper. While there is limited negative societal impact for such technical work, one idea could be that if we remove thinking, the model traces could be less interpretable, or model behavior could potentially be more brittle given the more fixed equality training metric.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the usefulness of ‘thinking’ during RL training for small MLLMs (sizes 2B, 7B). Their main findings are (a) thinking during RL finetraining outperforms SFT (b) however, thinking actually seems to hinder training. Instead, doing direct RL on actions (without thinking) is more sample efficient and requires considerably less wall-time  (c)  They find that adaptive thinking RL can perform marginally better (compared to No-Thinking RL). On some reasoning tasks, like mathematical problems, they do observe a gain for Thinking RL.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. Paper is well-written.  \n2. Experiments are conducted across range of datasets.  \n3. Paper has done a relatively ok job of disentangling some key hypotheses through creative experimentation.\n\n### Weaknesses\n\nNote to authors: Weaknesses are important to address. Either through including novel results, and/or including relevant discussion (e.g., in limitations), and/or convincing the reviewer that their critique is faulty. Improvements are things that I believe would be helpful to improve the paper but are not necessary to address (though addressing enough of them could mean I improve my score).\n\n1. The paper’s limitations section does not acknowledge its limitations in satisfactory way, and ought to be included in the main body.  \n2. The image classification datasets used are all highly popular ones. The paper does not discuss the ramifications of this. The dataset memorization could explain why no-thinking does better. I would ideally like the paper to test on novel synthetic image datasets (with potentially novel classes / objects) to mitigate this risk. Or if the authors don't think this is a risk, I would like to hear their explanation.  \n3. Image classification experiments results in the main paper are only presented for 2B model. 7B model results should also be in the main paper (currently table 10 in appendix) and done for all models. It seems gap shrinks significantly for 7B models between thinking / no-thinking RL (on OxfordFlowers it goes from 5 points to \\<.5 points).  \n4. The image classification setup reduces the number of classes. I imagine the number of classes as potentially a factor in model performance and would be interested in some analysis of this aspect. Would be interesting to understand how \\# of classes impacts model performance. Closest experiment to this is open-set classification experiments with results in Table 9 in which the gap shrinks significantly.  \n5. I am somewhat concerned that the RL training setup used in the paper is not optimal; no SFT warmup is used, no curriculum is used etc. So, another plausible explanation of the paper’s results could be that RL training was not done well. It is somewhat surprising to me that on reasoning tasks, thinking does not help (on classification tasks, it sort of makes sense that reasoning would be a distraction considering it is a recall based task).   \n6. Related to 5: \tLine 274-276 present two interesting hypothesis to explain why 7B model does not benefit from thinking on puzzle tasks. It would be interesting to try and disentangle which hypothesis is true. One way could be to give the LLM tools to perform visual reasoning in-context (e.g., zoom / crop etc.) and see if that helps model’s performance improve. “This may be because these tasks rely on visual rather than linguistic reasoning, and language-based reasoning might cause hallucinations, or the 7B model’s puzzle-solving capability may still be limited.”\n\n### Improvements\n\n1. The work provides prompts for all the settings which is good. It would be good to collect them within a single appendix section (instead of being spread throughout the appendix).   \n2. Create a distinct subsection in appendix where details of all datasets (and ideally one/two samples from the dataset) are provided. Some of the datasets were novel to me.  \n3. The appendix has many interesting results; would be helpful if authors could include some sort of table of content and preamble to the appendix to help readers navigate it better.  \n4. Doing some sort of analysis of what model parts change through the RL training would be insightful; e.g., are the changes concentrated in the vision encoder part or the LLM part?  \n5. Most of the results in the paper are presented in the form of table; it might be better to present them in terms of spider chart / radar plot where feasible. This presentation would be more amenable to a quick skim.\n\nQUESTIONS:\n1. How likely is that the model is only learning to ‘format’ the answer correctly to be parsable by the chosen parser than learning new capabilities?\n2. Reasoning training on small LLMs often requires some sort of warmup; did the authors try any sort of SFT warmup; or generally provided any curriculum for thinking?\n\nLIMITATIONS:\nLimitations should be in the main paper, and more thorough.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}