SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models
Authors: Hardy Chen
Haoqin Tu
Fali Wang
Hui Liu
Xianfeng Tang
Xinya Du
Yuyin Zhou
Cihang Xie
Venue: Accepted by TMLR ()

Abstract: This work explores two distinct approaches for enhancing reasoning abilities in Large Vision Language Models (LVLMs): supervised fine-tuning (SFT) and reinforcement learning (RL). To support the SFT approach, we curate a multimodal reasoning dataset with the complete reasoning trace guided by DeepSeek-R1. For the RL approach, we focus on GRPO and develop a training framework tailored to vision-language tasks with a composite reward system comprising four signals that address both visual perception and reasoning challenges. Our extensive experiments reveal that RL is a significantly more effective strategy than SFT for training reasoning VLMs. While SFT can assist models that initially struggle with following reasoning instructions, it often induces ``pseudo aha moments'' that degrade overall reasoning performance, implying that only a minimal amount of SFT data is necessary. In contrast, RL leads to substantial improvements, outperforming recent baseline models on a range of math reasoning tasks by at least 2% on average. We also present several intriguing findings --- \eg, combining SFT and GRPO also hurts the model performance, and stronger instruction-aligned LVLMs consistently lead to better results in RL. We hope these findings provide valuable insights into the development of reasoning-capable VLMs and guide future research in this area.

Published in Transactions on Machine Learning Research (11/2025)
SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models
Hardy Chen'*, Haoqin Tu Fali Wang", Hui Lin', Xianfeng Tang', Xinya Du?,
Yuyin Zhou', Cihang Xie!
University of California, Santa Cruz
University of Texas at Dallas
The Pennsylvania State University
Amazon Rescarch
Project Page: http:///raaaerraa.a.hthoooooooo.
7B Model: http:///aaaaaaaeerrooo/////////a
5VL-78
3B Model: http:///wwggigggee..//////////aa 3 5VL-3B
Dataset: http://wagggaaacccc/aaaassstsstt
Reviewed on OpenReview: http://ppppariiiinnm/m/m/miiiiia
Abstract
This work revisits the dominant supervised fine-tuning (SFT) then reinforcement learning (RL) paradigm for training Large Vision-Language Models (LVLMs), and reveals key finding: on visual math reasoning tasks, SFT can significantly undermine subsequent RL by inducing "pscudo reasoning paths" imitated from expert models. While these paths may resemble the native reasoning paths of RL models, they often involve prolonged yet. less informative reasoning steps. To systematically study this effect, introduce VLAA-Thinie Vhin-nhin new multimodal datasct designed to support reasoning in LVLMs. Constructed via six-step pipeline involving captioning, reasoning distillation, answer rewrite and verification, VLAA-Thinking comprises high-quality, step-by-step visual reasoning traces for SFT, along with a more challenging RL split from the same data source. Using this dataset, we conduct extensive experiments comparing SFT, RL and their combinations. Results show that while SFT helps models learn reasoning formats, it often locks aligned models into imitative, rigid reasoning modes that impede further learning. In contrast, building the Group Relative Policy Optimization (GRPO) with novel mixed reward module integrating both perception and cognition signals, our RL approach fosters more genuine, adaptive reasoning behavior. Notably, our model VLAAA-Thineer, based on Qwen2.5VL. 3B, achieves the best performance across six popular visual math reasoning benchmarkk among 4B scale LVLMs, surpassing the previous state-of-the-ar by 1.8%. We hope our findings provide valuable insights in developing reasoning-con capable LVLMs and can inform future research in this area.
1 Introduction
Large Language Models (LLMs) with strong reasoning capability have recently gained wide altention with the emergence of OpenAl's o1/o3 and Deepseek-Rt RR (Guo et al., 2025; Jaech et al., 2024). A common practice to empower models with reasoning abilitics comprises two steps: supervised fino-tuning (SFT) on reasoning data, followed by reinforcement learning (RL) to further boost performance. This successful paradigm has inspired efforts to extend these strengths beyond textual domains to Large Vision-Language Models (LVLMs) (Peng al., 2025; Chen et al., 2025a; Deng al., 2025b; Shen et al., 2025; Yang al., 2025b).
"Equal contribution.

---

Published in Transactions on Machine Learning Research (11/2025)
SFT
RL
Which operation is omitted in the equation?
OO
6 12
D
Choices: (A) + (B) (C) * (D)
<think> First, looking at the numbers So, adding two numbers together... exactly what is. Wait, maybe I'm making mistake. Adding 6 to gives 8 which is less than 12. Maybe I'm missing something. Alternatively,.. ...
<think> determine...
If use addition +:6 8, which is not equal to 12.
we use subtraction 6 = 4, which is not equal to 12.
multiplication *: * 2 12, which equal to 12. If we use division /: / 2 =, which is not equal to 12.
Hmm, think I need to go with the most likely answer, which B) ", because sometimes math problems, question marks might indicate subtraction. Maybe the answer is B), but I'm not entirely sure. </think <answerB<</answer>
should confirm that there is no error. Since 6* 2 12, the answer should be C. </think>
<onswer> (C)</answer>
Figure 1: Examples from LVLMs trained with different strategies for reasoning. Left: response from a model trained with reasoning-Se SFT, showing pseudo reasoning traces and a number of pseudo self-reflective cues (i.e., aha-moments) imitated from R1. Right: response from a model trained with RL, showing natine reasoning ability and authentic aha-moments emerged from RL training. Wrong reasoning steps are colored red and aha-moment moments are highlighted.
In this work, we take step further and examine whether the widely adopted "SFT then RL" paradigm similarly benefits the development of reasoning-capable LVLMs. Specifically, we ask: 1) What are the distinct effect of reasoning-SFT. and RL on visual math reasoning? and 2) Is this two-stage paradigm truly necessary for visual math reasoning in LVLMs? To systematically explore these questions, curate VLAA-Thinking the first comprehensive and high-quality image-text reasoning dataset. explicitly designed to support both SFT and RL. Unlike prior datasets, VLAA-Thinking includes detailed, step-by-step reasoning traces derived from the RI-style "think-(hen-speakk intermediate reasoning. We construct a dedicated SFT split featuring multimodal chain-of- tootugh (CoT)) examples suitable for visual instruction tuning, alongside a more challonging RL split curated from the same source to encourage deeper and more adaptive reasoning behaviors. To effectively transfer reasoning capabilities from texl-only models the multimodal domain, we construct our dataset through a six-stage pipeline: metadata collection, image captioning, R.l-bsee distillation, answer rewriting, verification, and split curation. Specifically, we input image captions and visual questions into DeepSeek-R1 to generate initial reasoning traces. These outputs are then rewritten for improved fluency and verified for correctness using GPT-based verifier, resulting in high-quality multimodal reasoning dataset for SFT and RL.
Next, we carefully ablate the role of SFT, RL and their combinations in multimodal reasoning using our VLAA-Thinking dataset. To better understand the role of SFT, we perform detailed analysis, systematically examining the impact of SFT data type (e.g., with and without the self-reflective 'aha moments'), dataset scale, and model capacity. To explore the polential of RL in the vision-language context, we design a novel mixed reward function within the Group Relative Policy Optimization (GRPO) (Shao et al., 2024) framework that involves both perception and cognition rewards to incentivize the model to produce well-reasoned answers. Specifically, our mixed reward signal blends 2 types of reward with 5 types of functions. For rule-based questions, there are functions for digil, multiple-choice, math and bounding box outputs. For open-ended questions, we adopt a competent reward model, XComposer- 22.5-255 (Zang et al., 2025), along with a reference-based reward method score an answer. We then closcly investigate the effects of different. reward functions, base models, and the interaction between SFT and GRPO to further optimize reasoning capabilities.
Our extensive experiments comparing SFT and RL reveal several noteworthy insights. First, we probe the contribution of SFT and RT. in multimodal reasoning: while SET improves performance on standard tasks
We SFT to represent reasoning-SFTT hereinafter for more concise terminology. SFT is used endow models with various abilities (e.g., basic instruction following ability which pretrained hase model lacks) while reasoning- SFT used specifically introduce reasoning patterns to models.
2

---

Published in Transactions on Machine Learning Research (11/2025)
Detailed Caption
Answer w/ CoT
The image displays detailed architectural plan development area, which includes various labeled sections and proposed improvements...
<think>Okay, this question. user wants to ...</thinkk> Based the caption my analysis, the architectural plan integrates green spaces...
DeepSeek
R1
User: How does the plan integrate green spaces, and what might be the benefits this integration for the area?
Visual Question
Rewriter
<thinloOkay, let's tackle this question. user wants </thn// Based on the Image and my analysis, the architectural plan integrates green spaces...
Assistant: The plan integrates green spaces interspersing them throughout the development. Notable areas include the "Toadstool Gardens"...
Ground Truth
Verifier
Rewritten Answer
Answer
w/ CoT
w/o aha moments
w/ aha moments
VLAA-Thinking-SFT
VLAA-Thinking-RL
Data generation pipeline. We first generate initial reasoning traces by feeding detailed captions and visual questions into DeepSeek-R1. These outputs are then rewritten for improved fluency and verified for correctness using a GPT--bsed verifier. the resulting data is split into VLAA SMA-Thinking-SF7 and WLAA-Thinking-RL
Figure 2:
over the base model, it falls short in enhancing complex reasoning. Merely imitating expert's thinking through SFT often induces "pseudo reasoning paths", superficial reasoning pattern which may contain "pseudo aha moments" (superficial self-reflcctvee cues), as illustrated in Figure 1. We show that these imitated reasoning patterns can hinder genuine reasoning advancement, i.e., 47% relative performance drop on 7B models. This observation also in line with recent studies highlighting the need for feedback and exploration signals to drive advanced rcasoning behaviors (Peng et al., 2025). Additionally, our ablations show that for based rewards, math and multiple-choice are more beneficial than others, and that a combination of both rule-based and open-ended rewards yields the best performance.
While prior work suggests that SFT followed by RL in LVLMs offers the best of both worlds (Guo et al., 2025; Yang et al., 2025b; Deng et al., 2025b)-firss first mimicking good reasoning format, then refining via RL feedback, we find that applying SFT before GRPO hurts performance on aligned models, with an average 12.7% drop, and even a smaller scale SFT leads to a similar decline. Regarding model size, larger models cannot immune from the degeneration brought by SFT, as 7B models share almost the same performance drop with their smaller counterparts. Finally, examining the training procedure, we observe little correlation between response length, reward, and performance-SFT. response yet underperform RL-trained ones, contrasting with the previous observation that better models usually produce longer answers with higher RL reward (Guo et al., 2025; Peng et al., 2025).
models get higher initial rewards and longer
To summarize, while SFT helps unaligned models follow instructions, it limits exploration during RL by promoting imitative reasoning. In contrast, learning directly from reward signals yields more effective and adaptable thinking behavior. Empirically, direct RL proves superior. Our model, VLAA-Thinker- Qwen2.5VL-33 achieves the top-1 performance on the Open LMM Reasoning Leaderboard among 4B-scale LVLMs, surpassing the previous state-of-the-art by 1.8%. Our case study further emphasizes these gains with more concise, elfective reasoning traces presented in model answers.
2
The VLAA-Thinking Dataset
To systematically evaluate the "SFT then RL" paradigm for developing reasoning capabilities in LVLMs, we construct VLAA-Thinking, a dataset that consists of two parts: 1) VLAA-Thinking-SFT which captures step- by-step reasoning grounded in visual inputs for SFT, and 2) VLAA SLAA-Thinking-PP RL which contains challenging samples designed specifically for RL. Our data gencration pipeline is designed to transfer reasoning capabilities from a powerful text-only model to the multimodal domain through a structured, multi-stage process. The entire pipeline, illustrated in Figure 2, consists of six key components:
3

---

Published in Transactions on Machine Learning Research (11/2025)
Name
Data Type
#Ori.
# #Pipeline
# Finnal SFT
#inal RL
Collected from Distilling R1
CLEVR-Mate
Closed-end
35,000
28,018
5,923
2,000
GeoQA
Closed-end
6,499
Math PUMA
Closed-end
30,000
26,672
19,258
6,696
ArxivQA
Closed-end
54,399
51,348
34,604
1,000
DoA
Closed-end
10,194
8,206
4,897
1,000
VizWiz
Closed-end
20,523
6,528
4,266
1,000
ALLaVA-LATON
Open-end
47,066
18,123
10,496
3,000
Collected from LLa VA-CoTT
COCO
Closed-end 3,000
3,000
8,727
2,000
VisualGenome
Closed-end
3,000
3,000
38,242
2,000
Total
Closed- & Open-end 203,182
144,895
126,413
25,195
Data statisties of VLAA-Thinking. We present the original volume of metadata, (#Ori.), the data size after the distillation pipeline (#Pipeline), the size of sampled examples for SFT (#Final SFT) and RL (#Final RL), respectively. Note that only GcoQA170K with verifiable answers for the RL split.
Table 1:
#1: Metadata Collection We collect metadata from 9 vision-language datasets featuring either closed- open-ended questions. Specifically, sample data containing unique images from CLEVR-Math (Lindström & Abraham, 2022), Math PUMA (Zhuang et al., 2024), ArxivQA (Li et al., 2024b), Doc (Mathew et al., 2021), VizWiz (Gurari et al., 2018), and ALLaVA (Chen et al., 2024a), and process them through our complete data pipeline. In addition, we directly adopt COCO and VisualGenome data from LLaVA-CoT (Xu et al., 2024). An exception is GeoQA170K (Gao et al., 2023), which we include only in the RL split due to persistent hallucination issues during captioning. Detailed statistics are in Table
#2: Visual Input and Additional Information Each sample begins with image, question, and its corresponding answer. To bridge the gap between the visual modality and language reasoning, we resort to GPT-4o to generate a detailed image caption describing the content in structured and semantically rich language (detailed prompts in Appendix A.1). During this process, we take full advantage of the provided knowledge in the data beyond just the GPT captions. In detail, we provide these dataset-specific information: (1) CLEVR-Math: Instructions for synthesizing the image from CLEVR (Johnson et al., 2017); ;); (2) Math PUМА: Textual description of math problems in the image from the dataset itself. (3) ALLAVA-LAION: Fino-grained and verified GPT-4V captions from the original datasct.
#3: Reasoning Answer Distillation We utilize a strong text-only reasoning model: DeepSeek-RI to generate thinking rationale and final answers. The model is provided with the image caption, the visual question, and additional information from certain datasets. It responds using a structured reasoning format that is between <think> and </think> lags and contains a sequence of logical steps leading to the final answer.
#4: Answer and Rewriting To enhance consistency and eliminate modality-specific artifacts, the reasoning answers generated by RI are passed through a rewriting module (f.e., GPT-4o (OpenAT et al., 2024) in our experiment). This module removes unnecessary phrases (c.g., references to "caption"), and ensures the answer adheres to a clean, instruction-folloving format based on the image. We further filter out samples with the sentence length gap larger than 15 words to ensure minimum modifications in this process.
#5: Automated Verification To assess whether the generated reasoning answers is correct regarding the groundtruth answer, we implement an automated verifier. This verifier compares the rewritten reasoning answer to the groundtruth of the visual question, determining whether the outputs correct or incorrect. Only the examples that are verified as correct are retained as the final training data.
#6: Curating Splits for SFT and RL The last step of our data generation pipeline is to curate two non-overlapped training sets for SET and RL, respectively. Inspired by Chu et al. (2025) which finds that RL is particularly effective in encouraging deeper reasoning on challenging cases, we aim to seloct more challenging samples for the RL split. To achieve this, we propose using the presence of self-reflective cues (i.e,, the "aha moments") in the distilled answers as indicator of sample's 's difficulty level (details are in
4

---

Published in Transactions on Machine Learning Research (11/2025)
Appendix A.2). For the SFT split, exchude samples with "aha moments", such samples may be too complex to fully imitate through finetuning. On the other hand, the harder examples with "aha moments" form the RL split, on which reward-driven learning may be better suited to elicit meaningful reflection.
Following these steps, our dataset adheres to the format (image, question, reasoning, answer), with reasoning and answer generated by DeepSeek-R1. We construct a high-quality multimodal reasoning dataset with 126,413 samples for SFT and 25,195 samples for RL. We provide results of quality evaluation in Appendix A.4.
3 Investigating The Role of SFT for Multimodal Reasoning
SFT has become the de-facto approach for training LLMs. Recent studies aim to extend the strengths of SFT to empower LVLMs with reasoning abilities by training on specially formatted Unlike prior methods that incorporate standalone textual descriptions of images (Xu et al., 2024), this direct strategy enables the model to develop grammatically coherent reasoning abilities, allowing it "think before speak." In recent vision-language reasoning systems, there is notable trend of complementing or even replacing SFT with RL to enhance complex reasoning abilities (Peng et al., 2025; Deng et al., 2025b). We follow this line and take it further by probing the underlying cause of this shift. Our finding suggests that self-reflection thinking ("aha moments") from the SFT process is overloaded with excessive and irrelevant reasoning, becomes what we call "pscudo aha moments" " and ultimately hurts performance. In this section, we explore 1) the model perform when SFT-ed on data with aha-moments and 2) the effect of SFT data size to model performance.
3.1 Experiment Setup
To investigate the effect of SFT training with aha-moments, we collect the distilled VQA pairs whose distilled answers contain thammoments, totaling 55K samples. To study the effect of SFT with different sizes of training sets, we use perplexity (PPL) filtering to obtain a smaller SFT datasct. Specifically, we compute using Qwen2VI-2B and Qwen2.5VL-3B, and sort all samples by their average PPL. scores over the two models. We keep the samples with high PPLs to obtain a total of 25K SFT samples, as these harder examples push models to learn more effectively and efficiently (Ankner et al., 2024; Li et al., 2024c).
the PPL score of each answer in WLA4-Thinking-SP7--22
We select four models for training: Qwen2VL. (2B and 7B)2, Qwen2.5VI. (3B and 7B). Fach model is trained with a batch size of 128 and their vision encoder frozen. We evaluate model performance with VLMEvalKit (Duan et al., 2024) on 6 math reasoning benchmarks hosted in Open LMM Reasoning Leader- board, which contains 6 challenging math reasoning benchmarks including MathVista (Lu et al., 2024), MathVision (Wang et al., 2024b), MathVerse (Zhang et al., 2024), DynaMath (Zou et al., 2024), WeMath (Qiao et al., 2024), LogicVista (Xiao et al., 2024). We present the percentage of relative performance drop of different models in Figure Detailed training and evaluation setup are in Appendix B.
3.2 Findings
Model
Avg.
SFT with Aha Moments Degrades Performance. We present results for the Qwen2.5VL-3B model trained under three different settings using our SFT data in Table 2. Somewhat unexpectedly, the model fine- tuned on 55K examples containing the aha moment performs significantly worse than the base model, with an average drop of 10.5%. This suggests that chasing the aha moment through SFT is unreliable, as SFT merely teaches the model to mimic rather than to generalize genuine self-reflective reasoning. Additionally, the table shows evidence that straightforward SFT using multimodal reasoning data also degrades performance, f.g., we observe an average drop of 10.2% and 19.1% when fine-tuning on 25K and 126K samples, respectively.
Qwen2.5VI.c
31.8
w/ aha-55K
21.3
w/ 25K
21.6
w/ 126K
12.7
Table 2: Average performance
over 6 reasoning benchmarks of Owen2.5VL-3E SFT-eed on different sizes of SFT data and data con- laining only examples with aha mo- ment (aha-55K).
2In this work, Qwen2VT..2R and Qwen2VT..7B refer to the instruction-tunai versions.
5

---

Published in Transactions on Machine Learning Research (11/2025)
Qwen2-VL-2B
Qwen2-VL-7B
8.1
4.3
3.5
% Performance Drop
8.6
-25
33.
-15.0
28.5
21.3
-25
-3.6633.6
20.
25.323.7
-27.0
29.7
50
-5..9
58.5
-56.
-75
-75
SFT-25K
-14.5
76.1
SFT-126K
94.5
MathVista
MathVision
MathVerse
DynaMath
WeMath
LogioVista
MaltiVista
MathVision
MathVerse
DynaMath
WeMath
LogicVista
Qwen2.5-VL-3B
Qwen2.5-VL-7B
Performance Drop
-25
-24
-20.5
-24.7
-1.5
-25
22.
H23.9
25.3
-50
50.7
-414
-50
33.6336.6
-30.837.3
47.11
59.3
00.2 2
68.0
-75
64.3
SFT-25K SFT-126K
100
100
MathVista
MatrVision
MaihVerse
DynaMath
WeMarh
LogioVista
MathVista
MathVision
MathVerse
DynaMarh
WeMarh
LogicVista
Figure 3: Delta percentage performance change of different models trained with supervised fine-tuning (SFT) only.
More SFT Data, Worse Performance. Counterintuitively, even five-fold increase in the supervised dataset (from 25K to 126K instances) often fails to improve performance and in most cases actually harms it. Models trained with 126K SFT samples suffer a relative performance drop of over average 14% compared to their 35K-trained counterparts over all model and task settings (e.g., 25K: 32.2% us. 126K: 47.0%). This degradation particularly evident on complex datasets such as WeMath and DynaMath, where the relative decrease reaches as high as 97.9% over Qwen2.5VL models on average. Even on mid-difficulty benchmarks like MathVision Vssin and MathVerse (L..,, model performance relatively higher), the 126K SFT models underperform, with an average drop of 28.6% compared to the untrained model over 4 models. These results suggest that simply scaling up SFT data does not boost gencralizable reasoning skills of LLMs, and may instcad suppress the model's capacity various reasoning lasks.
Larger Models Are Not Immune SFT Degeneration. Contrary to expectations, scaling up model size does not mitigate the adverse effects of excessive SFT, under heavier SFT they exhibit pronounced drops the most challenging evaluations. A larger 7B models fine-tuned on 126K examples experience drops nearly identical in magnitude to their smaller 2B or 3B counterparts: 47.2% for smaller models us. 45.4% for larger models compared with base models. Notably, despite the strong performance of Qwen2.5VVL-7B model (e.g., 68.1% on MathVista), it also suffers an average decline of 52.5% on these reasoning tasks when SFT-ed with 126K data.
These findings highlight the limitations of SFT as tool for enhancing multimodal reasoning. While it may be suitable for learning reasoning formats, it falls short of the expectations for fostering inherent self-reflection. Rather than simply scaling supervision data, results suggest for shift toward more advanced training methods like RL.
4 Improving Multimodal Reasoning with Mixed Rewards
The previous section shows that SFT is insufficient to transfer RI's ability to LVLMs on vision-language tasks. Therefore, it crucial to seck for other post-training methods elicit the reasoning ability of LVLMs. Since reinforcement. learning (RT) is effective in enhancing reasoning ability (Yang et al., 2025a; Kirk et al.,

---

Published in Transactions on Machine Learning Research (11/2025)
GRPO with Mixed Reward
Rule-based Digit CLEVR-Math (2K)
Rule-based MCQ
Open-ended
GeoQA170K (6.5k)
General Reasoning
ArXiQQA
becVQA
How many
What Choose from
VizWiz (1K)
Mixed Reward Module
cylinders are there?
A] Phone B) Apple c) Humon
ALLeYA (3K)
A:
Sampled Responses
Computed Rewards
Rule-based Math Math
Rule-based ToU
are they doing? The image shows beach
LLCCA-oT-CoCO (2K)
Reference
Policy
gathered near stage. people on the beach seem be watching him, and it looks like a lively
with
Model
Model
Find expression.
Locate the elephant [0.1 0.5]
A: 9==1
Penalty
Figure 4: The proposed Mixed Reward Module for GRPO training, comprising 2 reward formats (rule- based and open-ended) and 5 types of verifiable rewards (digit, MCQ, math, IoU and general reasoning).
2023), and GRPO has recently been proven more effective and efficient on textual math reasoning task (Shao et al., 2024; Jahin et al., 2025) than other methods like PPO (Schulman et al., 2017), it motivates us to apply GRPO training for vision-language reasoning tasks.
Mathematically, let be a query and {o,),= be group of G sampled outputs from the old policy model Wolds GRPO maximizes the following objective:
lo:1
(G) (o,
min (r(()) clip(ro(0), + +e)
BDLL (To keee)
G
and
(
To.. (0s,elg, 0s,zt)
where Ai, is the estimated advantage, 8 is the KL penalty coefficient and ma, Maaaa Troe are current, old, and reference policies, respectively.
4.1 GRPO with Mixed Reward
To better adapt GRPO to multimodal reasoning, in addition adopting the rule-based reward similar to the textual GRPO training, it is necessary to consider additional characteristies introduced by the vision modality. Inspired by (Fu et al., 2024) which benchmarks LVLMss by perceplion and cognilion (reasoning), propose a mixed reward framework for GRPO training, as illustrated in Figure 4. The reward system comprises five types of verifiable rewards with two formats, encompassing both visual perception and visual reasoning tasks.
Rulc-Based Reward There are types of rulc-based rewards, including digit matching, option letter matching and math expression matching and Intersection over Union for bounding boxes. For digit matching, single digit.
the model is asked to answer counting questions from CLEVR-Math whose groundtruths For option letter matching, the model is required to answer multiple-choice question (MCQ). For math expression matching, the model is asked to solve math question, such as finding a function expression or the volume of cone, and output its answers in latex format. We use the Math Verify² package to check for correctness. For bounding boxes, the model is prompted to output the bounding box coordinates of an object in the image, and an IoU score (range from to 1) computed as reward.
(Zang al., 2025) the scorer, denoted as S,(+), which takes image and QA pair as input, and outputs reward score. Following Muhtar
Open-ended Reward We leverage Internl MnternLM-XCe CoC.Coommm..--eeeeeeeee
http://ttthbb..cmmmmngggggeggaaa Verify

---

Published in Transactions on Machine Learning Research (11/2025)
et al. (2025), the reward for sampled response ÿ is computed as Ropen 1 exp(- - So())) 3) if To(0) > fo(y) else 0, where So(v) is the score of the reference answer, and / is a smoothing hyperparameter. Note that the open-ended reward is normalized into [0,1], which is consistent with the scale of rule-based reward, partially avoiding reward hacking during training.
Implicit Format Reward Unlike Guo et al. (2025) and its subsequent works which use a separate reward term for format correctness, we discard this format reward term and make the format reward supersede all other rewards. Namely, whenever would be O. We empirically find that by specifying the output format in system prompt, the model is able to generate answers with correct formats through trials and errors. The implicit format reward design simplifies the reward computation. Further, it may yield better performance since less restriction is imposed on the exploration process (Zeng et al., 2025).
unable to extract valid response from the raw answer, the reward
4.2 Effect of SFT on GRPO Training
MathVerse (vision-only)
DynaMath
GRPO Backbone
MathVista MathVision
WeMath Logic
Avg.
(worst)
Qw
59.6
19.8
33.9
15.2
30.5
36.0
32.5
Dven2VL-77-Innt-SSP
43.7
14.7
19.0
3.2
11.1
27.3
19.8,
39%)
Qwen2VL-7B--Bas 59.3
18.2
33.5
11.4
23.2
36.2
30.7
Qwen2VI-77D-asse-555 49.5
16.4
25.0
6.4
20.4
32.7
25.7 (61%)
Table 3: Benchmark results of models trained with GRPO on different backbones. SFT+GRPO yields performance degradation, indicating that SET NOT compatible with GRPO in multimodal reasoning.
SFT is NOT Compatible with GRPO in Multimodal Reasoning. Although we reveal in Section 3 that SFT alone leads to a performance drop in multimodal reasoning, it is still unclear whether SFT plays crucial role in aiding GRPO, like the golden key in DeepSeek-R1. We experiment with different backbones for GRPO training. Specifically, we adopt Qwen2VL-7B-Ban Basc and Qwen2VL-7B-nnn and perform SFT on them with 25K samples, followed by GRPO training.
From Table 3, we observe that models undergoing SFT before GRPO training perform worse than those trained with GRPO alone, presenting an average drop of 8.9% across Qwen2VL-Base and Qwen2VL-Inst compared their non-SFT counterparts. We also find that SFT introduces more degradation instruction models than base models without instruction-following capabilities. For instance, Qwen2VI.-Inst suffers 7.7% more drop in performance than Qwen2VL-Base post-SFT,, suggesting that SFT can compromise the cnteruction-oolloiinn ability crucial for effective GRPO training. Taken together, these results suggest that SFT is currently incompatible with GRPO in the context multimodal reasoning, impairing both base and instruction-tuie tundin LVLMs.
Smaller SFT Dataset Still Jeopardizes GRPO Performance. Since we reveal in Section 3.2 that more SFT data yields lower performance, we try to investigate the effect of downsizing the SFT train- ing set. Following the PPL, filtering method in Section 3, we select top-10K and top-5K samples from - finetune Qwen2.5VL-3B, followed by GRPO training. For comparison, we also conduct GRPO training without SFT.
We present the performance of Qwen2.5VI.-3B on each task in Figure 5. A clear observation is that applying SFT on 5K examples prior to GRPO significantly degrades performance compared to using GRPO alone, showing an average drop of 13.5%. Moreover, scaling up SFT data to 10K yields only marginal improvement of 0.8%. These results further support that SFT before GRPO can hinder the model's learning capability.
Response Length, Reward, and Model Performance are NOT Necessarily Related. Prior work in RL suggests that longer responses often correlate with better reasoning and higher RL rewards (Guo et al.,
8

---

Published in Transactions on Machine Learning Research (11/2025)
Performance on Math Benchmarks
60.80
60
GRPO only 5K SFT+GRPO
Accuracy (96)
42/4043.60
10K SFT-
39.15
35.86
Arg: 34.49
29.7691.54
22,72,1.32
Aug: 21,
20
15.6214.89
18.98
Avg: 20.98
14.10
9.62
5.79 5.19
MathVista
MathVision
MarhVerse
DynaMaath
WeMath
LogicVista
Figure 5: Impact of SFT with 5K and 10K samples before GRPO. Smaller-sized SFT datasets still jeopardizes GRPO performance.
Response Length over Training Steps
Reward over Training Steps
GRPO only
Response Length
300
5K SFRG
250
0.3
200
0.2
GRPO only
150
SK GRPO
0.1
SFT+GRDO
40
Training Step
Training Step
Figure 6: Response length (left) and reward (right) during training. Training with only GRPO yields the lowest response length and yet the highest final reward and best benchmark performance, indicating that response length, reward, and model performance are NOT necessarily related.
2025; Zhou et al., 2025; Chen et al., 2025c). However, our findings in Figure reveal that response length and reward in GRPO are not reliable indicators of reasoning ability. For instance, the 10K SFT+ model produces the longest responses but ends up with lower rewards than the GRPO-only model (~0.35 ~0.5) after training. Similarly, the 5K SFT+GRPO variant shows moderate length and reward but still underperforıs on downstream tasks.
Interestingly, both SFT-ed models start with higher initial rewards (e.g., ~020 for 10K SFT+GRPO us. ~0.05 for GRPO-only), which likely due to their carly learning experience with supervision since SFT and GRPO data share the same distribution. However, they exhibit limited reward improvement during training, whereas the GRPO-only model rapidly surpasses them. These trends further reveal that SFT solely provides higher "lower bound" for RL training, yet it may lower the "upper bound" since the reasoning SFT data constrains the model's exploration paths. Therefore, reasoning is native emerging ability that is more likely to be developed through R., not SFT. While SFT-ed models may appear to reason, their behavior is closer to pattern imitation a form of pseudo-reasoning that lacks the generalizable reasoning skills.
4.3 GRPO Training without SFT
Training Setup Following the findings in the previous section, directly conduct GRPO training which yields four models: SLAAA-Thinker-Qwee2VV...222
MeThhinkr--Qwen2..... We also train on a base model of Qwen2VL-7B, and the resulting model is named "LAA-Thinker Thinkee--Qwee2---77 Qwen2-7B-Zerr We sample 4 times for each query with temperature 0.8. Rollout and training batch size are set as 512 and 256, respectively. We train our model for 1 episode (outer loop) and 1 epoch per episode (inner loop) on 8*H100 GPUs with 49 steps. More details of training setup are Appendix C.I.
TLAAA-Thinkor-QweenV....
3B, WLAA-Thinker-Qwen22...mm
9

---

Published in Transactions on Machine Learning Research (11/2025)
MathVerse (vision-only)
DynaMath (worst)
Model
MathVista MathVision
WeMath Logic
Avg.
4B-scale LVLMs
Qwen2VL-2B
48.0
16.1
17.5
3.8
10.8
26.6
20.5
Qwen2.5VL-3B
61.2
21.9
31.2
13.2
22.9
40.3
31.8
TLM-R1-Math-0805
62.7
21.9
32.2
13.0
30.0
40.5
33.4
Taichu-VLR-3B
64.9
23.1
32.1
12.6
30.4
38.7
33.6
TLAA--hhiner--Owen2VWW.
43.6
14.8
19.0
3.4
12.6
30.4
20.3
VLAA-TThinkerr@wee25.V....
61.0
24.4
36.4
18.2
33.8
38.5
35.4
7B-scale LVLMs
MaVA-OOneVisson-7B
58.6
18.3
19.3
9.0
20.9
33.3
26.6
InternlA-XCompoee22h
64.0
17.8
16.2
8.2
14.1
34.7
25.8
Qwen2V1-7B
61.6
19.2
25.4
11.0
22.3
33.3
28.8
Qwen2.5VL-BB
68.1
25.4
41.1
21.8
36.2
47.9
40.1
InternVL2..-888
64.5
17.0
22.8
9.4
23.3
36.0
28.9 9
InternVI.3-8B
70.5
30.0
38.5
25.7
39.5
44.5
41.4
RI-OneVision
22.7
41.2
15.0
20.8
42.1
34.5
18.2
23.2
TLAA--Thinker-QQee22V----- VIAA-TTiinker-Qven22.Vi..
59.6
19.8
33.9
15.2
30.5
36.0
32.5
68.0
26.4
48.2
22.4
41.5
48.5
42.5
Table 4: Evaluation results of 6 math reasoning benchmarks on Open LMM Leaderboard. VLAA-Thinker
models significantly outperform baselines and other models.
Evaluation Setup We follow the identical evaluation setup described in Section 3.1. For 4B-scale baselines, have Qwen2VL-2B (Wang al., 2024c), Qwen2.5VL-3B (Bai et al., 2025), VLM-RI-Math- 0305 (Shen et al., 2025), and Taichu-VLRR-33 For 7B-scale baselines, we have I.aVVA-OneVisioo-TT (Li et al., 2024a),
(Zang et al., 2025), Qwen2VL-7B (Wang et al., 2024c), Qwen2.5VL-7B (Bai et al., 2025), InternVI.2.5-8B (Chen et al., 2025b), InternVL3-8B (Zhu et al., 2025), and R1-OneVision (Yang et al., 2025b). We present evaluation results Table and list our main findings below.
Direct GRPO Training Boosts Model Performance. Models trained directly with GRPO the VL-Thinking RL consistently outperform their respective base models. For example, at the 7B scale, two models trained on VI.-Thinking achieve an average score of 36.5%, marking a 2.0% improvement over their base model average of 34.5%. Moreover, our best- persoreeforming 7B model consistently outperforms other similarly sized LVLMs (e.g., InternVL2..-8B, reasoning-foeee focused model, VLM-RI-Math, by 1.1% on average. These results once again demonstrate that GRPO significantly enhances reasoning capabilities, even without additional SFT.
LaaVA-OneVision-711
while our 3B model surpasses the recent.
Stronger Instruction Model Leads to Better Post-GRPO Reasoning. An interesting observation is that model with better instruction luning generally performs better. The instruction--lignned Qwen2-7B model, after GRPO, outperforms its unaligned counterpart VLAA-Thinken Taa-Thih----en22--1-7- Qwen2-7B--ero by 1.8% on average (31.3% 29.5%), with notable gains harder tasks like DynaMath (5.0%) and WeMath (3.1%). Moreover, using stronger instruction-tumed model for GRPO further improves across both 3B and 7B scales - VTLAA-Thinke--ueen higher-quality instruction tuning leads to more effective post-RL reasoning.
surpasses
by 12.6% on average, confirming that
Emergence of Authentic Aha Moments. To show that our GRPO training can induce authentic self-reflection process, plot the frequency of four aha expressions ("alternatively", "double-check", "i should chock", "wait") for cach VI.AA-hhnker model in Figure 7. Since all models are trained using GRPO without being SFF-ed on distilled reasoning paths, all aha moments emerge from the GRPO process, demonstrating the model's self-developed reflective ability. Another finding that the number of aha moments is not directly correlate with overall model performance, as more aha moments do not necessarily translate to higher reasoning scores.
htttps///doss.. wartocc...iir.c htt.://accch//nttrouuciiiii
10

---

Published in Transactions on Machine Learning Research (11/2025)
VLAA-Thinker Models
Qwenn-22
Qwenz-77 7e-n2-78-Zero
Qwen2-78
Qwen2. S-38
Qwen2.5 5-78
Total
teematively
220
72
87
170
555
200
double-check-
16
150
Frequenc
100
should theck
18
0
4
3
1
26
-50
Total 28
271
90
146
187
Figure 7: Hcatmap of different. "aha" expressions generated by VI.AA-Thinker models during training.
4.4
Ablations
Row
Method
Digit
Math
MCQ
IoU
Open-ended
MVi
MVs
WM
0
Qwen2.5VL-3B
21.9
31.2
22.9
1
w/o Digit
23.5
34.6
28.8
2
w/o Math
21.4
32.7
27.0
3
w/o MCQ
21.5
33.9
18.4
4
w/o ToU
22.8
35.3
30.0
5 All Rule-Based
22.2
34.9
30.1
6 Mixed Reward
24.4
36.4
33.8
Table 5: Ablation of Mixed Reward on MVi: MathVision, MVs: MathVerse and WM: WeMath. A combination of rule-based and open-ended rewards yields significant boost in performance.
Mixed Reward. To demonstrate the effectiveness of our mixed reward strategy, we perform an ablation study on Qwen2.5VL-3B by selectively disabling individual reward components and evaluating performance across three math reasoning benchmarks, as shown Table The model trained with Mixed Reward achieves the best overall performance, with an average improvement of 6.2% over the baseline, demonstrating the effectiveness of reward design. Using only rule-based rewards (All Rule-Based) also yields consistent. gains (e.g., 29.1% vs. 25.3% baseline), while removing specific components especially MCQ (w/o MCQ) lcads to substantial drops. These results highlight the critical role of rule-based rewards in GRPO for multimodal reasoning tasks.
Hyperparameters To search for better hyperparameters, we experi- ment with different learning rates (LR) and KL divergence settings on Qwen2.5VL-3B. We start with a basic setting where LR anneals to zero following a cosine scheduler with no KL constraint. Results are shown Table 6. LR1 uses minimum learning rate of - with warmup ratio 10%, whereas LR2 uses minimum learning rate of 5e with warmup ratio 3%. Since LR2 performs slightly better than LR1, compare two KL settings on top of LR2. KL1 uses initial KL of 1e-2 and target KL of 5e whereas KL2 uses an initial KL coefficient. of le-3 and a target KL of 4 We find that introducing KL constraints significantly improves the performance on MathVersc and DynaMath by 1.1% and 3.2%, respectively, and that using a smaller KL can encourage the model to evolve.
Settings
MVs
DM
LV
Basic
31.7
15.0
38.5
Learning Rate
+ LRI
33.0
16.0
38.1
+ LR2
33.5
15.6
38.3
KL Coef.
+ KL1
34.4
18.8
37.8
+ KT.2
35.8
18.6
39.2
Table 6: Ablation LR and KL Coef. on MVs: MathVerse, DM: Dy- naMath and LV: LogicVista.
11

---

Published in Transactions on Machine Learning Research (11/2025)
4.5 Case Study
We provide an example showcasing the improvement of VLAA-Thinker over the original model in Appendix C.4. Qwen2.5VL-7B generates lengthy response with wrong reasoning traces. Although it outputs some self- reflective patterns like "ro-ovaluate", the final answer remains wrong. On the other hand, VLAA--hhinke Qwen2.5VL-7B is able to reason on the right track, with only a minor mistake near the end of its thinking process. Nevertheless, the high-level idea and reasoning process is overall correct, demonstrating strong capability of solving complex reasoning tasks.
5 Related Work
Vision- LaaiuugL Reasoning Models. Recent advances in vision-language (VL) reasoning models build on the success of text-only reasoning systems like OpenAT's ol (Jacch et al., 2024) and DeepScck-R1 (Guo et al., 2025). Earlier VL methods, such as few-shot prompting and chain-of-thought (CoT), offered limited visual reasoning (Brown et al., 2020; Wei et al., 2022). Recently, LLaVA-CoT (Xu et al., 2024) adopts an SFT approach a 4-step structured outputs to enhance model's reasoning, yet lacking flexibility due to its rigid output format. More recently, newer models incorporate more natural reasoning traces and reinforcement learning. VLM-RI (Shen et al., 2025) and R1-V (Chen et al., 2025a) align multimodal LLMs ShualThinkee-RR-Zeer (Zhou et al., 2025) goes further
using step-by-step reasoning and policy optimization. by training a 2B model via pure RL from scratch, achieving emergent inner reasoning. LMM-R1 (Peng et al., 2025) transfers CoT skills from language to vision through staged RL. Vision-R1 (Huang et al., 2025) combines reasoning trace supervision and RI. with correctness and format rewards to train a strong 7B VI. reasoner. Different from these concurrent works, we propose a high-quality multimodal reasoning dataset with R1-like reasoning traces for both SFT and RL, and provide comprehensive study on training paradigns.
Reward Modeling in Reinforcement Learning. Reward design plays central role in reasoning-oriented RL. While model-based rewards offer flexibility (Kwon et al., 2023; Wang et al., 2024a; Gao et al., 2024), they are prone to reward hacking (Eisenstein et al., 2023; Chen et al., 2024b; Fu et al., 2025), making them risky for reasoning tasks. Recent VL models prefer binary correctness rewards (Huang et al., 2025; Zhou et al., 2025) for math or QA tasks, dircctly reinforcing accurate outputs. Others apply rulo- rewards, enforcing structured formats or logie chains (Liu al., 2025; Deng al., 2025a). While recent studies deploy strong reward models for enhancing LVT.M reasoning, they are grounded by specific domains simpler tasks (Muhtar et al., 2025; Tu et al., 2025). GIPPO-tyle methods use relative ranking within output batches to guide optimization without value critics (Shao et al., 2024; Guo et al., 2025). Our Mix Reward objective combines the model-based and rule-based reward in four complex rewarding scenarios, yielding better performance than existing approaches.
6 Conclusion and Limitation
This work provides a comparative analysis on the effectiveness of leveraging SFT or RL. (more specifically, GRPO) to build LVLM with strong reasoning ability. We show by extensive experiments that distilling reasoning data and performing SFT is a deficient way to transfer reasoning ability across modalities. We then extend our dataset to GRPO training with a proposed mixed reward objective, which yields substantial improvement over the bascline models. We present several findings regarding combining SFT and GRPO and the correlation between reward, respond length, and final performance. These results indicate that reasoning is native emerging ability acquired from RL, rather than SFT, which merely equips the model with "pseudo-reasoning" ability.
The major limitation of this work lies in that we only conduct experiment on Qwen2/2.5VL series, which may limit the generalization of conclusion. Also, we only experiment with one RL algorithm GRPO, whose behaviour might be different from PPO (Schulman et al., 2017) and RLOO (Ahmadian et al., 2024). Our future works should focus on expanding the generalizability on different models, as well as comparing behaviours of difference RI. algorithms.
12

---

Published in Transactions on Machine Learning Research (11/2025)
Acknowledgement
We thank the Microsoft Accelerate Foundation Models Research Program for supporting computing needs. Hardy Chen and Xinya Du are supported in part by the National Science Foundation CAREER Grant IIS-2340435.
References
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in lims, 2024. URL https://arxiv: wwwg/aa/////////o/ 14740.
Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-basee data pruning with small reference models. arXiv preprint urXiv:2405.205, 2024.
Shuai Bai, Kegin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Penglei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL /etps://arxiie http://aaxxv...gabbeeesb/
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. NeurIPS, 2020.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gptav-syuthesized data for lite vision-language models. arXiv preprint 2xX::2402.11684, 2024a.
Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. Rl-v: Reinforcing super generalization ability in vision-language models with less than $3. https://github. 2025-02-02.
Agent/R1-V, 2025a. Accessed:
Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. arXie preprint arXiv:2402.07319 2024b.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuchui Wang, Qingyun Li, Yiming Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025b. URL https: / wwwp:iii.rrgg/baa. ottp///////2.././///// 05271.
Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, et al. An empirical study on eliciting and improving rl-like reasoning models. aixiv preprint or orXim:2503.0455 04548, 2025c.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint ar (xXi:2501.1716:: 2025.
Huilin Deng, Ding Zou, Rui Ma, Hongchen Luo, Yang Cao, and Yu Kang. Boosting the generalization and rea- soning of vision language models with curriculum reinforcement learning. or preprint arXiv:2508.0700 2025a.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Openvithinker: An early exploration to complex vision-language reasoning via iterative self-improvement. or arviv preprinl arXiv:2503. 1752,, 2025b.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedio, pp. 11198 100--12211, 2024.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' D'Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.099/ 0.2444, 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL /tapss//arxiv.... www.aaiwwwa.aaaaaaaaaaaeaoooeeee owg/ abs/ 133944 13394.
Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in rlhf. arxiv preprint orXiv:250.. 18770, 2025.
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang. Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint Xft:2332.113700 2023.
Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective reward at training time for Ilm reasoning. arXiv preprint 1tl:22410.11111. 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement lcarning. arXiv preprint arXiv:2501. 2025.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3608 0018-367, 2018.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143 2024.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision- rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.007 06749, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint ar 21.1:2212..67720 2024.
Afrar Jahin, Arif Hassan Zidan, Yu Bao, Shizhe Liang, Tianming Liu, and Wei Zhang, Unveiling the mathematical reasoning in deepsook models: comparative study large language models. arXiv preprint arXiv:2503.105% 2025.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Railcanu. Understanding the effects of rlhf on 1lm generalisation and diversity. arXiv preprint arXiv:2310.06452. 2023.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. Xix preprint arXi::23030000 2023.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. ArXio, abs/2408.003366, 2024a. URL
https://api.... http//api.semanniccccccch...seee wttpaatiiiiiittaaooooorrrrpplpaa ortp/OOrpsSSDD:2777199944
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang. Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprchension of large vision-language modcls. arXiv preprint arXiv:2403.0000 00200, 2024b.
Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfittrrng: Weak-to-strong data filtering for fast estruccioo-tuning arXiv preprint arXiv:2402.005: 2024c.
Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint 0rXiv2200800532 2022.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning, arXiv preprint urxi::2503.00 Sl::2503.01185, 2025.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Represenlations (ICTR), 2024.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docyqa: dataset for vqa on document images. In Proceedings of the IEEE/CVP winter conference on applications of computer vision, pp. 2200 2209, 2021.
Dilxat Muhtar, Enzhuo Zhang, Zhenshi Li, Feng Gu, Yanglangxing Hc, Pengfeng Xiao, and Xucliang Zhang. Quality-driven curation of remote sensing vision-language data via learned scoring models. arXiv preprinl arXiv:2503.007. 2025.
OpenAT, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex
Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Augela Jiang, Anloine Pelisse, Antonia Woodford, Amuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweigg Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Fastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Prochl, Elizabeth Yang. Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Hecwoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O'Connell Ian O'Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Πya Sutskever, Ingmar Kanitscheider, Ishaan Guirajani, Jacob Coxon, Jacob Mcnick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica. Gan
15

---

Published in Transactions on Machine Learning Research (11/2025)
Lee, Jessica Shich, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengehao Zhong, Mia Glacse, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofrr Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhiní Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Scan Grove, Scan Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. Gpt-4o system card, 2024. URL https: htaps:://arxiv.orr http///b////22///// 21276.
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rulc-based rl. arXiv preprint ar arXiv:2503.076: 077366 2025.
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint A0x::2207.01284, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alee Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint 160.::77770333 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepscckmath: Pushing the limits of mathomatical reasoning in open language models. arXiu preprint arXiv:2402 03300, 2024.
Haozhan Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vlm-rl: A stable and generalizable r1-style large vision-language model. https://github. www//oommoooa/aaa att.://maaaaaaaM-------- 2025. Accessed: 2025-02-15.
Haoqin Tu, Weitao Feng, Hardy Chen, Hui Liu, Xianfeng Tang, and Cihang Xie. Vilbench: A suite for vision-language process reward modeling. arXiv preprint arXim:203.220 20271, 2025.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arxiv preprinl 2rX1::200.000000 2024a.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In The Thirby-eight Conference on Neural Information Processing Systems Dalasels and Benchmarks Track, 2024b. URL https: ftapss://openreverr
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing language model's 's perception of the world at any resolution, 2024c. URL https: //arxiv. www.ix..oggaab////.o orggabs/24099 12191.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022.
Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal Ilm logical reasoning benchmark in visual contexts. arwiv preprint arXiv:2407.0497, 2024.
Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-bye byeee,, 2024. URL https://arxiv. http://ariiii.oogaab////2// orggabs///111..... 2001//140.............
Haoyan Yang, Ting Hua, Shanggian Gao, Binseng Xu, Zheng Tang, Jie Xu, Hongxia Jin, and Vijay Srinivasan. Dynamic noise preference optimization for Ilm self-improvement via synthetic data. arv preprint XrXiv:2502.04400 2025a.
Yi Yang, Xiaoxuan He, Hongkum Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503. 10615, 2025b.
Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, et al. Internln=xcoompoeee2 5-reward: A simple yet effective multi-modal reward model. arXiv preprint arXiv:2501. 2tt::2501..268,, 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models the wild. Xiu preprint arXiv:2503.18892 2025.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual Springer, 2024.
math problems? In European Conference on Computer Vision, pp.
Tianyu Zhang, Suyuchen Wang, Lu Li, Ge Zhang, Perouz Taslakian, Sai Rajeswar, Jie Fu, Bang Liu, and Yoshua Bengio. Ver: A task for pixcl-level complex reasoning in vision language models via restoring occluded text, 2025. URL https://arra org/abs//406.. http:://arrav..rr//aa//220.. 06466.
Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Chong, Tianyi Zhou, and Cho-Jui Hsich. R1-zcro's' aha moment' in visual reasoning on a 2b nou-sft model. arXiv preprint arXiv:2503. 05132, 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjic Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan Hc, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Gc, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. URL https://arxiv. org/ abs////ba/2..... 10479.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Wenwen Zhuang, Xin Huang, Xiantao Zhang, and Jin Zeng. Math-puma: Progressive upward multimodal alignment to enhance mathematical reasoning. arXiv preprinl arxi:2208..00 00000000006 2024.
Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang, Dynamath: dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint ar orXiv:2411.008 2024.
18

---

Published in Transactions on Machine Learning Research (11/2025)
A Data Generation
A.1 Prompt
We show the prompts for captioning (Figure 8), R1 answer distillation (Figure 9), rewriting (Figure 10) and verification (Figure 11).
Prompt for Captioning
### You are vision-language model generating highly detailed caption of image.
### Summarize the environment setting (Indoor/outdoor, surroundings).
### Describe visible objects, people, or structures (colors, shapes, textures, positions).
### Transcribe all text verbatim. For equations, LaTeX when appropriate but do not solve interpret them.
### If structured data (tables, charts) appears, Markdown formatting for clarity.
# ### Include labels, annotations, brand names, logos, any, otherwise don't mention them.
# ###
any visible expressions emotional tone factually, without speculation. details.
# ### Maintain logical order: from overall context
# ### Provide only the caption without extra context commentary.
# ## # unbiased and faithful your description, using natural language and Markdown only where relevant.
Figure Prompt for captioning with GPT-4-Turbo.
Prompt for Distillation
advanced visual perception abilities and can directly analyze images as if you are looking at them. You provided with detailed visual descriptions, but you should interpret them as they represent your actual visual understanding rather than text-based captions.
Answer qucstions as if you are visually pereeiving the scene, not reading a caption. Provide natural and confident responscs about objects, relationships, and numerical or spatial reasoning. Use a descriptive, visually grounded tone, avoiding mention of text.
Never mention that are reading text captions. Infer spatial relationships, numerical properties. and logical conclusions based on the perceived 'image." If information is unclear, respond naturally as if there are visual limitations (e.g., 't appears that...)) ").
Caption:
{caption)
Question: {question)
Figure 9: Prompt for distillation with Deepsock-R1.
A.2 A Aha-Moment Filtering
We use the following list of keywords identify aha moments: wait, again, double-check, hmm, mistake, alternatively, check, i should confirm. All answers are matched with the logic: has aha any [aha in text. (ower for aha in ahas])
To validate our idea of using "aha-moment" as proxy of sample difficulty, we first measure the sample difficulty by repetitively sampling from Swen-2.5-VI-3223 For each sample, we adopt the prompt in Figure 12 to generate 16 responses with temperature 0.6, top_p 0.95, max max_tokens 1024. Thus, we obtain a pass rate for each sample. We then compute the averaged pass rate (accuracy) for SFT and GRPO split, which yield 49.9% and 45.9%. The results indicate that the GRPO split is indeed harder than the SFT split, solidifying our proposal of using "aha-moment" a proxy of sample difficulty.
19

---

Published in Transactions on Machine Learning Research (11/2025)
Prompt for Rewriting
You will receive a snippet text that references a "description" or "caption" an image. Your task to produce a ** **naalyy identical** version of that text with **minimal** changes, focusing on the following:
1.
"Replace references to "description", "caption" and "rationale** with wording that references ***the image." "Fx For example, "The description says.." could become "The image shows..."
"The caption suggests.." could become "The image suggests..""
"Based on the rationale.." could become "Based on the image...
Make sure the replacement sounds natural but does **ot*** otherwise change the meaning.
2. **Preseree all line breaks, punctuation, and spacing** as much possible, and make **no additional edits outside of these replacements.
3. You should only output the rewritten content.
Here input: (iuput)
Figure 10: Prompt for answer rewriting with GPT-4-Turbo.
Prompt for Verification
You fair evalustor.
You will be given groundtruth and answer from model. If the answer aligns with the groundtruth, output "Yss", Otherwise, output 'No". Your output should only be "Yes" 'No'.
groundbruth:
{gold}
answer: (pred}
Figure 11: Prompt for verification with GPT-3.5-Turbo.
A.3
Sample Demonstration for VLAA-Thinnkng-SFFT-L1
We show several examples from MaM4-Thinkinn-SFT--222 in Figure 16, Figure 17, Figure 18, Figure 19 and Figure 20.
A.4 Quality Evaluation on SFT Split
GPT-4o rewriting quality. During the development stage, we eyeballed around 100 samples to optimize the prompts for rewriting. We also compute the Levenshtein distance between input and output texts, resulting in an averaged normalized similarity of 0.91. Both manual inspection and quantitative results indicate that the modification is minimal and proves that using gpt-4o for rewriting is a valid approach.
Final data quality. We evaluate our SFT data quality using GPT-5 100 randomly sampled examples (uniformly sampled from each data source) across 7 error dimensions with prompt shown in Figure 13. We find that the overall quality is 7.98 out of 10, and rating of 8.78/10 with 84.4% high quality for mathematical samples. The low error rates demonstrate that our caption+ QA approach for the text-only R.1 effectively transfers visual information. The performance gap between SFT and RL stems from fundamental differences in learning paradigms: SFT teaches format imitation while RL enables genuine reasoning through exploration and feedback.
20

---

Published in Transactions on Machine Learning Research (11/2025)
Prompt for repetitive sampling with Soenn225-VL--33
Solve the following problem by first giving brief stepiby-step analysis and then your answer. The answer in your response should be of the form # Answer: [image]
{question)
Figure 12: Prompt for repetitive sampling with Qwen-2.5-VI.-32.
Prompt for Verification
You are expert evaluator tasked with assessing the quality of reasoning traces in visual question answering tasks. Your job identify spocific types of errors the reasoning process.
**'ASK** Evaluate the following reasoning trace for visual question answering problem.
**UESTTON** {question)
*REASONING TRACE**; freasoning_freg trace)
"EVALUATION CRITERIA* **; Please evaluate the reasoning trace and count number crrors each the
following categories:
1. **FACTUAL FRRORS** Incorrect statements about the image content, object properties, or visual details
2. **TOGGCAL, FRRORS* Flawed logical reasoning, incorrect mathematical operations, or invalid deductions
3. *INCOMPLETE "RESSTING"",,." Missing steps, incomplete analysis, or failure to address all aspects the
question
4. **INCONSISTENCY ERRORS** Contradictory statements within the reasoning trace
5. "HALLUCINATION BORTAIIIIIOEERRSS****=. ERRORS* Describing objects, properties, or details that are not present in the image 6. **PROCESSING ERRORS* Errors in understanding the qucstion, misinterpreting instructions, or wrong problem
formulation
7. *VERIFICATION E Failure to double-check calculations, assumptions, conclusions
"INSTRUCTIONSS*
Carefully read through the cntire reasoning trace
Identify specific instances of each error type
Count the total number
in each category
precise and conservative your evaluation
no errors of particular type are found, return
category
Focus on the reasoning process, not just final answer
*OUTPUT FORMAT**
Return your evaluation as a JSON object with the following structure:
{[
'factual errors:: _rcousl <number>;
'logical_errone errors:: <number>,
<number>>
_errristeecc_errrrs..ss "inconsistency errors': <number>,
"hallucination errors': <number>, "processing errors': <number>,
"verification errors': <number>,
'overall quality_score': score:: <score from 1-10>,
'explanation': '<brief explanation of the main issues found> }}
*'MMPORTANT*,
Return ONLY JSON object, no additional
Ensure the JSON valid and properly formatted
The overall overall_quality quooe score should from (very poor) to 10 (excellent)
Be objective and fair your assessment.
Figure 13: Prompt for verification with CPT-3.5-Turbe
21

---

Published in Transactions on Machine Learning Research (11/2025)
B Details of SFT Experiments
B.1
Training
To enhance the instruction following ability, we append task-specific instructions (í.e. MCQ, short answer) to questions. The system prompt shown Figure 14 used. We use a global batch size of 128. Models are trained for 190 steps on 25K samples and 985 steps on 126K samples. All experiments are run on 8*H100 GPUs.
Interestingly, observe loss spikes for 25K SFT training Qwen2VL-7B which causes model collapse. Therefore, we run the settings for multiple times until we obtain normal loss curve, and use that checkpoint for evaluation.
You are VL-Thinking 3 a helpful assistant with excellent reasoning ability. A user asks you a question, and you should try to solve it. You should first think about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>.
Figure 14: System Prompt used for training and evaluation.
B.2 Evaluation
We adopt VLMEvalKiit (Duan et al., 2024) for all evaluation experiments. We set use_ False following the settings of most models in the toolkit. For higher efficiency, set max max_pixels to 256*32*32, and pax_new tokenss to 800. We also set system prompt the one used for training for a consistent training-test behavior. The other hyperparameters are default to the original toolkit.
to
We specify the split of datascts and metrics reported:
MathVista: The Test Mini split of MathVista dataset; overall accuracy.
2. MathVision: The Full test set of MathVision; overall accuracy.
MathVerse: The Test Mini split of MathVerse; accuracy of "Vision Only".
4. DynaMath: The Full test set of DynaMath; overall accuracy.
WeMath: The Test Mini split of WeMath; "Score (Strict)",
6. LogieVista: The Full test set of Logic Vista; overall accuracy.
C Details of GRPO Experiments
C.1
Training
We adapt our code from OLe framework (Hu et al., 2024). To suit for our need of deploying a reward model on the same machine, we offload the reward model to CPU and only move it to GPU when performing rollouts and scoring. This design saves valuable GPU memory which accelerate the training process.
We also perform datasct-specific inspcction and find some issues for several datascts. For example, although ArxivQA contains only MCQ, the answer format includes "A", "A)", "(a)", etc. And in the synthesis subset of Math PUMA, find that some solutions only contain the value of solved unknown variables when
22

---

Published in Transactions on Machine Learning Research (11/2025)
the questions ask to output the entire function expression. We fix these issues by rule-based filtering and CPT-assisted rewriting, aiming to improve the quality of the VI--Thnniig dataset.
C.2 Evaluation
We evaluate our models with an identical setting described in Appendix B.2.
C.3 Generalization Beyond Math Benchmarks
We evaluate model performance on two splits of VCR (Zhang et al., 2025), which is a complex reasoning task on restoring occluded lexts. Results are presented Table 7, which shows that on both 3B and 7B models, GRPO more effective than SFT+GRPO. Even though both settings show degradation on VCR-En-East for 7B models over the bascline, GRPO shows less degradation than SFT+GRPO.
Model
VCR-EN-Eas:
VCR-EN-Hard Avg
3B Models
Qwen2.5- Qwen2.55VII-3B-Imm VL-33BInnst
0.69
0.03
0.36
GFT+
0.23
0.12
0.18
GRPO (ours)
0.73
0.38
0.56
7B Models
Qwen2.5 Owen2.5-VVL-7B-Inst VL-7BBInst SFT+GRPO GRPO (ours)
0.85
0.21
0.53
0.58
0.38
0.48
0.66
0.49
0.58
Table Performance comparison on VCR-EN-Fasy and VCR-EN-Hard for 3B and 7B models.
C.4
Case Study
We present case demonstrating the improvement of TLAAA-Tinnee--Qeen225------ over its backbone in
Figure 15.
23

---

Published in Transactions on Machine Learning Research (11/2025)
pent Decranree C. coal 1a
Question in Image
As shown in the figure, if angle 36.0, then angle C is equal to 0 Choices: A:36" B:54 C:60 D:27"
Qwen2.5VL-78 solve problem, we need to use some properties of circles tangents.
1. Identify the given information:
-AB is tangent to circle at point B.
The angle Z AOB = 36'
2. Understand the properties:
-Since AB is tangent the circle at B the radius OB is perpendicular to the tangent AB Therefore, < OBA 900
In triangle OAB the sum of the angles is 180" So, can find L AOB:
A0B OBA L OAB
180"
36
90" 0AB
180'
OAB 54"
Determine the of C
BC, intercepts the same arc BC twice measure of inscribed angle OAB:
-Since OAR inscribed angle intercepts
of central angle ROC that
BOC x Z OAB
54" 108"
Now, triangle OBC, sum the angles is 180". So, we can
C:
4 OBC + 7 OCB + < BOC = 180*
90" + / OCB + 108" 180"
0CB = 180" - 198" = 18'
However, since < OCB is not directly given as an option, we need to re-evaluate the problem with the correct
interpretation of the angle C which is actually the angle subtended by the arc BC at the circumference, which is half the central angle:
L C = x BOC =
1 108 54"
Therefore, the measure 2 is The answer
VLAATTinnke-Qwen22.5V11-
<think>
To solve this problem, we need to use the properties of tangents and circles. Since AB tangent to circle O at point B, the radius OB is perpendicutar to the tangent AB at point B. This means that angle OBA is 90".
that angle A is 36", we can find angle AOB by subtracting angle A from 180" (since the sum of angles in triangle AOB must be 180%):
< AOB 180 36 90 54
Since angle AOB is central angle subtending arc BC, and angle C angle subtending the same BC, the measure of angle Cs half the measure of angle AOB:
C 54 27
Therefore, the correct answer 2727".
</think
<answer>
D:
</answer>
case from MathVerse testmini (sample index 20). Markdowns are rendered for illustration purpose. Wrong reasoning paths are colored rod.
Figure 15:
24

---

Published in Transactions on Machine Learning Research (11/2025)
GeoQA
layal lunge
VI-Thisking
sorrdd Themss
pemeee de
madiome
pooppt biing
Caeple dhod
Soriola Couage stte peu sward
witk andeitatti
Take
Figure 16: A VL-Thinking sample from GeoQA170K. 25

---

Published in Transactions on Machine Learning Research (11/2025)
Synthesis
Input Image
equation?
Original Answer: Step
poinz into finetion:: sin(-3 Step Simplify
T0
the equation di sint
Soep Slace sio: 1. ltave: at Step Solve for
Subottute
ortainal
siar-5
y.u -3 smm-5-30
VI-Thhiking
Fins, recogtine the finction gives y ar anc)-vi), and it passes through point
find the value e. subeittte cxmiirates of the point intu the cxpaaton..
Subeituting and 5, the cxpustio bosomee:
5 sain
Sampliiing the acuumnn the ene Ganction:
So, the oquation now
(-f)
thatsin (-f) =- subsditsting m--)
Solving fic
Finilly substiittig book into orginall Basction gives: stat-5a)
detemiin exuation of finction umi-ba] that pass dhrugh peina &
5). follow
aepo:
Subetitute the Point into Equation:
oquatio::
T0
2. Simplify the Argunent the Sine Function:
So. ouuation becomes
sit
3. Evaluste the Sine Function:
Subsemuunng thss bads ano the equnoot:
Sulve
Final Equation:
oegginal Sauction:
eni-Se)
Staplify Negative (Opiionat):
sacP), equation
5- siai)))))
Final Ansmer:
Figure 17: A VL-Thinking sample from Math PUMA (subset Synthesis).
26

---

Published in Transactions on Machine Learning Research (11/2025)
CLEVR-MATH
Input Image
Question: Subtract balls. How many objects are left?
Original Answer:
VL- WL-Thinking
<think> Okay, let's see. The user is asking to subtract all balls and determine how many objects are left. First, I need to figure out which objects are considered "balls" in the image. The image shows a large blue metal sphere. sphere is three-dimensional circle, which is essentially a ball shape. the blue metal sphere is definitely ball.
Next, surrounding objects are two small metal cylinders single small brown metal cylinder foreground. Cylinders are balls; they have different shape circular bases and straight sides. So those would not be subtracted. The question says to subtract all balls, so only the sphere (ball) needs to be removed.
Originally, there's the sphere (1 object), two red cylinders (2 objects), and one brown cylinder object). Total objects before subtraction: + 2 = 4. Subtracting the one ball leaves 3 objects: the two red cylinders and the brown cylinder.
Wait, but should make there other balls mentioned. The image only refers the sphere ball-like object. The cylinders are elearly not balls. So subtracting the sphere leaves three cylinders. The answer should be 3.
</think>
In the scene, there is one large blue sphere ball-shaped object) centrally positioned, The remaining objects consist of two small red cylinders flanking symmetrically and single brown cylinder in the foreground. Removing the sphere leaves only the three cylindrical objects visible.
Answer: There are 3 objects remaining.
Figure 18: VL-Thinking sample from CLEVR-Math.
27

---

Published in Transactions on Machine Learning Research (11/2025)
ArxivQA
Input Image
2010
Question: In figure (b), how does the repetition rate affect the ratio of 9th to 1 Itth harmonic intensity for the 1030 nm wavelength? Choose from the following options A) The ratio increases the repetition rate increases. B) The ratio decreases the repetition rate increases. C) The ratio remains constant the repetition rate increases. D) The ratio fluctuates unpredictably with changes in repetition rate.
Original Answer:
VL-Thinking
<think>
Okay, I'm looking figure here, and need to figure out how the repetition rate affects the ratio of 9th to 11th harmonic intensity for the 1030 nm wavelength. Let me start by understanding what's in the image.
First, the image a scatter plot with the x-axis being the repetition rate 7th and 11th harmonic intensity on logarithmic scale. There are two sets of data points: dots and blue squares. The red dots probably representing the 1030 nm wavelength since in panel (a), the red curve was labeled as 1030 nm. The blue squares would then correspond to the 515 nm wavelength.
the y-axis being the ratio
The question specifically asks about the 1030 nm wavelength, so focus on the dots. Looking the scatter plot, as the repetition rate increases from left to right the x-axis, need see what happens to the y-axis values for the red dots.
From what can see, as the repetition rate goes up, the red dots trend downward. That means the ratio of 9th to 11th harmonic intensity decreasing. There's no indication of an increase or a constant ratio; instead, clear downward trend. The red dots don't fluctuate unpredictably either; they follow decreasing pattern repetition rate increases.
So, putting it all together, the ratio decreases as the repetition rate increases for the 1030 nm wavelength.
That would correspond to option
</think>
The image shows that the ratio of 9th to 1tth harmonic intensity for the 1030 nm wavelength decreases as the repetition rate increases.
Answer: The ratio decreases the repetition rate increases.
Figure 19: A VL-Thinking sample from ArxivQA.
28

---

Published in Transactions on Machine Learning Research (11/2025)
ALLaVA-LAION
SJORE
Input Image
Question: Considering the proximity of Preston Store to other landmarks, what could be its significance in the local arca?
The Preston Store's significance in the local area, suggested by the map, could be quite fact that has a dedicated Facebook page marker. appears situated near major roads and commercial entities like KFC, Harvey Norman, and Dan Murphy's, which indicates likely well-known establishment, possibly retail store or social venue, that serves as a local landmark gathering place. The specific lype of store cannot he determined from the map alone, but its central location among various retailers and close park suggests it may cater wide range of customers and possibly holds key place in the local community.
Original Answer:
central given its prominent placement
VL-Thinking
<think>
Okay, let's see. The user is asking about the significance of Preston Store in the local area based on its proximity to other landmarks. First, need to visualize the map shown. The Preston Store is labeled prominently with red marker and a Facebook 'f' logo, so it's probably a key location.
Looking at the nearby landmarks, there's KFC, Harvey Norman, Dan Murphy's, and Max Kirwan Mazda. Those are all big-name retailers and services. Then there's Target, Myer Northland, and Anaconda Preston. These are major stores, which suggests the area commercial hub. The store being near these places means it's likely in a busy shopping district, making it a central spot for locals.
Also, there's TW Blake Park and Darebin Creek Forest Park nearby. Green areas mean that Preston Store might be place people visit before after outdoor activitics. Maybe it's convenience community spot where people gather. The mix of retail, dining, and parks indicates it's multifunctional area, so Preston Store could serve both daily shopping needs and act as meeting point. The Facebook marker suggests they have an online presence, maybe promoting local events or services, enhancing community engagement. Since it's near major roads like Bell St and Murray Rd, accessibility is good, increasing its importance as central landmark. </think>
Preston Store appears centrally positioned within a bustling commercial and recreational hub, making it likely focal point for local activity. prominent placement near major retailers like Target, Myer Northland, and Harvey Norman suggests could serve a practical anchor for shoppers-perhape offering essentials services that complement these larger stores. The proximity to fast food (KFC) and leisure destinations (Darebin Creek Forest Park, TW Blake Park) positions it as potential pitstop for families or individuals transitioning between errands, dining, and outdoor activities. The red Facebook marker implies community engagement, possibly acting as an information node or local business with social media visibility, Its accessibility via intersecting roads like Bell St and Murray Rd further underscores its rolc convenient landmark in the area's daily flow.
Figure 20: A VL-Thinking sample from ALLANA-LAION
29


Reviews:
[{"review_id": "0JRzlC1xt0", "reviewer": "Reviewer_M1xC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed VL-Thinking dataset represents a valuable contribution, providing a useful resource both for fine-tuning vision-language models (VLMs) and for systematically studying the effects of different fine-tuning strategies.\n- The experimental results offer notable insights into model behavior under varying fine-tuning approaches.\n- The paper is well-written, clearly structured, and easy to follow, which makes the contributions accessible.\n\nWeaknesses:\n- The choice of models across experiments sometimes appears inconsistent or arbitrary. For example, the first two findings in Section 3.2 rely only on Qwen2.5-VL-3B, while Section 4.2 presents results for Qwen2-VL-7B (Instruct and Base) before returning to Qwen2.5-VL-3B only. While the high computational cost of such evaluations is understandable, the presentation could be clearer about which models are evaluated for which findings.\n- The findings are often generalized too broadly, as if they would necessarily hold across all VLMs during fine-tuning. However, no discussion of limitations is provided. For instance, results could vary with different training data, evaluation datasets, or base LLM architectures. Prior work (Shao et al.) already suggests that such differences can be significant.\n\nShao, Rulin, et al. \"Spurious rewards: Rethinking training signals in rlvr.\" arXiv preprint arXiv:2506.10947 (2025).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate the prevalent \"SFT-then-RL\" paradigm for training vision language models. They construct a large-scale dataset suitable for both supervised fine-tuning (SFT) and reinforcement learning (RL) using an automated data collection pipeline. Through experiments with Qwen2 and Qwen2.5-based vision language models, they find that applying RL without prior SFT achieves competitive state-of-the-art results, improving average performance on math-focused vision language benchmarks compared to similarly sized models. Their analysis further shows that SFT alone can significantly degrade model performance. Notably, on Qwen2-VL-7B and Qwen2.5VL-3B, RL without SFT consistently outperforms the conventional pipeline of SFT followed by RL.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical points:\n\n- Update the framing and implications: The conclusions are currently drawn solely from experiments on Qwen-based models. Some of the observed behaviors may not generalize to models built on other base LLMs, and this limitation should be explicitly acknowledged.\n- Clarify model usage across experiments: The presentation of results does not always make it clear which models are evaluated in each section. Improving transparency here is essential for interpreting the findings.\n- Discuss the influence of fine-tuning data: How might the fine-tuning dataset impact the benchmark results? For example, the most notable improvements occur on MathVerse and WeMath, while other benchmarks show marginal or even negative changes. What might explain this? One hypothesis could be that the model’s ability to handle synthetic inputs improves, given that these datasets primarily consist of synthetic black-and-white images. A discussion along these lines would strengthen the paper considerably.\n\nNon-critical points\n- Extending the experiments to another model that is not Qwen-based would be very insightful. However, I understand that it could go beyond the scope.\n- One of the rewards is called \"Rule-based reward\". It appears to function more like a supervised, verifiable reward rather than a truly rule-based one (which suggests execution of explicit rules on top of model outputs). Refining this terminology would avoid confusion.\n\nSmall remarks:\n- The abbreviation MCQ is never introduced.\n- The color scheme from Fig. 5 could be used for Fig. 3 as well.\n- Table 4, lower part, misses the best performance marked for MathVista.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lizX52BSqG", "reviewer": "Reviewer_6XCA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n+ The proposed VL-Thinker model outperforms similarly sized models on the Open LMM Reasoning Leaderboard.\n+ The RL approach adopted in this work (i.e., the use of GRPO and the reward design described in Section 4.1) is reasonable.\n+ The paper is well-written and easy to follow.\n\n## Weaknesses\n\nWhile the paper presents interesting insights, its central claims suffer from overgeneralization and questionable assumptions. In its current form, the work may not be suitable for journal publication.\n\nFirst, the central hypothesis that \"SFT can significantly undermine subsequent RL\" is problematic. According to Section 2, the datasets used for SFT and RL are disjoint, and the RL dataset was explicitly designed to be more challenging. If the training materials for the two differ substantially, then comparing SFT and RL is not fair. Favoring the more difficult examples for RL puts SFT at an unfair disadvantage. Furthermore, Table 3 shows that applying GRPO to Qwen2VL-7B-Instruct yields better performance than applying it to Qwen2VL-7B-Base. If the former includes SFT in its construction of Qwen2-VL-7B-Instruct, this result contradicts the claim that SFT impedes RL. It is thus plausible that the issue lies not in SFT per se, but rather in the specific SFT dataset used in this study. While I see that it is difficult to ensure fair conditions when comparing SFT and RL, the decision to base the entire paper on such a hypothesis may have been inappropriate.\n\nSecond, the claim that \"reasoning is a native emerging ability that is more likely to be developed through RL, not SFT\" seems overly broad. Since the experiments are conducted solely on vision-language models, it would be more appropriate to support this claim with additional experiments in the text-only setting. Without this, generalizing the conclusion to reasoning models at large is potentially misleading.\n\nThe explanation of the VL-Thinking dataset (Section 2) states that DeepSeek-R1 was used for Reasoning Answer Distillation and GPT-3.5 for Answer and Rewriting. However, the paper does not justify the choice of these LLMs for reasoning data synthesis, nor does it evaluate the quality of the resulting dataset.\n\nThe paper also states in Section 2 that \"we propose using the presence of self-reflective cues (i.e., the 'aha moments') in the distilled answers as an indicator of a sample's difficulty level.\" If this is indeed a proposal, empirical validation is needed, but none is provided.\n\nIn Section 3, the authors write that \"Our finding suggests that self-reflection thinking ('aha moments') from the SFT process is overloaded.\" However, if the SFT dataset was constructed from relatively simple examples with few self-reflective cues, it is unsurprising that the model trained on it would produce limited or unhelpful self-reflection outputs.\n\nFinally, while achieving top scores on the Open LMM Reasoning Leaderboard is a contribution, the RL method employed (GRPO with outcome-based reward) does not appear to offer significant novelty compared to the work of DeepSeek-R1.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This study aims to answer two central questions: \"What are the distinct effects of SFT and RL in multimodal reasoning?\" and \"Is this two-stage paradigm truly necessary for reasoning in Large Vision Language Models (LVLMs)?\" To this end, the authors build reasoning models for vision-language models (VLMs) using both supervised fine-tuning (SFT) and reinforcement learning (RL). They introduce a new multimodal dataset, VL-Thinking, which is constructed through a six-stage pipeline including captioning, reasoning distillation, answer rewrite, and verification. The dataset contains more challenging examples for RL than for SFT. The authors claim that while SFT helps models learn reasoning formats, it often locks aligned models into imitative, rigid reasoning modes that impede further learning. In contrast, they report that reinforcement learning with a mixed reward module and Group Relative Policy Optimization (GRPO) enables more natural and flexible reasoning, achieving higher performance. Their model, VL-Thinker, based on Qwen2.5VL 3B, achieves state-of-the-art performance across six popular visual math reasoning benchmarks among 4B-scale LVLMs.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please reconsider the main claims of this work (see the weaknesses of this paper).", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fJWIuMAApi", "reviewer": "Reviewer_7XcF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper addresses a fundamental question in LVLM training of whether of SFT or RL leads to reasoning capabilities in Large models.\n- The paper introduces a new multimodal reasoning dataset namely VL-Thinking. This dataset will be of much value for future research in multimodal reasoning.\n- The paper proposed mixed reward module for GRPO training, that combines rule based rewards, open ended rewards, and implicit format reward.\n- The paper shows response length and initial reward values do no correlate with reasoning capability in RL trained models. \n\nWeaknesses:\n- The VL-Thinking dataset reasoning traces are distilled from DeepSeek R1. SFT can imitate suboptimal reasoning patterns from DeepSeekR1. A thorough analysis on this is crucial to make claims in this paper more accurate. \n- The paper does not show any comparisons with SOTA RL based LVLm reasoning methods.\n- While the empirical findings in this paper show that SFT is detrimental to reasoning capability of the LVLM but the authors do not explore why it might be the case. Can it be related to catastrophic forgetting and if yes what are potential methods to avoid this issue?\n- The split between SFT and RL is crucial and must be explained in more detail.\n- Its not clear from the paper, whether the authors propose for SFT related degradation. Do the authors suggest to get away from SFT to RL completely or can the authors suggest any strategy to avoid the detrimental effect of SFT?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the two stage procedure of LVLM training for reasoning tasks: SFT followed by RL. The paper does that by systematically introducing a multimodal reasoning dataset “VL-Thinking” dataset that leverages GPT-4o, DeepSeek-R1, GPT-3.5 at different steps of dataset curation. The dataset is split into spfecific SFT and RL sets such that complex samples are used for SFT to avoid imitation learning. The authors conduct experiments comparing SFT, RL and combination of both using Qwen2VL, Qwen2.5VL on various visual math reasoning benchmarks. The paper also introduces mixed reward module consisting of rule based rewards, open ended reward, implicit format reward. The paper presents key findings such as SFT before GRPO leads to worse performance than GRPO alone, and SFT constrains exploration paths for reasoning models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Apart from addressing the weakness above. I would suggest the authors to address the following points:\n\n- Majority of the benchmarks are maths reasoning benchmarks, does the findings from this paper hold for common sense reasoning etc,\n- Explain the VL-Thinking dataset curation in more detail, especially the key details related to the DeepSeek-R1 based reasoning traces and SFT/RL split based on “aha moment”.\n- I would suggest the authors to also put limitations and future work for the current study.", "broader_impact_concerns": "N/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)