VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning
Authors: Haozhe Wang
Chao Qu
Zuming Huang
Wei Chu
Fangzhen Lin
Wenhu Chen
Venue: NeurIPS 2025 spotlight ()
Keywords: Vision-Language Models
Reasoning
Reinforcement Learning
Primary area: deep_learning

Abstract: Recently, slow-thinking systems like GPT-o1 and DeepSeek-R1 have demonstrated great potential in solving challenging problems through explicit reflection. They significantly outperform the best fast-thinking models, such as GPT-4o, on various math and science benchmarks. However, their multimodal reasoning capabilities remain on par with fast-thinking models. For instance, GPT-o1's performance on benchmarks like MathVista, MathVerse, and MathVision is similar to fast-thinking models. In this paper, we aim to enhance the slow-thinking capabilities of vision-language models using reinforcement learning (without relying on distillation) to advance the state of the art. First, we adapt the GRPO algorithm with a novel technique called Selective Sample Replay (SSR) to address the vanishing advantages problem. While this approach yields strong performance, the resulting RL-trained models exhibit limited self-reflection or self-verification. To further encourage slow-thinking, we introduce Forced Rethinking, which appends a rethinking trigger token to the end of rollouts in RL training, explicitly enforcing a self-reflection reasoning step. By combining these two techniques, our model, VL-Rethinker, advances state-of-the-art scores on MathVista, MathVerse to achieve 80.4%, 63.5% respectively. VL-Rethinker also achieves open-source SoTA on multi-disciplinary benchmarks such as MathVision, MMMU-Pro, EMMA, and MEGA-Bench, narrowing the gap with OpenAI-o1. We conduct comprehensive ablations and analysis to provide insights into the effectiveness of our approach.

VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning
Haozhe Wang 00 Chao Qu Zuming Huang', Wei Chu', Fangzhen Lin Wenhu Chen Vt
HKUST9, University of Waterloo 9 INF!, Vector Institute
Corresponding to: jasper.whn.ouutooooooooo
wenhuchen@uwateriee
Project Page: https://tiger-ai-lea htw.aagitttbbbooot-RRiiieeeee
Performance Comparisons on Multimodal Benchmarks
VL-Rethinker-728
Gemini-2 0-Flash
OpenAt-GPT-4o
Claude-3.5-Sonnet
OperAl-o1
80.4
73.9
67.7
00.0
60.3
62.4
55.5
Scores
11.551 15.55.5
44.9
40
35.5
15 15.602,7 15.1
20
MathWista testmini
MathVerse testmini
MathVision test
MMMUU Pro overall
EMMA full
Figure Performance comparison between VL-Rethinker and other SoTA models on different multimodal reasoning benchmarks.
Abstract
Recently, slow-thinking systems like GPT-ol and DeepSeek-RI have demonstrated great potential in solving challenging problems through explicit reflection. They significantly outperform the best fast-thinking models, such GPT-4o, on various math and science benchmarks. However, their multimodal reasoning capabilities remain par with fast-thinking models. For instance, GPT-ol's 's performance on benchmarks like Math Vista, Math Verse, and Math Vision is similar to fast-thinking models. In this paper. showcase how enhance the dow-thinkiing capabilities of vision-language models using reinforcement learning, to advance the state of the art, without relying on costly distillation. First, we adapt the GRPO algorithm with novel technique called Selective Sample Replay (SSR) to address the van- ishing advantages problem. While this approach yields strong performance, the resulting RL-trained models exhibit limited self-reflection. To further encourage slow-thinking, we introduce Forced Rethinking, which appends a rethinking trigger token to the end of rollouts in RL training, explicitly enforcing a self-reflection reasoning step. By combining these two techniques, our model, VL-Rethi nnee, advances ofte-of-the-art scores on Math Vista, Math Verse to achieve 80.4%, 63.5% respectively. VL-Rethinker also achieves open-source SoTA on multi-disciplinary benchmarks such as Math Vision, MMMU-Pro, EMMA, and MEGA-BBencs nar- rowing the gap with OpenAl-ol. We conduct comprehensive ablations and analysis provide insights into the effectiveness of approach.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Recently, slow-thinking systems such as OpenAI-ol (Jachh al., [024]. DeepSeck-RI [Guo et al. 2025 Kimi-1.5 [Team et al. 2025), Gemini-Thinking [Team et al. [023]. and QwQ/QvQ [Bai et al. 2025] have significantly advanced the performance of language models in solving challenging math and science problems. These models engage in extended reasoning and reflection before arriving at final answer, in contrast to fast-thinking models like GPT-4o [Hurst et al. [024] and Claude-3.5-Sonnel [Anthropic), 2024], which produce answers rapidly without such deliberation. Through this reflective process, slow-thinking
models outperform the best fast-thinking models by over 30% on math datasets such as AIME24 and AMC23 [Hendrycks et al.). and by around 10% on general science benchmarks like GPQA (Rein et al.
2024].
However, their multimodal reasoning capabilities remain on par with fast-thinking models. For example, GPT-ol achieves 73.9% on Math Vista [Lu al. 2023 and 57.0% on MathVerse [Wang et al.] 2024a], which is slightly worse than Qwen2.5-VL-722 Wang et al. 2024b) scoring 74.8% and 57.2% on the same benchmarks. This raises an important research question:
How can we effectively incentivize multimodal slow-thinking capabilities in Vision-Language Models?
To address this, we explore how to effectively train multimodal reasoning models through reinforce- ment learning (RL), without relying on costly distillation from stronger teacher models (Yang et al..
2025 Deng al. 2025]. Our main contributions are follows:
GRPO with SSR: We construct a dataset of 38,870 queries covering a diverse range of topics for training our vision-language model (VLM). We adapt the Group Relative Policy Optimization (GRPO) algorithm [Guo et al. 2025], which computes advantages by comparing responses within the same query group and normalizes rewards to guide policy updates. However, we identify key challenge with GRPO: the vanishing advantages problem. This occurs when all responses in group receive identical rewards (either all correct or all incorrect), leading to zero advantage signals and ineffective gradient updates. This reward uniformity exacerbates instability as training progresses, hindering the model from exploring deeper reasoning.
To mitigate this, we introduce Selective Sample Replay (SSR). which enhances GRPO by integrating an experience replay mechanism that samples high-value experiences from past iterations. SSR augments the current training batch with rehearsed samples that previously indicated large magnitudes of advantages. This strategic experience replay embodies the principles of curriculum learning [Team] al., [025] in an online and active fashion [ightman et al.] [2023), which dynamically adjusts the training focus towards high-value experiences situated near the model's ' decision boundaries. While this approach demonstrates strong empirical performance across several multimodal reasoning benchmarks, we observe that the resulting models still exhibit limitations in explicit reflective behavior, suggesting avenues for further improveee improvement.
To encourage explicit reflections, we propose simple yet effective technique called forced rethinking. We append a textual rethinking trigger the end of roll-out responses and train the model using the same RL setup. This strategy prompts the model to engage in self-reftection and self- verification before producing the final answer. We name the resulting model VL-Rethinke As shown in Fig. [ VL-Rethinker significantly outperforms GPT-ol on mathematical benchmarks such as Math Vista, MathVerse Furthermore, on general- pprppose multimodal benchmarks like EMMA and MMMU-Pro VL-Rethinker achieves a new open-source state of the art performance, closely approaching GPT-ol's ''s performance.
Forced Rethinking:
Observations: We observe a notable discrepancy between modalities: while RL training often induces slow-thinking behaviors such as longer reasoning traces math-focused tasks [Zeng et al. 2025] Wen et al.) 2025]. vision-language tasks rarely exhibit such development. In fact, our analysis reveals that accuracy improvements in VL-Rethinker do not correlate with an increase in the length of the reasoning chain. This finding leads us to a key conjecture: the current bottleneck in multimodal reasoning may not be the depth of logical deliberation, but the accuracy of initial perception. This suggests the "slow-thinking extending abstract logical chains, the model learns to use its rethinking steps to perform perceptual verification and self-correction (e.g.. "Wait, let me double-check that number in the chart"). The balance between perceptual accuracy over reasoning depth is a critical insight for improving vision- languae models and opens new avenue for future research.
incentivized by our method is qualitatively different. Rather than

---

In summary, our contributions are threefold: (a) We propose and validate a simple, direct RL approach for enhancing VLM reasoning, offering viable alternative complex supervised fine- tuning and distillation pipelines. (b) We introduce Selective Sample Replay (SSR) to mitigate the vanishing advantages in GRPOO-based RI. for VLMs.. (c) We propose Forced Rethinking, lightweight yet powerful strategy to incentivize self-refiection in VLMs. Our final model, VL VL-Rethinker, sets a new state of the art on key multimodal reasoning benchmarks, demonstrating the value slow- thinking reinforcement in vision-language modeling.
2 Preliminaries
Problem Formulation We define the multimodal reasoning task follows: given multimodal input consisting of one or more images and textual query Q. the goal to generate a textual response that correctly answers the query by reasoning over both visual and textual information.
Let denote the visual input space and T the textual input space, The input is denoted as æ V T, where (1, Q) captures both modalities. The output is a textual response E V. where y represents the response space. The challenge lies in building a vision-language model (VLM) that can integrate multimodal information and perform deep. multi-step reasoning eespecilly for complex queries requiring extended deliberation or external knowledge.
Our goal is to improve the reasoning capabilities of an instruction- tuned VLM that initially exhibits fast-thinking behavior, i.e., producing shallow, immediate responses. We aim shift the model toward slow-thinking behavior [Wang et al.) 2025a engaging in deeper, more deliberate reasoning- to significantly improve performance reinforcement learning (RL), which encourages the generation of accurate, thorough, and well- reasoned responses by assigning higher rewards to such outputs.
downstream multimodal tasks. We achieve this via direct
Formally, we train a policy vo(y)æ²), parameterized by 0, to maximize the expected reward r(y, a) for generating response y given input The reward function r(y, x) is designed to prioritize correctness. The learning objective is: max Eyo
where 2> is dataset of multimodal queries and their corresponding answers. Consistent with Deepseek RI Guo et al. 22025). we adopt a binary reward function: r(y, x) if y is correct for input x, and r(y, 2) O otherwise.
Group Relative Policy Optimization (GRPO) variant of PPO (Schulman al. [017]. J.
estimates the advantages of language model generations by comparing responses within query- specific group. For given input x = (T, Q), the behavior policy Thw generates group of G candidate responses {y:)e The advantage for the i-th response time step is computed by normalizing the rewards across the group:
A r(z,3.) mean y1) v1)...., r(.. vc)1)
std((r(æ, v....... r(z, vG)})
3
Our Method
This section outlines our contribution, including Selective Sample Replay (SSR) and Forced rethink- ing, two techniques to incentivize slow-thinking capabilities.
3.1
Vanishing Advantages in GRPO
We identify a critical limitation in GRPO, which we term the "Vanishing Advantages" problem. In GRPO, simple binary reward signal is used to indicate the correctness of response y to given vision-language query x. When all responses within query group are uniformly correct or uniformly incorrect, the calculated ad advantages become zero for every response in that group. Consequently, such examples cease to provide effective policy gradients, as the gradient signal relies on non-zero advantages to guide learning.
This issue becomes increasingly pronounced as training progresses, especially for high-capacity models. As illustrated in Fig. 2 tracking the training of Qwen2.5- VVL-72B reveals steady decline in the percentage of examples exhibiting non-zero advantages, falling from approximately 40% the start to below 20% after 16 x 16 gradient steps. This decline is symptom of the policy's 's tendency
3

---

Rañoo of Effective Quefies
Neptnnok&Cormt &'rectve Qurlic Tone Respotsondd drcemnutt
Figure 3: Method Overview. We present two-stage RL method based Figure 2: The Vanishing Advan- on Qwen2.5-VL -L-nsruuttt The first stage enhances general reasoning tages problem. Training of 72B through GRPO with Selective Sample Replay (SSR), which retains ex- rapidly saturates, leading to signifi- plored trajectories with non-zero advantages and selectively replay samples cant decrease of effective queries based on their advantages. The second stage promotes deliberate reasoning only 20% within 256 steps.
using forced rethinking, where we append specific rethinking trigger.
converge towards generating responses that yield uniform rewards within group over time. As the policy improves and generates more consistently correct and incorrect responses within a query group, the reward diversity (variations) necessary for calculating meaningful advantages diminishes, thereby intensifying the problem. We notice that similar trends have been concurrently observed in GRPO training on text-based LLMs (Yu et 2025].
The "Vanishing Advantages" phenomenon undermines the goal of fostering deliberate, complex reasoning in VLMs. As more query groups yield zero advantages, the effective batch size for training shrinks, causing training instability. This instability increases the risk of premature convergence to shallower reasoning traces, discouraging the model from exploring deeper reasoning pathways.
3.2 Selective Sample Replay (SSR)
To counteract the Vanishing Advantages problem and maintain training efficiency, we introduce Selective Sample Replay (SSR). SSR enhances GRPO by integrating an experience replay mechanism that strategically samples bigh-value experiences from past iterations, similar Prioritized Experience Replay (Schaul et al. 2015] in Temporal Difference learning.
SSR maintains a replay buffer Beplay that persists for storing tuples (Ic, yi, Å.). Critically, the buffer exclusively stores samples for which the corresponding query group exhibited non-zero (lÅyll > 0). The effective training batch is augmented at each training step by incorporating rehearsal samples drawn from Breplay. The sampling is prioritized based on the absolute magnitude of the advantages, thereby emphasizing the rehearsal of experiences that previously indicated significant positive or negative advantage signals. Specifically, sample j from the buffer is selected with probability:
P(select j) = |4
Eke
Agg
where a is a hyperparameter that governs the intensity of prioritization. We provide an algorithm diagram for SSR in the appendix.
By selectively sampling valuable experiences, SSR counteracts the issue of vanishing ad advantages and provides more consistent gradient signals. This stabilizes training and prevents premature stagnation,
as further substantiated in the ablation studies (Fig. 3). Furthermore, SSR embodies the principles of curriculum learning [Team et al., [225 Wang et al. [022] in an online and active fashion [ightman et al. (2023). Instead of relying on static, offline data curriculum, SSR dynamically prioritizes experiences that lie near the model's s decision boundaries. This dynamic focus directs training efforts towards improving performance on challenging queries associated with large positive advantages (signaling promising reasoning pathways) and penalizing incorrect solutions corresponding to large negative advantages (often relating trivial queries).

---

An Example Forced Rathinking
right
options"
provided
mistake recheck
inage
Value
Queetton diegrwee mitelll
terghh
configur
Reppanee: To soke the lingth or segrere V. to the glver stangle
eecheprnnt,
propertics irseggncca
unid
text
optión
structure
NBOSE Logic
SOLd BSt answe
angle
Accooding to ee relhegren feseere 1/ [H Micciiie 1181 Nares KL 1 mediply boch sales of sxsat 2t K Mirns 2 V KL- that nightee misaae the ineeuoration midiegrent
sotions
given
solution
mondlly
snd
might
seee
Indeed
correct
count
pooded hull to longly aud KL V, Su we should hove 1f Sheelt Yaree LS Sensoometion reght Sox the kongi of UM Vaiee Vieealh0) N.. but the londiee V.KL respens add inkeraaionn
double
naber
check
disecty glven mittogeent
interpretatio
problen
Trac
length
Calculation
PORST alptopptingg
Figure 4: An example of Forced Rethinking (Left). VL-Rethinker discovers a flawed problem via rethinking upon its hallucinations. The word cloud of VL-Rethinker (Right) shows the learned rethinking pattern of self-verification, self-correction and sell-questioning.
3.3 Forced Rethinking
While GRPO with SSR improves optimization stability, we observe that complex, deliberate thinking patterns, such as explicit self-correction, did not consistently emerge as a direct result of standard RL on VLMs, divergence from trends observed in large text-only models. Specifically, the base model, Dwen2.5-VL-Instrucc, did not intrinsically generate reasoning processes incorporating self-reflection. To explicitly cultivate deliberate reasoning within our VLM framework, introduce training technique termed Forced Rethinking. This method aims to proactively encourage the model to engage in more extensive internal deliberation before producing final answer.
Forced Rethinking employs two means to stimulate the model's s deliberate reasoning. The first, a straightforward means, involves hint within the instruction prompt itself, e.g., "regularly perform self-reffection on your ongoing reasoning". This contextual cue serves increase the model's propensity for generating rethinking sequences. The core principle of Forced Rethinking, however, lies in a targeted intervention within the RL rollout procedure, as depicted in Fig. 3 Following the VLM's initial generation of a response Y1 to a given input 2, we append specific textual "rethinking trigger" to y1. This augmented sequence is then fed back into the model, urging it to generate subsequent response segment y/. Consequently, the complete generated sequence becomes y y1 + trigger + y2. To elicit a diverse range of reasoning behaviors, we designed three distinct categories of triggers: self-verificatii self-correction, and seff-quustioninng Detailed descriptions of these rethinking triggers are provided in the appendix.
We apply forced rethinking to a fraction 0 < 9 < 1 of the generated responses, and retain only those rethinking trajectories that lead to a correct final answer. Based on these successful forced rethinking trajectories, we incorporate additional imitation loss (the SFT loss), which directly incentivizes the model to generate the desired deliberate thinking patterns.
Our method shares similarities in forced prompting with inference-tine budget forcing in SI Muen nighoff et al., 2025], but it serves as a training intervention to incentivize deliberate reasoning. This approach also constitutes key distinction from methods (Deng et al., 2025 Yang et al.) [025] that rely on SFT distillation from existing thinkinin systems. Our VL ---Reehhikkr.. trained with this strategy, does not necessitate rethinking step for every query. Instead, it learns to strategically en- gage this process only when it implicitly determines efficient inference. Intriguingly, as illustrated in the example provided in Fig. 4 our VL-Rethinker demonstrates the capability to even identify flaws the given problem when checking its initial reasoning through rethinking, showcasing form of emergent metacognitive ability (similar to the findings in Wang et al. (025cc) )
be necessary, potentially leading to more
Rethinking as Perceptual Verification. A primary takeaway from our work is the qualitative difference in "rethinking" between text-only and vision-language models. In our analysis, we found no strong correlation between improved accuracy and the length of the generated reasoning chain. This suggests that the "slow-thinking" fostered by VL-Rethinker does not improve performance by significantly enhancing logical deliberation, but about improving perceptual accuracy. The model to "look twice." " The "rethinking" " tokens are used to trigger a re-evaluation of the visual input against the initial reasoning step. This insight reframes the challenge of VL.M reasoning: the primary bottleneck the current stage is often not only the reasoning itself, but the synergy of the visual perception and reasoning.
Discussion:
does not learn to think "longer"; learns
5

---

Model
Math-Related
Multi- Discipline
Real-World
Vist testmini
Verhe testinini
Mathvision test
MMMU-Pro overall
MMMU
EMMA full
MEGA
val
core
Proprietary Model
OpenAl-ol
73.9
57.0
60.3
62.4
78.2
45.7
56.2
OpenAI-GPT-4o
60.0
41.2
30.6
51.9
69.1
32.7
52.7
Claude-3, 3.5d--3,,-Soee
67.7
47.8
33.5
51.5
68.3
35.1
52.3
Gemini-2.0-Plash
73.4
54.6
41.3
51.7
70.7
33.6
54.1
Open-Source Source Models
Llama4-Scout-109B
70.7
52.2
69.4
24.6
318
IntennVL-2.5-78B
72.3
51.7
34.9
48.6
61.8
27.1
44.1
QvQ-72B
71.4
48.6
35.9
51.5
166.7
32.0
8.8
LLava-OV-728
67.5
39.1
30.1
31.0
56.8
23.8
29.7
Qwen2.5-VL-32B
74.7
48.5
38.4
49.5
159.4
31.1
13.3
Qwen2.5-VI.722B
74.8
57.2
38.1
51.6
167.0
34.1
49.0
WL-Rethinker-S2B
78.8
56.9
40.5
50.6
65.6
37.9
19.9
VL NL-Rethinner-72B
80.4
63.5
44.9
55.9
68.8
38.5
51.3
À (Ours Open SoTA)
+5.6
+6.3
+6.5
+3.7
-0.6
+4.4
+2.3
Table I: Comparison between our 72B model and other statc-of-thc-an models. The notation of indicates reproduced results using our evaluation protocols.
4 Experiments
In this section, we first outline the training and evaluation settings, and then examine the key factors for effectively fostering deliberate reasoning VLMs.
Training Data and Benchmarks. Our training data was compiled by integrating publicly available
datasets [Du et al. 2025 Yang et al. 2025 Meng al. 2025 with novel data collected from the web.
This initial "seed" query set underwent rigorous cleaning and augmentation pipeline, yielding high-quality dataset of approximately 38,870 queries. Analysis of training dynamics (Fig 2 revealed that RL training on the seed queries quickly reached saturation, so we strategically curated different query subsets for training models of varying scales based on query difficulty. This procedure resulted in specialized subsets: approximately 16,000 queries for 7B model training and 20,000 queries for 32B and 72B model training, representing a spectrum performance levels for each corresponding model. A detailed description of our data preparation methodology is provided in the appendix.
For evaluation, employ diverse of challenging multimodal benchmarks:
Math-related reasoning: Math Vista [Lu et al. 2023 Math Verse
[Zhang et al.
2024). and
Math Vision [Wang al.) [2024a].
MMulti disti-dissciplie understanding and reasoning: MMMU Yue et al.
2024a
MMMU-Pro
Yuc
et al. 2024b), and EMMA (Hao al., 2025].
scale long-tailed tailed real-world tasks: MegaBench [Chen et al.
2024a]
This benchmark suite covers wide range of complex multimodal reasoning challenges. We report the @ass@1 accuracy using greedy decoding.
Baselines and Implementation. We compare against several categories of models:
Proprietary models: GPT-4o Hurst al. 2024 ol [Jacch al. 2024 Claude 3.5 Sonnet
An
Thropic 2024 Gemini-2.0- 2.-ii.2- Team al. 2023
State-of-the-an open-source models: Qwen2.5
VL-72B Bai et al.
2025
QvQ-72B Wang et al. 2024 Laama Scout
2024b),
CCen et al. 2024b). Iava-Onevision
et al.
and Kimi-VL TTem al. 2025
Representative open-source reasoning-focusel models: Open [Dngg et al. 2025
R1-One (Yang et al. 2025), RI-VL [Zhang al.) 2025 and MM-Bureka Meng al.
2025
These models are mainly trained on multimodal reasoning dataset.
Our VL-Rethinker-725 was trained using OpenRLHF for maximum of 3 epochs 8 sets of x A800 (80G) for approximately 60 hours. We include training details in the appendix, and will release code, models and our high-quality 39K dataset.

---

Model
Math-Related
Hullii-Dssipline
Seal-World
Mith lestmimi
Verse tesimini
Vishinsion test
MMMU-Pro
MMMU
EMMA full
MEGA
overall
core
General Vision- Language Models
lntem
17.4
29.0
51.2
19.8
26.0
V125588
64.4
39.5
19.7
34.3
56.0
30.4
Qwen2-VL-7B
58.2
16.3
30.5
54.1
20.2
34.8
Qwcn2.5-VI-7B
68.2
46.3
25.1
36,9
154.3
21.5
35.0
Llava-OV-7B
63.2
26.2
24.1
48.8
18.3
22.9
Kimi-VL-16B
68.7
44.9
21.4
155.7
Vsion-Language Reasoning Models
MM-Eurcka-8B (IntemVL)
67.1
40.4
22.2
27.8
49.2
RI-VL-7B
40.0
24.7
7.8
44.5
8.3
29.9
R1-Onevision-7h
46.4
21.6
20.8
27.1
Op
70.2
47.9
25.3
37.3
52.5
26.6
12.0
VL R--RRhhinker-71
74.9
54.2
32.3
41.7
56.7
29.7
37.2
A (Ours Prev SoTA)
+4.7
+6.3
+2.4
+4.4
+0.7
+3.1
+2.2
Ablations (Incrementally Ablated from VL VL-Rethinker-775
VL-Reasoner-78 w/o SSR DAPOO) Filter (=GRPO)
72.4
53.2
29.8
40.9
29.5
72.0
50.0
28.5
40.0
26.3
50.8
27.4
39.2
26.4
Table 2: Comparison between our 7B model and other general and reasoning vision- laagua-l models. means that the results are reproduced by us.
Neword on 3ill St:
TINAO
0.4
AAPD Hiter
0.44
0.42
1 0.42
040
Figure 6: Comparisons of training batch advantage distribution. Standard GRPO and GRPO-Filter has biased advantage distribution, with mass centered around In contrast, GRPO-SSR re-distribute probability mass over training examples evenly across different advantage values.
30080
Figure 5: Comparisons of training dynamics of GRPO, GRPO-Filter and GRPO-SSR. GRPO baseline exhibits significant overfit, and GRPO-Filter are more stabilized. GRPO-SSR achieves the best convergence.
4.1
Main Results
Our approach demonstrates significant performance gains, as evidenced by the quantitative results. For the 72B models (Table [ VL-Rethinker--221 achieved significant improvements base model, Qwen2.5-VL. L--22-.. 72B. Notably, VL VL-Rethinker-7211 achieved state-of-the-an results on math-related benchmarks among all models, including OpenAI-ol. For the 7B models (Table 2). VL-Rethinker-7B outperforms competitor 7B models that also employ RL, e.g., Open VLThinker, RI RI-One Vision, by large margin. These results underscore the effectiveness of our proposed approach in enhancing performance across various challenging benchmarks.
the
4.2 Ablation Studies
Ablation on Data. We include an ablation data compositions in the appendix.
Ablation on Selective Sample Replay (SSR). To address vanishing advantages, we introduce Selective Sample Replay (SSR) based on GRPO. GRPO-SSR filters out queries causing zero ad- vantages and perform selective sampling with probability proportional to the absolute advantage. To investigate the impact of filtering and selective replay, we establish two corresponding baselines for comparison against VL--Ressoner (a baseline without "Forced Rethinking"), (a) the baseline without selective replay retains the filtering, denoted as GRPO-Filter (also used in DAPO YYu
al. 2025) ); (b) the bascline with neither sample replay nor filtering, identical to standard GRPO.
7

---

Roner-5 (forced)
VL- VL-Beasaner-PP 78 (hound)
VL-Rathinkar-78
Rethirking Ratio
(33.68)
13.2275
80
Relstive Improvement (96)
(12.27)
8.46%
40
6-
5.04%
774 94
3:955
155,24)
3.77%
BSK
(30.0 69
2.49%
156.
20
.84%
(72.90)
0.16%
0.50%
(5355))
(41 15
53%
72.20
MathVerse
MMMUU
Figure 7: Relative Improvement with Different Re-thinking Strategies. We compare: (a) VL-Reasoner (forced), which is foreed to rethink at test time; (b) VL-Rcasoner (bound), represents the upper bound of test-time forced re-thinking; and (c) VL-Rethinker is trained for self-reflection. The results indicate that forcing VL-Reasoner rethink test time yields positive performance gains. Training for self-reflection significantly enhances performance, achieving closer results the upper bound of forced re-thinking. The overlaid line plot shows the rethinking ratio (right y-axis) of VL VL-Rethinker across different benchmarks, showing VL-Re thinker adaptively performs re- thinking, unlike the fixed forced re- re-thinking strategy.
The results presented in Table. highlight the effectiveness of our proposed components. The models trained with the full GRPO-SSR algorithm consistently achieves superior performance compared to ablated versions, strongly supporting benefits both filtering and selective replay.
Further insights into the behavior of these algorithms are revealed by analyzing the training dynamics, as shown Fig. [ the GRPO baseline exhibits the most pronounced overfitting, eventually leading to performance degradation. This can be attributed to the vanishing ad advantages problem, where the number of training examples with near-zero advantages increases as training progresses. These examples provide minimal learning signal, effectively reducing the batch size and destabilizing the training process. In contrast, GRPO-SSR demonstrates a more stable training process and achieves better convergence compared to HIRPO-Fitee. suggesting the beneficial role of SSR.
The underlying reason for these differences is illuminated by the advantage distributions during training (Fig. 6 Standard GRPO displays a highly skewed distribution, with pronounced peak at zero advantage, confirming that large fraction of samples provides ineffective gradients. GRPO- Filter alleviates the extreme peak at zero, yet it still retains a strong central bias, indicating that many examples with very small advantages persist.
Conversely, GRPO-SSR significantly alters the advantage distribution by redistributing the probability mass away from zero and placing greater emphasis examples with large absolute advantages. These examples, such as a correct response to a challenging query or an incorrect response to a simple one, intuitively more informative as they likely lie closer to the decision boundary. By selectively replaying these high-advantage examples, GRPO-SSR ensures a more balanced and effective learning process, ultimately leading to improved convergence evidenced by the reward curves.
Analysis on Forced Rethinking. To evaluate the effectiveness of our Forced Rethinking training compared its impact against baseline models and
technique in fostering deliberate reasoning, theoretical limits, as illustrated in Fig. 7 Our primary objective was to examine whether training with Forced Rethinking encourages VL VL-Rethinker to develop internal metacognitive awareness, enabling it to strategically decide when rethinking is beneficial, rather than applying it rigidly.
Fig. 7 compares the performance of VL WL-RRthinker against several configurations. The baseline is "w/o Forced Rethinking", which we dub VL-Reasoner. We first assessed the inherent potential of rethinking via VI--Rasooner (forced), where the baseline model is compelled to perform rethinking step test time for every instance. The results (blue bars) show positive relative improvements across all benchmarks. This indicates that the baseline model already possesses latent rethinking capabilities

---

that can lead to correct answers. However, this approach is suboptimal, as the baseline struggles to effectively leverage this ability, sometimes even corrupting initially correct answers through flawed rethinking. We also compute upper bound, VI-Reasoner (bound) (yellow bars), which represents the maximum achievable improvement if test-time rethinking is only applied to the wrong outputs.
Crucially, VL-Rethinker (red bars), trained using our Forced Rethinking technique, consistently outperforms the VL-Reasoner (forced) baseline. For example, on Math Vision, VL-Rethinker achieves an 8, 8.46% relative improvement, significantly higher than the 2.49% gained by passively forcing the baseline to th-nhink. This demonstrates that integrating rethinking into the training phase markedly enhances the model's 's capacity for effective self-reflection.
Importantly, the analysis highlights the adaptive nature of the learned rethinking behavior. The overlaid line plot (right y-axis) shows the "Rethinking Ratio" for VL-Rethinker - the fraction of test instances where it spontaneously engaged in the rethinking process. This ratio varies substantially across benchmarks, in stark contrast to the rigid, 100% application in the VL--Reasoner (forced) scenario. It suggests that VL-Rethinker has learned selectively trigger re-thinking based on the query's s perceived difficulty or its initial confidence, embodying the targeted metacognitive awareness rather than relying on fixed, potentially inefficient strategy.
5 Related Work
5.1 Multimodal Instruction Tuning
Instruction tuning has become central technique for aligning large language models (LLMs) with human intent, enabling them to better follow open-ended natural language instructions. In the multimodal setting, however, aligning both language and vision modalities presents unique challenges. Building upon the success of unimodal instruction tuning methods such as FLAN [Wei let al. 2022], Self-Instruct (Wang et al. 2023], and Direct Preference Optimization (DPO) [Rafailov et al. 2023], J. researchers have extended these strategies to vision- vision-language models (VLMs). These models must reason over visual semantics, resolve cross-modal references, and produce grounded, coherent responses-2a all within the framework of natural language instructions.
Initial efforts such as InstructBLIP (Dai et al. 2023 LLaVA [Liu et al. 2023]. and MiniGPT-4 (Zhu et al. 2024) demonstrated the feasibility of aligning VLMS using instruction- wow.wiwnnooooooinggooooooooo data. More recent advances, including Llava-OV [Li al., [024]] Infinity-MM [Gu et al.) [2024), MAmmoTH- (Jia et al. 2025), show that scaling up instruction tuning datasets and introducing diverse tasks can significantly enhance generalization across wide range of multimodal benchmarks.
VL (Guo et al. 2024], and VisualWebInstruu
5.2 Reasoning with Reinforcement Learning
The release of GPT-o1 (Jacch et al. 2024) and DeepSeek-R1 [Guo et al. 2025 has sparked renewed interest in incentivizing reasoning capabilities LLMs via reinforcement learning (RL). Recent works like SimpleRL-Zoo [Zeng et al. 2025) and Open-Reasoner--erro [Hu et al. 2025 explore direct RL fine-tuning from base models without relying on additional supervised instruction-tuning phases. Building on this foundation, approaches such as DeepScaler [Luo et al. 2025 and Light-R1 [Wen)
al. 2025 incorporate cold-start datasets specifically designed to promote long-form reasoning and step-by- st-p thought processes.
In parallel, efforts such as DAPO et. 2025 and Dr GRPO [Lu al. 2025 aim improve the original Group Relative Policy Optimization (GRPO) algorithm, refining reward structures and advantage estimation more effectively elicit deep reasoning behaviors from LLMs during training.
5.3 Multimodal Reinforcement Learning
There is a growing body of work focused on bringing Rased reasoning into the multimodal domain (Deng et al. 2025 Yang et al.) [025] Huang et al. 2025 Peng et al.) 2022]. Inspired by models like RT.,Seek-RT, these approaches typically follow a multi-stage pipeline. A common practice involves first performing supervised fine-tuning (SFT) on vanion-language data that has been annotated or augmented with detailed reasoning traces, often derived from strong text-only LLMSs after converting visual inputs into textual descriptions.

---

Following the SFT stage, reinforcement learning is used to further enhance the model's 's reasoning capabilities. While effective, these pipelines often require complex and resource-intensee processes, including visual captioning, teacher model distillation, and tightly coupled SFT+RL orchestra- tion [Wagg et al. 2025c]. In contrast, our work investigates a more direct and lightweight Rl.-olly approach, aiming incentivize slow-thinking behavior without relying on large-scale supervision or teacher-based distillation.
Conclusion
In this paper, investigated how more effectively incentivize the reasoning capabilities of multimodal models. Our proposed approaches have shown effectiveness multimodal reasoning benchmarks. However, models still lagging behind human expert performance general multimodal tasks like EMMA and MEGA-Bench. We conjecture that this is due to lack of high-quality multimodal training dataset. In the future, endeavor further improve the data quality to improve multimodal reasoning capabilities.
more
10

---

References
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in lims 2025.
via reinforcement learning. arXiv preprint arXiv:2501.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.12559) 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint MXXi::2502.1392 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Anthropic. Claude 3.5 sonnet model card addendum, 2024. URL https::/www.nttroppcccoco -oonne-----oonee--- coonnet-model-ee addenduum
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Sort, 2(4): 0-6.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint trXiv:2310.02255. 2023.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095-95166 169,,9,,9-669,-- 2024a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024b. URL /ttps://s//// org/ abs////s////// 22119111
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. RI Ro-ovevision: Advancing generalized multimodal reasoning 2025.
through cross-modal formalization. arXiv preprint arXiv:2503.
Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Openvithinker: An early exploration to complex vision- rision-language reasoning via iterative self-improvement. arXiv preprint arXiv:2503.174 17352, 2025.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Πya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 700: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar 20.. 2025.
11

---

Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and for long cot from scratch and beyond. arXiv preprint arXiv:2503. 10460, 2025.
Haozhe Wang. Haoran Que, Qixin Xu, Minghao Liu, Wangchunshu Zhou, Jiazhan Feng, Wanjun Zhong, Wei Ye, Tong Yang, Wenhao Huang, et al. Reverse-engineerg reasoning for open-ended generation. arXiv preprint MKKi::5509.06666 2025a.
Haozhe Wang, Qixin Xu, Che Liu, Junhong Wu, Fangzhen Lin, and Wenhu Chen. Emergent hierarchical reasoning in Ilms through reinforcement learning. arXiv preprint arXiv:2509.038. 03646, 2025b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:/707.000 1707.06347, 2017.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. 000003//4476, /4476, 2025.
Tom Schaul, John Quan, Toannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv: I. 1.0555,, 2015.
Haozhe Wang, Chao Du, Panyan Fang, Shuo Yuan, Xuming He, Liang Wang, and Bo Zheng. Roi- constrained bidding via curriculum-guided bayesian reinforcement learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4021-4031 2022.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501. 2025.
Haozhe Wang, Long Li, Chao Qu, Fengming Zhu, Weidi Xu, Wei Chu, and Fangzhen Lin. Learning autonomous code integration for math language models. arXiv preprint arXiv:2502.0066 2025c.
Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Virgo: A preliminary exploration on reproducing ol-like milm. arXiv preprint arXiv:2501.0 000004 2025.
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning, arXiv preprint arXiv:2503.07335, 2025.
Renrui Zhang, Dongzii Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169-186, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: massive multi- discipliie multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9556-9567, 2024a.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.028811 2024b.
Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can milmss reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.0544. 2025.
Jiacheng Chen, Tianhao Liang, Sherman Siu, Zhengqing Wang, Kai Wang, Yubo Wang, Yuansheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, et al. Mega-bench: Scaling multimodal evaluation to over 500 realdworld tasks. arXiv preprint arXiv:2410.1056: 2024a.
12

---

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu. Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024b.
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint MrXi:2240.033320 2024.
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. RI-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503. 12937, 2025.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Dai, and Quoc V Le. Finetuned language models are zero-shot learners,,,,, In International Conference Learning Representations, 2022.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generate instructions. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pages 13484-13508, 13508, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53777 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIIP: Towards general-puroose vision-language models with instruction tuning. In Thirth-seventh Conference on Neural Information Processing Systems, 2023. URL httppst://ppeeeeiiiiiieeee http:://pppaneieee..eeeeee//nnnr rett/forumniddwwoowyyg...
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024.
Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Infinity -mm Scaling multimodal performance with large-scale and high- high-quaitt instruction data, arXiv preprint arXiv:2410.188: 18558, 2024.
Jarvis Guo, Tuney Zheng, Yuelin Bai. Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint arXivv2412.05233 2024.
Yiming Jia, Jiachen Li, Xiang Yue, Bo Li, Ping Nie, Kai Zou, and Wenhu Chen. Visualwebinstruet: Scaling up multimodal instruction data through web search. arXiv preprint arX5::2503.10882. 2025.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning the base model. arXiv preprint 2rXii:2503.24296. 2025.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Sur- passing ol opevvvw with 1.5b model by scaling https: https::/prettt b75. notion. si site/ DeepScaleR SeeeScalee-SSrppssiinng
00680000220000000333333330030000 3a4e2 waa-wwithhhttie oy 101
2025. Notion Blog.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective, arXiv preprint arXiv:2503.20783, 2025.
13

---

Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao. Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arX1v:2503.06749, 2025.
Yingzhe Peng. Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xim Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint (tt.xiv:5003075530 2025.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Sco, Hamnaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images, 2016. URL
07396
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal Digital Libraries, 23(3):289 301, 2022.
14

---

Appendix
A Limitations and Discussions
In this work, we studied how to effectively cultivate slow-thinking via Reinforcement Learning. proposed Selective Sample Replay to mitigate the vanishing advantages GRPO, and em- ployed forced rethinking to foster deliberate reasoning. While we achieve state-of-the-an results on math-related benchmarks, our models still lag behind human expert performance on more general multimodal benchmarks like EMMA and MEGAA-Bench. This reveals that our model is still limited in high-quality training queries. While we show that a direct RL approach without costly distillation can outperform existing RL-based VLMs that involve costly distillations, remains open question what conditions SFT can indeed help the subsequent RL, phase for VLMs.
B
Training and Implementations
B.1
Training Dataset
Our initial seed query set was constructed by aggregating publicly available multimodal datasets Yang
al. 2025 Meng al., 2025 Kembhavi et al. 2016 Saikh al., 2022 Du al. 2025 with
novel queries gathered from the web. This aggregated dataset exhibits broad topical diversity, as visually represented in Fig. 8 Given our reliance on rule-based reward mechanisms for subsequent
Reinforcement Learning (RL.) training, crucial first step involved filtering the seed queries. We retained only those queries with reference answers that were programmatically verifiable by our defined rules. From this verifiable subset, an augmented query set was systematically generated through the rephrasing of questions and permutation of multi-choice options. This augmentation strategy was designed to facilitate knowledge re-occurrence and reinforce learning across variations of the same core information. This rigorous data preparation pipeline culminated a final training set comprising 38,870 queries.
Topics
(Non-Geo) Math 38.33%
0.31%
Social Science
Others 9.45%
2.46%
Spotial Roasosing
Phys/Chem/Bio 8.94%
(Geometric) Math 27.35%
Charts Charts/Dlagrama Diagrams 15.92%
(Non Geo) Math (Geometric) Math
Charts/Diagrams Phys/Chem/Bio
Spotial Reasoring
Social Science Commonsense
STEM Topics
Figure 8: Our training data contains a diverse collection of topics, including eight major categories.
Utilizing this comprehensive query set. we proceeded to train models at different scales. To ensure efficient training and leverage each model's inherent strengths, we selected subsets of queries tailored to their initial capabilities. Specifically, for each model scale, we curated training subset consisting of queries where the initial checkpoint of that model demonstrated a non-zero PassRate@8. This selection criterion ensured that the models were trained queries falling within their potential
15

---

Algorithm Selective Sample Replay (SSR)
1: Input: Buffer Breplay. raw training batch Paw = {(r,, ye. A.)3, intensity a 2
2: Output: Training batch Dirain updated buffer Breplay
3: Let Nbathh = [Draw]
4: Initialize list for effective current samples Detlective
0
5: for each sample (ar4, Vi. A.) in Drw do
6: Add (2x, Vi. A.) Dellective when |Á,| >
7: end for
8: Update buffer: Breplay <-
Breplny U Deffective
9: Let Neffeectve = effectiive
Calculate number of samples needed from buffer: 7erom buffer max(0, Nhatch Reffectivee
11: Initialize list for samples from buffer Pirom_bufta f- 0)
12: if Vatrom buffer > 0 then
13: Calculate sampling probabilities P(select) j) for all j Broplay according to Eq.
14: Form Dimom by drawing Pfrom_bulfe samples from Beplay
15: end if
16: Dirain - Detfective J D Pirom_bur buller
competence range, allowing the RL process to refine and enhance existing, albeit nascent, abilities rather than attempting to instill knowledge from scratch.
B.2 Algorithms
We provide diagram for Selective Sample Replay in Alg.
B.3 Implementations and Training Details
Our WL-Rethinker-72BB was trained using OpenRLHF for a maximum of epochs 8 sets of 8 x A800(80G) for approximately 60 hours. The final checkpoint was selected based on the mean reward achieved on held-out validation set. We employed near on-policy RL paradigm, where the behavior policy was synchronized with the improvement policy after every 1024 queries, which we define as an episode. The replay buffer for SSR persisted for the duration of each episode before being cleared. For each query, we sampled 8 responses. The training batch size was set to 512 query-response pairs. We accept at most two correct rethinking trajectories for each query. We set the priority hyperparameter in SSR to a 1.0 in the experiments. We released code, models and our high-quality 39K dataset to support further research.
B.4 Prompts Used for Training
Default Instruction Prompt
(question)
Please reason step by step, and put your final answer within \bboxed).
During the first stage RL training with SSR, we use the default instruction prompt as above.
Rethinking Instruction Prompt
(question) Guidelines:
Please think step by step, and exregularly perform self-questioning, self -verification, self-correction to check your ongoing reasoning**, using connectives such as "Wait moment", "Wait, does it seem right?", etc. Remember to put your final answer within Wbboxed[).
16

---

During the Forced Rethinking training stage, we use the above prompt to encourage self-reflectio and use three types of rethinking textual triggers.
Rethinking Triggers
self_questioning " "An'nWait, does it seem right?" self_correction = "An'nWait, there might be mistake" self_verification "An\nWait, let's s double check"
Additional Experiments and Analysis
C.1 Experiments
We conducted an ablation on the data compositions. Our training queries are comprised three major genres: math-related Assionlanguaae queries, science-related queries and text-only ones. We conducted ablation studies on these components. As shown in Table. removing text-only queries does not cause significant differences. As we further remove queries from the broader scientific domains, we observe more pronounced drop in performance. This significant reduction underscores the importance of scientific data in improving the model's general reasoning ability.
Model
RL-Algo
Data
Misionsion
Math
Verse
MMMU-Pro
EMMA
WL-Rethinker-77B
SSR
16K
32.3
74.9
54.2
41.7
29.7
w/o Forced--Retn Rethinkinn
SSR
16K
29.8
72.4
53.2
40.9
29.5
no SSR no SSR& Filter
Filter
16K
28.5
72.0
50.0
40.0
26.9
GRPO
16K
26.0
70.9
51.4
38.8
26.2
Text Text
SSR
13K
29.1
73.5
53.5
41.1
28.7
SSR
11K
28.0
71.6
50.3
39.7
28.0
Table 3: Ablation Results to show the impact of SSR and Data Mix.
We performed sensitivity analysis to the prioritization intensity hyperparameter in SSR. We plot the final averaged eval reward of models trained with different hyperparameters. As shown in Fig. 9 the performance does not vary significantly regarding different hyperparameters, and the best value is around I.O. We observe that models trained with different hyperparameters will always reach similar peaks sooner later.
Hyperparameter vs. Averaged Reward
Averaged Reward
2.1
0.5
03
3.
0.5
3.1
1.2
Hyperparameter
Figure 9: Sensitivity Analysis of the Hyperparameter SSR
C.2 Cases
Below we show few cases where VL-Rethinker shows the ability rethinking. This ability can help achieve correct answers more often than not, but sometimes still fails with challenging questions.
Listing 1: successful casel
17

---

the question requiring integer and provide the final value, e.g., 1, 2, What is the highest amount this class measures?
Q: Hint: Please
GT: [60x)
A: To determine the highest amount this beaker measures, let 's analyze the image step by step:
1. The beaker is labeled \"600m11" at the bottom, which indicates its + total capacity.
2. The markings on the side of the beaker show volume levels: 100, 200, 300, and 400 ml.
3. The highest marking visible on the beaker is 480 ml.
Wait a moment, does the highest marking represent the total capacity?
+ No, it doesn't 't. The beaker is labeled \"600m1 at the bottom, + which means it can hold to 600 ml of liquid.
Wait, does seem right? Yes, it does. The beaker is designed
hold 600 ml, even though the highest marking is 400 ml. The remaining capacity above 400 ml is not marked but is still part of the beaker total capacity.
Therefore, the highest amount this beaker measures is 600 (660
Listing 2: successful case2 Q: In the triangle with hypotenuse length 20, find
+ y. : AnA: : :0
GT: \boxed(D)
A: find the length
WV y W) in the given right triangle with hypotenuse of 20 and an angle u0lueabe, we can use 4 trigonametric relationships.
1. **Identify the trigonometric =*
In a right triangle, the sine of an angle is the ratio of the length the opposite side to the hypotenuse.
Here, WC Asiin u00b)) = (\frac(y) W)).
2. **Recall the value of AV (1sin(50) **)*** - (Nsin
Wifrac(N\sart(333133 W).
3. **Set up the equation:**
WIAn (se = Wfrac(y)(20)) Wimplies (Nfrac((ssgrt333332 = 1\frac(y3(20NN WIAnAn4. **Solve for W =** \Il 20 Wedot (frra((\sqrt(3))2) 1011sart(3)An WV
**Wait mament**, let double check the problem statement and the choices. ** The problem asks for \\O \1) and the choices are 4, 8, 10. It seems there might be mistake in the problem setup or the choices provided. However, based on the trigonometric calculation, the correct value of VV \)) is WV 1011sqrt W),, which is approximately 17.32, not matching any of the given choices.
Given the choices, the closest integer value to A 10\Nsqrt(3) AV) is
4 10, but it is not exactly 18.
**Wait, does it or the choices might be incorrect.
right?** The problem might have different context
Since the problem asks for the closest value and 10 is the closest to AV 10)lsart(3) W) the answer is:Nn\n\\boxedd0)
18

---

Listing 3: failure case
Q: In the figure there are nine regions inside the circles. The numbers + 1 to should be written in the regions so that the sum of the
+ numbers in each circle is exactly 11. Which number has to in
the region with the question mark?<imagel>
GT: \boxed(6)
A: To solve this problem, we need to place the numbers 1 to 9 in the nine regions such that the of the numbers in each circle is exactly 11. Let denote the regions as follows:
AVC A W is the region in the top left circle.
WV \1) is the region in the top middle circle.
\/( C \)) is the region in the top right circle. \VC \1) is the region in the bottom left circle. (/( W1) is the region the bottom middle circle. A F W) is the region in the bottom right circle.
VV( V1) is the region the middle left circle.
AI( H W) is the region in the middle right circle.
VVC W is the region the middle circle (the one with the question mark).
The circles and their regions be described follows:
The top left circle contains VV A, 6, \\).
The top middle circle contains AV G, B, H Ai).
The top right circle contains VV H, C, F W)).
The bottom left circle contains AV D, E, I IA)..
The bottom right circle contains VV I, F, AD)
We need to satisfy the following equations:
1.
AV A + + D = 11 W
2.
AV(
11
W
3.
AVC
11
W1
4.
AVC
+
I
=
11
W)
5.
WV I + 11 W)
Let's start by adding all five equations:
WE (A G D) (G + H) (H C) (D E + (I F E)
+ 55 WV
This simplifies to:
WE A
C 20 2E + 2F 2G 2H + 21 = 55 AV]
Since WV
C
45 \\) (the sum of the
numbers to 9), we can substitute A 45 + + = 55 W), which simplifies to: WJ
+
WE D + D +
Now, let's s try to find solution by trial and error, keeping in mind + that each circle must sum to 11.
Assume W I A)):
19

---

Then + 11 \1) and \V( 1 + + 11 \1), \1( D E 10 V)) and V F 10 \)). This means AV = \1) which is not possible since all numbers must be distinet.
Assume 1V( I \))
Then \V( D E = 11 WV) and AV + = 11 W)), AV D +
11) and W F = \)). This means AV \1), which is not possible.
Assume AV = \1):
Then VV( 11) and \V( F E W1). This means V D \)), which is not possible.
\) and W + W 90 W
Assume \V( Then \V( D E 4 \V) and W 4 F W), so W
\\
11) and A F + = W) This means \ D \)), which is not possibll.
Assume WV( Then \V( D E 5 11 W) and W + W), so \ D + +
W)
11) and A F + E W) This means \ D = \)), which is not possible.
Assume WV Then \V( D E + 6 = 11 W) and A F 11 W), so W D +
W)
11) and A F = W1). This means \ D = \)), which is not possible.
Assume WV( Then \V D E 7 = 11 W) and A 7 F 11 V), so \ D +
W)
11) and \ F E = W. This means A D = F \)), which is not possible.
Assume AV( Then \I D E 8 = 11 W) and A 8 F 11 W), so W D E
W)
11) and \ F 3 W This means \ D F \)), which is not possible.
Assume A I W)
Then \V D E
11 W) and A F 11 1 W D = 11) and W F = W This means \ D = W)) which is not possible.
After trying all possibilities, we find that the only number that works 4 is W == W)
*sWait, does it seem right? Let's verify:*x
If WV == W)) then AV D + E +E 10 W1) and W F E = 10 W) Let 4 try WV
6 (W and AY E == A1), and \V F 6 11) and \ E possible. Let's try WC D 5 11) and W E = W1). This
4 AW) This is
AV and W( Let's try WV D == = W) and N(E \), and 11 \1) and
AV), and FVF
not possible.
AV E W)). This is not possible. Let's try \1( D \1) and
W E \)) and WCF== F W) and E 11). This is not AD) and W F =
possible. Let's try AV D \1) and W) and W W. This not possible.
After all these trials, we find that the only number that works is W I = W)
Therefore, the number that has to go the region with the question 4 mark is (111))
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have ensured the claims are accurately reflected.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in the conclusion and the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
We have detailed the assumptions and problem formulation.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have detailed the training and evaluation settings. We include further details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification:
We have provided code, data and models for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.cc/publl htto:l//c/iiiee//oeeuibbiieee
wwi.mmmissiooooooiiooooccc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided the training and evaluation details in experiment section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We did not provide error bars due to the high cost of experiments. The results have surpassed existing models by large margins, sufficing support effectiveness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided the information.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics //teur/p/////rpppp
Answer: [Yes]
Justification: We have reviewed and fully adhered to the NeurIPS Code of Ethics throughout our research process, including data usage, experimental design, and reporting. No ethical concerns were identified the course of this work.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work has potential positive societal impacts by advancing the capabilitics of multimodal models, which could benefit applications such as assistive technologies and education. However, like other general-purpos. AI systems, also poses risks of misuse, such as generating misleading or harmful content. We encourage responsible use and open discussion regarding the deployment such technologies.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Our work does not involve the release of pretrained language models, image generators, or scraped datasets that pose high risk of misuse. Therefore, no specific safeguards are required.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets and code assets used in our work are properly cited in the paper, along with their respective licenses. We ensured compliance with the terms of use of each asset, and included license details where applicable.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper thcode. com commdaaasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce assets in the form of code and data used in experiments. These assets are accompanied by clear documentation, including usage instructions, data schema descriptions, and license information.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: None.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve any research with human subjects crowdsoureed participants, and therefore IRB approval is not required.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

for Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
We have detailed in the appendix.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer our LLM policy what should or should not be described.
27


Reviews:
[{"review_id": "KwXmuEV067", "reviewer": "Reviewer_UJQZ", "summary": "This paper introduces a method to facilitate reasoning in vision-language models. Authors propose to add selective sample replay to avoid the issue of diminishing advantages in GRPO, which in turn stagnates RL training, preventing the models from learning more complex reasoning patterns. Additionally, to encourage more reasoning steps, authors also propose to append a rethinking token to encourage longer reasoning rollouts. These two changes allow the model to excel in visual reasoning on a variety of benchmarks, with impressive performance on MathVista and MathVerse in particular.", "questions": "- What is the definition 'effective queries' in Figure 2?\n- How do you build figure 6? Is this at the end of training? Or average throughout training? I think a better way to show this would be to show a plot with update steps or epochs on the x axis, and either fraction of low-advantage queries or average absolute advantage on the y axis, then show lines for GRPO, GRPO filter, and GRPO SSR. Your claim in the paper is that without SSR, model quickly reaches very low advantage values, but you don't show that anywhere with an experiment. This plot would address this problem.\n- How do you get the word cloud in Figure 4? I don't think this figure adds any value to the paper, I think you can just remove it.\n- In line 169, you say you incrorporate the additional SFT loss. Is training with this loss happening at the same time as the RL stage, or is it separate?\n\n### Typos and nitpicks\n- The text in Figure 3 is too small\n- Line 132: exhibited non-zero : I think you missed 'advantage' here.\n- Line 134: Breplay - replay should be in subscript like it is in line 131.", "limitations": "See comments above", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n - I like the intuitive simplicity of the proposed ideas\n- Thorough experiments including large-scale models (72B), and impressive performance across a variety of benchmarks\n\n**Weaknesses**\n- The authors claim that without SSR, the model quickly reaches very low advantage values, but this is not confirmed anywhere in the experiments.\n- Using methods akin to prioritized experience replay can lead to known issues in RL, such as oversampling outliers or outdated samples from earlier in the training. These issues may become more prononunced as RL stage is scaled further.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their patience of reading the paper and their constructive feedback. The suggestions have greatly helped us improve the clarity and impact of our paper. Below, we address each of the questions raised.\n\n**1. Definition of 'effective queries' in Figure 2.**\n\nWe appreciate the reviewer's request for clarification. In the context of Figure 2, \"effective queries\" refers to queries for which the corresponding training batch contains at least one non-zero advantage value, thus effectively contributing to the gradient update.\n\nMore specifically, the GRPO method computes advantages within each query group. A key scenario arises when all responses within a single query group receive an identical reward (e.g., all rewards are 0 or all are 1). In this case, the calculated advantages for all samples in that group become exactly zero. Consequently, these samples do not provide a learning signal (i.e., they contribute zero gradient) during the policy update.\n\nCurrently the direct definition of \"effective queries\" is missing, which can lead to readers' confusions. Since the notion of \"effective queries\" is indeed equivalent to the number of \"effective training samples\" (query-responses with non-zero advantages), in our revision, we will use the more unambiguous term \"effective training samples\" to avoid any confusions.\n\n**2. Clarification and Improvement of Figure 6.**\n\nWe thank the reviewer for this suggestion. The current Figure 6 was generated via a post-hoc analysis: we used a static set of training rollouts and applied three distinct filtering methods (no filtering, zero-advantage filtering, and our SSR) to this fixed dataset to plot the resulting advantage histograms.\n\nWe agree that a dynamic plot tracking advantage metrics over the course of training would be far more compelling and would directly substantiate our claim that advantage values diminish rapidly without SSR. In the revised manuscript, we will replace the current figure with a new plot following the reviewer's suggestions.\n\n**3. Rationale for Figure 4.**\n\nThe word cloud in Figure 4 was generated by sampling responses from our trained VL-Rethinker on the validation set and visualizing the most frequent words. We agree with the reviewer's assessment that this figure is primarily illustrative and does not contribute significant technical value to the paper's core claims. To streamline the paper and focus on the most impactful results, we will remove this figure in our revision.\n\n**4. The Role of the Additional SFT Loss.**\nThe SFT loss is an auxiliary loss that is applied concurrently with the RL objective during the RL training phase. Its purpose is to directly incentivizes the policy to adopt the \"forced rethinking\" responses. While the model can learn this behavior through the RL objective alone, we found that the learning process is significantly slower. This is because the \"wait\" token initially has a low probability in the policy's native distribution, causing the policy ratio in the PPO objective to be frequently clipped. The auxiliary SFT loss thus provide more direct gradients that accelerates the adoption of this rethinking token.\n\n**5. Typos and Nitpicks**\n\nWe are grateful to the reviewer for their careful reading and for catching these issues. We will correct them in the revised version."}, {"review_id": "hiGco0114i", "reviewer": "Reviewer_Zs28", "summary": "This paper presents VL-Rethinker, a method that leverages reinforcement learning (RL) to enable vision-language (VL) models to exhibit slow-thinking capabilities in VL reasoning tasks. By introducing Selective Sample Replay (SSR) to address the vanishing advantages problem in GRPO and Forced Rethinking to encourage explicit self-reflection, the model achieves state-of-the-art (SOTA) performance on multiple multimodal benchmarks.", "questions": "I must note that I am not an expert in this field. The following questions are what I am not clear about after reading the paper.\n1. If slow-thinking (even forced reasoning or thinking mechanisms) can enhance VL reasoning capabilities, I'm curious if the training process for those VLMs (e.g. Qwen2.5-VL) already uses a similar approach. If not, why?\n2. In Figure 1, VL-Rethinker-72B outperforms OpenAI-o1 on the first three datasets (e.g., EMMA full, MathVerse testmini) but lags behind on the latter two (e.g., MMMU-Pro overall, overall). What factors contribute to this inconsistent performance across different benchmarks?\n3. The paper mentions that \"Understanding why RL incentivizes reflection differently in multimodal contexts versus math-only settings is an important avenue for future work.\" Do the authors have any initial hypotheses or speculative explanations for this observed difference?\n4. It would be valuable to see the performance of the proposed approach across additional benchmarks, including MMStar[1], M3CoT[2], AI2D[3], ChartQA[4].\n\n[1] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024\n\n[2] Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M3Cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. arXiv preprint arXiv:2405.16473, 2024\n\n[3] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. Springer, 2016\n\n[4] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022", "limitations": "See in Questions.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths\n- The identification and mitigation of the \"Vanishing Advantages\" problem in GRPO through SSR is a valuable technical insight, addressing a critical limitation in RL for VLMs and improving training stability.\n\n- Forced Rethinking is a novel and lightweight strategy that effectively promotes deliberate reasoning, as evidenced by the model's ability to self-correct and verify answers, leading to state-of-the-art results on key benchmarks.\n\n# Weaknesses\nSee in Questions.", "was_revised": true, "final_justification": "The reviewer answered all my review comments and used experiments to prove the effectiveness of their method, so I increased the score accordingly.", "rebuttal": "We thank the reviewer for their thoughtful feedback and insightful questions. We are glad to address them below.\n\n**1. Regarding the inclusion of \"slow-thinking\" data in the original Qwen-VL training.**\n\nThis is an excellent question. Based on our analysis of the base model (Qwen2.5-VL), the model does not intrinsically generate reflective reasoning process (see line 152), which is precisely the motivation for \"forced rethinking\".\n\nThere are several potential reasons for not including \"slow-thinking\" data in Qwen2.5-VL. First, the release of Qwen2.5-VL is within one month after the release Deepseek-R1, so the training data might not include much slow-thinking data. Besides, Qwen-VL are engineered for comprehensive performance across a wide spectrum of tasks, including many \"fast-thinking\" perception-focused benchmarks (e.g., OCR, grounding) where slow-thinking is less necessary. Second, creating a well-balanced instruction-tuning dataset that effectively mixes \"slow-thinking\" data for reasoning with \"fast-thinking\" data for perception is a significant challenge. In fact, a very recent work, MiMo-VL, highlights this difficulty and proposes a complex four-stage tuning process to empower VLMs with slow-thinking capabilities, underscoring the non-trivial nature of this problem.\n\n**2. On the performance gap between VL-Rethinker and OpenAI-o1 on certain benchmarks.**\n\nThe reviewer correctly observes the performance discrepancy on benchmarks like MMMU-Pro and MEGA-Bench. These benchmarks are exceptionally broad, covering a massive range of multidisciplinary knowledge and long-tail, real-world visual-language tasks. The sheer diversity of domains and instructions in these tests requires vast world knowledge that is not well-represented in publicly available academic datasets.\n\nProprietary models like OpenAI's GPT-4o are trained on datasets of a scale and quality that are orders of magnitude larger than what is available to the public. This extensive, high-quality private data likely provides them with a decisive advantage on these benchmarks. \n\n\n**3. On the speculative reasons why RL incentivizes reflection differently in multimodal contexts.**\n\nOur primary hypothesis, mentioned briefly in the paper (line 152), is that the base VLM generate insufficient reasoning processes that include self-reflection in its native sampling distribution. In more technical terms, during the RL exploration phase, the model's rollouts overwhelmingly consist of direct, concise answers rather than multi-step, reflective thought processes.\n\nThis creates an exploration bottleneck. Since the model rarely, if ever, spontaneously produces the \"slow-thinking\" behaviors we want to reward, the RL algorithm has few opportunities to learn and reinforce these complex reasoning pathways. This contrasts with text-only settings, where the base models have seen slow-thinking data more frequently in pre-training and instruction tuning, and samples these reflective responses during RL more frequently.\n\n**4. Regarding evaluation on additional benchmarks.**\n\nWe appreciate the reviewer suggesting these highly relevant benchmarks. Our selection of the seven benchmarks used in the paper was deliberate, aiming to cover a representative set of reasoning skills. For instance, MathVerse, MathVista, MathVision assess quantitative reasoning skills similar to those in **MMStar**. Likewise, MathVerse, EMMA, MMMU involves evaluating diagram and chart understanding capabilities that overlap with **AI2D** and **ChartQA**. \n \nWe acknowledge the value of broadening our evaluation and will certainly consider including these benchmarks in future iterations of our work to provide an even more comprehensive assessment."}, {"review_id": "bvNvu5soLc", "reviewer": "Reviewer_J7nB", "summary": "This work proposes the VL-Rethinker model and associated training method for slow-thinking capabilities in Vision-Language Models. VL-Rethinker uses selective sample replay (SSR) to improve standard GRPO training, which suffers from a vanishing advantages problem due to the advantage normalization. SSR maintains a buffer of past high-value learning experiences. VL-Rethinker also uses forced rethinking, which triggers the model to extend its thinking process during RL training. VL-Rethinker demonstrates strong performance across multimodal reasoning and understanding tasks compared to both open-source and proprietary models. The paper ablates the importance of SSR and forced rethinking and analyzes how they impact the trained model.", "questions": "1. How does VL-Rethinker performance vary as a function of the number of thinking tokens per-problem setting?\n\n2. How is the performance impacted by the SSR buffer size? \n\n3. Can forced rethinking work if it is just included in the RL training without the SFT loss? In this setup, the model is trained with GRPO and the SSR technique, but the generated responses used in RL also include the forced rethinking. Is the SFT loss necessary as opposed to just end-to-end RL?\n\n4. How robust is SSR to the choice of alpha? What values of alpha does SSR degrade in performance for? How should the best alpha value be selected?\n\n5. How important is the rethinking fraction and the rethinking prompts to the performance of VL-Rethinker?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n\n1. VL-Rethinker demonstrates strong empirical performance across a variety of competitive benchmarks against many models. VL-Rethinker outperforms the best \"slow-thinking\" models like OpenAI-o1 and open-source models like QvQ-72B. \n\n1. SSR is a novel technique for fixing the important problem of vanishing advantages in GRPO. Several recent works, such as DAPO [1] and LOOP [2], also observe similar findings for text-based environments. However, SSR is a novel solution that replays high-value data points based on the advantage values. The paper empirically demonstrates this replay-based strategy performs better than a purely filtering-based strategy like DAPO. Furthermore, VL-Rethinker demonstrates this for multimodal tasks.\n\n1. Forced rethinking is also a novel technique for training the model to output reasoning chains. Just like the novelty of SSR, several recent works like S1 have explored similar ideas. However, the forced rethinking technique from this paper is different in that it is applied at training time. As Figure 7 demonstrates, this results in VL-Rethinking being able to adaptively control when it rethinks based on the problem difficulty.\n\n1. The paper has comprehensive analyses ablating the components of VL-Rethinker. First, the paper confirms the vanishing advantages problem, which was previously explored in text-based tasks, also occurs for multimodal tasks. The paper quantifies the impact of the vanishing advantages problem on the effective batch size in Figure 2. Table 2 also confirms the empirical importance of filtering non-meaningful advantages and including the SSR buffer in Table 2. Analysis in Figures 5 and 6 illustrates why these techniques are important to reduce overfitting and prevent a biased advantage distribution. The paper also conducts a similarly detailed investigation for the forced rethinking by comparing to no rethinking or forced rethinking every time. The upper bound of rethinking on only wrong answers also helps ground the results in the possible improvement that can be gained from thinking. This analysis demonstrates how forced rethinking dynamically rethinks based on the problem difficulty.\n\n1. The paper addresses the important problem of reasoning for multimodal tasks, where previously, multimodal reasoning capabilities were similar to non-reasoning models.\n\n1. The paper claims that all the data and training code will be released. The high-quality dataset of 38,870 queries is also valuable for future work to build off of. The paper also provides sufficient implementation details in Appendix B. \n\n**Weaknesses**:\n\n1. The SFT loss for the forced rethinking is not justified. The rest of the training pipeline relies on end-to-end RL, yet the forced rethinking stage requires an additional SFT loss to distill the extended thinking traces. However, this forced rethinking could have also be included in the regular RL training, which is a simpler algorithm and implementation. See my question about the importance of this choice in the \"Questions\" section.\n\n2. The paper does not include an analysis of the reasoning outputs outside of the rethinking ratio in Figure 7 and qualitative examples in Appendix C.2 Since the paper's primary contribution is improving the reasoning process, it is important to further analyze this. How does the model's performance improve as a function of the number of thinking tokens? How many thinking tokens does the model output for different types of problems, like based on problem difficulty?\n\n3. The paper does not investigate how the size of the SSR buffer impacts performance. For example, a large buffer over a long training run could result in the model learning from stale, very off-policy data, which could negatively impact performance.\n\n4. Figure 9 compares the sensitivity of $\\alpha$ in the performance of VL-Rethinker, but does so for a seemingly narrow range of $\\alpha$ since the performance of VL-Rethinker is relatively unaffected. \n\n5. The forced rethinking requires several decisions with the selection of the rethinking triggers and rethinking fraction. It's unclear how robust these decisions are to different problem settings.\n\n6. What is the training setting for Figure 2? Is this training the base model on the collected dataset with regular GRPO?\n\n**Minor weaknesses**:\n\n1. What does the bold in Table 1 mean? For MMMU, why is VL-Rethinker bolded, but Llama4-Scout is the best-performing model?\n\n1. I suggest referring to specific sections in the Appendix to make it easier for readers to find the associated sections (L138, L166, L192, L222).\n\n1. There is a missing subscript for $\\mathcal{B}_\\text{replay}$ on L134.\n\n**References**:\n\n[1] Yu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" arXiv preprint arXiv:2503.14476 (2025).\n\n[2] Chen, Kevin, et al. \"Reinforcement Learning for Long-Horizon Interactive LLM Agents.\" arXiv preprint arXiv:2502.01600 (2025).", "was_revised": true, "final_justification": "Overall, this paper introduces novel techniques in VL-Rethinker for reasoning capabilities in Vision-Language models. VL-Rethinker shows strong empirical performance against baselines. The rebuttal states several additional important analyses will also be added, which will further strengthen the paper.", "rebuttal": "We are very grateful to the reviewer for their positive and exceptionally thorough feedback. We are thrilled that they recognized our work as \"excellent\" and appreciated the novelty of our contributions. Their insightful questions and constructive criticism have been invaluable in helping us identify areas for improvement and strengthen the paper.\n\nBelow, we address the weaknesses and questions raised.\n\n\n### 1. **On the Justification of the SFT Loss**\n\nThank you for raising this important point. We realize that the role of the auxiliary SFT loss was not sufficiently justified in the manuscript.\n\nThe SFT loss is applied concurrently with the RL objective (GRPO) during the training phase. Its primary purpose is to **accelerate the learning of the \"forced rethinking\" behavior**. While the model could learn to generate the thinking process through the RL objective alone (we tried), we observed that this approach is significantly slower. This is because the trigger tokens for rethinking (e.g., \"wait\") initially have a very low probability under the policy's distribution. As a result, the discrepancy distribution on the forced tokens is large in the initial training phase and gets clipped. It is until an extended training process that the policy adopts the rethinking tokens. In contrast, the auxiliary SFT loss provides a more direct and stable gradient signal, guiding the policy to adopt the rethinking syntax much more efficiently. We will add a dedicated justification to clarify the motivation for the auxiliary SFT loss in our revision.\n\n\n\n### 2. **On the Analysis of Reasoning Outputs**\n\nThis is an excellent suggestion. We agree that a deeper analysis of the model's reasoning process is crucial to fully demonstrate the benefits of our approach.\n\n  **Revision Plan:** We will add a new analysis section to the Appendix. Due to time constraints during the rebuttal period, we have completed preliminary analyses and will include a comprehensive version in the revision. This new section will include:\n  1.  **Rethinking Efficacy:** A quantitative breakdown of rethinking success and failure rates across different tasks (which we have already completed a preliminary study as per .Reviewer EyWX's suggestions)\n  2.  **Error Analysis:** A comparative error analysis of final answers with and without rethinking to pinpoint the types of mistakes our method corrects.\n  3.  **Reasoning Length vs. Difficulty:** A correlation study between the length of the thinking trace (i.e., number of tokens) and problem difficulty and types, providing insight into how the model dynamically allocates computational effort.\n\n\n### 3. **On the Impact of the SSR Buffer Size**\n\nThe reviewer raises a valid concern regarding the SSR buffer size and the risk of learning from stale, off-policy data. A very large buffer can indeed degrade performance by increasing the discrepancy between the behavior policy and the current policy. While a full ablation on buffer size was not feasible during the rebuttal period due to computational limitations, we recognize its importance. We commit to including this ablation and a discussion on it in the final version of the paper. \n\n### 4. **On the Robustness of Hyperparameters ($\\alpha$ and Rethinking Fraction)**\n\nWe appreciate the feedback on Figure 9 and the general question of hyperparameter robustness.\n\n* **Alpha ($\\alpha$) Sensitivity:** The range for $\\alpha$ in Figure 9 was selected to demonstrate performance stability across a set of reasonable values. But we realize the current manuscript does not justify this range is reasonable by connecting it to the sharpness of the temperature-annealed softmax distribution. We will refine this in our revision.\n* **Rethinking Hyperparameters:** we will add a brief discussion in the Appendix on the sensitivity to the rethinking fraction and the selection of triggers, providing practical guidance for applying our method to new problem settings.\n\n### 5. **Clarifications and Minor Points**\n\n* **Figure 2 Training Setting:** Thank you for requesting clarification. This figure illustrates a 72B-Instruct model trained with standard GRPO on publicly available datasets. We will update the caption and text to make this clear. The figure's purpose is to motivate our work by showing that even large models can quickly saturate their performance on publicly available VL reasoning queries that are potentially noisy, and easy-to-learn for larger models.\n* **Minor Weaknesses:** We are grateful for these detailed corrections. We will remove the incorrect bolding of MMMU in Table 1, add the missing subscript for $\\pi$ on L134, and add direct cross-references to the Appendix sections as suggested.\n\nThank you once again for your constructive and valuable feedback."}, {"review_id": "CqNhhav3RA", "reviewer": "Reviewer_EyWX", "summary": "1. The paper introduces VL-Rethinker, a novel framework to train vision-language models (VLMs) for adaptive, reflective, and step-by-step reasoning using direct reinforcement learning—without any supervised fine-tuning, chain-of-thought supervision, or teacher model distillation. The approach is built on Qwen2.5-VL-Instruct (7B, 32B, 72B scales).\n2. A key novelty is the Forced Rethinking mechanism, where natural language triggers like “Wait, is this right?” are inserted into the model’s own generation to initiate a reflective second-stage response, which is reinforced only if the final answer improves.\n3. To enable stable and effective RL training in this long-form, sparse-reward setting, the authors introduce Group Relative Policy Optimization (GRPO) with Selective Sample Replay (SSR), which reuses high-advantage trajectories to boost learning efficiency and policy robustness.\n4. VL-Rethinker learns to rethink selectively—invoking reflection when needed and skipping it otherwise—and often revises incorrect reasoning steps into correct ones, demonstrating emergent metacognitive behavior.\n5. The model is trained on a curated set of ~39K rewardable multimodal QA examples sourced from MathVista, MathVerse, EMMA, MEGA, and other datasets, using programmatic reward functions based solely on final answer correctness.\n6. Evaluation is conducted via greedy decoding (no sampling or self-consistency) across 7 benchmarks, including: Math-heavy: MathVista, MathVerse, MathVision, Scientific/diagram: MMMU, EMMA, General open-ended: MEGA. Accuracy is measured via pass@1 based on boxed final answers.\n7. VL-Rethinker achieves state-of-the-art open-source performance, outperforming or matching teacher-distilled baselines like OpenVLThinker and R1-OneVision, with the largest gains on math and diagram reasoning tasks. The model shows limited but consistent improvements on scientific (EMMA, MMMU) and real-world (MEGA) tasks, indicating potential for generalization, though the largest gains are in structured tasks with deterministic rewards.\n8. The paper releases pretrained models (7B, 32B, 72B) and training data publicly, offering a reproducible and scalable framework for studying self-reflection and direct RL in both vision - language and potentially text-only reasoning models.", "questions": "1. Can the authors provide a measure of correctness of reflective steps  - quantify when the model chooses to reflect, how often it helps, and whether reflective steps are more accurate than initial ones? Statistics on reflection frequency vs. task difficulty, and success rates of revised answers compared to original answers, would support claims of emergent metacognition.\n2. Failure case analysis or reward shaping details could reveal whether the system is robust or prone to unnecessary revision loops. Does rethinking ever lead to degraded answers or hallucinations? How does the model learn to suppress unhelpful reflection?\n3.Even a small-scale experiment on a text-only task would help test the modality-agnostic claim and broaden the paper’s applicability.\n4. A breakdown of errors corrected through rethinking—separated into vision-specific (e.g., perceptual misinterpretation) and logic-specific (e.g., math or inference errors)—would clarify whether reflection leverages the visual modality meaningfully. It would help to clarify whether the rethinking mechanism helps the model improve visual perception (e.g., object recognition, spatial relationships), or if it primarily corrects symbolic/logical reasoning based on already-processed visual features?", "limitations": "Limitations are fairly discussed in supplementary material.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths -\n1. The paper proposes a novel direct reinforcement learning (RL) framework to induce reflective, step-by-step reasoning in vision-language models (VLMs) without relying on supervised CoT data, rationales, or teacher model distillation—marking a significant advance in self-improving LLM training.\n2. It introduces Forced Rethinking, a simple but effective mechanism using natural-language triggers to teach the model to re-evaluate its own answers, enabling reflective behavior to emerge purely through reward-driven optimization.\n3. The method scales robustly across model sizes (7B, 32B, 72B) and works across domains (math, science, diagrams, general QA), demonstrating scalability and domain generality without additional supervision.\n4. VL-Rethinker offers a generalizable training recipe that is modality-agnostic in principle, paving the way for reflective reasoning in both vision-language and future text-only LLMs using direct RL approaches.\n\nAreas of Improvement -\n1. While Selective Sample Replay (SSR) improves efficiency, the paper does not sufficiently discusses the potential downsides of this approach. It may introduce bias toward early high-reward trajectories, which may reduce exploration and limit the discovery of diverse or novel reasoning strategies.\n2. The evaluation primarily focuses on final answer accuracy (pass@1) and does not include direct metrics for rationale quality, correctness of reasoning steps, or effectiveness of reflection—missing an opportunity to quantify reflective behavior.\n3. Although the approach could generalize to text-only models, this is not demonstrated—limiting the scope of applicability despite the technique’s modality-agnostic potential. \n4. Additionally, VL-Rethinker primarily seems to corrects symbolic or logical errors, which are common in text-only models, but does not clearly demonstrate improvements in vision-specific reasoning such as object grounding, spatial understanding, or visual attribute correction. This raises a question about whether the rethinking mechanism truly leverages visual inputs or merely enhances textual reasoning with such approach. Further error analysis on VL-Rethinker can demonstrate the major error types and patterns, providing direction for further research on visual reasoning.\n5. The method relies solely on binary end-of-sequence rewards (i.e., answer correctness), which leads to potential credit misattribution—the RL algorithm may reinforce flawed intermediate reasoning steps that coincidentally lead to the correct final answer. Further analysis on such behavior would be helpful to understand the limitations of proposed approach.", "was_revised": true, "final_justification": "The rebuttal responses clarify some of the important questions including the qualitative analysis of rethinking and rethinking success across multiple difficulty levels and impact of SSR in modality agnostic/ text settings. \nThe two questions remain open, 1) error analysis of rethinking trigger to correct different categories of error, 2)downside of SSR approach. But, overall the quality of the paper is good and rebuttal response further helps to update the rating.", "rebuttal": "We sincerely thank the reviewer for their thorough evaluation and insightful feedback. The comments are highly constructive and have helped us improve the quality of our work. \n\nBelow, we conducted additional experiments and analysis to address the questions and provide revision plans for the areas of improvement.\n\n### **1. On Quantifying Reflective Behavior and Metacognition (Weakness 2, Question 1)**\n\nThis is an excellent point. We were aware of this qualitative analysis of reflections but encountered challenges in properly defining reflections and recognizing reflections in the responses: in many occasions, model reflections cannot be detected by pre-defined reflective phrases. \n\nBelow we attempted a qualitative analysis by focusing on the reflections triggered by our \"forced rethinking\". We conducted our analysis using VL-Rethinker-32B inferred responses on the validation data, selecting a subset of queries that keep the correctness of responses balanced. We defined problem difficulty based on the mean accuracy of responses within the same query group. We then measured the frequency of rethinking (i.e., generating the \"Wait\" trigger), the success rate (initial answer was wrong, final answer is correct), and the failure rate (initial answer was correct, final answer is wrong).\n\nOur findings are summarized in the table below:\n\n| Problem Difficulty (Baseline Accuracy) | Rethinking Frequency | Success Rate of Rethinking | Degradation Rate of Rethinking |\n| :--- | :--- | :--- | :--- |\n| **Easy** ($>75\\\\%$) | 44.8% | 27.3% | 4.8% |\n| **Medium** ($37.5\\\\%-75\\\\%$) | 68.2% | 14.7% | 3.1% |\n| **Hard** ($<37.5\\\\%$) | 78.5% | 6.9% | 0.8% |\n\nWe draw the following key observations:\n* **Adaptive Reflection:** The model’s rethinking frequency is positively correlated with problem difficulty. It adaptively triggers reflection more often on harder problems where its initial answer is more likely to be incorrect.\n* **Effective Rethinking:** While the success rate of rethinking is higher for easier problems and lower for harder problems, rethinking is overall beneficial. The model consistently improves its answers across all difficulty levels via rethinking, with very low degradation rates. The degradation rate is low because the model tends to use self-verification for correct answers, and the outcome reward distinguishes the degraded rethinking response from a beneficial rethinking response.\n\nHere we acknowledge the limitation that problem difficulty is bootstrapped from our model's own performance. We will refine this analysis using more objective judgement in our revision. Nonetheless, these statistics provide quantitative support for our claim of **emergent metacognitive behavior**. The model shows evidence of *when* to rethink and this reflection leads to consistent, positive performance gains. \n\n\n### **2. On the Modality-Agnostic Claim of SSR (Weakness 3, Question 2)**\n\nWe thank the reviewer for asking for clarification on this point. The \"vanishing advantages\" problem is general in GRPO **when training queries is easy to master**, leading to vast majority of training rollouts being zero advantages. This issue is particularly pronounced for vision-language reasoning domains because there are still a lack of publicly available high-quality challenging reasoning-related queries. In the text-only math reasoning domain, the vanishing advantages problem is mitigated by a few high-quality challenging query dataset, for example, DeepMath. The table below shows the performance improvement on GSM8K and Math500 when training with different collection of training queries (easier-to-master `Math8K` vs. the higher-quality challenging `DeepMath`). The values shown are absolute pass@1 improvements.\n\n| Data | GSM8K | Math500 |\n| :--- | :--- | :--- |\n| Math8K | +1.7 (84.4->89.3 vs 87.6) | +3.2 (60.4->71.4 vs 68.2) |\n| DeepMath | +0.3 (84.4->89.5 vs 89.2) | +1.7 (60.4->78.2 vs 76.5) |\n\n**Key Observations:**\nThe benefit of SSR is more pronounced on the `Math8K` dataset. This is because the problems in `Math8K` are easier to master, making the baseline GRPO more susceptible to saturation. On the more uniformly challenging `DeepMath` dataset, the baseline is stronger, but SSR still offers a reliable, albeit smaller, improvement.\n\n\n\n### **3. On Error Analysis of Rethinking (Question 3)**\nThis is a crucial analysis to understand whether rethinking improves visual perception or primarily corrects textual-symbolic reasoning. Due to the time-intensive nature of manually inspecting and annotating numerous generation rollouts, we were unable to complete this analysis during the rebuttal phase.\n\nHowever, we have a clear plan to incorporate this into the final version of our paper. The analysis will involve manually categorizing corrected errors into groups such as:\n 1.  **Logic-Specific Corrections:** The initial visual perception was correct, but the model corrected a flaw in its mathematical or logical reasoning.\n\n 2.  **Vision-Specific Corrections:** The model initially misinterpreted or failed to locate a key visual element, and the reflection process corrected this perceptual error.\n\n 3.  **Combined Corrections:** Both the visual interpretation and the subsequent logic were flawed and were corrected.\n\n\n### **4. On the Areas of Improvements**\n\n- **Outcome Reward and Rationale Quality.** We thank the reviewer to pointing out this critical point. Since this work is a preminimary study of slow-thinking in VLMs, the paper follows the prevailing setting of using outcome reward. However we agree this will be a important direction for furture work and we will add this point in the limitation and future work section in the revision. \n\n- **Downsides of SSR.** We agree with the reviewer that discussing the potential downsides of SSR is important. In the meantime, we would like to clarify a detail of our implementation designed to mitigate this risk: SSR prioritizes trajectories based on **absolute advantage**, not just raw rewards. This prevents the policy from merely overfitting to simple, high-reward samples and instead encourages it to focus on trajectories where the policy's performance significantly exceeded its expected value.\n    \n    However, we acknowledge that the full impact of SSR on the policy's long-term exploration remains an open area for investigation. We will add a discussion in the paper to elaborate on this potential limitation and frame it as a direction for future research into balancing exploration and exploitation in VLM-based RL.\n\n\nWe thank the reviewer once again for their valuable and actionable feedback, which has undoubtedly strengthened our paper."}]

Author rebuttal:

Meta-review: All reviewers agree on the paper's acceptance.

- The proposed Selective Sample Replay (SSR) and Forced Rethinking, are novel and effective.
- Strong empirical effectiveness.
- Thorough experiments.
Decision: Accept (spotlight)