OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles
Authors: Yihe Deng
Hritik Bansal
Fan Yin
Nanyun Peng
Wei Wang
Kai-Wei Chang
Venue: NeurIPS 2025 poster ()
Keywords: Vision-language reasoning
iterative improvement
distillation
reinforcement learning
Primary area: deep_learning

Abstract: We introduce *OpenVLThinker*, one of the first open-source large vision–language models (LVLMs) to exhibit sophisticated chain-of-thought reasoning, achieving notable performance gains on challenging visual reasoning tasks. While text-based reasoning models (e.g., Deepseek R1) show promising results in text-only tasks, distilling their reasoning into LVLMs via supervised fine-tuning (SFT) often results in performance degradation due to imprecise visual grounding. Conversely, purely reinforcement learning (RL)-based methods face a large search space, hindering the emergence of reflective behaviors in smaller models (e.g., 7B LVLMs). Surprisingly, alternating between SFT and RL ultimately results in significant performance improvements after a few iterations. Our analysis reveals that the base model rarely exhibits reasoning behaviors initially, but SFT effectively surfaces these latent actions and narrows the RL search space, accelerating the development of reasoning capabilities. Each subsequent RL stage further refines the model's reasoning skills, producing higher-quality SFT data for continued self-improvement. OpenVLThinker-7B consistently advances performance across six benchmarks demanding mathematical and general reasoning, notably improving MathVista by 3.2\%, EMMA by 1.4\%, and HallusionBench by 2.7\%. Beyond demonstrating the synergy between SFT and RL for complex reasoning tasks, our findings provide early evidence towards achieving R1-style reasoning in multimodal contexts.

Open VLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles
Yihe Deng, Hritik Bansal, Fan Yin Nanyun Peng, Wei Wang, Kai- Chang University of California, Los Angeles
Abstract
We introduce Open of the first open-source large vision-language models (LVLMs) to exhibit sophisticated chain-of-thought reasoning, achieving notable performance gains on challenging visual reasoning tasks. While text- based reasoning models (e.g., Deepseek R1) show promising results in text-only tasks, distilling their reasoning into LVLMs via supervised fine-tuning (SFT) often results in performance degradation due to imprecise visual grounding. Con- versely, purely reinforcement learning (RL)-based methods face a large search space, hindering the emergence of reflective behaviors in smaller models (c.g., 7B LVLMs). Surprisingly, alternating between SFT and RL ultimately results in significant performance improvements after few iterations. Our analysis re- veals that the base model rarely exhibits reasoning behaviors initially, but SFT effectively surfaces these latent actions and narrows the RL search space, accel- erating the development of reasoning capabilities. Each subsequent RL stage further refines the model's reasoning skills, producing higher-quality SFT data for continued self-improvement. Open VLennkthhinee-775 consistently advances per- formance across six benchmarks demanding mathematical and general reasoning, notably improving Math Vista by 3.8%, EMMA by 2.4%. and HallusionBench by 1.6%. Beyond demonstrating the synergy between SFT and RL for com- plex reasoning tasks, our findings provide early evidence towards achieving RI- style reasoning in multimodal contexts. The code, model and data held attps://git htt.bb/ttttt.t//.// hhom://hhedeegg////e//.......iaa
80
Math Reasoning
General Reasoning
Percoption
Average
Model
CPT-4o
Cloude33 3.sul-3.5-Sormet
60
Qownd S-L-78
R1-1-78
Rl Onevision- 7B
Thinhinkrr 78
OpenVVT
20
MarhM
MathWerse
MethVision
MMMU-IFr
EMMMA
Hulluberch
Avg
Figure 1: Our OLenneLThn (in red) performs competitively to large proprietary multimodal models such GPT-4o and Claude-3.5 (in gray), especially in Math and Perceptron tasks. outperforms VL base models at the same scale (in blue) and other recently released VL reasoning models (in purple).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Proprietary large language models (LLMs), notably OpenAT's 0-series [30] and Google's Gemini-2.5 Pro [18], have demonstrated impressive multi-step reasoning abilities of planning, reflection, and verification. Recent open-wen weeeht models [53. 28, 47, 84, 83] (e.g.. DeepSeek-Ri -1 [20] and smaller LLMs like S1 [49] and QwQ-32B [63]) show that reinforcement learning (RL) with verifiable rewards effectively reproduces these advanced capabilities, significantly boosting performance on challenging mathematical and logical tasks.
Unlike text-only L.Mss, it remains unclear whether open-source large vision-language models (LVLMs) can effectively adopt similar sophisticated reasoning strategies. Modern LVLMs such as LLaVA-NeXT [37] and Qwen2.5-VL [3] benefit from extensive vision-language pretraining and demonstrate strong visual instruction-follwwiii capabilities. However, they rarely demonstrate advanced reasoning behaviors like GPT-ol or DeepSeek-RI.
Moreover, it is known that reasoning capabilities can generally be distilled from larger L.MMs to smaller ones through supervised fine-tuning (SFT) on chain-of-though demonstrations [35, 32] for text-only tasks. This recipe has been recently applied in distills demonstrations from DeepSeek-RI (LIMO [78]. S1 [49] and OpenThinker [62]) followed by optional RL fine-tuning [79]. However, adapting this method to LVLMs does not work. Proprietary LVLMs, such as OpenAl's o1/o3 and Google's Gemini, do not expose their internal reasoning paths, making their outputs unsuitable for distillation. Therefore, most recent attempts are focusing on improving LVLMs through distillation from text-only R1 reasoning models (see discussion in Section 2.2). Unfortunately, our experiments show that naively fine-tuning LVLMs on reasoning paths generated from text-based DeepSeck-RR - with image captions leads to a non-trivial performance drop (see Figure 3), primarily due a lack of precise visual grounding. Similar observations can be found in [6, 76].
In this paper, we present Open WLThinker-TT 7B, one of the first open-weight LVLMs that exhibit complex reasoning capabilities in complex vision-language tasks. Specifically, it is trained by iterating between the following two steps:
1. Lightweight SFT. In the first iterations, we distill CoTs using a text-only Deep-Seek R1 given the task question and the corresponding generated image caption. These CoTT traces provide demonstrations of reasoning actions, although they do not immediately improve LVLM's accuracy. For later iterations, we use the LVLM from the previous iteration to produce CoTSs on 3,000 data points. This small dataset is sufficient to progressively enhance the model's reasoning depth. 2. Curriculum RL. In subsequent iterations, we further enhance the LVLM's reasoning through RL exploration with Group Relative Policy Optimization (GRPO) [56], which splits training into two rounds to form a smooth curriculum.
We found that while the initial step of SFT leads a performance drop, iteratively alternating between SFT and RL eventually gradually yields a significant performance gain on both reasoning depth and answer accuracy (Figure 2).
Our further analysis shows that the inference-time reasoning behaviors are often triggered by specific tokens (e.g.. "wait"). SFT serves as an inductive prior that highlights these reasoning actions. Specifically, it demonstrates the tokens such as "first", "wait", "check", that trigger the model's s planning, reflection, and verification behaviors. Without this SFT step, launching RL from scratch forces the model to search through a prohibitively large space, making reflective behaviors slow to emerge - if they emerge at all. On the other hand, RL plays the critical role in learning the reasoning hehaviors, generalizing from training data, and offering a better foundation for the next SFT iteration. The iterative cycle between SFT and RL. collaboratively optimizes LVLM's performance.
We highlight our contributions as follows:
We introduce Open TLThinker-7BB one of the first open-source LVLMs to demonstrate reliable self self-reflection, planning, and correction in visual contexts.
We present a simple yet effective iterative SFT-RL loop that enables R R--stye reasoning into multimodal domains and steadily self-improves without requiring massive datasets.
We analyse linguistic markers of complex reasoning and show that SFT can steer RI. exploration toward highlighted reasoning actions.
On six challenging benchmarks, including Math Vista and Math Verse, Open presents remarkable improvements while reducing hallucination on HellusionBench.
2

---

1650
Iterative SFT-RL Cycle GRPO-Iter()
SFT-Iteri
1000
RL
636
Exploration
Gererttion wi outcoms verification
and generalization
Progressively chailenging question source
for
1400
chec
arkdee Wa
k
1958
diget
Guide RL search space
plan
retectioon
aitemste
Detter Prior
may maybe asking
be
Base Model
SFT-Iter(i-1) Data
Owwn2.5-VVL-P
recal
Ere
eve
mayle
Figure 2: Illustration of Open VLThinnke--7B5 training process. We iteratively apply SFT and GRPO to refine the LVLM using reasoning data generated from previous iterations. The data sources are also progressively evolved introduce more challenging questions over time.
2
Related Work
2.1 Complex Chain-of- Thouughtt Reasoning
Since the introduction of OpenAl's O1 model [30]. researchers have shown strong interest in reproducing and enhancing the complex reasoning capabilities LLMs [53, 28, 47, 84, 83], partly due to its superior performance on mathematical benchmarks. [20] introduce the open-source DeepSeek- R/ model and investigate how RL with verifiable rewards can promote advanced chain-of-thought reasoning and reflective behaviors. This development inspired line of research focused on open- source reproduction [45, 25, 85, 41, 62] and the analysis of such complex reasoning in mathematical problem solving [79, 73, 78, 10]. In parallel, several recent studies have similarly explored the effects of test-time scaling on encouraging more complex model reasoning behaviors [49, 55, 39, 17, 87, 60]. However, the majority of research have significantly advanced text-based reasoning, and development of languae reasoning much more initial.
2.2 Vision-Language Reasoning Model
Recent advancements large vision-language models (LVLMs) stem from open-source LLMs [65, 66, 14, 75] and text-aligned image encoders [54, 36]. Integrating these components has enabled
LVLMs to follow diverse visual instructions and generate meaningful responses [38, 15, 16, 11, 37. 3]. Parallel to the model development, researchers have also been interested in eliciting CoT reasoning chains from LVLMs via prompting [89, 90, 48, 26] or fine-tuning [21, 74, 64, 13]. These reasoning models remain mostly on a shallow level of common dee--by-step prompting, without self-reflcctions or seff-verifications.
Concurrent work. Very recently, many studies have started exploring how equip LVLMs with R R1-like reasoning capabilities through distillation from text-only reasoning models [5, 76, 27, 33) or directly rely on RL [91, 43] for self-exploration. Further advancements [57, 71, 6, 46, 72, 68, 40, 86] have focused on improving performance in visual math reasoning, which marks the transition from carly-stage exploration more effective complex vision-language reasoning. Please note that most of these works are within the two months before the submission date, and some of them do not even have associated technical reports available yet. Our work aligns with these studies and contributes unique insights into the role of SFT for complex reasoning, along with an iterative SFT-RL framework to further advance research in this direction.
3 Preliminaries
An LLM is defined by probability distribution Pe. parameterized by model weights 0. Given prompt sequence x = [x1....,,,,] J. the model generates response sequence y [yi...., Vmn where v; and yj represent individual tokens. The response y is sampled from the conditional distribution (-(+))), factorized as po (y)x = [ Po :1j-).)
Supervised Fine- Mone-Tuning (SFT). SFT is typically applied to specialize L.MMs for a particular task or domain. This process updates the model parameters 0 by providing example responses of desired
3

---

72
ethinle
dkay, let's see. The question is askdnng what happens to the supply of salmon if the number of ferns decreases. First, need look the connections in the food web diagran. The ferns So
70
food night have less salson can eat fish. the salmon's is food sources pretty diverse. len, let check Becrease sounds right because fener ferns means less Food for salnon. Can't tell is out because we have clear connection. Stay sane is unlikely because the temms are direct food source. Increase is the opposite of shat ne're looking for. </thinke-
there's more. diagran also
68
other options.
Owen2.5-VL1.
Qwen2VL-728 (reparted)
The fems directly shown food source for salmon the number of ferns Food available, their supply.
64
the salmon will Likely <answer=(A)=< aaaswers>> would
OpenVLTTinker-778
Distill
Iterl
Iter2
Iter3
Figure 4: An example of Open WiThinker-7 reasoning. Question: In the diagram of the food web shown, if the number of ferns decrease, the supply of salmon will most likely? (A) decrease (B) can't 't tell (C) stay same (D) increase. Corre- sponding image is shown in Figure 12.
Figure 3: Iterative performance improvement of our model on Matta.sta. We note that Iier(i) is always fine-tuned from the base model Qwen2.5-V1.-7B, with its training data gener- ated from Iter(i-I).
behavior to the imput instructions. Concretely, Given a dataset D = {(x(1, ( -()) where x() the prompt sequence and y( is the desired response sequence. We update 0 to maximize the likelihood of producing (() given x()) Formally, (sFr(0) = Σ log po (y(0) (x).. By minimizing the loss, the model learns produce responses more aligned with the labeled examples.
Reinforcement Learning (RL). RL approaches fine-tune LLMs via human preferences modeled under the Bradley- model [50, 12, 56, 1]: p(y. > yi x) o (r(x, Yu and y, denote preferred and dispreferred responses, respectively, and at)) 1/(1 + e e-t) the sigmoid function. The common RL objective under the Bradley-Terry assumption of the reward model r r(x,y y) thus
r(x, yt)). where
max
Ex.y~po y)] B Ex
where B > is the KL penalty coefficient. Under this framework, [56] introduced Group Relative Policy Optimization (GRPO) by sampling a group of response trajectories {o:] from the old policy model Oold for each query x, with the objective as maximizing:
G
lo.1
Po (
x, Oi,ct)
cllpp
Pa (otr X, oi,ce) 1 e, 1 +() A..)]
Po(or, x, Oi,ce)
min
Oj
Poola (Oitt x, Oi,ce)
1=1
- 3 DKL po
po...]
where e 0 is a hyperparameter bounding the clipping range, B balances the KL-penalty term DKL. [To #ref] against the advantage- policy update, and Outd the old policy model. Here, the advantage A,,, Pi = (ra - mean(r)) (st))))((r) is set as the normalized reward at group level.
OpenVeThinker: Iterative Self-improvement on Curriculum Data
In this section, we first amalyze how SFT and RL affect the occurrence of reasoning-relatte keywords, which serves to motivate approach. We then introduce the proposed iterative approach to enhancing complex reasoning capabilities in Open VLThinker-7B with SFT-RL cycles. At last, we propose a source-based curriculum RL. 4.1
The Role of SFT and RL
The initial SFT data. The standard distillation approach used for text-only reasoning cannot be directly applied because the RI model does not support visual input, and other proprietary LVLMs, such as OpenAl's ol/o3, do not expose their internal reasoning paths. To learn reasoning behaviors from RI, we instead use the target model as captioning model, prompting it to generate detailed textual descriptions for each image. Subsequently, these captions serve proxies for the images when input into a text-based RI reasoning model, QwQ-32B [63], which then generates candidate reasoning chains. Among these candidates, select the shortest reasoning chain that correctly

---

Qwen2.5-VL-7B
SFT-Iterl
GRPO-Iterl
1400
1200
1200
1200
MadVVisza Accuracy (N)
68
1000
digest
1000
1000
87
plan
reflection
86
600
600
600
400
400
400
03
64
idenniyy
recal
prosided
first
chrect
mayoe
Idenify
recall
provided
frrst
wait
eteck
maybe
identily
recal
given
provided
firsr
wast
cherk
moybe
SFf
Figure Occurrences of reasoning keywords when solving Math Vista with the base model, SFT-Iterl model, and PRPOO-Ieer model. The most significant distribution shift occurs after SFT, while the scale remains largely unchanged after GRPO, despite notable performance improvements.
arrives at the final answer to avoid excessive reasoning length after SFT (further details in Section 5.2). The overall procedure is summarized in Figure 5.
Impact of SFT and RL Model Reason- ing Actions. Complex reasoning behaviors in LLMs have been described using various terms, including long CoT [79] and aha mo- ments [20]. At their core, these behaviors reflect autonomous planning. reflection, and verification steps that occur during inference. actions,
Question:: According the table, what was the pate of change between 2018
Teat-beaa Rezsoning Morkd (QwQ-32B)
Arsomm::
Peacentag Anowe
Peasening Ansre
Captooning
We refer to them as inference- which are often triggered by specific tokens such as "wait". To examine how SFT and RL influence these reasoning actions, we identify eight representative keywords corresponding to perception, question comprehension, plan- ning, reflection, and seeking alternatives.
5-V--17)
Preseezng Arome
Caption: The image is table Here the data presented table:
Wertiy
SFT-lurl Dat::
2888
5
(Imagn Question Reasoning, Answer)
2009
11
2016
Figure 5: Curation SFT-Iterl data from text-based reasoning models based on image descriptions.
As illustrated in Figure 6, the base model seldom exhibits planning, reflection, alternative-solttinn actions. However, SFT guided by text-based R1-like reasoning traces effectively surfaces these behaviors. As shown in the third and fourth subplots of Figure 6, subsequent GRPO-based RL training following SFT-Iterl substantially enhances model performance Math Vista by 5.2%, yet largely maintains the initial reasoning action distribution, with minor refinements such as reduced repetitive reflections.
Conversely, direct RL training without prior SFT struggles efficiently induce reasoning behaviors, exemplified by the absence of reflection keywords (e.g.. "wait"") even after an equivalent training volume. Concurrent research by [68], which solely relies on RL, addresses this by explicitly appending relevant keywords during training rollouts. These observations support our argument that SFT plays critical role in highlighting desirable reasoning actions, providing an efficient and effective foundation for RL to build upon. In contrast, RL primarily serves to further refine and enhance performance.
4.2 Iterative Improvement
The model obtained after the first iteration (GRPO-Iterl) demonstrates enhanced complex reason- ing capabilities and improved reliability in processing visual inputs compared to methods based on image-to-text conversion. This advancement positions GRPO-Iterl an effective source for generating higher-que gigher-quality reasoning demonstrations, Consequently, we propose an iterative self- improvement strategy, inspired by established methodologies such as iterative SFT in ReST-EM [59] and iterative direct preference optimization (DPO) schemes [81, 51]. both of which have shown substantial effectivenees iterative training processes and fall under the Expectation- Expectation-Maximizaati framework [59].
Specifically, each iteration, we sample a new set of enhanced reasoning traces using the model trained in the preceding iteration. These refined demonstrations then utilized retrain the base
5

---

model', thereby progressively elevating its reasoning performance. The overall iterative pipeline is illustrated in Figure 2, and the consistent incremental performance gains achieved through successive iterations are depicted in Figure 3.
4.3 Two-Stage Source-Basee Curriculum RL
To ensure effective exploration during reinforcement learning (RL), we assess the difficulty of data sources, aiming to provide data that is challenging yet appropriate for the model's proficiency level. Specifically, we utilize GPT-4o to rate the difficulty of five representative examples drawn from various data sources such as FAgureQA [31], MapQA [4], and GeoQA [7], in similar fashion to the text-based evaluation in DeepMath- 103K [23]. Additionally, we employ the base model, Qwen2.5-VL-7B', to obtain its error rates as a complementary difficulty indicator. We standardize independently using z-score normalization for both the GPT-4o rating and base model error rates and compute the average of the two. Based on this composite score, we categorize the data sources into Easy, Medium, and Hard groups via k-means clustering in ld space. With these categories, we construct two difficult--seecinn
datasets: PRI((Medium)) and (Hard) Our curriculum training thus proceeds in two stages within one iteration. sequentially training on (Mediuu)) and D DRI (Hard))
5 Experiments
Training setup. We take Qwen2.5-VL-7B [3] as the base model and perform three iterations of the SFT-RL cycle as illustrated in Section 4, applying full fine-tur tne-tuning for both SFT and RL. Our training framework is based on LLaMA- -aMMoory for SFT and EasyR1* for RL. We source our training data from the established VLaV-OneeVisio [34] and specifically consider the 14 data sources in overlap with Math [58] (Table 4). Based on our preliminary experiments, we equally draw 500 examples from each source to form the SFT seed dataset of 7K examples, where for each iteration we collect distillation data via rejection sampling, resulting in final 3K SFT data. We then classify the data sources into easy, medium and hard (as detailed in Table 4). We construct the 3K medium-level RL training data from the 5 sources that we identified as medium difficulty. Finally, hard-level RL training data from the most difficult sources, summing up to 12K data in total for each iteration that trains from the base model. We defer the training hyperparameters Appendix C.
construct 6K
Evaluation. Our evaluation employs exact matching and a grader function from MathRuler'. We use the same inference hyperparameter as suggested by Qwen and recovered Qwen2.5-VLL-7B's reported results on Math Vista at 68.5%. The hyperparameters are detailed in Table 12. We employ six established benchmarks to examine model's ability thoroughly:
Math reasoning: Math Vista [44], Math Verse [88] and Vision [69]. The three benchmarks evaluate how LVLMs interpret and reason with diagrams in visual math problems through both
multiple-choice and free-form questions.
General reasoning: MMMU-Pro [82] and EMMA [22]. MMMU-Pro spans 30 subjects across 183 subfields, including business, medicine, and science. EMMA evaluates in physics, chemistry, coding, and math.
Perception: HallusionBench [19], designed to evaluate LVLMs' susceptibility to language halluci- nation and visual illusion.
Baselines. We evaluate the non-reasoning base model Qwen2.5- VL as primary baseline to demonstrate the improvements introduced by our method. Additionally, we include the reported performance of proprietary models, including GPT-40 [29] and Claude-3.5-Sonnet [2], alongside open-source LVLMs such as Mulberry-7B [77], V1.2.5888 [9], Kimi-VL-16B [61], and Qwen2- VL-7B [70], as reference points. Crucially, to highlight the effectiveness of our iterative SFT-RL training strategy, we compare our model with concurrent approaches employing a single round of SFT distillation and RL at the same model scale (7B), yet utilizing significantly larger training datasets. These concurrent models include RI-VL-7B [86]. RI-Onevision-775 [76]. and VLAA- Thinker-Qwen2.5VL--B [6]. Notably, RI-Onevision and VLAA-Thinker- S Qwen2.5VL-7B also start from the same base model (Qwen2.5-VL-77B as ours, using 165K and 150K total data, respectively.
'To maintain stability, we retrain the model from scratch at each iteration with the newly generated dataset, as similar to some iterative approaches in text-only domain [59, 24]. In alignment with previous RI reasoning research [79, 781, we choose the base model from Qwen2.5 family for their strong general capability obtained in pre-training.
htttps://githut httt:p::/ggiaaa....o///h/gg///// com//iiyogg//LLMM......... Factory
'nttpa://githe httgg//a...o//hiiogg//aaagg htteps://eittbb..co////eeeeeeee/ hwwphh/..../ohiiooggg//hhhhhleee

---

Table Evaluation results across visual math reasoning benchmarks (Math Vista, Math Verse, Math Vi- sion). general visual reasoning benchmarks (MMMU-Pro EMMA), and perception (HallusionBench). We include the reported performance of proprietary models and open-source Vision-Language models as references. *Performance of the base model Qwen2.5-V -ren2.5-VIL-7B VL-7B and concurrent reasoning models are evaluated by us under the same setting and hardware as OpenVL.Thinker The bold numbers indicate the best results among the open-source models and the underscored numbers represent the second-ber results.
Math Reasoning
General Reasoning
Visual
Model
Data
Math- Vista
Math- Verse
Math- Vision
MMMU- Pro
Hallu- Bench
Avg
EMMA
Proprietary Model
GPT-4o
63.8
50.2
30.4
51.9
32.7
55.0
47.3
Claude-3.5-Somee
67.7
47.8
33.5
51.5
35.1
55.0
48.4
Open-source Vision-Language Language Model
Mulberry-7BB
63.1
39.6
54.1
Intern VL2.5-8B
64.4
39.5
19.7
34.3
Kimi-VL-
68.7
44.9
21.4
Qwen2-VL-7B
58.2
32.5
16.3
30.5
20.2
50.6
34.7
Owen2.5-VL-7B*
68.5
46.8
24.0
42.3
24.4
67.5
45.6
Concurrent Vision- Language Reasoning Models
RI-VL-7B
270K
63.5
40.0
24.7
7.8
8.3
54.7
33.2
RI RI-Onevision-7BB
165K
64.1
46.4
29.9
21.6
20.8
65.6
41.4
VLAA-Thinker-795 7B*
150K
70.0
48.6
24.9
42.0
25.5
68.4
46.6
OLenVL.Thnnee=-77
12K
72.3
50.3
25.9
42.9
26.8
69.1
47.9
In contrast, our model achieves better performance with only 12K training samples from the base model.
5.1 Main Results
We present our main results in Figure 1, with detailed performance across datasets shown in Table 1. As il- lustrated, Open WLThnkkern consistently achieves ei- ther the best or second-best scores among open-source LVLMs of comparable scale across all six benchmarks, including concurrent reasoning models. On average, Thhnnkrrr attains accuracy of 46.6%, repre- senting a 2% improvement over the base model and performance comparable to proprietary models such as GPT-4o. Notably, Open VLThnkkr exhibits fewer hallucinations and more precise perception than its base model on HallusionBench, improving accuracy by 2.7%. Compared to concurrent reasoning meth- ods that utilize substantially larger datasets for single-iteration SFT and RL, our iterative approach achieves superior results while utilizing only 1/10 of the data scale as used in concurrent works with a single-iteration SFT-RL pipeline.
Table 2: Performance of 3B models on Math Vista.
Model
Accuracy (%)
R1-VL-2B
52.1
Intern
60.5
Qwen2.5-VL-3B
62.3
VLAA-Thinker-3B
61.0
OpenVLThinker-3B
63.4
Open VLThinker-31 We additionally train 3B model using single iteration of the SFT-RL pipeline, where the training process distills from our 7B model. Table 2, we compare the performance of our 3B model against current representative models at the same scale, including our base model, Qwen2.5-VL VLL-3,, and the reasoning model T.AA--Thinee-33: which is trained from the same initial checkpoint ours. Open VLThinker-3B achieves the best performance Math Vista and outperforms state-of-the-arn 3B reasoning models.
5.2 Analysis
Distillation at iteration 1.
At SFT-Iterl, we utilized the base model Qwen2.5-VLL-7B to generate image descriptions and obtained R1-like reasoning from QwQ-32B through rejection sampling. A
7

---

Table 3: Performance on the MathVista benchmark comparing different SFT data-filtering strategies. Removing the most repetitive keywords in data can mitigate repetitive reflections after SFT.
Table 4: Categorization of data sources by com- posite difficulty score using k-means with k=3. The geometry question sources all fall into the hard category.
Model Variant
Accuracy (%)
Easy
Medium
Hard
Qwen2.5-VL-7B
68.5
ChartQA
FigureQA
UniGeo
Vanilla
57.5
IconQA
CLEVR
GEOS
Filtered
58.7
VizWiz
A-OKVQ
Geometzy3K
TabMWP
SuperCLEVR
GeoQA
Truncated (SFT-Iterl)
63.4
DVQA
MapQA
dves
MathVista
80
5.0
SFT-Iter2
Qwer2.5 1... 11-TB-1 BBinstruct Accurscy (96
70-
4.5
GRPG-ter2
60
GRPO-Iter itert
iconso theg
1000 Bugeo
(Medium)
3.5
40
50
3.0
Mathiision-Min
40
2.5
SFe-
g0eaa
GRPO-Iter2
Mesiu
gons
20
Ontetrykk
1.5
GRPG-ter2 (Meiium
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
25
Composite Difficully (higher harder)
Figure 8: Performance at GRPO-Iterl using data from different difficulty sources, at the same scale of 3K.
Figure 7: Data source difficulty based on base model accuracy and GPT-4o rating.
common problem for distillation observed in text-only math reasoning the overly long reasoning length coupled with unnecessary repetitions of reflections [79, 42]. We observed similarly that these initial reasoning traces were often excessively verbose, partly due to information loss during conversion. Consequently, post-SFT reasoning became increasingly repetitive with unproductive self-reflections (see Appendix for an illustration). To address this, we evaluated two filtering strategies: (1) discarding samples with reasoning traces exceeding 500 words, and (2) truncating reflections by splitting traces of at specific keywords that were overly repetitive in data ( "Wait," "But wait," and "But the question") and discarding subsequent segments while preserving the final answer. The latter approach was ultimately adopted to prevent the model from internalizing reflection loops. while preserving the reasoning action reasonable frequency. Table 3 compares models trained on original versus processed data.
Data source difficulty. We conducted a quantitative analysis to categorize the data sources based on difficulty. Applying k-means clustering (with & = 3) to our composite difficulty score described in Section 4 allowed us to clearly identify three distinct difficulty pools, as shown in Table 4. We visualize the difficulty scores for each source in Figure 7. In Figure 8, show the performance of GRPO-Iterl when drawing 3K data from either (1) 10 data sources classified as either Easy or Medium, (2) 5 data sources classified Medium. We observe that RL training with easy-level data results in ineffective performance gain as compared to sourcing from medium-level data only. This finding aligns with concurrent algorithmic efforts such as DAPO [80] in the text-only domain for improving GRPO by dynamically filtering out overly-easy examples.
Curriculum RL to maximize utilization of challenging data. Figure 9 investigates the impact of incorporating challenging training data (e.g., geometry datasets) at iteration 1. On the left panel, we illustrate the absolute performance gains transitioning from SFT-Iterl to GRPO-Iterl (medium difficulty), and subsequently from GRPO-Iterl (medium) to GRPO-Iter] (hard). Training on these harder datasets yields substantial improvements more difficult benchmarks, such as Math Vision, while not significantly affecting performance on easier benchmarks like Math Vista. On the right panel, further compare two-stage, source-based curriculum RL approach against training solely
8

---

hard data. The results indicate that initiating RL with moderately challenging (medium difficulty) data and subsequently progressing harder datasets provides optimal performance improvements.
5.5
5.2
GRPO (Medium) SFT
GRPO (Curicu lum) SFT
Absoluts Improvement
GAPO (Hard) (Medium)
[HardOnlyl
3.4
3.0
2
2.0
1.3
MathVista
MathVision Mini
MathVista
MathVision Mini
Figure 9: Absolute performance gain achieved at iteration 1. The round 2 RL training on hard data
provides more significant performance gain on harder benchmarks such as Math Vision. Moreover, RL training with the hard data only yield less improvement than our two-stage curriculum RL.
Reasoning Lengths All
Reasoning Lengths
Incorrect
400
80
Ieer]
200
40
100
20
500
1000
1500
500 1000
1500
Figure 10: Distribution of reasoning length (number of words) across iterations of training. While our trained reasoning model across iterations all tend to reason longer than the base model, iterative training resulted gradually more concise length, possibly due to reduced repetitive reflections.
Iterative progression. Building upon the performance improvements shown in Figure we further analyze changes in reasoning length across iterations, as illustrated in Figure 10. Our results indicate that the reasoning model consistently utilizes more words inference time compared the base non-reasoning model, without becoming excessively repetitive. Notably, the largest increase in reasoning length occurs Iteration 1, with subsequent iterations gradually adopting more concise reasoning. This progression suggests an increasingly efficient utilization of reflective reasoning, engaging reflections primarily when beneficial. In Appendix D (Figure and 14), we show reasoning examples that our SFT-ed model was incorrect while our RL-ed model was correct.
Design choice on restarting iterations Restarting training from scratch at each iteration is standard practice in iterative self-improvement methods [59, 24). This design choice ensures training stability and prevents overfitting, especially when the data scale is relatively small, thus maintaining better generalization to unseen tasks. As noted in [59], re-training from the base model provides comparable task-specific performance and significantly better transfer to held-out tasks compared to continued training.
In our iterative re-training approach, the SFT data is refined and improved across iterations, while the base model parameters are reinitialized to prevent error accumulation. To further substantiate this design choice, we conducted an additional comparison between (a) re-training from scratch and (b) continuing training from the previous checkpoint.
The results indicate that continued training leads performance degradation on HellusionBenchh suggesting potential overfitting to the previous iteration's 's data. Hence, restarting from the base model offers more robust and generalizable learning trajectory across iterations.
Additional evaluation benchmarks. Our main paper evaluates Open ViThinker across six widely- used vasiun-language benchmarks covering mathematical reasoning, general reasoning, and perceptual

---

Table 5: Comparison between re-training from scratch and continued training.
Method
Math Vista
EMMA
HallusionBench
Open VLThinkrr (re-training)
71.7
25.8
70.2
Open OThhnkrr (continue training)
71.8
25.1
66.8
reliability. These benchmarks are consistent with those used in recent reports on both proprietary (e.g., GPT, Gemini) and open-source (e.g.. Qwen-VL, Intern-VL) models. This setup aligns with concurrent works [27, 6] that also employ these benchmarks with emphasis on reasoning ability.
To further clarify benchmark coverage. Table 6 provides subset-level EMMA results, showing performance across Math, Chemistry, Physics, and Code categories.
Table 6: Subset performance EMMA benchmark.
Model
EMMA-Math RMMA-Chemisee EMMA-Physics EMMA-Code
Qwen2.5-VL
24.6
21.9
29.5
28.0
VLAA-Thinker
28.1
22.3
28.8
27.3
Open VLThinker
28.8
22.6
32.7
26.8
In addition, we expanded the evaluation to include two recent benchmarks, MM-Star [8] and We- Math [52]. MM-Star assesses six major LVLM capabilities, including fine-grained perception, mathematics, science & technology, and logical reasoning.
Table 7: Results on newly included benchmarks MM-Star and WeMath.
Model
MM-Star
WeMath
Qwen2.5-VL
53.9
61.9
VLAA-Thinker 55,4
62.4
OLThnknhr 61.9
64.1
Together, these expanded evaluations across eight comprehensive benchmarks demonstrate the robustness and generalizability of our approach across multiple reasoning domains.
6 Conclusion
In this work, proposed new perspective LLM reasoning as actions at inference time, signified by keywords such as "wait". We thus interpret the roles of SFT as action highlighting that efficiently surfaces desired actions by distilling reasoning model's demonstrations. On the other hand, RL makes improvement on basis provided by SFT. Based on this intuition, we introduced Open VLThiker 7B, LVLM enhanced through iterative self self-improving process combining SFT and RL to enable complex CoT reasoning. Our results demonstrate that integrating R RI-style reasoning into LVLMs effectively boosts their multimodal reasoning performance across benchmarks. With only three SFT-RL cycles and 12K training examples, the model raises average accuracy on six diverse visual- reasoning benchmarks 46.6 %, with % absolute gain over its base model and par with proprietary systems such as GPT-4o.
Limitations. Our experiments span six established benchmarks, yet they do not exhaustively probe robustness in other tasks or real-world settings. In addition, we validated the method only on a 7B model as proof of concept. While the approach should scale larger backbones (e.g., 32B) and likely yield further gains, such exploration requires substantially greater computational resources.
Acknowledgments
We thank anonymous reviewers for their helpful comments. This work was partially supported by U.S. DARPA ECOLE Program No. 12390060 ONR grant 100011-23-1-27888 DARPA ANSR program FA8750- 23-2-0004, Amazon, and Apple. Chang and Peng were supported in part by grant from DARPA to the Simons Institute for the Theory of Computing. Bansal was supported in part by AFOSR MURI grant FA9550022-11-0800 Wang was supported by National Science Foundation (2106859, 2200274, 2312501), National Institutes of Health (U54HG0125177, U24DK097771, U540D03647)) NEC, and Optum AI.
10

---

References
[1] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: reinforen for learning from human feedback in 1lms. arXiv preprint arXiv:2402.14740 2024.
reinforce style optimization
[2] Al Anthropic. The claude model family: Opus, sonnet, haiku, claude-3 model card. 2024.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[4] Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler-Lussier, and Ningchuan Xiao. Mapqa: A dataset for question answering on choropleth maps. arXiv preprint arXiv:2211 1.08545, 2022.
[5] Hardy Chen, Haoqin Tu, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. VI-thinking: An rl-derived visual instruction tuning dataset for thinkable Ivims. https: //github. com/UCSC- VLAA/V /VL- Thinking. 2025.
[6] Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training rl-like reasoning large vision-language models. arXiv preprint arXivv2504. 1468, 2025.
[7] Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingho Liu, Eric P Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint arXiv:2105. 14517, 2021.
[8] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-lane lasoon-language models? arXiv preprint vrXiv:2403.203300 2024.
[9] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, arXiv preprint ar200:2412.0527,, 2024.
[10] Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, Lei Fang. Zhongyuan Wang, and Ji-Rong Wen. An empirical study on eliciting and improving r1-like reasoning models, 2025.
[11] Wenliang Dai, Jumnan Li, Dongxu Li, Anthony Meng Huat Tiong. Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose lision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[12] Hanze Dong, Wei Xiong, Bo Pang. Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang. Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405. 0776666663, 2024.
[13] Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Virgo: preliminary exploration on reproducing ol like milm. arXiv preprint a0X11:2501.01944, 2025.
[14] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 2024.
[15] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304. 15010, 2023.
[16] Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, Wenqi Shao, Chao Xu, Conghui He, Junjun He, Hao Shao, Pan Lu, Hongsheng Li, and Yu Qiao. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. In International Conference on Machine Learning (ICML), 2024.
11

---

[17] Jonas Geiping, Sean McLeish, Neel Jain. John Kirchenbauer, Siddharth Singh, Brian R. Bar- toldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach, 2025.
[18]
Google.
Gemini 2.5 pro, May 2025.
[19] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang. Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14375 14385, 2024.
[20] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in IIms via reinforcement learning. arXiv preprini arXi::2501.12948, 2025.
[21] Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vi. Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprins arXiv:2412.05237, 2024.
[22] Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang. Lijuan Wang, and Yu Cheng. Can milms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.0544: 2025.
[23] Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang. Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang. et al. Deepmath- 103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456 2025.
[24] Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint 20X.:22402.06457, 2024.
[25] Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung-Yeung Shum Xiangyu Zhang. Open-reasoner-zert An open source approach to scaling reinforcement learning on the base model, 2025.
[26] Yushi Hu, Weijia Shi. Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. arXiv preprint 20..::::06609400000 2024.
[27] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability multimodal large language models, 2025.
[28] Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey part Surpassing ol opervvw through simple distillation, big progress or bitter lesson?. 2024.
[29] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
[30] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camney, al. Openai ol system card. arXiv preprint arXiv:2412. 2024.
[31] Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017.
12

---

[32] Seungone Kim, Se Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. The CoT collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13685-12708, Singapore, December 2023. Association for Computational Linguistics.
[33] Sicong Leng. Jing Wang, Jiaxi Li, Hao Zhang. Zhiqiang Hu, Boqiang Zhang, Hang Zhang, Yuming Jiang, Xin Li, Deli Zhao, Fan Wang. Yu Rong, Aixin Sunt, and Shijian Luf. Mmrl: Advancing the frontiers of multimodal reasoning. https:// //gpt:/b//t.t...//.e com/LengSicong/ 2025.
[34] Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint MrXiv:2408003222 2024.
[35] Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. Symbolic chain-of-thoue distillation: Small models can also "think" step-by-step. In ACL, 2023.
[36] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[37] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[38] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems (NeurIPS), 36, 2023.
[39] Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can lb Imm surpass 405b IIm? rethinking compute-optimal test-time scaling, 2025.
[40] Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shich. Noisyrollout: Reinforcing visual reasoning with data augmentation. arXiv preprint 250i::2004.30555 2025.
[41] Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, and Min Lin. There may not be aha moment in rl-zero-like training a pilot study, 2025. Notion Blog.
[42] Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, and Min Lin. There may be aha moment in rl rl-zero-like training - a pilot study. sstps://oatlls notion, site/ oat- zero, 2025. Notion Blog.
[43] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025.
[44] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
[45] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing 01 preview with a 1.5b model by scaling rl, 2025. Notion Blog-
[46] Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang. Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rale- large-scale reinforcement learning. arXiv preprint arXiv:2503.07365, 2025.
[47] Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Imitate, explore, and self-improve: A reproduction report on slow-the sow-thinking reasoning systems, 2024.
13

---

[48] Debjyoti Mondal, Suraj Modi, Subhadarshi Panda, Rituraj Singh, and Godawari Sudhakar Rao. Kam-cco: Knowledge augmented multimodal chain-of -hain-of-thoughts reasoning. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 18798-18888 2024.
[49]
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer. Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025.
[50] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[51] Richard Yuanzhe Pang, Weizhe Yuan. He He. Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37:116117-1166377 2024.
[52] Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprini arXiv:2407.0128. 2024.
[53] Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and Pengfei Liu. O1 replication journey: strategic progress report part 1, 2024.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 8748-8763, PMLR, 2021.
[55] Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or rl is suboptimal, 2025.
[56] Zhihong Shao, Peiyi Wang, Qihao Zhu Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 2422:22402003000 2024.
[57] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vim-rl: A stable and generalizable r rl-style large vision-language model. arXiv preprint arXiv:2504.00 076/5, 2025.
(58) Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See- See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math-Ilava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.17294 2024.
[59] Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.000 2023.
[60] Nishad Singhi, Hritik Bansal, Arian Hosseini, Aditya Grover, Kai-Wei Chang, Marcus Rohrbach, and Anna Rohrbach. When to solve, when to verify: Compute-optimal problem solving and generative verification for Ilm reasoning. arXiv preprint wwXiv:2504.01000 2025.
[61] Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.00 074911 2025.
[62] OpenThoughts Team. Open Thoughts. https://pen-thougghee January 2025.
[63] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025.
[64] Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Absan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav-ol: Rethinking step-by-step visual reasoning in llms. arXiv preprint 20X.:250010.666. 2025.
14

---

[65] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[66] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini urXiv:2307.09266 2023.
[67] Rohan Wadhawan, Hritik Bansal, Kai-Wei Chang, and Nanyun Peng. Contextual: Evaluat- ing context-sensitive text-rich visual reasoning in large multimodal models. arXiv preprint arXiv:2401.13311 2024.
[68] Haozhe Wang, Chao Qu, Zuming Huang. Wei Chu, Fangzhen Lin, and Wenhu Chen. VI- rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint M.XKv:2504.08833 2025.
[69] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095-95169, 2024.
[70] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 1219, 2024.
[71] Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao,, Shenglong Ye, Xizhou Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503.10291 2025.
[72] Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Bhung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self improvenne..... arXiv preprint arXlv:2504.07934 2025.
[73] Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing I1m reasoning with rule-based reinforcement learning, 2025.
[74] Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-ol: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440, 2024.
[75] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1rxiv:2412.15115, 2024.
[76] Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, Bo Zhang, and Wei Chen. R R1-ovevision: Advancing generalized multimodal reasoning through cross- formalization, 2025.
[77] Huanjin Yao, Jiaxing Huang. Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering mllm with ol-like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412.18319 2024.
[78] Yixin Ye. Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning, 2025.
[79] Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning 1lms, 2025.
[80] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. /4476, 2025.
[81] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024.
15

---

[82] Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint 20Xiv:240.02811,, 2024.
[83] Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: Lim self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735-6644 647722, 2025.
[84] Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, et al. Llama-berry: Pairwise optimization for ol-like olympiad-level mathematical reasoning. arXiv preprini arXiv:2410.02884, 2024.
[85] Hanning Zhang, Jiarui Yao, Chenlu Ye, Wei Xiong, and Tong Zhang. Online-dpo-rl: Unlocking effective reasoning without the ppo overhead, 2025. Notion Blog.
[86] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint (txXiv:503.129:33 2025.
[87] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint a4Xiv:2408. 15240, 2024.
[88] Renrui Zhang, Dongzii Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169-186. Springer, 2024.
[89] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Mul- timodal chain-of-thought reasoning language models. arXiv preprint arXiv:2302.00923, 2023.
[90] Ge Zheng, Bin Yang, Jiajin Tang. Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct of prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5168-511 1191, 2023.
[91] Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1-zero's' r aha moment" in visual reasoning on 2b non-sft model. arXiv preprint a3X1v:2503.05132, 2025.
16

---

A Additional Experiments
A.1
Computational Cost of the Iterative SFT-RL Loop
We provide a detailed breakdown the computational cost for each stage of the iterative SFT-RL training process. The experiments were conducted 8xH100 (or equivalent) GPU node.
Table 8: Approximate GPU hours per stage for iterative SFT-RL training.
Stage SFT
GPU Hours (per GPU)
Total (8 GPUs)
0.06 (3 min 30 s)
0.48
GRPO-Medium
2.01 (2 h 35 s)
16.08
GRPO-Hard
4.57 34 min 26 s)
36.56
The SFT stage incurs minimal computational cost due to the small dataset size (3k examples). For RL training on medium-difficult data, the number of epochs is reduced maintain efficiency. Although the hard-stage RL incurs the highest cost, the overall compute remains comparable to contemporary RL-based post-training methods.
Importantly, the preceding SFT and medium RL stages accelerate convergence during the final RL stage. Despite the iterative nature introducing additional overhead, the total compute remains practical and resource-efficient for academic-scale training. We plan to include precise GPU-hour estimates and discuss scalability trade-offs future versions.
A.2 Stagle-Stage SFT-Only and RL.-Only Baselines
To isolate the effect of iteration beyond simply combining SFT and RL, we conducted experiments with single-stage SFT-only and RL-only baselines trained on the same 12K examples used in Open VLThinker.
RL-only: Qwen2.5-VL. trained exclusively with GRPO on the full 12K dataset.
SFT-only: Qwen2.5-VL trained solely via SFT on iteration-2 trajectories generated by Open VLThinkr. Instances with no correct reasoning within
samplings were filtered.
Both baselines were trained full convergence, and checkpoints were selected based on validation performance. Open VL.Thinker was trained using the same initialization but followed the iterative SFT-RII loop.
Table 9: Comparison of single-stage baselines and iterative Open VLThinker.
Method
Vista
EMMA HallusionBench
RL-Only
71.3
24.5
66.8
SFT-Only
71.1
22.3
65.4
(Iterative)
71.7
25.2
70.2
Training the RL-only baseline on the full 12K dataset required approximately 16 GPU-hours using an 800 node-ccmpaaable to the cumulative training time of Open VLThinker. While RL- only surpassed SFT-only, the iterative Open VLThinkr consistently achieved the best performance, demonstrating that the improvement stems from iterative refinement rather than merely combining SFT and RL.
A.3
Impact of Caption Quality Iterative Training
The quality of caption-based SFT data significantly influences reasoning performance in the iterative training loop. During iteration constructed the dataset using captions generated by QwQ-32B, filtered by final-answer correctness. Higher-quality captions increased the likelihood of correct reasoning traces, thus expanding the effective training pool.
To analyze this effect, we compared two variants: one using captions from the weaker Qwen2.5-VL 3B model and another using Owen2.5-VL-7B. both under identical rejection-sampheee conditions (k = 4). The performance evolution across iterations on the Math Vista benchmark is shown below.
Better caption quality improves visual grounding and reasoning trace precision in early stages, yielding higher-quality data for subsequent iterations. Consequently, richer initial captions amplify
17

---

Table 10: Effect of caption quality across training iterations on Math Vista.
Caption Source SFT-Iterl GRPO-Iterl
SFT-Iter2 GRPO-Iter2 SFT-Iter3 GRPO-Iter3
3B Caption 62.5
65.6
66.1
69.4
69.0
70.2
7B Caption 63.4
66.6
67.5
70.9
69.5
71.7
the benefits of the iterative framework, leading to more consistent improvements in reasoning performance.
B Additional Empirical Study
Does Complex Reasoning Matter for VQA?
We additionally investigated whether complex, multi-step reasoning provides sigmificant performance gains over standard (non-R1) reasoning in visual tasks. In this study, we use the ConTextual [67] validation set of 100 VQA examples, aiming to disentangle the roles of image grounding and textual reasoning. similar to our first distillation process, we separately employ a vision-language model for caption generation and pure-text model for reasoning. The image description generated by the captioning model is then fed into one of two text-based models: DeepSeek-RI-Diee -R--issil--- 14B (an R RI-style reasoner) or Qwen2.5- 4-IIntrucctt (a standard instruction-ture model). This setup allows us to isolate the impact of RI reasoning from the effects of the underlying vision encoder.
We further explore how different levels of caption quality influence final accuracy by comparing two caption generators, LLa VA-v1.6-344 and GPT-4o. Additionally, we vary the number of sampled reasoning paths (k: 2, 4) and compute pass @K k accuracies for each condition. As a baseline, include direct QA outputs from 3LaVA-v1.6-347 without any intermediate text description (i.e., the model sees images directly). Figure 11 summarizes these results. In experiments, we find that R RI-style reasoning provides consistent benefits:
(1) RI reasoning outperforms standard methods. When provided with identical captioned inputs, DeepSeek-RI --Dit-Rl---iiil--- achieves higher accuracy than 1etm2255-144-Istteeee Moreover, its performance can match (or even surpass) the direct QA accuracy of its own captioning model (LLaVA (LL4VA-v1.6-34B), despite potential information loss from translating the image into text.
(2) Sampling benefits complex reasoners. Increasing the number of sampled reasoning chains (k:
4) leads to larger performance gains for RI models than for standard Qwen models, indicating that the multi-step reasoning approach can more effectively converge on correct solutions when multiple hypotheses are explored.
(3) Image grounding quality matters. We observe that richer and more precise captions signif- icantly enhance final VQA accuracy. When captions are more detailed (e.g., from GPT-40), the improvements from complex reasoning especially pronounced.
Caption by LLaVA-v1.6 (34B)
Caption by GPT-4o
65
Taxt-based Reasoning: Owan2, 5-44-rrrttuc
Text: based Reasoning: DeepSeek Rittilll-11
Accuracy (%) Passak
55
Direct from LLaWA-1.6 (348)
50
40
35
30
25
Figure 11: Pass@ k accuracy of different reasoning models based on captions generated with different vision-language L.LMs.
18

---

Table 11:
VQA accuracy after a single round caption refinement. While pass@4 increases slightly, pass @1 and pass @2 remain largely unchanged.
Caption Type pass @ pass @ pass
Original
33
37
44
Refined
29
35
46
Caption Refinement via Feedback: Limited Effectiveness..
We also investigated whether one-round feedback loop could improve the quality of captions and thus final VQA performance. Concretely, DeepSeek-FR R1 -Distill-14B was prompted to list missing ambiguous details in the initial captions generated by LLaVA-v1.6-34BB The captioning model then re- re-generated a "refined" description incorporating this feedback. Table shows that the refined captions did not produce major accuracy improvements, suggesting that single feedback pass is insufficient for significantly enriching image descriptions.
Overall, although the idea of iterative caption refinement has intuitive appeal, preliminary tests suggest that more elaborate or repeated feedback cycles might be necessary to achieve substantial gains. Even so, the primary finding remains that RI-style reasoning robustly boosts performance relative to standard instruction-turs tuned reasoning, underscoring the importance of multi-step logic in VQA tasks.
C Experiment Details
We thank LLaMA-Facttryy and EasyR1? for open-sourcing the training framework that we used for SFT and GRPO. Table 13 and 14, we detail the hyperparameters that we used for SFT, GRPO and inference. We further lay out the prompts we used for generating image captions. Experiments were conducted on GPU clusters to the similar level of NVIDIA H100 80GB GPU. SFT/Distillation requires 30 minutes and RL requires 20 hours for each iteration. In addition, distillation data generation with verification requires about hours.
Table 12: Inference hyperparameters.
max_newaar new_tokens
2048
top_p top k 0.01
0.001
temperature
repetition_penalty
1.0
Table 13: Supervised fine-tume fine-tuning hyperparameters.
Data type
bf16
Learning rate
5e-7
Global batch size
32
Scheduler
Cosine
Warmup ratio
0.1
Num train epochs
Image max pixels
262144
Prompt for image description generation
Please provide detailed description of this image that could help another Al model un- derstand it completely. Be specific and comprehensive while maintaining natural language flow.
19

---

Table 14: GRPO hyperparameters.
Rollout batch size
512
Global batch size
128
Max grad norm
1.0
Data type
bf16
Learning rate
le-6
Weight decay
le-2
Warmup ratio
0.0
Rollout temperature
1.0
GBr
Figure 12: The image to the demonstrated reasoning example in section 1.
D Output Examples
Repetitive reasoning example after SFT without filtering
But the question is asking for the route and endpoint. So, the route would be the highway number, and the endpoint would be the destination.
Wait, but the signs arranged vertically. So, the top signs the routes, and the bottom ones are the destinations. So, if you're re at the junction and you't 're facing the signs, you can both. So, the route is the highway number, and the endpoint is the destination.
But the question is phrased "the route and endpoint reached by continuing in straight line." So, if you'te re at the junction and you't re facing the signs. you can see both the routes and the destinations. So, the route would be the highway number, and the endpoint would be the destination.
Wait, but the instruction is a bit ambiguous. It says "continuing in a straight line," which might mean following the route indicated by the sign. So, you're at the junction and you't 'r're facing the signs, you see both the routes and the destinations. So, the route is the highway number, and the endpoint is the destination.
But the question is asking for the route and endpoint. So, the route would be the highway number, and the endpoint would be the destination.
Wait, but the signs arranged vertically. So, the top signs the routes, and the bottom ones are the destinations. So, you're at the junction and you't 're facing the signs, you can both. So, the route is the highway number, and the endpoint is the destination.
But the question is phrased "the route and@endpoint reached by continuing in straight line." So, if you'te re at the junction and you't re facing the signs, you can see both the routes and the destinations. So, the route would be the highway number, and the endpoint would be the destination.

---

10
100 2X 3
-5
10
Figure 13: The image to the visual reasoning task that we show for SFT-Iter3 and GRPO-Iter3 output below.
Example of SFT-Iter3 incorrect output and GRPO-Iter3 Correct output the same question
Question:
The derivative of f(x) x=2 is that at x=5 Choices: (A) larger than (B) equal to (C) smaller than
SFT-Iter3
<think>
Okay, let me see. The question is asking about the derivative of the function f(x) = 12x at x=2 compared to x=5. First, I need to recall what the derivative represents. The derivative gives the slope of the tangent line at point on the function. So I need to look at the slopes at those two points. </think>
The function f(x) 12x 31 + is absolute value function, which means it has V-shape. The vertex of this V-shape occurs where the expression inside the absolute value is zero, i.c., ( 2x = =0) which gives x 3 ). The derivative of f(x) will be at the vertex because the function changes direction there. For (x 3 the function is decreasing, and for ( x 2 respectively. At x = (which is to the right of the vertex), the function is increasing, so the derivative positive. At 5 (which is further to the right), the function also increasing, so the derivative is positive but steeper because the function is increasing faster. Since the function is increasing faster at x 55 than at x = ), the derivative at (x 5 is larger. canswer>(C<<answer>
3 ), increasing. The slopes on either side of the vertex are negative and positive,
GRPO-Iter3
<think>
Okay, let's The question is asking about the derivative of the function f(x) 12x at x=2 compared to x=5. First, I need to recall what the derivative represents. The derivative point gives the slope of the tangent line that point. So need to look at the slopes x=2 and x=5. </think>
The function f(x) = 12x 1) is an absolute value function shifted up by 1. The absolute value function 12x 31) has a V-shape with a vertex at ( [x=] ). The derivative of 12x 31 ) is (2) for (x _ and (-2) ) for (x < Adding to the function doesn't 't affect the derivative's magnitude but shifts the y-values up. At (x = 2) ) which is greater than 3
2 the derivative is ( At (x=5) ), which is also greater than ? the derivative is also ( (2). Since the derivative is the same at both points, the slopes are equal. <answer>BB<<answerr
Ground truth: (B)
21

---

W
Z
52
38'
Y
X
Figure 14: The image to the task that we show for SFT-Iter3 and GRPO-Iter3 output below.
Example of SFT-Iter3 incorrect output and GRPO-Iter3 Correct output to the same question
Question: Find m23. Choices: (A) 28 (B) 38 (C) 52 (D) 62
SFT-Iter3
<think>
Okay, let me see. The problem is asking for the measure of angle 3 in the given quadrilateral. The hint suggests looking at the angles provided, which are degrees and 38 degrees. Hmm, the quadrilateral is a rectangle, me. Let me check the options again. The angles 52 and are given, but they 're not the ones I need to find. The question about angle 3, which is opposite angle Since opposite angles in a rectangle are equal, angle should be the same as angle 1. But angle is not given directly. Let me see, the hint is saying look at the angles provided. The 52 and 38 are supplementary angles, which means they add up to 180 degrees. But angle is not supplementary to 52 or 38. It's the opposite angle, it should be the same as angle 1. Since angle is not given, need to use the options. The only angle that could be the same as angle 1 is the that's not given directly. The options A, B, C, D. The only that's not supplementary to 52 or 38 is 62. So angle 3 must be 62 degrees.
all angles should be 90 degrees. But the hint is confusing
</think>
The problem states that the quadrilateral is a rectangle, and all angles in a rectangle are 90 degrees. The hint provided the angles 52" and 38", which are supplementary angles (520 + =00). However, angle 3 is the opposite angle angle 1, which means they are equal because opposite angles in rectangle are congruent. Since angle is not given directly, we need look at the options provided. The only angle that fits the properties of opposite angles in rectangle is the one that is not supplementary to the given angles. The options are A (28), (38), (52), and D (62). Since angle is not given, as angle 3 is the one that is not supplementary to 52 or 38". The only angle that fits this criterion is 62%, as it the only one not listed as supplementary to the given angles. <answer>(DD</answer
only angle that could the same
GRPO-Iter3
<think>
Okay, let's see. The question is asking for the measure of angle 3. The image provided is rectangle with some angles labeled. The hint suggests looking at the angles given, which 52 degrees and 38 degrees. Hmm, the rectangle has right angles, each angle should be 90 degrees. The angles 52 and 38 are supplementary because they add up to 90 degrees (90 52 38). But angle 3 not so it should be the same. Wait, the question is about angle 3, which is the one opposite the 38-degree angle. So it should be 38 degrees too, right? </think>
of those. It's the angle opposite the 38-degree angle,
The problem states that angle is opposite the 38- 38-degree angle. In rectangle, opposite angles are equal. Since angle opposite the 38-degree angle, angle must also be 38 degrees.
<answer>(B)</answerr
Ground truth: (B)
22

---

E
Impact Statement
Our work introduces Open OpenVLThinker, an open-source large vision-language model (LVLM) designed explicitly for complex reasoning tasks through a novel iterative cel--mmprveeeett process integrating supervised fine-tuning (SFT) and reinforcement learning (RL).
Positive societal impacts include significant improvements in multimodal reasoning, beneficial for applications in education, assistive technologies, accessibility, and content moderation, potentially enhancing decision-making processes involving visual and textual data.
However, improved generative capabilities inherently pose potential negative societal impacts, such as misuse for creating deceptive or manipulative content or misinformation. There are also fairness and privacy considerations, as biases in training data or misinterpretaation of visual information could unfairly affect specific groups.
To mitigate these risks, responsible deployment practices such as gated model releases, comprehensive documentation, usage guidelines, and continuous monitoring for misuse and biases are essential.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims presented in both the abstract and introduction clearly and accurately reflect the paper's contributions and scope. They precisely describe the novel iterative self-improvement cycle combining SFT with RL, clearly articulate the performance improvements across multiple benchmarks as shown in Table I, and highlight the conceptual novelty of reframing SFT as action- action-highlightie highlighting. These claims consistently align with the detailed methodological and empirical evidence provided throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We included discussion limitation in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of the iterative seff-improvement methodol- ogy, explicitly outlining the SFT process, RI. learning (RL) stages, and model architectures. We clearly specified datasets, benchmarks, evaluation protocols, and experimental conditions (c.g.. the iterative steps, curriculum RL, and data generation strategies) necessary to repro- duce its main experimental findings. This level of detail adequately supports reproducibility for validation of the primary claims and conclusions, independent of direct code data availability.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
25

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The paper commits providing open access the code. We provided code in the anonymous GitHub link in page footnote.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ httpii/gguidde//odSSuueaae www.suummissoooooiiooo.ooo. for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ tnips.cc/publl wwwpc//guu///eeeeee CooeeSubeeeeeeee wwtaaniissiiooooiiiiiiioi
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detailed our experiment setting in Section 5 and our experiment hyperpa- rameters and prompts in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Evaluations in this work are all deterministie. fully reproducible and do not incur error bars. The computational expense of fine-tuning large language models or generating 2M data multiple times is prohibitive, so did not include error bars with
26

---

regard to training. We acknowledge this limitation and ensure that the reported results are consistent and reliable based on our experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We detailed the compute we used Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt////euripppppee htrppulic//thicauudeeee.
Answer: [Yes]
The research conducted the paper adheres to the NeurlPS Code of Ethies, ensuring ethical standards are maintained in data usage, experimental procedures, and reporting of results. The paper preserves anonymity and considers the implications of the research on society.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
27

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We included an impact discussion in Section E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We will ensure the responsible release and usage of our dataset by implement- ing detailed usage guidelines.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use open-sourced models and datasets for our experiments, of which are properly cited and used under their original licenses.
28

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce any new assets, this question is not applicable. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicablle as well as details about compensation (if any)?
Answer: [NA]
Justification; The paper does not involve crowdsourcing or research with human subjects, so this question is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
29

---

Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects, so this question is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: This paper focus research directly related LLMs and described the usage of LLMs in methodology in Section 4.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: anttps:/neuurps... http://uripsccc/oofeeeeeeeeeeeee cc/Conferencess22255 /LLM) for what should or should not be described.
30


Reviews:
[{"review_id": "XuBaNrXaCb", "reviewer": "Reviewer_6vDT", "summary": "This paper introduces an open-source Large Vision-Language Model (LVLM), OpenVLThinker. The model adopts an iterative training strategy that alternates between Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), gradually introducing and enhancing complex Chain-of-Thought (CoT) reasoning capabilities on top of a non-reasoning vision-language base model. Results show that OpenVLThinker achieves certain performance improvements across six benchmark tasks that require mathematical and general reasoning abilities.", "questions": "Please see weaknesses for details.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths：\n1、\tThe study proposes an iterative SFT-RL loop. The resulting model outperforms open-source vision-language models of comparable scale on multiple reasoning benchmarks and achieves performance comparable to proprietary models such as GPT-4o. Furthermore, compared to models trained with SFT or RL alone, the proposed approach attains superior reasoning capabilities while using only about one-tenth of the training data.\n2、\tThe paper investigates the synergy between SFT and RL in acquiring reasoning capabilities. Specifically, the SFT stage helps narrow the RL search space, guiding the model toward reasonable reasoning traces. RL then further optimizes and enhances the model’s performance.\n3、\tThe study also explores the impact of data difficulty during the RL stage. The results show that a progressive two-stage training strategy—starting with medium-difficulty samples followed by hard ones—more effectively enhances the model’s complex reasoning ability than using simple or uniform difficulty data.\n\n## Weaknesses：\n1、\tHow were the representative keywords for inference-time actions selected? For example, why was \"maybe\" chosen as the representative keyword for \"seeking alternatives,\" and why was \"identify\" selected, given that its usage frequency significantly decreases after supervised fine-tuning?\n2、\tEach training iteration restarts from the base model rather than continuing from the intermediate model of the previous iteration. This design choice might be further explained.\n3、\tThe evaluation benchmarks are relatively limited. Expanding the diversity of benchmarks would better demonstrate the model’s general reasoning capabilities.", "was_revised": true, "final_justification": "The concerns have been addressed, thus I raise my score.", "rebuttal": "We appreciate your feedback and make clarifications with additional experiments as below to address the raised questions.\n\n---\n\n### **W1: How and why were the representative keywords selected.**\n\nWe selected keyword candidates by analyzing word cloud visualizations generated from reasoning outputs across various checkpoints, including QwQ-32B. Final representative keywords were chosen based on their frequency in the statistics and further categorized through manual inspection. For instance, we categorized \"maybe\" as indicative of the action \"seeking alternatives\" due to its frequent co-occurrence with reflective reasoning patterns, such as \"Maybe I should consider...\", \"Maybe I can consider...\" or “Or, maybe, …”. The co-occurence frequency of “Maybe I” is 28.1% for OpenVLThinker at iteration 1 and 7.9% for QwQ-32B at data generation. Similarly, “identify” is frequently associated with “First, I need to identify …”\n\nThis manual selection approach follows established practices in recent reasoning-model literature [1-3]. Specifically, keywords such as \"wait,\" \"check,\" and \"maybe\" are recognized reasoning indicators in studies of text-based reasoning models. Notably, [1] references the keyword \"maybe\" in Appendix B, while [2,3] similarly identify keyword sets through manual analysis (detailed in their appendices).\n\nEventually, the effectiveness of particular keywords in triggering specific reasoning behaviors depends on the reasoning model (in our case, QwQ-32B). Therefore, we prioritized the most frequent keywords highlighted by the word cloud analysis. We will clarify this in our revision.\n\n[1] LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters! \n\n[2] SEAL: Steerable Reasoning Calibration of Large Language Models for Free\n\n[3] Speculative Thinking: Enhancing Small-Model Reasoning with Large Model Guidance at Inference Time\n\n---\n\n### **W2: Design choice on restarting training iteration needs explanation**\n\nThank you for the suggestion. As clarified in footnote 2 (page 6, line 177) of our manuscript, **restarting training from scratch at each iteration is a standard practice in iterative self-improvement methods [1, 2]**. This design choice ensures training stability and prevents overfitting, thus maintaining better generalization to unseen tasks. Notably, ReST^EM [1] explicitly highlights that re-training from the base model provides comparable task-specific performance and significantly better transfer to held-out tasks compared to continued training (Section 3, page 5, Differences with ReST). We emphasize that, in our iterative re-training approach from the base model, **the SFT data is refined and improved across iterations**.\n\nTo further substantiate this design choice, we conducted an additional experiment comparing our approach of re-training from scratch versus continued training from the previous checkpoint. We evaluated these approaches on three representative benchmarks: MathVista (math reasoning), EMMA (general reasoning), and HallusionBench (perception reliability).\n\n|   | MathVista | EMMA | HallusionBench |\n|---|------|----|------|\n| OpenVLThinker (re-training)   | 71.7 | 25.8 | 70.2  |\n| OpenVLThinker (continue training) | 71.8 |  25.1  | 66.8  |\n\nThe results clearly indicate that continued training from the previous iteration leads to noticeable performance degradation on HallusionBench, despite similar performance on MathVista and EMMA. We will emphasize this in our revision by elevating the footnote into the main text and including these additional results.\n\n[1] Beyond human data: Scaling self-training for problem-solving with language models\n\n[2] V-star: Training verifiers for self-taught reasoners\n\n[3] Reinforced self-training (rest) for language modeling\n\n---\n\n### **W3: Evaluation benchmarks are relatively limited**\n\nThank you for your suggestion. In the paper, **we selected six widely-used vision-language benchmarks covering diverse reasoning aspects, including mathematical reasoning, general reasoning, and perceptual reliability**. These benchmarks are consistently employed in technical reports of proprietary models (e.g., GPT, Gemini) and open-source models (e.g., Qwen-VL, Intern-VL). Our evaluation setup further aligns with concurrent works [1,2] on vision-language reasoning, which similarly employ 5–7 benchmarks with a particular emphasis on mathematical reasoning. \n\nNotably, our evaluation benchmarks are comprehensive and all span multiple evaluation subfields. We reported the overall performance to maintain conciseness. For example, EMMA evaluates Math, Chemistry, Physics, and Code through both multiple-choice and free-form questions. To provide clarity on evaluation coverage, we include detailed subset performances on EMMA below: \n\n|   | EMMA-Math | EMMA-Chemistry | EMMA-Physics | EMMA-Code |\n|-----|------|------|-------|------|\n| Qwen2.5-VL | 24.6   | 21.9   | 29.5 | 28.0  |\n| VLAA-Thinker | 28.1  | 22.3   | 28.8 | 27.3  |\n| OpenVLThinker | 28.8  | 22.6   | 32.7 | 26.8 |\n\nAdditionally, in response to your concern, we have expanded our evaluation by including two recent benchmarks: MM-Star and WeMath (both released in 2024). MM-Star is particularly comprehensive, assessing six core LVLM capabilities: fine-grained perception, coarse perception, mathematics, science & technology, logical reasoning, and instance reasoning.\n\n|  | **MMStar** | **WeMath** |\n|---|---|---|\n| Qwen2.5-VL| 53.9 | 61.9 |\n| VLAA-Thinker | 55.4 | 62.4 |\n| OpenVLThinker| **61.9** | **64.1** | \n\nWe believe our expanded evaluation across these eight comprehensive benchmarks sufficiently demonstrates the effectiveness and generalizability of our approach.\n\n[1] Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models\n\n[2] SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models"}, {"review_id": "O45g0ueL1D", "reviewer": "Reviewer_5ZWN", "summary": "This paper introduces OpenVLThinker-7B, one of the first open-source large vision-language models (LVLMs) to demonstrate sophisticated chain-of-thought reasoning capabilities similar to proprietary models like GPT-o1. The authors propose an iterative training methodology that alternates between supervised fine-tuning (SFT) and reinforcement learning (RL) stages. The key insight is that SFT acts as an \"action highlighting\" mechanism that surfaces latent reasoning behaviors (triggered by keywords like \"wait\", \"check\"), which RL can then effectively optimize. Starting from Qwen2.5-VL-7B, the model is trained through multiple iterations using only 12K examples per iteration, achieving competitive performance across six challenging benchmarks.", "questions": "What is the total computational cost of the iterative training process compared to standard approaches? Could you provide FLOPs or GPU-hours comparisons?\n\nCan you provide results for single-stage SFT-only and RL-only baselines trained on the same 12K examples? This would clarify the benefit of iteration versus simply using both techniques.\n\nHave you tested the approach on other base models (e.g., LLaVA, InternVL)? Is the success tied to specific properties of Qwen models?\n\nCan you elaborate on the connection to EM algorithms mentioned in line 173? A more formal analysis of why iteration helps could strengthen the paper.\n\nIn Table 3, \"Vanilla\" SFT shows a large performance drop (68.5→57.5). Could you analyze what specific aspects of the distilled reasoning traces cause this degradation?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe iterative SFT-RL approach is innovative and well-motivated. The authors provide compelling analysis (Figure 6) showing how SFT surfaces reasoning keywords that are rarely present in the base model, while RL refines performance without dramatically changing the action distribution.\n\n Despite using only 12K training examples, OpenVLThinker-7B achieves competitive or superior performance compared to concurrent models using 10x more data (150-270K examples). The 2% average improvement over the base model and 2.7% reduction in hallucinations on HallusionBench are notable.\n\nThe paper includes insightful analysis of reasoning behaviors, including keyword frequency analysis and the role of SFT in guiding RL exploration. The curriculum RL approach based on data difficulty is also well-designed.\n\nThe paper is generally well-written with clear figures illustrating the methodology and results. The examples in Figure 4 and Appendix C effectively demonstrate the model's reasoning capabilities.\n\nWeaknesses\n\nThe paper doesn't discuss the computational requirements of the iterative training process. With RL requiring 20 hours per iteration (mentioned in Appendix B), the total training cost could be substantial. How does it compare to baseline methods? I think the improvements seem marginal considering the great cost.\n\nThe paper primarily compares against the base model and concurrent methods. More comparisons with standard vision-language training approaches (e.g., single-stage SFT or RL) would strengthen the evaluation.\n\nThe initial SFT data generation process using image captions and QwQ-32B is somewhat ad-hoc. The impact of caption quality on final performance could be better analyzed beyond the brief discussion in Appendix A.\n\nAll experiments use models from the Qwen family. It's unclear whether the approach generalizes to other model architectures or if the benefits are specific to Qwen's pretraining. See the paper \"ognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs\".", "was_revised": true, "final_justification": "Thank you for answering my question; I decided to keep my original score.", "rebuttal": "Thank you very much for your support, careful reading and the constructive feedback that helped us improve our work. Please see our detailed response with additional experiments below.\n\n### **W1. Discuss the overall computational cost of the iterative SFT → RL loop (e.g., FLOPs or GPU‑hours)**\n\nThanks for pointing this out. In appendix B, we roughly estimated the training time on 2-4 GPUs. Here, we rigorously detail the GPU hours for the SFT and RL processes below with a 8xH100 (or equivalent) node:\n\n| Stage | GPU Hours    |\n|---------|-------------|\n| SFT   | 0.06 (3 min 30 s) * 8 |\n| GRPO-Medium  | 2.01 (2 hrs 35 s) * 8 |\n| GRPO-Hard  | 4.57 (4 hrs 34 min 26 s) * 8 |\n\nThe computational cost of the SFT stage is minimal, primarily due to the small dataset size of 3k examples. For RL with medium-difficulty data, we limit the number of training epochs to further manage resource usage effectively. Although the RL stage using hard data (6k examples) incurs the highest computational cost, our method's overall compute demand remains comparable to contemporary RL-based approaches. Crucially, the preceding SFT and medium RL stages expedite convergence during the final RL (hard) stage, and the relatively smaller dataset size reduces the overall computational burden. Nevertheless, we acknowledge that iterative approaches inherently introduce additional computational overhead, and we will explicitly note this in our revised manuscript and update the more precise GPU hours estimate.\n\n---\n\n### **W2. Add single‑stage SFT‑only and RL‑only (trained on the same 12 K examples) to show that iteration, not merely using both techniques, brings the lift**\n\nThank you for suggesting these critical baselines. We conducted the requested experiments under the following settings:\n- RL-only baseline: Qwen2.5-VL trained solely with GRPO on the entire 12K dataset used in OpenVLThinker.\n- SFT-only baseline: Qwen2.5-VL trained solely with SFT on iteration 2 trajectories generated by OpenVLThinker for the same 12K dataset. Similar to the original setup, we filter out examples where the model fails to reach a correct answer within k=4 samplings.\n\nBoth baselines were trained until full convergence, with checkpoints selected based on validation performance. In comparison, OpenVLThinker involves both SFT and RL stages starting from the same Qwen2.5-VL initialization.\n\n|     | MathVista | EMMA | HallusionBench |\n|-----|------|------|-------|\n| RL-Only | 71.3 | 24.5 | 66.8 |\n| SFT-Only | 71.1 | 22.3 | 65.4 |\n| OpenVLThinker | 71.7 | 25.2 | 70.2 |\n\nTraining RL-only on the complete 12K dataset required approximately 16 hours to finish using an 8xH100 (or equivalent) node, comparable to the cumulative training time across the three iterations of OpenVLThinker. Results indicate that the RL-only approach outperforms the SFT-only baseline, especially on held-out benchmarks like HallusionBench. Nevertheless, OpenVLThinker achieves superior performance compared to both baselines.\n\n---\n\n### **W3: How does the quality of the caption‑based SFT data affect final accuracy, and can you provide a deeper analysis on the caption quality?**\n\nOur training data from QwQ-32B was selected based on final-answer correctness. Thus, the quality of captions directly influenced the quantity and richness of the initial dataset available for the first-stage SFT. Higher-quality captions increased the likelihood that the text-based reasoning model would correctly derive final answers instead of random guessing due to lack of information, thus expanding the pool of useful training samples for iteration 1.\n\nIn the original setup, we employed Qwen2.5-VL-7B to generate captions and performed rejection sampling from QwQ-32B with k=4 attempts, discarding instances if none yielded correct reasoning. To provide deeper insights, we conducted additional experiments using Qwen2.5-VL-3B, a weaker captioning model, under identical rejection sampling conditions (k=4). The comparative performance on MathVista is detailed below:\n\n| Iter-1 Caption Type | Qwen2.5-VL | SFT-Iter1 | GRPO-Iter1 | SFT-Iter2 | GRPO-Iter2 | SFT-Iter3 | GRPO-Iter3 |\n|--------------|------------|-----------|------------|-----------|------------|-----------|------------|\n| 3B Caption   | 68.5       | 62.5      | 65.6       | 66.1      | 69.4       | 69        | 70.2       |\n| 7B Caption   | 68.5       | 63.4      | 66.6       | 67.5      | 70.9      | 69.5      | 71.7       |\n\nAs demonstrated, better caption quality provides more precise visual grounding at the very first data generation step, enabling the reasoning model to produce more accurate and reliable traces. This in turn improves the initial training dataset and enhances the model's reasoning capabilities across iterations. Eventually, the initial data quality orthogonally complements our iterative training framework, positively influencing final model performance.\n\n---\n\n### **W4: Does the method transfer to architectures outside the Qwen family?**\n\nThank you for raising this important point. To ensure a fair comparison with existing baselines (all trained from Qwen2.5-VL-7B-Instruct), we adhered to the same experimental setting in our paper.  Currently, open-source RL frameworks for vision-language models (EasyR1, OpenRLHF, VeRL) only support models within the Qwen-VL family. Extending these frameworks to other architectures for GRPO is a non-trivial effort beyond the scope of our rebuttal timeframe. To demonstrate that our iteratively evolved dataset maintains its effectiveness beyond the Qwen-based models, we have conducted additional experiments validating the iterative SFT data (distillation) with Gemma-3-4B-IT at iterations 1, 2, and 3.\n\n|     | MathVista |\n|----|-----------|\n| Gemma3-4B   | 50.1 |\n| Gemma3-4B (SFT-Iter1) | 46.1 |\n| Gemma3-4B (SFT-Iter2) | 48.5 |\n| Gemma3-4B (SFT-Iter3) | 50.6 |\n\nWe also acknowledge the findings presented in [1] and agree that base-model capabilities significantly influence RL (GRPO) performance due to variations in pre-training quality. Consistent with this idea, we highlight SFT’s crucial role as an intermediate training step (priming) that explicitly surfaces beneficial reasoning behaviors, thereby enhancing subsequent RL effectiveness. We will incorporate this discussion with [1] into our manuscript revision.\n\n[1] Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs\n\n---\n\n### **Q1: The paper mentions an EM‑style perspective; could you formalize this connection?**\n\nAs similar to ReST^EM (an iterative SFT algorithm [1]), our method also decouples data collection (E-step) and policy optimization (M-step) in one iteration. To put this formally,\n\n- Generate (E-step): We generate a SFT dataset $D_i$ at iteration i by sampling reasoning traces from the current policy model $p$:  $𝒚^{i} \\sim p_{\\theta} (𝒚 |𝒙_{i})$\n\n- Improve (M-step): We leverage the new dataset $D_i$ obtained from the Generate step to improve the policy model p via minimzing the SFT (log-likelihood) loss: $𝐽(\\theta) = 𝔼(𝒙,𝒚)∼D_𝑖 [𝑟(𝒙, 𝒚) \\log 𝑝_{\\theta} ( 𝒚 |𝒙)]$. Additionally, the model is further improved through an unified RL step via the GRPO objective on a fixed dataset $D_{RL}$.\n\n[1] Beyond human data: Scaling self-training for problem-solving with language models\n\n---\n\n### **Q2: Table 3 shows a sharp drop for vanilla SFT. What in the distilled reasoning traces causes this degradation, and how might it be mitigated?**\n\nThe challenge of small-scale LLMs (<7B) struggling to learn effectively from lengthy chains-of-thought (CoTs) generated by larger models is common to both LVLMs and text-only LLMs. [1] specifically highlights that smaller models benefit more when learning from teachers of a similar scale, rather than substantially larger models. In our observations, repetitive or overly verbose reflections become reinforced during the SFT training, causing looping or redundant behaviors, as demonstrated in Appendix C.\n\nFor the vision-language tasks examined in our paper, the degradation is further caused  by the information loss when relying solely on image captions rather than direct visual inputs. Consequently, the teacher model’s reasoning may rely on incomplete or ambiguous information. Even with post-processing, this issue notably impacts performance. For example, QwQ-32B attempts to reason from captions that lack crucial visual details:\n\n> Question: What is the main color of the can's top surface?\n\n> QwQ-32B: [think]\\nOkay, let's tackle this question. The user is asking for the main color of the can's top surface based on the image provided. First, I need to go through the image carefully to find any shows of colors.\\n\\nLooking at the image details: *The can's top is shown with a pull-tab mechanism. The surface is on a dark wooden table. The pull-tab is in the center with a metallic ring. The instructions on the tab are shown, but no color is specified for the text.* …... Metals can be silver, but maybe the overall top is a different color.\\n[/think]\\n\\n[answer]Silver[/answer]\n\nThe mitigation to such performance degradation is the exact motivation of our iterative approach. By iteratively refining the teacher model through cycles of SFT and RL, we produce progressively clearer and more precise reasoning demonstrations, which in turn yield higher-quality distillation data in subsequent iterations."}, {"review_id": "cUja3XWLDb", "reviewer": "Reviewer_iVRi", "summary": "This paper introduces OpenVLThinker-7B, an open-source Large Vision-Language Model (LVLM) based on Qwen2.5-VL-7B, designed to exhibit complex chain-of-thought (CoT) reasoning. The core methodology is an \"iterative SFT-RL cycle\", where Supervised Fine-Tuning (SFT) is alternated with Reinforcement Learning (RL) using Group Relative Policy Optimization (GRPO). The authors propose that SFT helps \"surface latent reasoning actions\", and RL subsequently refines these. In each iteration, the base model is retrained using data generated by the model from the preceding iteration. The paper reports performance gains on six benchmarks, such as +3.2% on MathVista and +2.7% on HallusionBench, claiming these are achieved with high data efficiency (12K samples per iteration) compared to concurrent methods.", "questions": "- The paper asserts the Qwen2.5-VL-7B base model \"seldom exhibits reasoning behaviors initially\". This contrasts with findings from Chen et al. (VLAA-Thinker), who note reflective patterns in the same base model family prior to RL. Could you please discuss this discrepancy? \n\n- VLAA-Thinker also suggests SFT can induce \"pseudo reasoning paths,\" contrasting with your view of SFT effectively surfacing actions and narrowing the RL search space. How do you reconcile these differing perspectives on SFT's role? \n\n- Given that SFT on text-based CoTs (from DeepSeek-R1 and captions) is stated to cause a performance drop due to \"a lack of precise visual grounding\", could you provide a detailed explanation of the specific mechanism by which GRPO, an RL algorithm optimizing policy based on textual output preferences, directly rectifies this multimodal visual grounding issue? \n\n- \"Iterative Self-Improvement\" involves retraining from the base Qwen2.5-VL-7B model in each iteration using data from the previous iteration's model. Why is this termed \"iterative self-improvement\" of a single evolving model rather than an iterative data refinement strategy that produces a sequence of distinct, newly trained model instances? \n\n- What are the advantages of this full retraining approach versus continuously fine-tuning the Iter(i-1) checkpoint to create Iter(i), especially if aiming for \"self-improvement\"?", "limitations": "yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "$$\\textbf{Strengths}$$\n\n- The paper claims data efficiency using only 12K training samples per iteration to achieve its results, which is considerably less than several concurrent 7B models using single-iteration SFT+RL (e.g., 150K-270K). \n\n- Provides an open-source LVLM (OpenVLThinker-7B) capable of enhanced reasoning is valuable for the research community. \n\n- The process of iteratively generating better training data using the previous iteration's model to train a new model instance from a base checkpoint is systematic. Figure 3 shows progressive improvement on MathVista, supporting this data refinement strategy. \n\n- The attempt to analyze how SFT and RL affect the occurrence of specific keywords (e.g., \"wait,\" \"check\") is a good step towards understanding model behavior changes, forming the basis of the \"action highlighting\" hypothesis. \n\n$$\\textbf{Weaknesses}$$\n\n- The paper states the model is retrained \"from scratch at each iteration\" or from the \"base model Qwen2.5-VL-7B\" using data from the prior iteration. This is more accurately an iterative data refinement and curriculum generation strategy for training new model instances rather than the \"self-improvement\" of a single, continuously evolving model. \n\n- The SFT \"action highlighting\" hypothesis, while interesting, is not clearly demonstrated to be mechanistically distinct from SFT providing a good initialization or behavioral prior for RL. \n\n- Some effort could help address conflicting findings from relevant concurrent research. For instance, the paper's premise that the \"base model seldom exhibits reasoning behaviors initially\" is challenged by Chen et al. (VLAA-Thinker), who report observing reflective reasoning patterns in the same Qwen2.5-VL base model before RL. This discrepancy undermines a key motivation for the proposed SFT role. Similarly, the paper's positive framing of SFT (surfacing latent actions, narrowing RL search space) contrasts with VLAA-Thinker's caution that SFT can induce \"pseudo reasoning paths\" detrimental to subsequent RL. Given VLAA-Thinker is a cited concurrent model using the same base architecture, this lack of discussion is a major omission. \n\n- The paper argues that initial SFT on text-based CoTs (with image captions) can degrade performance due to \"imprecise visual grounding\". It then claims the iterative SFT-RL cycle, where RL \"refines the model's reasoning skills\", rectifies this. Can you provide some clarity on RL's Mechanism for Improving Visual Grounding.\n\n- There is insufficient explanation of how the RL component (GRPO), which primarily optimizes policies based on ranked preferences of textual outputs, directly addresses or improves multimodal visual grounding. This link is critical but underspecified.", "was_revised": true, "final_justification": "The authors successfully addressed all major criticisms with a rebuttal, including crucial new ablation studies that validate their iterative SFT-RL methodology's superiority over single-stage approaches. They clarified the method's mechanistic underpinnings and reconciled their findings with concurrent work, resolving initial concerns about the paper's scientific rigor. The paper's claims are now substantially better supported and its contributions more clearly articulated.", "rebuttal": "Thank you for the detailed feedback. Please find our responses below. We hope that our clarifications and additional experiments resolve the concerns and any potential confusions.\n\n---\n\n### **W1: Why term the process “iterative self‑improvement” when each cycle trains a fresh model from the base checkpoint? Please justify full restarts versus continuing fine‑tuning and explain any resulting gains.**\n\nThank you for the suggestion. As clarified in footnote 2 (page 6, line 177) of our manuscript, **restarting training from scratch at each iteration is a standard practice in iterative self-improvement methods [1, 2]**. This design choice ensures training stability and prevents overfitting, thus maintaining better generalization to unseen tasks. Notably, ReST^EM [1] explicitly highlights that re-training from the base model provides comparable task-specific performance and significantly better transfer to held-out tasks compared to continued training (Section 3, page 5, Differences with ReST). We emphasize that, in our iterative re-training approach from the base model, **the data is iteratively refined and improved across iterations**.\n\nTo further substantiate this design choice, we conducted an additional experiment comparing our approach of re-training from scratch versus continued training from the previous checkpoint. We evaluated these approaches on three representative benchmarks: MathVista (math reasoning), EMMA (general reasoning), and HallusionBench (perception).\n\n|   | MathVista | EMMA | HallusionBench |\n|---|------|----|------|\n| OpenVLThinker (re-training)   | 71.7 | 25.8 | 70.2  |\n| OpenVLThinker (continue training) | 71.8 |  25.1  | 66.8  |\n\nThe results indicate that continued training from the previous iteration leads to noticeable performance degradation on HallusionBench, despite similar performance on MathVista and EMMA. We will emphasize this in our revision by elevating the footnote into the main text and including these additional results.\n\n[1] Beyond human data: Scaling self-training for problem-solving with language models\n\n[2] V-star: Training verifiers for self-taught reasoners\n\n[3] Reinforced self-training (rest) for language modeling\n\n---\n\n### **W2: Can you supply evidence that SFT’s “action‑highlighting” effect is mechanistically distinct from a simple good initialization for RL**\n\nWe appreciate your comment and realize there might be a misunderstanding. We clarify that our arguments explicitly differentiate between two mechanistically distinct roles that SFT plays:\n\n1. **Highlighting reasoning actions**: SFT explicitly promotes specific reasoning behaviors, such as reflections and verifications, by emphasizing reasoning-trigger keywords. Crucially, without prior highlighting from SFT or inherent reasoning tendencies from the base model, RL alone does not consistently induce these behaviors spontaneously for VL models. To substantiate this, we analyzed the keyword frequency of the RL-trained MM-Eureka (7B) model across 1,000 MathVista examples: (a) \"wait\": 0 occurrences, (b) \"check\": 52 occurrences, (c) \"maybe\": 0 occurrences. On the contrary, SFT effectively and efficiently influence reasoning behavior patterns, distinct from mere performance benefits.\n\n2. **Initialization for RL**: Independently, the quality of SFT-generated data directly affects the model’s initial performance, facilitating more effective subsequent RL training. This quality-driven initialization is precisely the motivation behind our iterative training approach, which progressively refines the quality of SFT data while retaining the desired reasoning behaviors.\n\nAs demonstrated in Figure 6 of our manuscript, iteration 1’s SFT does not provide superior initialization in terms of accuracy; in fact, its performance is lower than the base model. Thus, the primary contribution of SFT at this iteration is not performance initialization but rather the clear distillation of reasoning behaviors, evidenced by significantly increased frequencies of reflective keywords. These behaviors persist through subsequent RL training, remaining stable while RL enhances overall model performance. Consequently, the RL-enhanced model serves as an improved reasoning demonstration source for the following iteration, retaining desired behaviors while benefiting from RL-driven performance improvements.\n\n---\n\n### **W3: Reconcile with VLAA‑Thinker’s reports that (a) the same base model already shows reflective reasoning and (b) SFT can create harmful pseudo‑reasoning paths?**\n\nThanks for the question. We believe this perceived contradiction also roots from the misunderstanding of our iterative approach as well as our arguments toward the role of SFT. We clarify that our findings **align rather than conflict with concurrent papers**.\n\n(a) We must note that several concurrent works have made similar observations as ours. VL-Rethinker [1] explicitly noted that “complex, deliberate thinking patterns, such as explicit self-correction, **did not consistently emerge** as a direct result of standard RL on VLMs.” This is exactly what motivated their approach in forcing the model to rethink during RL by explicitly appending the keyword “wait”. Similarly, our findings suggest these reflective behaviors are latent and require targeted approaches (like iterative SFT-RL cycles) to reliably surface and refine.\n\n(b) We fully acknowledge VLAA-Thinker's observation regarding harmful pseudo-reasoning paths caused by SFT. Our manuscript explicitly states (lines 47–48) that CoT traces from initial SFT provide demonstrations of reasoning actions but **do not immediately improve LVLM’s accuracy** Additionally, we highlight (line 54) that the initial SFT step \"**leads to a performance drop**\", which is consistent with VLAA-Thinker. In Appendix C (page 18), we specifically illustrate examples of ineffective, repetitive reasoning behaviors post-SFT. Precisely this harmful initial impact motivated our iterative approach: each RL stage refines these reasoning behaviors, generating **progressively improved data** for subsequent SFT steps.\n\nIn short, our iterative SFT-RL methodology explicitly addresses and leverages the phenomena described by VLAA-Thinker, rather than contradicting them.\n\n[1] VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning\n\n---\n\n### **W4: By what specific mechanism does GRPO repair the imprecise visual grounding introduced by caption‑based CoTs?**\n\nIn the paper, we mentioned the imprecise visual grounding in the context of the very first SFT stage, where reasoning traces are derived from text-based LLMs (QwQ-32B) conditioned on captions generated by a vision-language model (Qwen2.5-VL-7B). Since captions inherently lose some visual information and precision, the reasoning traces from QwQ-32B may contain inferences or educated guesses based on incomplete information. For example, consider the reasoning of QwQ-32B below, based solely on captioned inputs:\n\n> Question: What is the main color of the can's top surface?\n\n> QwQ-32B: [think]\\nOkay, let's tackle this question. The user is asking for the main color of the can's top surface based on the image provided. First, I need to go through the image carefully to find any shows of colors.\\n\\nLooking at the image details: *The can's top is shown with a pull-tab mechanism. The surface is on a dark wooden table. The pull-tab is in the center with a metallic ring. The instructions on the tab are shown, but no color is specified for the text.* …... Metals can be silver, but maybe the overall top is a different color.\\n[/think]\\n\\n[answer]Silver[/answer]\n\nGRPO improves upon this by progressively shifting the model away from caption-based inference (that it learns during SFT iteration 1) toward reasoning on direct visual input, as it allows the model explore better reasoning paths with verifiable reward. Consequently, OpenVLThinker after iteration 1 already reasons directly from visual inputs, substantially reducing the potential ambiguity in the reasoning traces it generates as compared to the QwQ generated data. Further iterations continue refining the ”teacher model” for next iteration, systematically improving reasoning demonstrations while retaining beneficial reasoning behaviors demonstrated by QwQ-32B.\n\nWe will emphasize this point in our next revision to avoid confusion."}]

Author rebuttal:

Meta-review: The paper received three expert reviews. The authors provided a rebuttal that attempted to address the concerns raised in the reviews. The reviewers read the rebuttal and engaged with the authors. After the rebuttal and discussion, the reviewers gave final recommendations of borderline accept, borderline accept, and borderline accept.  The reviewers unanimously like the paper and recommended accept. The area chair agreed with the recommendation and decided to accept the paper. 

Congratulations! Please see the reviews for feedback on the paper to revise the final version of your paper and include any items promised in your rebuttal.
Decision: Accept (poster)