Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning
Authors: Junhao Shen
Haiteng Zhao
Yuzhe Gu
Songyang Gao
Kuikun Liu
Haian Huang
Jianfei Gao
Dahua Lin
Wenwei Zhang
Kai Chen
Venue: NeurIPS 2025 poster ()
Keywords: Large vision-language model
Slow-thinking reasoning
Primary area: deep_learning
TL;DR: This paper introduces a simple and scalable semi-off-policy reinforcement learning method, i.e., SOPHIA, to enhance LVLMs’ ability to perform visual slow-thinking reasoning.

Abstract: Enhancing large vision-language models (LVLMs) with visual slow-thinking reasoning is crucial for solving complex multimodal tasks. However, since LVLMs are mainly trained with vision-language alignment, it is difficult to adopt on-policy reinforcement learning (RL) to develop the slow thinking ability because the rollout space is restricted by its initial abilities. Off-policy RL offers a way to go beyond the current policy, but directly distilling trajectories from external models may cause visual hallucinations due to mismatched visual perception abilities across models. To address these issues, this paper proposes **SOPHIA**, a simple and scalable **S**emi-**O**ff-**P**olicy RL for vision-language slow-t**HI**nking re**A**soning. SOPHIA builds a semi-off-policy behavior model by combining on-policy visual understanding from a trainable LVLM with off-policy slow-thinking reasoning from a language model, assigns outcome-based rewards to reasoning, and propagates visual rewards backward. Then LVLM learns slow-thinking reasoning ability from the obtained reasoning trajectories using propagated rewards via off-policy RL algorithms. Extensive experiments with InternVL2.5 and InternVL3.0 with 8B and 38B sizes show the effectiveness of SOPHIA. Notably, SOPHIA improves InternVL3.0-38B by 8.50\% in average, reaching state-of-the-art performance among open-source LVLMs on multiple multimodal reasoning benchmarks, and even outperforms some closed-source models (e.g., GPT-4.1) on the challenging MathVision and OlympiadBench, achieving 49.08\% and 49.95\% pass@1 accuracy, respectively. Analysis shows SOPHIA outperforms supervised fine-tuning and direct on-policy RL methods, offering a better policy initialization for further on-policy training.

Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning
Junhao Shen 1,2
Haiteng Zhao
Yuzhe Gu
Songyang Gao?
Kuikun Liu? Kai Chen
Haian Huang?
Jianfei Gao
Dahua Lin
Wenwei Zhang
Shanghai Jiao Tong University *Shanghai AI Laboratory
MMLab, The Chinese University of Hong Kong {shenjunhao, zhangwenwei, chenkai]@pjjab org.
Abstract
Enhancing large vision-language models (LVLMs) with visual slow-thinking rea- soning is crucial for solving complex multimodal tasks. However, since LVLMs are mainly trained with lsioon-anguaag alignment, is difficult to adopt on-policy reinforcement learning (RL) to develop the slow thinking ability because the rollout space restricted by its initial abilities. Off-policy offers a way to go beyond the current policy, but directly distilling trajectories from external models may cause visual hallucinations due to mismatched visual perception abilitics across models. To address these issues, this paper proposes SOPHIA, simple and scalable Semi- Off-
Off-Policy RL for vision-language slow-tHlniing reAsoning. SOPHIA builds semi-off-policy behavior model by combining on-policy visual understanding from a trainable LVLM with off policy slow- now-thinking reasoning from language model, assigns outcome- based rewards to reasoning, and propagates visual rewards backward. Then LVLM learns slow- fow-thinkiing reasoning ability from the obtained reasoning trajectories using propagated rewards via off-polic RI. algorithms. Ex- tensive experiments with Intern VL.2.5 and Intern VI.3.0 with 8B and 38B sizes show the effectiveness of SOPHIA. Notably, SOPHIA improves Intern VI.3.0-38B by 8.50% in average, reaching state-of-the-ud performance among open-source LVLMs on multiple multimodal reasoning benchmarks, and even outperforms some closed-source models (e.g., GPT-4.1) on the challenging Math Vision and OlympiadBench, achieving 49.08% and 49.95% pass@ @ accuracy, respectively. Analysis shows SOPHIA outperforms supervised fine-tuning and direct on policy RL methods, offering better policy initialization for further on-policy training.
InternVL3.0-308 SOPHIA
InternVL...0-08
InternVL330-58 MPO
IaternVL3300338 GRPO
Qwen2,5-VL-72BB
GFF441
18.50
+24.45
+16.15
55.51
40.51
50.96
$1.74
55.07
49.08
52.60
46
49
40.58 49.6
57.57
20.01
$7.95
30.24
96
35.39
38.29
25,50
Average Pass@l Accuracy
OlympiadBench (Puss@.1)
Math Visho test (Pass@:1)
MMMU Pro (Pass
The average pass @1 accuracy of SOPHIA and some representative models across eight benchmarks, as well as their pass@1 accuracy on OlympiadBench, Vision test, and MMMU Pro. Intern VL3.0-L3.00338 SOPHIA significantly improves over its base model, Intern VL3.0-38B (Instruct), achieving state-of-the-art performance on most of the benchmarks.
Figure 1:
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Visual Undderttandig
Visual Q&& Query
Query Rollout
Data Sampling Path
Spatial Layour
Offir compreeonsttrr
érealdoom
Reward Propagation
Sematic Relationship
Fine-grained Visual Detuils
mugitwrring cleveisrry.
Question
precaze fasse ghes stsual ingormattio..
Policy Updating
Image
Semi-off-Policy Behavior Model
Trajectory y, (lhes nu Agoon ihe therem parses the shapas. Frint, need to steaeliee the imsgee The image descr theal live with fire curreer
Vision-I Vision-Ianguage Model πT
Reasoning Language Model
Va Volog z. v)
VoJ(O)
R(
R(c
Figure 2: An overview of SOPHIA. SOPHIA samples reasoning trajectories y, and calculates rewards for both reasoning R(y,) and visual understanding R(c,). Finally, it updates the LVLM policy via an off-policy method to cultivate visual slow-thinking reasoning abilities.
1 Introduction
Leveraging reasoning abilities to solve complex problems is a crucial step toward artificial general intelligence [1. 2]. Recent large language models (LLMs), e.g., OpenAl o-series [3] and DeepSeek- RI [4], have excelled at complex problem solving by their slow thinking capabilities [5, 6]. These advances inspire the community explore large vision-language models (LVLMs) for complex multimodal problem solving (e.g., geometry problems) with visual slow thinking [7-10].
Early explorations mainly improve the reasoning capability of LVLMs through pipeline-generated reasoning trajectories [1 [11-14] or distillation [15], yet, these methods lead to pattern memorization rather than genuine improvement in visual reasoning [16]. Inspired by the recent success of reinforce- ment learning (RL) for LLMs reasoning, especially DeepSeek-R1 [4], recent works explore on- on-policy algorithms (e.g. GRPO (17]) in LVLMs and try elicit the reflection behavior in training [18-21]. However, since most LVLMS are trained through vision-text alignment data [22, 23] that lack slow- thinking reasoning trajectories, it is hard to sample slow-thinking behaviors from the output space of LVLMs. Consequently, the performance of on-policy RI are fundamentally bounded by the initial policy distribution, as they essentially only amplify existing behaviors inside LVLMs [24, 25].
Given the limitation of on- on-policy RL, off-policy learning offers a promising solution. By using alternative policy models to generate trajectories, it bypasses the limitations of the current policy distribution and enables learning beyond their initial abilities [26, 27]. However, the visual features involved in off-policy slow-thinking reasoning trajectories may not be aligned with the own visual understanding abilities of LVLMs, leading to a conflict between the optimization direction and the visual understanding of LVLMs [28-30], which severely exacerbates issues like hallucinations in LVLM visual slow-thinking reasoning [31, 32].
To overcome the limitations of both on-policy and off off-policy RL for LVLMs reasoning, this paper proposes SOPHIA, Paaii-Off-Pooccy RL for laionnlanguage slow- reAsoning. As shown in Figure 2, SOPHIA consists of two steps: semi-off- offioicf-policy sampling and policy optimization with propagated rewards. Specifically, SOPHIA builds a semi-off policy behavior model by combining the on-policy visual understanding from the trainable LVLM with the off- off-policy slow thinking from open-source reasoning LLMs (e.g., QwQ [33]). During trajectory collection, SOPHIA first samples a batch of image understandings (i.e., detailed descriptions of images) from LVLMs for the same query, and then adopts reasoning models for sampling dow-thinking reasoning trajectories.
In the policy optimization process, SOPHIA first calculates rewards for the slow-thinking reasoning trajectories of LLMs based on the verification results of the reasoning outcome. Because outcome- based rewards may cause the model to overlook the accuracy and completeness of visual content, we assign each visual understanding a reward via reward backpropagation to strengthen the link between visual understanding and reasoning. Such design not only enhances the slow-thinking abilities of LVLM but also optimizes the use of visual understanding in reasoning with visual rewards. Furthermore, SOPHIA is scalable as it uses the propagation of outcome- based rewards instead of human or closed- source model annotations for ensuring the quality of visual understanding.
Extensive experiments show that SOPHIA effectively enhances the ability of performing slow thinking on challenging visual tasks. Experiments on both Intern VI.2.5 and Intern at 8B and 38B show that SOPHIA improves performance across various benchmarks. Specifically, Intern VL300-38B +

---

SOPHIA reaches ttate-of-the--eee results on several benchmarks. which achieves an average gain of +8.50% and attains 49.08% and 49.95% pass@1 accuracy on the challenging Math Vision [34] and OlympiadBench [35] datasets, outperforming leading open-source models such as Qwen2.5-VI. 72B [36], as well as closed-source models like GPT-4.1 [37]. Further results show that SOPHIA outperforms both supervised fine-tuning and on-policy RL. methods. Continued on-policy RL. training with SOPHIA leads to substantial improvements, achieving performance comparable to Claude-3.7- Sonnet general college-level questions. Moreover, in-depth analysis and ablation studies including diverse settings and training data further verify the generality and scalability of SOPHIA.
2 Related Work
Visual Slow-thinking Reasoning Tasks. Large vision-language models (LVLMs) are thriving in
various visual tasks. Both closed-source models (e.g., ChatGPT 4o [38] and Claude 3.7 Sonnet [39] and open-source models (e.g.. Qwen VL [36] and Intern VL [40]) have made key progress in image captioning [41], object detection [42] and video understanding [43]. Recent efforts focus on complex visual reasoning tasks which require both slow-thinking and visual understanding [7-9]. However, it remains challenging to equip LVLMs with robust slow-thinking reasoning. Although Chain-of- Thought (CoT) techniques enhance reasoning of LLMs [44], applying them directly to LVLMs often fails to improve the textual and visual reasoning [45, 46.. This is largely because most LVLMs rely on vision-text alignment during pre-training and fine-tuning [22, 23], which hinders the development of genuine visual slow-thinking reasoning.
Supervised Fine-tuning for LVLM Reasoning, Previous efforts to endow LVLMs with reasoning abilities often rely on supervised approaches which involve constructing reasoning trajectories or structured pipelines to generate training data. For instance, M [11] constructs CoT using a multi-step workflow, and Mulberry [14] employs Collective Monte Carlo Tree Search to generate COT. More recently, large datasets are synthesized by leveraging LLMS to generate visually-infoma reasoning trajectories based on image captions [15]. However, generated data often includes visual hallucinations [47] and these pipelines rarely consider it. Training on such noisy data may degrade the performance [48]. Additionally, supervised fine-tuning tends to encourage models memorize specific reasoning patterns rather than develop robust and generalizable visual slow-thinking reasoning capabilities [16]. Compared these methods, SOPHIA offers better scalability and generalization.
Reinforcement Learning for LVLM Reasoning. Reinforcement learning (RL) offers better gen- eralization and is regarded
more fundamental and effective training approach [16]. Extensive research has explored the use of on-policy RI. to enhance slow-thinking reasoning abilities, e.g., RL.O [49, 50]. PPO [51], GRPO [17] and DAPO [52]. Recently, inspired by the success of L.LMs, some studies apply on-policy RL to LVLMs. MM-Eureka [20] and Vision-RI 1 [19] mainly leverage GRPO to improve the reasoning abilities of LVLM. However, on-policy methods are fundamentally limited by existing policy distributions of LVLM, reinforcing suboptimal behaviors and hindering further progress. While off-policy RL is promising, its application LVLMs remains challenging: the visual features in off-policy slow-thinking trajectories may misalign with the inherent visual understanding of LVLM, creating optimization conflicts that exacerbate hallucinations in visual reasoning [28-32]. Compared to these methods, SOPHIA enables the model to better perceive visual features from trajectories and more effectively leverage its visual understanding during reasoning.
3 Preliminaries
Policy Model. Causal large vision-language model (LVLM) and large language model (LLM) are adopted to produce the trajectory. A causal rision-language models produces output trajectory = (y1, V2. y),,) given a textual input prompt x = (æ1, Ø2, LL.) and image input v = (v1, U2, ... VL.)). where T.y: I,,, Tap represent the length of the token sequence, and each entry of these sequences belong to the token space of LVLM. A language generation policy in an auto-regressive model is characterized by a conditional probability distribution parameterized by 0 as
Jny no (I 21,-1,2,00)
T v)
(1)
3

---

with the convention V1:0 = 0 and M1:--1 (V1, 32. .t-1). Similarly, a causal LLM f is
Ly
T σ) = II
x (y:
(2)
Off-policy Optimization. In our method, we focus on off-policy settings [53] and adopt policy gradient methods based on importance sampling [54, 55], which maximizes the expected cumulative returns. The objective function is formulated as
J(0) A y)
To x. M
R(x, v, v))
x, (
where D is the off-policy dataset containing samples (æ, v,y) along with their rewards R(x, v, y). Ty is the policy model (i.e, LVLM) with learnable parameters 0 and ju is the behavior policy that generates the offline data. Unlike on on-policy RI,, off-policy RL learns from trajectories generated by different behavior policy, enabling more efficient exploration beyond the model's current abilities.
4 Semi-off-Policy Reinforcement Learning
An overview of SOPHIA is shown Figure 2. In the first stage (Section 4. SOPHIA builds semi- off-policy behavior model by combining the on on-policy visual understanding from the trainable large vision language model (LVLM) with the off-policy slow thinking from an open-source reasoning LLM, which is used to sample a number of image descriptions and description-derived reasoning trajectories. Next (Section 4.2), SOPHIA assigns outcome-base rewards to reasoning and propagates visual rewards backward. Finally (Section 4.3), we use off- off-policy RL to update the LVLM, with theoretical analysis showing that importance sampling involving the semi-off-pone polfff-olccy behavior can be efficiently approximated via our reasoning and visual rewards.
4.1 Semi-off-policy Sampling
We construct a oemi-off-policy behavior model by combining on on-policy visual understanding from the trainable LVLM To and off-policy slow-thinking reasoning from open-source language model म. Other existing multimodal models (e.g., GPT-4o) are not employed because current LVLMs have limited visual understanding. As result, visual features involved in off- pol-colicy reasoning trajectories may not be fully perceived or understood by the target LVLM. Thus, directly using trajectories containing visual features extracted from other behavior LVLMs can lead conflicts between the gradient direction and the LVLM's inherent visual understanding biases. Even when these reasoning trajectories yield correct results, they still face bottleneck during training.
Specifically, given a visual training dataset denoted as Duain = {(4,, Tyg. a;)li = 1,2, ... Derain } where 4; is the i-th image, Tg is the i-th query, a; is the i-th golden answer 2g and | calculates the size of set, the LVLM first performs visual understanding in an on-policy manner. For each image I, we first extract its visual tokens v; V(I.) using the pretrained visual encoder V inside the LVLM. Then, prompt engineering [56] making the LVLM try its best to capture spatial layouts, semantic relationships among objects, fine-graine visual details and on (Details in Figure A2 of Appendix E). Then, we can obtain a conditional distribution of image description e; To (-(-aa,,, v;))
After sampling descriptions from the on-policy LVLM,, given an off-policy reasoning language model म. as well as a query Zy. and its corresponding descriptions c,, we can leverage prompt engincering to construct it based on the query and description. This enables the reasoning model to simulate LVLM that can see the image based on the provided description (see Figure A3 in Appendix E for details). As result, we can sample the slow-thinking reasoning trajectory y; x(.(2g.)
We assume that the generated e; can effectively encode the visual cues required for reasoning. Prior studies show that, in LLaVA-style LVLMs, adapter layers project visual embeddings into text-embedding subspaces, aligning continuous visual features with their textual representations [57]. Hence, visual understanding (whether expressed as embeddings tokens) serves the conditioning context for autoregressive reasoning. When this context captures spatial layouts, object relations and fine-grainee details accurately, the reasoning becomes more coherent and logically consistent.

---

4.2 Reward Evaluation and Propagation
Reward for Slow Thinking. When LLM has generated tokens 91.1, the policy model is in state E S, where S is the finite state space and / = 0, 1,2, Ly. Since fine-grained supervision over intermediate reasoning steps is typically unavailable, a practical solution is to extract the final answer [4, 58] and evaluate its accuracy using rules models, assigning outcome-baset reward at the end. We adopt this outcome-basee reward strategy. Thus, the episodic reward for a reasoning trajectory is defined as Eq. (4), where r(.) calculates the reward for each state.
{
if = Ly and the answer is correct otherwise.
R(y) = - r(s),, r(si)
Reward for Visual Understanding. To propagate the correctness signal to the related visual descriptions, define the reward of each description R(c,) based the average outcome score of the reasoning trajectories. Specifically, we sample N reasoning trajectories from the semi-off- ooff-ollicyy behavior model 5 g.. conditioned the visual input, and compute
R(c,) N Σ Rly!")
(5)
Reward for Off-Policy Dataset. After obtaining both reasoning and visual reward, along with number reasoning trajectories regarding the image and query, we use them to construct the off-policy dataset D, and use their rewards to guide the policy updating. The reward is defined as
{1
if R(c,) > R(y.) = 1, y: min (T v, ~ D, R(argu v.y) 1}
R(Z U, y.)
bn
(6)
otherwise
where a E (0. is the threshold for the visual reward. Eq. (6) ensures that the chosen reasoning trajectory contains both sufficient correct visual information and correct final answer. At the same time, we aim to sample the shortest possible trajectory since there exists issue of overthinking in these reasoning language models [59], and opting for the shortest response not only help models learn a more concise reasoning structure but also mitigate redundant repetitions in the trajectory.
4.3 Policy Updating
We adopt off-policy optimization and the object function is Eq. (3), By differentiating the Eq. 3 with respect 0, we obtain the policy gradient Eq. (7) for optimization.
To T xx. v)
[o) = VolBaa,v.))>>
R v, y)
(u ar v)
Te T v) pu (2)
= (a
V
R(x, v.
R(ax, v,
= ((
V Vomo(yyla
p(ylar, v)
log deriveiiivaiee trick
Te x v) (uyla,
R(x, v. v. Vo
(7)
Ly
Ey
To at, v) Mlyla, xc, v)
R(x, U. v)
Va log (oggoounl
Try
Lemma A.3
Ei (..
R(x, v. y)
Vo log To(nn 912-1,
Note that off-policy RL results in distribution mismatch between the behavior policy and the current LVLM policy T&. Standard off-policy RL corrects this mismatch and reduces bias via importance sampling (IS). In the last step, we apply Lemma A.3, which ensures directly approximating the oof-yy objective using the average reward introduces only a small and controllable bias. Detailed proof is provided in Appendix A.
5

---

Implementation
5.1 Policy Initialization
We utilize InternVL2.5 and Intern VL3.0 in 8B and 38B scale base models, all of which is instructed. Initially, we fine-tune the base models using general visual question-ama -aniiorinnn data, covering image description and caption. These warmed-up models then serve the initialization for the policy model To in SOPHIA framework. We also explore directly initializing policy models with base models without warming up and discuss the influence of different initial policy models in Appendix D. The warm-up training data consists of the open-source RLAIF-V V [60] and WildVision [61] datasets. For RLAIF-V, select the accepted samples, and for WildVision, the reasoning language model, the policy model क is initialized by QwQ-32B-Pree [33] for easy questions and DeepSeek-RI [17] for difficult ones.
choose the best response. For
5.2 Reinforcement Learning
Data Preparation. The visual question aartonn-nswweiing (VQA) training dataset Duain is private and consists of around 80K questions from higher educational mathematics cources in both Chinese and English, as well as vocational education subjects, covering fields like finance and management. We also analyze SOPHIA on open-source training dataset Math V360K [7] Section 6.4. To prevent data leakage, we filtered out any queries overlapping with the benchmark using keyword matching and LLMs.. Additionally, our policy updates are mainly guided by visual reasoning trajectories, with supplementary text trajectories from Numina, MATH, and past AMC/AIME (excluding AIME 2024), and general VQA data from RLAIF-V and WildVVisio.
Reward Signal. Since reasoning language models typically produce well well-structured responses, we directly extract answers using regular expressions and evaluate their correctness using the rule-based verifier from OlympiadBench [35], providing binary rewards as defined in Eq. (4).
Algorithm and Settings. The policy gradient for optimization is described in Eq. (7), and omit the KL. regularization term during optimization, following recent practices for instruction-tume models to improve response length [20]. Detailed algorithms, hyperparameter configurations and prompt design strategies used during sampling and policy updates are provided in Appendix B.
Experiment
6.1 Evaluation Setup
Baseline. We evaluate multiple baselines on benchmarks. When evaluating the SOPHIA on VL2.5-3L and Intern [40]. the baseline models include closed-source models
such GPT-4.1 [37] and Claude-3.7-Sonnet [39], well open-source models like VL2.55
38B-MPO, Intern V133.0338--MPO [46], BetrmmLL...---------
MM-Eureka-Qwen--
[20],
Qwen2.5-VL-72BB [36]. WvQ-72B-previce [62] and Virgo-72B [63]. When evaluating 8B scale, the
baseline models include open-source models like Intern LL..--B--MPPP
-0..cmVL-3.0-B-MME. [46],
Mulberry-8B [14], MiniCPM-o-8B [60], MM-Eureka-8B [20] and Qwen2.5-VL-7B [36].
Evaluation. To comprehensively evaluate the performance, our benchmarks include MMMU, MMMU Pro [64], Vista [65], Verse [66], DynaMath [67], Math Vision[344.MVVMMMM [68]
and OlympiaBBench [35]. We use pass@ accuracy as the metric for evaluation under the zero-shot
setting. Detailed evaluation setup is in Appendix C.
6.2 Main Results
Overall Comparison. Table 1 presents the performance of SOPHIA on Intern VL2.5-38B, EnterVVL3.0-333 and both open-source and closed-source models across eight benchmarks. Focus- ing on Base Model and Fine-tuned Ones, SOPHIA achieves substantial gains over the base models, with average improvements of 10.94% on InternVL.2.5-38B and 8.50% on VL.3.03 challenging benchmarks such as Math Vision and OlympiadBench, it boosts performance by 26.166 16% and 24.45% respectively, highlighting its strong problem-solving abilities. Furthermore, SOPHIA surpasses the mixed preference optimization (MPO) method, which is trained on 3 million examples,
On

---

Collego-level Question
Math-rclated Question
Challenging Sclentific Reasoning
MMMU
MMMU
MathVista
DynaMath
OlymppiaBBench
Average
Clesrd-soury Model
GPT-4.1
73.78
51.37
70.20
49,49
72.06
47.53
39.52
30.38
55.07
Cluade333 Sonoete
70.44
36.01
69.30
47,46
70.04
42.43
36.24
36.25
53.52
Representative Open-source Madel
Owen2 5-VL-228
66.44
49.65
74,20
49.37
67.13
38.29
38.62
30.24
51,74
QuQ-72B
Virgo-728
40.98
MMi-Eureka- Qwenn-3B
64.00
47.51
72.90
47.37
66.41
39.24
35.39
34.42
Bure Model and Fine timed Oner
InneraVL2.5388
61,22
42.
69.50
37.44
52.95
31.32
27.08
23.06
45.09
+MPO
63.11
46 41
74.80
45.18
60..00
35.56
20.72
26.66
47.74
49.65
74.70
45.30
72:20
AMPO
69.33
49108
GGROO
66.67
49.88
75.00
43.02
68.22
42.23
29.82
32.96
50.98
ASOPHIA
69.00
$2.60
75.20
47.46
65.35
49.08
35.44
49.95
58.51
SSPPHIA+GRPO
70.22
56.19
76,00
49.11
05.87
30:40
37.03
51.79
57.28
Table I: @ass@1 accuracy of SOPHIA in 38B scale, other fine-tuning strategies and baselines, In each column, An entry is in bold if its value is the best among open-source models, and underlined it ranks second (exclude the last line "SOPHIA+GRPO").
College level Question
Math-relater Question
Challenging Scientific Reasoning
Model
MMMU
MMMU Pro
MathVerse VathVista
DynaMath
Math
MV-MATII OlympiadBench
Average
Mullerry-88
19:31
20159
3:27
24.92
MiniCPM-a-8B
45.44
29.60
66.90
27.79
42.38
18.19
22:30
8.22
32.61
MM 4MM-Eureka-8B
51.22
23.35
67.40
29.82
39.46
21.97
20.76
14.97
34.87
Owea2.5.VL. VB
52.00
38.61
68.90
38.58
54.35
26.38
28.42
16.28
40.44
48
21,02
20.11
12.90
32.70
MPO
54.33
32.54
69.40
29.82
46.07
22.11
23.05
14.25
36.45
SSOHHA
57.00
22.02
61.10
30.84
46.51
22,66
22.00
18.53
36.33
VL...8L33.0888
$1.11
33.99
67.80
33.12
47.34
25.66
21.95
19.40
37.55
-MPO
58
72.60
39.72
35.40
20.63
43.41
-SOPHIA
60.67
38.27
64.40
35.91
55.30
37.89
26.48
38.67
34.71
Table 2: Pass@ accuracy of SOPHIA in 8B scale and baselines in small scale. In each column, An entry is in bold if its value is the best, and underlined if it ranks second.
showing more efficient data utilization and greater effectiveness than pipeline-basse supervised fine-tuning. Overall, MnterVLL3.00 + SOPHIA achieves the best or second-bess pass@1 accuracy most benchmarks and demonstrates ofttt-----he-rrr performance in terms of the average of these benchmarks, coming close to the performance the closed-source model GPT 4.1. Notably, our model achieves remarkable pass @ accuracy of 49.08
Vision and 49.95 on OlympiadBench.
On-Policy RL. On-policy RL is commonly used to enhance slow-thinking reasoning. The policy initialization includes both the original Intern VL3.0-38B 3.B-3BB and the model trained using SOPHIA. The training queries are drawn from a subset the VQA training dataset used in SOPHIA. Detailed GRPO configurations are in Appendix B.2. As shown in Figure 3, track pass @ accuracy for and Intern VL3.00388 00-00-88---0 +SOPHIA on MMMU Pro and Math Vision over 480 training steps. The solid lines show each model's 's performance; dashed lines indicate base model results from Table 1. While performance of VL.30- declines with GRPO, SOPHIA not only prevents this drop but also improves accuracy. This highlights the limitations of on-policy RL for LVLMs (Section 1) and shows that our method enables stronger policy initialization for further on-policy RL.
Moreover, we evaluate the checkpoint with the highest average score on Math Vision and MMMU Pro after 100 steps. As shown in Table I, SOPHIA outperforms GRPO across multiple benchmarks, showing stronger visual dow-thinkinn reasoning, Further on-policy RL boosts performance beyond direct RL, achieving Beude--.7--ommee on college-level tasks and surpassing most closed-source models on challenging scientific reasoning.
Overall Performance in Small Scale. The performance SOPHIA on 8B scale and other open- source models are shown in Table 2. At the 8B scale, SOPHIA outperforms the base model across most benchmarks. It significantly surpasses the MCTS-based Mulberry and remains competitive with both the GRPO-based MM-Fureka 8B and MasO-based methods. Notably, Intern VL 3.0 8B+SOPHIA achieves the best on average performance and excels on college-level MMMU and challenging OlympiadBench. In contrast, SOPHIA performs worse Math Vista. We attribute this to

---

Primiss vL3 505 501550AA
20-
Tuinin
Tuining Step
(a) MMMU Pro Performance
(b) Vision Performance
Figure 3: The performance of InternVL3.0-33 0-338B and Intern VL3.0-38B on MMMU Pro and Vision 480 training steps, where the green and blue solid lines represent their respective pass@1 I accuracy, while the dashed lines indicate the base model performance.
DynaMath
Math Vision
Olympiad
Model
Plane Geo
Solid Geo
Plane Geo
Solid Geo
Plane Geo
Solid Geo
GPT-4.1
59.87
54.00
50.75
43.08
33.08
13.65
QurvieB-preview
58.05
55.33
40.77
30.74
33.49
24.32
InternVL.2.533B
43.90
25.33
33.99
26.23
12.71
4.16
+SOPHIA
62.08
67.33
56.95
45.49
50.55
52.54
InternVL3.00333
51.43
46.67
36.92
30.74
22.64
12.61
+SOPHIA
62.99
72.00
58,26
43.85
56.25
62.81
Table 3: @ass@1 1 accuracy of SOPHIA and baselines in geometry questions. "Plane Geo" in Math Vision includes metric geometry-ange metric geometry-ane and metric geometry-length. "Plane Geo" in OlympiadBench includes geometry and plane geometry. In each column, An entry is in bold if its value is the best, and underlined if it ranks second.
MathVista
Olympladllenh Average
Mechod
MMMU
MathVista MathVision OtynptodBeechh Average
IotcaVL2..5388 SOPHIA
Breese VIT
unfreese VTT
74.70
72.40
47.96
49.53
59.01
59.55
-Raddam
+SOPHIA w/o
64.11
06.10)
09.80
08.20
40.35
37.33
39.39
35.10
$3.54
51
lnteraV1.3.0 38B SOPHIA
SOOHHA whs
46.33
70.00
41.04
40:86
56 74
freezo ViT
69,00
75.20
49.08
49.05
08.1 99.74
SSOPIIA
65.44
70,90
42.73
41.46
55.13
unfizzore VIT
67.67
71.00
49.34
50.51
Table Ablation study on reward design. "Ran- dom" randomly samples trajectories from D; "SOPHIA w/o CR" removes the caption reward; "SOPHIA w/o S" drops the shortest trajectory constraint. Best value bold.
Table 4: Ablation study on VIT freezing when using SOPHIA. bold entry indicate better performance compared to the alternative under the same base model.
the training data's 's limited coverage of certain problem types, such as age-difference reasoning. Due the smaller constrained generalization capability of model, struggles to develop robust reasoning strategies for these underrepresented cases, resulting in lower performance.
Geometry Performance. Geometry challenging math tasks and requires both visual understand- ing and reasoning [69]. We evaluate SOPHIA in 38B scalre as well as top models and the base model on DynaMath, Math MathVision Vision, and OlympiadBench geometry problems. As shown in Table 3, SOPHIA outperforms other open- open-source models, especially on complex OlympiadBench tasks, validating its effectiveness in enhancing the abilities of leveraging the visual understanding during reasoning.
6.3 Ablation Study
We assess the impact of weight freezing, and the effectiveness of our proposed reward and warm-up. Further experiments on response style, teacher models, and caption as training data are detailed in Appendix D.
Frozen/Unfrozze VIT. We conduct experiments on Intern VL22553888 and Intern VL.30038B to explore the effect of freezing the visual transformer. As shown in Table 4, freezing the visual transformer consistently improves average performance, with notable gains on general-purpose benchmarks such MMMU, We attribute this to the possibility that long reasoning trajectories may interfere with

---

Collego-teeel Question
Math-rclated Question
Challenging Seientifie Reasening
Method
MMMU Pro
MathVerse
DynaMath
MathVVsion
MV-MATHI
OlympiadBlench
Average
InteraV1
52.95
27.08
23.06
+SOPIIA wio warn up
63.06
43.96
70.90
45.56
61,08
41.21
34.60
48.10
51.88
+SOPHIA
66.33
49.65
74.70
45.30
63.13
47.96
35.94
49.22
54/03
InieraVL2.5-88
48.22
28.61
64.30
26.78
38.14
21.02
20.61
12.90
32.70
+SOPHIA wio wam--up
51.44
30.98
60.10
MLS4
45.35
23.75
17.47
17.54
34.68
+SOPHLA
57.00
32.02
30.84
46.51
22.66
22,00
18.53
36.33
Table 6: Ablation study of warm-up IntemVL2..53388 and InternLL2.5-8D A bold entry indicate best performance compared to others under the same base model.
vision-language alignment. Therefore, we freeze the visual transformer during training to not only improve performance but also enhance training efficiency.
Reward. In Eq.6, the reward includes not only for the correctness of the reasoning trajectory but also for the quality of visual information. To evaluate the effectiveness, we conduct ablation studies on VL.er To ensure the validity of our results, no additional general visual or textual data are used, As shown in Table5, "Random" refers to trajectories randomly sampled from off- off- policy dataset D; "SOPHIA w/o CR" excludes the caption reward and uses only the outcome reward; "SOPHIA w/o S" removes the constraint that we select shortest one. We observe that: 1) The Random strategy is a strong baseline; its modest gains on Vision and OlympiadBench suggest that even imperfect reasoning can help models learn reasoning patterns. 2) Using outcome reward promotes correct logic, but unfiltered visual errors still limit performance on difficult datasets. 3) Selecting the shortest trajectories helps avoid repetition and redundancy, yiclding notable gains on challenging datasets.
Warm up. To investigate the impact of the warm-up on the performance, we set up experimental group including "SOPHIA w/o warm-up" and "SOPHIA w/o warm-up"" The policy initialization in the experimental group is directly Intern VL2.5-388 (Instruct) and Intern VL2.5-8B (Instruct) without a warm-up phase. The control group includes models trained following the standard procedure. The experimental results are shown in Table 6. We observe that models with warm-up consistently outperform those without warm-up. This likely due to the emphasis of warm-up data on visual description and grounding. Such training enhances the captioning ability of model, thereby improving the quality of trajectories collected by the behavior model.
6.4 Analysis of Training Data
We further evaluate SOPHIA on different train- ing datasets and data scales. Additional exper- iments including caption data mixing, different data sources, and the Keep-N strategy, are in Appendix D.
Method
MMMU
MathiVist Vasion Olympiadllech Average
L2teraVL22-338
61.22
69.50
31:32
23.06
46.28
+ -SOPEIA
19.17
MaaliV 50%
67.33
66.90
42.14
40.38
Private
65.44
70.90
42.73
41,46
55.13
Versatility. To verify that performance improve- ments are not solely due to higher data qual-
Table 7: Versatility analysis of SOPHIA.
ity, we also evaluate our method on the public V30K dataset. Specifically, using the same settings as with the private dataset (80K), we train Intern VL2.5-38B on 10% (36K) and 50% (180K) subsets of Math V360K to explore the trade-off between data quality and quantity. No additional general visual or textual data are used ensure fair comparison. Results are shown in Table 7. We observe that SOPHIA still brings performance gains when trained on open-source datasets. However, despite containing more questions than private data, these datasets do not lead to notable improvements on Math Vision and OlympiadBench, even degrading on Math This suggests that the quality and difficulty of questions are more critical than the quantity.
Scaling Law. We examine how SOPHIA scales with data size by training Intern VL2.5-387 on datasets of 5K, 10K, 20K, 40K, and 80K questions. To ensure fair comparison, no additional general-purpose visual or textual data are used. The results are shown in Table 8. We observe that: 1) Average performance improves with more data, showing rapid gains initially and diminishing returns thereafter; 2) Larger datasets yield significant improvements on more challenging benchmarks: 3) For MMMU and MMMU Pro, performance is sensitive to data size only at smaller scales, with limited gains beyond that; 4) On Math Vista, Math Verse and DynaMath, increasing the dataset size does not consistently lead to performance gains and may even be detrimental in some cases.
9

---

College-level Question
Math-related Question
Challenging Scientific Reasoning
#Data
MMMU
MMMU Pro
MathVista
MathVerse
DynuMath
MuthVisson
MV-MATII
OlympiadBench
Average
59.22
44.22
67,20
35.26
28.42
28.50
10K
65.22
46.47
70.10
38.83
60.35
37.50
30.66
34,80
47.99
20K
66.56
60.82
31.61
40K
66.11
47.57
70.40
46.45
58.40
41.41
32.45
39.59
50.30
80K
65.44
46,93
70.90
45.18
59.14
42.73
34.15
41.46
50.74
Table 8: Pass@1 1 accuracy of SOPHIA on InternVL 2.5 38B in different data scale.
6.5
Qualitative Analysis on Hallucinations and Logical Inconsistencies
We conduct manual evaluation on 50 randomly sampled cases from MMMU Pro before and after SOPHIA training, covering diverse disciplines to as- sess both visual hallucination and logical inconsis- tency. We report the proportion of responses with visual ballucinations and logic inconsistency. As shown in Table 9, SOPHIA significantly reduces hallucinations and logic error, consistent with im- provements in overall correctness, which supports our motivation that on on-policy visual understanding training enhance the visual understanding of the model while reducing hallucinations.
Vioual Halluciration Logle Inconsistency
Table 9: Qualitative analysis of SOPHIA.
Conclusion and Future Work
This paper presents SOPHIA, a simple and scalable semi-off-polee offf-poliyy reinforcement learning framework to enhance the visual slow-tinkiing reasoning capabilities of large vision-language models (LVLMs). By leveraging semi-off- ofni-off-policy behavior model that integrates on-policy visual understanding with off-policp slow-thinking reasoning, SOPHIA not only enhances the LVLM's slow-thinking abilities but also uses visual rewards to guide the LVLM in optimizing the strategy of leveraging its own visual understanding during reasoning without requiring closed-source or buman-annotated data. Extensive experiments across eight multimodal benchmarks show that SOPHIA achieves state-of-the-ane performance among open-source LVLMs and even surpasses some closed-source models on challenging benchmarks like Math Vision and OlympiadBench. SOPHIA also outperforms both supervised fine-tuning and on-policy RL methods, serving a better initialization for further on- on-policy reinforcement learning, cnabling continual improvement without performance degradation.
Despite these advances, SOPHIA still struggles with long-range visual dependencies and fine- grained or low-level recognition complex scenes. Also, performance may suffer when facing underrepresented domains or noisy samples. Issues such as hallucinated visual content, redundant reasoning, and the binary nature of reward signals still persist. Future work will explore stronger visual encoders, adaptive curriculum learning, and more robust data augmentation to further improve generalization, efficiency, and reasoning robustness across broader domains.
Acknowledgements
We would like to thank the anonymous reviewers and area chairs for their comprehensive and constructive reviews. This work is supported by the Shanghai Artificial Intelligence Laboratory. The authors would like to thank Yiming Zhang, Chengqi Lyu and Jiangning Liu to their valuable suggestions and comments on this work.
References
[1] Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, al. Towards large reasoning models: survey of reinforced reasoning with large language models. arXiv preprint vrXii:2501.000 00966, 2025.
[2] Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, et Evaluation of openai ol: Opportunities and challenges of agi. arXiv preprint arXiv:2409. 18486, 2024.
10

---

[3] OpenAl. Learning to reason with 1lms. 2024.
[4] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin
Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen. Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingehang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. I.. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang. Kuai Yu. Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang. Wenjun Gao, Wenqin Yu, Wentao Zhang, W. I.. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li. Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You. Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng. Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, 7. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. DeepSeek-RI: Incentivizing reasoning capability in LLMs via reinforcement 2025.
learning, arXiv preprint arXiv:2501.
[5] Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey-part part 2: Surpassing opepveve through simple distillation, big progress or bitter lesson? arXiv preprint arX.1.2411.16489 2024.
[6] Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.09413. 2024.
[7] Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math-LLaVA: Bootstrapping mathematical reasoning for multimodal large language models. In The 2024 Conference on Empirical Methods in Natural Language Processing, pages 4663-4680, Miami, FL, 2024.
[8] Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang. Multi- math: Bridging visual and mathematical reasoning for large language models. arXiv preprint arXiv:2409.00147, 2024.
[9] Wenwen Zhuang, Xin Huang, Xiantao Zhang, and Jin Zeng. Math-puma: Progressive upward multimodal alignment to enhance mathematical reasoning. arXiv preprint arXiv:2408. 2408.08640. 2024.
[10] Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal Ilms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition 2024, pages 13084-13094, Seattle, WA, 2024.
[11] Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M&CoT: novel benchmark for multi-domain multi-step multi-modal th In Proceedings of
11

---

the 62nd Annual Meeting of the Association for Computational Linguistics, pages 8199-8224 Bangkok, Thailand, 2024.
[12] Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. LLaVA-CoT: Let vision language models reason step-by-step. arXiv preprint arXiv:2411. 2024.
[13] Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2025, Nashville, TN, 2025.
[14] Huanjin Yao, Jiaxing Huang. Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, and Dacheng Tao. Mulberry: Empowering MILM with ol-like reasoning and reflection via collective monte carlo tree search. arXiv preprint 1eXiv:2412118111 2024.
[15] Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng. Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, Bo Zhang, and Wei Chen. RI-onevision: Ad- vancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint Stf.::5003.061155 2025.
[16] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sit memorizes, rl generalizes: comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161 2025.
[17] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
[18] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-RFT: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.0178. 2025.
[19] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-R1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749, 2025.
[20] Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. arXiv preprint arXiv:2503.07365, 2025.
[21] Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1 R1-Zero's 's "Aha Moment" in visual reasoning on 2b non-sft model. arXiv preprint arXiv:2503.051132, 2025.
[22] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5625- 5644, 2024.
[23] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems 36, pages 34892-34916, New Orleans, LA, 2023.
[24] Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi. Erie Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint 2434422403.04642 2024.
[25] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in lmm beyond the base model? arXiv preprint arXiv: arXiv:2504.1388 2025.
12

---

[26] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
[27] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484 2016.
[28] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models, arXiv preprint arXiv:2402.00035 2024.
[29] Ailin Deng, Zhirui Chen, and Bryan Hooi. Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding. arXiv preprint arXiv:2402.15300 2024.
[30] Wenbin An, Feng Tian, Sicong Leng, Jiahao Nic, Haonan Lin. Qian'ing Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vision-language models with assembly of global and local attention. arXiv preprint Stt::2406.127188 2024.
[31] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models, arXiv preprint arXiv:2310.0077 2023.
[32] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object ballucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[33] Qwen Team. QwQ: Reflect deeply the boundaries of the unknown, November 2024.
[34] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In Advances in Neural Information Processing Systems 37, pages 95095-95169, Vancouver, Canada, 2024.
[35] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-leve bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 3828-3850, Bangkok, Thailand, 2024.
[36] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.
[37] OpenAl. Introducing GPT-4.1 in the api, 2024.
[38] OpenAl. Hello GPT-4o, 2024.
[39]
Anthropic. Claude 3.7 sonnet and claude code. 2025.
[40] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024, pages 24185-24199 Seattle, WA, 2024.
[41] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In Proceedings of the 18th European Conference on Computer Vision, pages 370-387, Malmö, Sweden, 2024.
[42] Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with multimodal large language models. International Journal of Computer Vision 133:825-844 843, 2024.
13

---

[43] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A simessensitvee multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition 2024, pages 14313, Seattle, WA, 2024.
[44] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia. Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models, In Advances in Neural Information Processing Systems 35, pages 14824-24835 New Orleans, LA, 2022.
[45] Wei Liu, Junlong Li, Xiwen Zhang, Fan Zhou, Yu Cheng, and Junxian He. Diving into self-evolving training for multimodal reasoning. arXiv preprint 1/Xiv:2412.1455,, 2024.
[46] Weiyun Wang. Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:241 1.0442, 2024.
[47] Wen Huang, Hongbin Liu, Minxin Guo, and Neil Gong. Visual hallucinations of multi-modal large language models. In Findings of the Association for Computational Linguistics 2024, pages 9614-9631, Bangkok, Thailand, 2024.
[48] Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Shishir G Patil, Matei Zaharia, Joseph Gonzalez, and Ion Stoica. LLMs can easily learn to reason from demonstrations structure, not content, is what matters! arXiv preprint arXiv:2502.07374 2025.
[49] Keinosuke Fukunaga and Donald M. Hummels. Leave-one-out procedures for nonparametric error estimates. IEEE transactions on pattern analysis and machine intelligence, ((()):21-425)
1989.
[50] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in lims. arXiv preprint arXiv:2402. 14740, 2024.
[51] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (tti:::7777.34777 2017.
[52] Qiying Yu, Zheng Zhang, Ruofei Zhu Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang. Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source 11m reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476. 2025.
[53] Scott Fujimoto, David Meger, and Doina Precup. Off- deep reinforcement learning without exploration. In Proceedings of the 36th International Conference Machine Learning, pages 2052-2062, Long Beach, CA, 2019.
[54] Herman Kahn and Andy W Marshall. Methods of reducing sample size in monte carlo compu- tations. Journal of the Operations Research Society of America, (5):263-278) 1953.
[55] Reuven Y Rubinstein and Dirk P Kroese. Simulation and the Monte Carlo method. John Wiley & Sons, 2016.
[56] Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, and Philip Torr. A systematic survey of prompt engineering on vision-language foundation models. arXiv preprint arXiv:2307. 12980, 2023.
[57] Jiaqi Liao, Yuwei Niu, Fanqing Meng, Hao Li, Changyao Tian, Yinuo Du, Yuwen Xiong, Dianqi Li, Xizhou Zhu, Li Yuan, et al. Langbridge: Interpreting image as combination of language embeddings. In Proceedings of the I 19th International Conference on Computer Vision, Honolulu, HI, 2025.
14

---

[58] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:241 15124, 2024.
[59] Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol ol-like llms. arXiv preprint arXiv:2412.21187 2024.
[60] Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405. 2024.
[61] Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wildvision: Evaluating vision- language models in the wild with human preferences. arXiv preprint arXiv:2406. 1069, 2024.
[62] Qwen. QVQ: To see the world with wisdom. 2024.
[63] Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, and Ji- Ji-Rong Wen. Virgo: A preliminary exploration on reproducing ol llike mllm. arXiv preprint arXiv:2501.0190 2025.
[64] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: massive ditliiidssiiiieee multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024, Seattle, WA, 2024.
[65] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning foundation models visual contexts. In Proceedings of the 12th International Conference Learning Representations, Vienna, Austria, 2024.
[66] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-We Wei Chang, Yu Qiao, et al. Mathverse: Does your multi- Ilm truly see the diagrams in visual math problems? In Proceedings of the 18th European Conference on Computer Vision, pages 169-186, Malmö, Sweden, 2024.
[67] Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. In Proceedings of the 13th International Conference on Learning Representations, Singapore. Singapore, 2025.
[68] Peijie Wang, Zhongzhi Li, Fei Yin, Dekang Ran, and Chenglin Liu. Mv-math: Evaluating mul- timodal math reasoning in multi-visual contexts. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition 2025, Nashville, TN, 2025.
[69] Spyridon Mouselinos, Henryk Michalewski, and Mateusz Tomasz Malinowski. Beyond lines and circles: Unveiling the geometric reasoning gap in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6192-6222. Miami, Fl., 2024.
[70] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35, New Orleans, LA, 2022.
[71] Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under kl-constraint. In Proceedings of the 4 sss International Conference on Machine Learning, Vienna, Austria, 2024.
15

---

[72] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.55111 15115 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we have comprchensively summarized and presented our methods and contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our proposed method in Appendix G.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
We analyze the approximation of important sampling and prove it in Ap-
pendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We have provided comprchensive experimental details in the appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer:
[Yes]
Justification:
We will release the ppen-source modela and training script immediately this
paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
Beliiii/guides//oeeSuuaaae soibboiisin for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CCodeSubmiis
httpps.cccpubiic///ecc S ssi wwwinnsssonniiiiiieoeiooo sesionnyocy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We give all experimental settings and details in Section 5 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We have validated the effectiveness of our method through large number of experiments in Section 6 and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided comprehensive experimental details the Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / htt////nurrp...eee httpp//ttss.cppppiit///////////t detp://///ittttt............eee.
ctcuidel ines?
Answer: [Yes]
Justification:
Our research complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impacts of our proposed method in Appendix G
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our method does not involve these risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have provided detailed citations for the datasets and other models used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release the dataset immediately this paper is accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is the standard component in this research and we only use it for writing, editing, or formatting purposes.
Guidelines:
The answer NA
that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https ht/p//rrps..cc/oofeeeeeeeeeee / for what should or should be described.
23

---

The appendix is organized as below:
Appendix presents the proof of lemma related Eq.
Appendix B describes the detailed settings of SOPHIA and GRPO.
Appendix explains the detailed evaluation setup.
Appendix D shows in-depth analysis on warm-up, response style, involving caption data, data
composition and keep-NV strategy.
Appendix E introduces all prompts use.
Appendix discusses the limitations SOPHIA.
Appendix G discusses the broader impacts of this research.
A Theoretical Analysis
We first explore the properties of trajectories in off-policy dataset D.
Lemma A.1. There exits set y such that Vy $ V, either ply)ānn v) or R( 0,9) O.
Proof. Let the set y be defined as
y (Z v, D, R(E.g v. =
(A1)
We aim to show that for any trajectory y & Y and any context (ă.g v, xg), either (zy(d, = or R(zq. U, y) O. Consider arbitrary trajectory y such that y V, and an arbitrary context (ă,, v, 2,). (3/ (t2) = 0, the condition of lemma is immediately satisfied.
Alternatively, assume मt (g)(d))) > O. We then need to demonstrate that R(æ.. n,,) 0. If the specific tuple (I v. y) (for which we assume F ( > 0) is indeed sample in D. Then, because & }. by the definition of Y) this sample (æ,, v, y) cannot satisfy Rax, v, y) 1. If it did,
would be in V. which contradicts our premise. Since the reward R((-) can only or 1, it follows that W, 0, y) O.
This completes the proof.
The proof gives a special subset of all trajectories D, i.e., = (y v, v,y) D, R(xgg v. y) = 1}, whose property results from appropriate sampling strategy and reward design. Thus, Assump- tion A.2 holds. This assumption ensures the coverage (n(Y (.g v) > 0) and that ny and f do not deviate significantly from each other within y
Assumption A.2. Vy ),, p v) > 0, and the following boundedness condition holds
To To xg. v) M Fqq
- 8
(A2)
where 8 0 is a small constant.
Finally, we present Lemma A.3 to address aforementioned issues.
Lemma A.3. Given the expectation using IS and without (i.e., treating the ratio as 1)
Tee Teyy v) R1 R( v. v.
Gıs B( the bias from ignoring IS is bounded by |G1s G1 < 5.
G E(a [R( v, v,y)l
(A3)
plyl Bg. v)
Proof. The difference between Gis and G1 is
( ( xgg v.)
IGis Gil =
18 ()))
R(x v v)
(1 (x.g. 1))
(A4)
To / D.y 2)
E D
|R v, v)
1 Øgg v)
Since y & y contributes nothing to the expectation, we only need to consider y E y. Using an indicator function 1 1 ))). have
To TT x. v) p v)
Gis G1 < E (
1(y E ))
RR(ag y)
(AS)
24

---

Applying the bounds [R(2g. g. v. u)1 I and Assumption A.2, we have
Gis G1 x E w.) [ 3/88 1] 6E ".))) (1yy [1 3)]
(A6)
This completes the proof.
Lemma A.3 shows that, under the above assumption, directly approximating the off-policy objective using the average reward introduces only a small and controllable bias.
B Implementation Details
B.1 SOPHIA Setup
Algorithm. The reinforcement learning training procedure shown Algorithm To implement the proposed, we perform single round of policy update after collecting all trajectory samples, aiming to ensure training stability, improve computational efficiency, and mitigate excessive distributional shift when captioning images.
Algorithm 1 The Semi-off- Policy Reinforcement Learning Algorithm
Inputs: Training dataset Durinn policy model (LVLM) wa. reasoning language model म, number of rollouts per image K, number of rollouts per caption N.
2: Initialize policy Te and Section 5.1.
3: for 7 E Dimin do
4: Obtain its image tokens v.
5: Generate K caption samples {c(1), c(22),... c(()) where (kk) m((l v).
6: for k = 1,2, ... K do
7:
Construct prompt b based on (kk), Tg.
8: Sample N reasoning trajectories (1) (2) ;.. y( where (()) a (
9: Calculate the reward of each trajectory with Eq. (4).
10: end for
11: Calculate the reward of each caption with Eq. (5).
12: end for
13: Construct the off-policy dataset D with their reward based Eq. 6.
14: Update ne with Eq. (7).
15: Return: The optimized policy model n²
KL Regularization. Some earlier studies on RLHF [70, 71] indicate that incorporating a KL regularization term helps prevent the model from overly prioritizing reward maximization, which could otherwise compromise the linguistic patterns acquired during pretraining. However, recent research on on-policy RL--based methods shows that omitting the KI. regularization term during fine-tuning an instruction-tuned model reduces computation and increases response length [20]. Considering that the policy initialization is instruction- model, follow this approach.
Hyperparameters. During reasoning-rewarded sampling, we set K N = 8. The max length of each caption and rollout trajectory is 32678 tokens. During policy updating, the threshold of caption reward a = 0.75, and only the weights of language backbone of the LVLM are unfrozen. The policy model is trained with batchsize of 512, learning rate of 2 10 weight decay of 0.05 and employs a cosine annealing learning rate schedule, decaying to 1/4 of the initial learning rate over time. We optimize the policy model using the AdamW optimizer. When training, apply different system prompts to distinguish data from different sources (see Appendix E for more details).
B.2
GRPO Steup
Data Preparation. All data used in this section are drawn from our VQA training set, without incorporating any general visual or text-only training data. Each question independently sampled 16 times using the Intern VL3.0-38B model to estimate its correctness rate. To improve the effectiveness of on-policy RI., we retain only questions with correctness rates between 0.4 and 0.6, filtering out those that are too easy too difficult.
25

---

Training Details. We adopt Qwen2.5-72B In--ructt as the generative verifier and use a binary outcome reward signal for GRPO. During training, each batch contains 64 questions, with 8 rollouts per question and maximum trajectory length of 16384 tokens. The correctness scores across rollouts are averaged to compute pass rate; questions with a pass rate of exactly 0 or are excluded, using 0.5 as the threshold for incorrectness. The policy model is trained with learning rate of 5 x 10- with all other settings consistent with Appendix B.
B.3 Computational Costs
During the full training process, the computational cost of training the Intern VL.300-38B model serves as the baseline. Under the same data scale, the proposed GRPO-based training consumes roughly three-quarters of that baseline's total computation. For inference, the SOPHlA-trained model follows the same algorithmic procedure as the base LVLM (i.e., next-token prediction). However, due to its slow-thinking and step-by-step reasoning design, requires longer reasoning sequence to reach the final output. Empirically, the inference process of SOPHIA is about 3-4 times slower than that of the base LVLM on standard reasoning benchmarks. This moderate increase aligns with the "test-time scaling" paradigm in reasoning models
C Evaluation Details
Benchmark. To comprehensively evaluate the performance of model, our benchmarks include
college-level question, math-related question and challenging scientific reasoning. The specific split of datasets is shown as below.
MMMU [64]: the valiation split of MMMU dataset.
MMMU Pro [64]: the vision split of MMMU Pro dataset, where the input image contains both the
visual content and the question, while the text query excludes the question.
Math Vista [65]: the testmini split of Math Vista dataset.
Math Verse [66]: the testmini and vision-only split of Math Verse dataset.
DynaMath [67]: the full test set of DynaMath dataset.
Math Vision [34]: the full test set of Math Vision dataset.
MV-MATH [68]: the full test set of MV-MATH dataset.
OlympiadBench [35]: the full test set of OlympiadBench dataset, including visual and texture questions and excluding all proof questions.
Metrics and Evaluation. We use pass@1 I accuracy as the metric for evaluation under the zero-shot setting. Since greedy decoding may lead to performance degradation and endless repetitions for reasoning model, models are evaluated with temperature of 1.0, top-p of 0.8 and the output token length of 16384. To ensure optimal performance for all models, response hint is added before the original query the benchmark provides it. For all benchmarks across models, answers are extracted using Qwen2.5-722 5-725----7ttccc [72] with greedy decoding (see Figure A5 in Appendix E for the extraction prompt). For benchmarks where answer formats are difficult to match with rules (i.e., MMMU, Math MathVerse Verse and MV-MATH), answers are evaluated using Qwen2.5-725 1ee.2..577B-Instteeee InIstrcctt (see Figure A6 in Appendix for evaluation prompt); for others, verifier from OlympiadBench is used for strict equivalence checking.
D Additional Experiment on SOPHIA
Response Style and Teacher Model. For the behavior policy, we evaluate different slow-thinking reasoning styles and their impact on SOPHIA performance. As shown in Fig.AI, the QwQ Preview tags. while the style presents the full reasoning process without summary reasoning from summary. Experi- DeepSeek-RI style is more decisive and uses these tags to separate ments are conducted Results are shown in Table A1. We observe no significant differences between two different models in terms of quality (DeepSSee--RI focus at trajectories should be on the correctness and logical consistency of trajectories rather than style or format, as well as SOPHIA can achieve strong performance without an extremely powerful teacher model. In practice, we combine both response styles: QwQ Preview is used for simpler questions, while DeepSeek-F RI handles more complex ones.
Intern VI.2.5-38B and additional general visual textual data used.
MivQ-32--Previee
and size (658B 32B), indicating that the
26

---

Aleighs For you this gronvery probive hore. Lat's me it's ashog The quertion is about mssgll ABC, and mentionn tuo ongle ilsectoos OR and OC, which ustervect al poiee O it gives we that ungic A 216 dogress, and need to Sid the wousare of ongle BOC. The choicss or 135", 140", 145 15P.
cstint-
Okny let's tucele this problew sup by sup, The question tr: Sub- mart alf lagg rubler spherer Satevactal all big shony cyllnders How many odjects are leff? Hom first I Isod idcutit all the objects the spheres, shits cysisdors and
and culleuriie them into long ralber
Firnt nood moke moee unterston wrap. kow gli ABC, ungles at wertiees A. R, onglc Mirectoss of angles Bl and C. respecmely, and they weer- veet or point O inzide the trisngle.
OC are
Looking the image diseription: therw's goon lurge rubler sphere, perple lorge rabber cilindrr a gray larg mesal cahe, yellene large rabier coke. a yellow lorge secaa cylinder sad perple malt rabber sphere.
Hait a eniusse angic BOC &s the.. Wait minute angie BOC the... Alternatieyy
propartic...
Mall, make sare dide's mis objects deserigiio sts so yeu, Subtracting the two scoutonne odjects giver So atswer should be <Abinnt>
So, the answer should be 145", which coonspoonds to sprine C.
chhtt that's solid Pore ased can different mothads to arrive of the mome aurave, which giver confideore in the mead
To deteviine how muny objects are left ofter rabonarring the lorge rabler yheer and the shicy cfttnaer follow these suppe... **/nswor *0 4
"nPiaal Aupuget (ibores/()) ()]
(a) QuQ Previow Style
(b) DeepSeck-RI Style
Figure A1: Examples of different response styles. (a) is the QwQ Preview response style and (b) is the DeepSeek-RI response style.
Caption as Training Data. SOPHIA generates a
Methed
MMMM
MorthNiste
MathViddn
(lymptottivnch
Average
large number of image-related captions during sam-
pling. We explore whether incorporating these cap- tions into model fine-tuning improves performance. Experiments are conducted on Vn2e5mVL.2.5-38B. us-
UxU Phestee Deeplock-111
65.49
67:20
4121
Table Al: Ablation study of the response style for SOPHIA. "QwQ Preview" represents the
ing only captions with reward score of R(c)= = 1.0,
QwQ Preview response style (Figure Al (a)) and "DeepSeek-R1" is the DeepSeek-R1 re-
without adding any additional general visual or tex- tual data. As shown in Table A2, "SOPHIA DI" refers to training data composed of the query, image, and caption. "SOPHIA H" denotes a setting that the sampled caption is prepended to the reasoning trajectory and connected by "An)m to construct a new trajectory for policy optimization. We observe that incorporating captions offers significant benefit, and placing them before reasoning trajectories can even hurt performance. This degradation stems from two factors: 1) captions consume part of the output token budget. 2) their distribution diverges from that of reasoning trajectories. Moreover, reasoning trajectories already embed the necessary visual understanding. Thus, caption data is not incorporated into the training dataset.
sponse style (Figure A1 (b)).
Data Composition. To examine the interaction be- tween different data sources, we incorporate gen- eral visual data (GV) and pure textual data (Text) during policy updates. We conduct experiments on InternVL..55333 VL2.5-38BB and the results are summarized in Table A3. In the table, "GV" and "Text" indicate the general visual and textual data, respectively, while "RC" represents the trajectory and update strategy from SOPHIA. A / denotes that the corresponding data type is used in the experiment. We observe that text trajectories alone can provide moderate improve- ments in reasoning abilities; however, the absence of visual understanding limits their effectiveness. Intro- ducing general visual data or SOPHIA alongside text leads to further gains, with the best results achieved when all three sources are combined.
Mathoel
MMMU
Math
MathNidden
(kyngtadllechh
Anarage
61.2
69.50
31.32
23.86
40.25
SUPHAR SOPHIA
6539
67.00
40.0
14.27
63:44
78.00
42.73
41.86
55.15
Table A2: Ablation study of caption as train- ing data for SOPHIA. "SOPHIA DI" repre- sents training data composed of the query, image, and caption. "SOPHIA H" represents that sampled caption is prepended to the rea- soning trajectory to construct a new trajectory. Best value is in bold.
Keep-N. In our trajectory selection, we prioritize the shortest valid trajectory, which effectively implements a Keep-1 strategy A Keep-N strategy selects the top-N shortest trajectories for each image-query pair, as long as they are correct and their caption rewards exceed the threshold. This approach allows up to N high-quality trajectories to be used for policy updates. This strategy may enhance learning in visual scenarios, and we evaluate its effectiveness on Intern Intern VL2.5-38B. Here, no additional general visual or textual data are introduced. Results are presented in Table A4. We find that the Keep-N strategy can improve model performance some extent, e.g., on OlympiadBench and Math Vision. But the gains are marginal relative to the overall model abilities and come at
'Due to the length of trajectories and the use of sampling, duplicates are virtually impossible
27

---

Training Data
MMMU
Math Vista
MathVision
OlympiadBench
Average
GV
Text
RC
59.67
69.20
31.28
24.95
28
66.56
73.90
42.86
44,66
57.00
70.90
42.73
41,46
55.13
67.78
71.90
43.62
44.84
$7.04
65
72.00
42.89
46.05
56.60
66.33
74.70
47.96
49.22
59.55
Table A3: Pass accuracy of SOPHIA 38B scalre and baselines. "GV" and "Text" indicate the general visual and textual data, respectively, while "RC" represents the trajectory and update strategy from SOPHIA. A / denotes that the corresponding data type is used in the experiment.
College-level Question
Math-related Question
Challenging Scientilic Reasoning
Method
MMMU
DynaMalb
OlympiadBench
Average
Keep-1
46.93
70.00
45.18
59.14
42.73
34.15
41,45
50.74
Keep-2
64.22
45.83
72.40
44,67
60.34
42.82
30.81
43.91
50.63
K
61.56
45.61
72.00
45,95
61,22
44,77
31,71
45.71
51.45
Table A4: @ass@) accuracy of SOPHIA on InternVL 2.5 38B in different N (N = 2,
higher computational cost. Therefore, considering the trade-off between performance and efficiency, adopt the Keep-1 strategy.
E
Prompt
Image Caption Prompt. Figure A2 is the prompt for describing images to provide comprehensive captions including objects, color, text and so on. We do reveal the question associated with the image to the vision model, as doing so may prompt it to answer the question directly, leading to potential reward hacking when evaluating captions.
Simulating Visual Reasoning Prompt. Figure A3 is the prompt for enabling reasoning model to simulate LVLM that see the image based on provided caption. We build sample responses for the model, and they are typically generated automatically and then refined manually with additional visual operations (e.g., "look back"), which enables in-context learning that produces replies with consistent formatting and greater suitability for visual tasks.
Thouugh Reasoning. Figure A4 is the system prompt for the slow-thinking LVLM.. During training, we use this system prompt when the trajectory contains <thinks..<</thinn tag: otherwise, the default system prompt is applied. This approach helps the model adapt to data from different distributions. At inference and evaluation time, we consistently apply this system prompt to our model.
Slow-thinking
Answer Extracting Prompt. Figure A5 is the prompt for extracting answer from the response. Examples are provided to prevent the LLM from answering questions instead of extracting the original answer.
Answer Evaluation Prompt. Figure A6 is the prompt for evaluating whether the answer is correct.
F
Limitations
Despite the effectiveness of SOPHIA in enhaneing visual slow-thinking reasoning, several limitations remain. First, LVLMs still struggle to retain long-range visual dependencies, especially in multi-hop or complex reasoning tasks, which can lead to partial or fragmented understanding of intricate visual scenes. Second, the current visual encoder lacks sufficient fine-grate -ha-iid recognition capability for complex scenes. We acknowledge that some low-level features, such as gradients and pixels, are not captured by semantic representations, since they are largely independent of semantics, which restricts the fidelity of visual perception and, consequently, the precision of downstream reasoning tasks that rely heavily such features. Third, although the reward design in SOPHIA effectively filters out flawed trajectories, persistent issues such as hallucination and reasoning redundancy are not fully resolved, evidenced by occasional overthinking repetitive outputs in challenging
28

---

benchmarks. Addressing these challenges in future work could involve integrating stronger visual grounding techniques, adaptive trajectory pruning, and multi-stage curriculum learning further improve long-range visual reasoning and visual fidelity.
G Broader Impacts
This paper presents SOPHIA, a semi-off offiioff-policy reinforcement learning framework which advances vision- lision-language models by improving both visual understanding and reasoning without relying human or closed-source annotations. By enabling scalable, automated training and providing open evaluation benchmarks, SOPHIA supports the development of more reliable and generalizable AI systems for complex multimodal tasks. We hope our open resources and methods will foster further research and broader real-worke impact, particularly in education, science, and assistive technologies.
29

---

Image Caption Prompt:
Provide highly detailed and precise description of every visible element within the image, capturing all visual and textual aspects exactly as they appear.
For images containing text, thoroughly extract and transcribe all visible words, symbols, numbers and time, including labels, annotations, legends, and numerical values. Pay close attention to font style, size. color, and positioning of each text element. Describe any formatting or emphasis, such as bold italic text, and note any alignment or proximity between text elements.
For the connections between nodes, describe the type of line arrow used, its direction, and any associated labels, symbols, or annotations that clarify the meaning of the relationship. If there are multiple types of connections, make sure to distinguish between them and describe how they are represented visually (e.g., solid vs. dashed lines, different colors, or arrowheads). Provide context for the meaning of each connection, especially in terms of the relationships or interactions they represent between the nodes.
If the image includes any type of statistical chart or graph, such as bar charts, line charts, scatter plots, pie charts, any other type of diagram, describe each element in detail. For geometric shapes, lines, points, or bars, specify their labels, colors, positions, and exact measurements values when visible. For bar charts, include each bar's label, color, height or value, category labels, legends, and axis labels, carefully noting the numerical scale on the and any increments intervals. For line charts, detail the data points, lines, their color, and any markers or trends, indicating the specific values at key positions, note that every data point should be interpreted with numerical value. For scatter plots, describe each point's position, color, and any patterns or clusters. For pie charts, describe each segment's s label, color, proportion, and the percentage value represents. Always ensure that any numerical scale, axis labels, and unit measurements are clearly noted, including any legends or other graphical annotations that may aid in understanding the data.
In complex scenes, describe every object in exhaustive detail, thoroughly extract and transcribe all visible words, symbols, numbers and time, noting its color, texture, shape, size, and relative position within the scene. Start with the objects in the foreground and gradually move towards the background, maintaining clear and logical sequence. Provide information on spatial arrangements, including any overlapping, layering, or alignment among objects. Mention any shadows, highlights, lighting effects that influence the appearance of objects. When describing the background, include its color, patterns, or gradients, and any additional visual elements that contribute to the scene, ensuring that the focus shifts progressively from the front to the back. This methodical, from-from front-tooooooo boock approach will help provide a coherent and structured visualization of the scene.
If the image depicts realistic human figures, provide detailed description of each individual's posture, physical appearance, clothing, facial expressions, and interactions with the surroundings. If the image includes well-known figures, make every effort to identify them by their facial features, attire, or contextual clues. identification uncertain, describe potential characteristics or identity hints in detail.
Lastly, aim to describe the image as comprehensively possible, ensuring no detail is overlooked. Capture every nuance in the image, providing an accurate, in-depth account of each element and its precise attributes, arrangement, and interactions within the scene.
Figure A2: Prompts for comprehensively captioning images.
30

---

Simulating Visual Reasoning Prompt:
Please read the following example carefully.
<Begin of example>
Question: (hhe example of visual question)
Visual information:
(The example of image caption]
Your response:
(The example of response from reasoning model)
<End of example>
Offer comprehensive breakdown of your analytical process based the given visual information, detailing each step, the reasoning behind your decisions, and how you integrated various pieces of information. If you find you overthinking this, you can stop analysis and determine the final answer. If you can directly find answer from visual information, just copy the original text, and no need to explain what is the meaning of the text or provide more context. Present your final answer at the end after "#nal a and wrapped with boxed( J. You should assume that you can see this image, and you must use 'image" refer to the visual information.
Question:
(Provided visual question]
Visual information:
[Provided image caption)
Your response:
Figure A3: Prompts for simulating LVLM to answer questions.
31

---

System Prompt:
You expert mathematician with extensive experience in mathematical competitions.
You approach problems through systematic thinking and rigorous reasoning. When solving problems, follow these thought processes:
## Deep Understanding
Take time to fully comprehend the problem before attempting solution. Consider:
What the real question being asked?
What are the given conditions and what do they tell us?
Are there any special restrictions or assumptions?
Which information is crucial and which is supplementary?
## Multi Multi-angle Analysis
Before solving, conduct through analysis:
What mathematical concepts and properties are involved?
Can you recall similar classic problems or solution methods?
Would diagrams or tables help visualize the problem?
Are there special cases that need separate consideration?
## Systematic Thinking
Plan your solution path:
Propose multiple possible approaches
Analyze the feasibility and merits of each method
Choose the most appropriate method and explain why
Break complex problems into smaller, manageable steps ## Rigorous Proof
During the solution process:
Provide solid justification for each step
Include detailed proofs for key conclusions
Pay attention to logical connections
Be vigilant about potential oversights
## Repeated Verification
After completing your solution:
Verifyy your results satisfy all conditions
Check for overlooked special cases
Consider if the solution can be optimized or simplified
Review your reasoning process
Remember:
1. Take time to think thoroughly rather than rushing to answer
Rigorously prove each key conclusion
3. Keep an open mind and try different approaches
Summarize valuable problem-sol soovenn-oovvng methods
5. Maintain healthy skepticism and verify multiple times
Your response should reflect deep mathematical understanding and precise logical thinking, making your solution path and reasoning clear to others. When you're ready, present your complete solution with:
Clear problem understanding
Detailed solution process
Key insights
Thorough verification
Please put your thinking process within </hink>..</hhnk>> tags.
Provide answers in the same language as the user asking the question. You have ##32768**
tokens to complete the answer.
Figure A4: The system prompt for long CoT reasoning.
32

---

Answer Extracting Prompt:
Please read the following example. Then extract the answer from the model response and type it at the end of the prompt.
Hint: Please answer the question requiring an integer answer and provide the final value, c.g.,
1, 2, 3, at the end. Question: Which number is missing?
Model response: The number missing in the sequence is 14.
Extracted answer: 14
Hint: Please answer the question requiring floating-point number with one decimal place and provide the final value, e.g., 1.2, 1.3, 1.4, the end. Question: What is the fraction of females facing the camera?
Model response: The fraction of females facing the camera is 0.6, which means that six out of ten females in the group are facing the camera.
Extracted answer: 0.6
Hint: Please answer the question requiring Python list as an answer and provide the final list, e.g., [1, 2, 3]. [1.2, 1.3, 1.4), at the end. Question: Between which two years does the line graph saw its maximum peak?
Model response: The line graph saw its maximum peak between 2007 and 2008.
Extracted answer: [2007, 2008]
Hint:
(Provided hint}
Question: (Provided question)
Model response: (Provided response from the model)
Extracted answer:
Figure A5: Prompts for extracting the answer from response.
33

---

Answer Evaluation Prompt:
Below answers to math question. Question [Question], [Standard Answer] the standard answer to the question, and [Model_ane _answer] is the answer extracted from model's output to this question. Determine whether these two answers are consistent.
Please note that only when the [Model_add answer] completely matches the [Standard Answer] means they are consistent. For non-multiple-choice questions, the meaning expressed in the same way, it is also considered consistent, for example, 0.5m and 50cm. they are consistent, Judgement I; if they are different, Judgement is
[Question]: Write the set of numbers represented on the number line in interval notation.
[Standard Answer]: (-2,1]
anddee]:: Extracted Answer: ((-2, 1))
Judgement: 0
[Question]: Find the domain and range of the function f using interval notation. [Standard Answer]: domain: [-4, 0) and range: (-3, 1]
MModel_answert Range: ((-4, 1])
Judgement: 0
[Question]: Given the graph of the ellipse that intersects with x-axis at 9 and -9 and with y-axis 3 and -3, determine its equation.A. [Standard Answer]: A =² Judgement: कT + #
I x² डT
+
B. Can not determine.
[Question]:
[question]
[Standard Answer]: (gt)
[Model _ander]_ : [extraction]
Judgement:
Figure A6: Prompts for evaluating whether the answer is correct.
34


Reviews:
[{"review_id": "AmhmuTPlVu", "reviewer": "Reviewer_xBLR", "summary": "This paper proposes a semi-off-policy reinforcement learning framework to\nenhance slow-thinking reasoning in large vision-language models. The approach\nworks by combining on-policy visual captioning from a trainable LVLM with\noff-policy slow-thinking trajectories generated by a reasoning LLM. Visual and\nreasoning rewards are computed and used to fine-tune the LVLM using off-policy\nRL. Experiments on different benchmarks demonstrate the effectiveness of the\nproposed method.", "questions": "See Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\n* The studied problem is important and interesting.\n\n* The motivation for improving the on-policy and off-policy LLM RL is clear.\n\n* The proposed method outperforms supervised fine-tuning and direct on-policy RL methods\n\n# Weaknesses\n\n* A central assumption in this work is that image content relevant to reasoning\ntasks can be effectively captured through textual descriptions. While this may\nhold for certain structured scenes, it overlooks a fundamental difference\nbetween modalities: images are continuous and often encode fine-grained spatial\nand visual information that discrete language space may fail to represent fully.\nRelying entirely on language-based captions generated by an MLLM risks omitting\ncritical visual cues, especially in tasks where spatial relations or low-level\nvisual details influence the reasoning outcome. This paper would benefit from a\nmore explicit discussion of this and an analysis of how potential information\nloss in the captioning step might affect downstream performance.\n\n* The current formulation of the proposed method appears particularly\nwell-suited for tasks in which visual inputs already contain structured symbolic\ncontent, such as text-labeled diagrams or schematic illustrations. In these\ncases, the conversion from image to language retains most of the task-relevant\ninformation, enabling the reasoning LLM to perform effectively. However, for\nscenarios that require more perceptual visual understanding, e.g., interpreting\nnatural images, or abstract style and spatial patterns, this kind of assumption\nmay no longer hold. This limitation is also reflected in the choice of\nevaluation benchmarks: most of the datasets used (e.g., MathVision,\nOlympiadBench, DynaMath) focus on math, science, or diagram-based reasoning\ntasks where textual abstraction is relatively reliable. There is a lack of\nevaluation on more diverse, perception-centric benchmarks that stress visual\ngrounding beyond captionable content.\n\n* The performance gains achieved by SOPHIA are largely the result of\nincorporating reasoning trajectories from stronger language-only models such as\nQwQ-32B-Preview and DeepSeek-R1. Since QwQ and R1 has much stronger inherent\nreasoning capabilities than InternVL, part of the improvement may come simply\nfrom distilling a more capable teacher. While this is a valid and useful\nstrategy, it complicates the interpretation of the results: it is difficult to\ndisentangle the contribution of the semi-off-policy framework itself from that\nof the teacher model’s strength. An ablation study using different teacher\nmodels of varying quality or complexity could help clarify this point.\n\n* Another limitation relates to the training data used to support the method’s\nperformance. A significant portion of the results is based on a private dataset,\nwhich is not released or described in sufficient detail. While the authors take\ncare to evaluate SOPHIA on public benchmarks as well, the strongest gains appear\non the internal dataset, whose construction, domain coverage, and difficulty\nlevel remain unspecified. This makes it challenging to fully assess the\ngenerality of the results or to reproduce them independently. More transparency\non dataset characteristics or representative examples would strengthen the\ncontribution.\n\n* Additionally, the entire experimental study is conducted on InternVL models\nonly, across both 8B and 38B scales. While InternVL is a strong baseline, it\nwould be valuable to test the framework on other MLLM families with different\narchitectures or pretraining approaches. \n\n* What is the time and computation resouces overhead from the proposed method?", "was_revised": true, "final_justification": "Most of my concerns are addressed.", "rebuttal": "We thank the Reviewer xBLR for your comprehensive feedback. We are delighted to answer reviewer's concerns below to refine our paper.\n\n> **Review**: The paper assumes that textual descriptions can fully capture image content for reasoning tasks, but this may overlook fine-grained visual details inherent in images, potentially leading to information loss that warrants further discussion and analysis.\n\n**Response**: Our work builds upon the LLaVA architecture LVLM, which aligns visual features with language semantics.  Some studies [1] show that adapter layers project visual embeddings into subspaces spanned by corresponding text embeddings, supporting our core assumption. While converting images to language inevitably loses some low-level details, some elements such as color, shape, texture, and layout have natural language annotations and can be implicitly encoded within the semantic representations during pretraining [2, 3]. \n\nTherefore, as shown in Figure A2 (page 22), we can use deliberately designed prompts to guide the LVLM to obtain the most comprehensive visual descriptions covering object color, texture, shape, size, spatial relationships, foreground/background, and object interactions within the scene. The \"critical visual cues including spatial relations or low-level visual details\" you mentioned are, for the most part, addressed by our instructions. \n\nOur experiments also show that the model achieves noticeable improvements in spatial reasoning and visual understanding (see our second response for more details). We will further revise the paper to discuss this issue and provide a more detailed analysis of how potential information loss in the captioning step may affect downstream performance.\n\n> **Review**: The proposed method excels at tasks involving structured, text-labeled visual inputs where image-to-language conversion preserves essential information, but its effectiveness is limited for perception-driven scenarios, as reflected by the evaluation's focus on diagram-based rather than perceptual benchmarks.\n\n**Response**: We would like to emphasize, as we mentioned in response to the previous question, that our framework is also capable of capturing some low-level visual features by the deliberately designed prompt. Some benchmarks we evaluated in this work also require perception abilities of low-level visual features. For example, in MMMU Pro, many subsets such as art, history and design comprehensively assess visual grounding and perception, on which SOPHIA improves InternVL 3.0 38B pass@1 acc from 46.47 to 52.60. Additionally, the MathVision you mentioned contains numerous spatial reasoning tasks (e.g., identifying the side view of a given cube in combinatorial geometry). And on this kind of tasks, the untrained InternVL 3.0 38B achieves an average of 21.75 pass@1 acc,  while the SOPHIA-trained model reaches 32.79, showing a significant improvement. We will explore more perception-centric visual slow-thinking reasoning in future studies.\n\n> **Review**: The observed performance gains of SOPHIA may be primarily due to distillation from stronger language-only teacher models rather than the semi-off-policy framework itself, making it challenging to isolate the framework’s true contribution without further ablation studies using teachers of varying quality.\n\n**Response**: We agree that SOPHIA leverages the reasoning capabilities of strong language-only teacher models. However, as shown in our ablation studies (Table 5), semi-off-policy framework contributes independently to performance gains (line SOHPIA), above and beyond raw teacher strength (line SOHPIA w/o CR). We further explore different teacher models in Appendix C.3 (Table A2), finding that no significant differences between two different models in terms of quality (DeepSeek > QwQ-32B-Preview) and size (658B > 32B), indicating that SOPHIA can achieve strong performance without an extremely powerful teacher model.\n\n> **Review**: The reliance on an unreleased and insufficiently described private dataset, which accounts for the strongest results, limits the reproducibility and generalizability of the findings, highlighting the need for greater transparency regarding dataset characteristics.\n\n**Response**: Indeed, we use internal datasets to train the model, as currently available large-scale visual reasoning datasets are generally of low quality. Our dataset is composed of diverse and challenging multimodal questions from K12 and higher education, spanning science, math, finance, and social sciences, and is carefully curated to ensure a balanced difficulty distribution and prevent data leakage.\n\nWe also conduct experiments on public visual datasets, including MathV360K (mentioned in paper), M3CoT, llava-CoT, etc., but due to issues such as lower difficulty levels and limited diversity in question types in these datasets, the performance was not as strong as with our internal dataset. Despite these weaknesses of public datasets, SOPHIA still shows improvements (Table 6) and the public datasets are competitive with private datasets. To support reproducibility, we will release our internal training dataset upon acceptance.\n\n> **Review**:  Additionally, the entire experimental study is conducted on InternVL models only, across both 8B and 38B scales. While InternVL is a strong baseline, it would be valuable to test the framework on other MLLM families with different architectures or pretraining approaches.\n\n**Response**: Thanks for this advice. Our method is not tied to a specific architecture and can be applied to other LVLM families with minimal modification. To quickly validate its generality, we further conduct research on Qwen2.5-VL in both 7B and 32B scales with a subset of the training data without warming-up and mixing data. The results on four benchmarks are shown in the following tables.\n\n||MMMU|MathVista|MathVision|OlympiadBench|\n|---|---|---|---|---|\n|Qwen2.5-VL-7B|52.00|68.90|26.38|16.28|\n|Qwen2.5-VL-7B+SOPHIA|57.22|68.50|33.26|26.59|\n|Qwen2.5-VL-32B|65.33|72.80|38.98|38.92|\n|Qwen2.5-VL-32B+SOPHIA|68.22|73.70|43.68|41.99|\n\nWe can observe that SOPHIA is also effective for the QwenVL family. We will present related results after revision.\n\n> **Review**: What is the time and computation resouces overhead from the proposed method?\n\n**Response**: The sampling complexity of our method during the training phase is $O(MN)$, where $M$ represents the sampling size for visual understanding, and $N$ represents the sampling size for visual reasoning. During the entire training period, for the InternVL3.0-38B model we use, total GPU hours during the training phase are about 2048 hours on A800 GPU. Note that for the same data scale, the GRPO training requires about 1536 hours on A800 GPU.\n\n---\n**References**\n\n[1] Jiaqi Liao, et al. LangBridge: Interpreting Image as a Combination of Language Embeddings. In ICCV 2025. \n\n[2] Rui Xiao, et al. FLAIR: VLM with Fine-grained Language-informed Image Representations. In CVPR 2025. \n\n[3] Michael Tschannen, et al. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. arXiv 2502.14786."}, {"review_id": "xHPs1kAIN2", "reviewer": "Reviewer_oeeq", "summary": "This paper proposes SOPHIA, a semi-off-policy reinforcement learning framework designed to enhance slow-thinking reasoning abilities in large vision-language models (LVLMs). SOPHIA combines the on-policy visual understanding of a trainable LVLM with off-policy reasoning trajectories generated by a language model, enabling more robust and scalable learning. Through extensive experiments on InternVL2.5 and InternVL3.0 (at both 8B and 38B scales), SOPHIA achieves state-of-the-art performance on multimodal reasoning tasks, outperforming both open-source and some closed-source models.", "questions": "In general, I believe this paper is solid and the results are impressive. I have some questions that needs further clarification\n\n1. Can the authors provide qualitative examples or visualizations of improved visual grounding after SOPHIA training?\n2. What is the inference latency introduced by using SOPHIA compared to the base LVLM?\n3. Could SOPHIA be applied to other modalities or tasks beyond math reasoning?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "First of all, my research primarily focuses on 3D vision. RL and VLM are out of my expertise. Therefore, I can only provide a general assessment rather than insightful and critical review.\n\n**Strengths:**\n\n- SOPHIA’s semi-off-policy design smartly combines stable on-policy visual understanding with the flexibility of off-policy reasoning from a stronger language model. This setup addresses key issues like limited exploration in on-policy RL and hallucinations in off-policy RL.\n- The method shows clear performance gains on tough multimodal benchmarks like MathVision and OlympiadBench, outperforming other open-source models and even GPT-4.1 in some cases.\n\n**Weaknesses:**\n- Analysis on visual hallucinations: While SOPHIA aims to reduce hallucinations through visual reward propagation, qualitative examples or failure modes would be benficial to strengthen the claim.\n- Binary reward signal: As acknowledged in the limitation section, the reward mechanism is based on a simple binary outcome and a preference for the shortest correct trajectory. This is a coarse signal that may not capture the nuances of a high-quality reasoning process and could prematurely penalize more complex but valid reasoning paths.\n- Limited generalization in some cases: The results show that performance can degrade on certain tasks or data distributions not well-represented in the training data (e.g., MathVista for the 8B model). This suggests that while powerful, the method's robustness may be sensitive to the domain and quality of the reasoning trajectories it learns from.", "was_revised": true, "final_justification": "I appreciate the additional experiments and encourage the authors to incorporate them in the revision. My concerns are properly addressed.", "rebuttal": "We thank the Reviewer oeeq for your positive feedback. We are happy to address the reviewer’s concerns and will incorporate all feedback to improve the paper.\n\n> **Review**: Analysis on visual hallucinations: While SOPHIA aims to reduce hallucinations through visual reward propagation, qualitative examples or failure modes would be benficial to strengthen the claim. Can the authors provide qualitative examples or visualizations of improved visual grounding after SOPHIA training? (**Cons1&Q1**)\n\n**Response**: Thanks for the suggestion. We conduct additional manual evaluation on 50 randomly sampled cases from MMMU Pro before and after SOPHIA training, covering diverse disciplines to assess both visual hallucination (including visual grounding error). We report the proportion of responses with visual hallucinations. As shown in the table,  SOPHIA significantly reduces visual hallucinations, consistent with improvements in overall correctness.  This further supports our motivation that on-policy visual understanding training can enhance the model's visual understanding capabilities while reducing hallucinations.\n\n|Model|Visual Hallucinations|\n|---|---|\n|InternVL3.0-38B|38%|\n|InternVL3.0-38B+SOPHIA|16%|\n\nDue to space constraints, we are unable to provide specific qualitative examples in the rebuttal. We will present representative cases during the discussion phase and include these examples in the revised paper to better illustrate our motivation and the effectiveness of SOPHIA in reducing visual hallucinations.\n\n> **Review**: Binary reward signal: As acknowledged in the limitation section, the reward mechanism is based on a simple binary outcome and a preference for the shortest correct trajectory. This is a coarse signal that may not capture the nuances of a high-quality reasoning process and could prematurely penalize more complex but valid reasoning paths.\n\n**Response**: We would like to first denote that our training supervision is not inherently limited to binary outcome rewards. With the fine-grained reward signal for visual understanding computed by reward propagation strategies, it is possible to employ more detailed feedback to the visual understanding during the visual reasoning process. \n\nAdditionally, we agree that for the reasoning stage, the binary reward signal is a simplification. This design choice follows standard practice in current reasoning model training (e.g., DeepSeek-R1) [1], which is motivated by the challenges of reliably assigning stepwise or token-level rewards in complex reasoning. Specifically, some works propose using such process reward design [2], but fails in terms of long Chain-of-Thought due to value initialization bias and reward signal decay [3]. While outcome-based rewards may lack granularity, they prove effective in incentivizing correct reasoning without introducing substantial noise. We agree that exploring richer reward signals is important future work but is beyond this paper's scope.\n\n> **Review**: Limited generalization in some cases: The results show that performance can degrade on certain tasks or data distributions not well-represented in the training data (e.g., MathVista for the 8B model). This suggests that while powerful, the method's robustness may be sensitive to the domain and quality of the reasoning trajectories it learns from.\n\n**Response**: We appreciate this observation. We agree that model performance may decline on tasks underrepresented in the training data, but this issue is significantly alleviated as the model size increases. In Table 1, the performance of the 38B model on MathVista still surpasses these base models. Therefore, we acknowledge that data distribution can influence the effectiveness of the method to some extent, especially for smaller models, which are more sensitive to data during fine-tuning, but for larger models, this issue is greatly mitigated.\n\n> **Review**: What is the inference latency introduced by using SOPHIA compared to the base LVLM?\n\n**Response**: The inference algorithm of SOPHIA is no different from the base LVLM. However, its slow-thinking capability requires more tokens to complete the reasoning process. SOPHIA’s inference time is 1.64 GPU hours on A800 for MathVista benchmarks. For comparison, the inference cost for InternVL3.0-38B is 0.46 hours. Although SOPHIA introduces moderate additional inference latency compared to the base LVLM, this also reflects that SOPHIA is designed to cultivate slow-thinking, step-by-step reasoning, which is an ability absent from standard LVLMs. The increase in inference time is comparable to that observed in other recent reasoning LVLMs and LLMs, and follows the paradigm of test-time scaling in reasoning models. We believe this latency trade-off is reasonable given the significant gains in complex reasoning.\n\n> **Review**: Could SOPHIA be applied to other modalities or tasks beyond math reasoning?\n\n**Response**: Thanks for this question. While our main focus is on mathematical and scientific reasoning, the SOPHIA framework is modality- and domain-agnostic: it can in principle be extended to other tasks that require complex, step-wise reasoning, such as audio reasoning. In our experiments, we also evaluate SOPHIA on benchmarks such as MMMU and MMMU Pro, which cover wide range of fields beyond mathematics, including law, medicine, art, social science and engineering, and observe consistent performance gains (46.47->52.60 on MMMU Pro for InternVL3.0-38B).  We believe SOPHIA provides general improvement in slow-thinking reasoning.\n\n---\n**References**\n\n[1] Zhihong Shao, et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv 2402.03300.\n\n[2] Hunter Lightman, et al. Let's verify step by step. In ICLR 2023.\n\n[3] Yufeng Yuan, et al. What’s Behind PPO’s Collapse in Long-CoT? Value Optimization Holds the Secret. arXiv 2503.01491."}, {"review_id": "21j7pteWN4", "reviewer": "Reviewer_F2hY", "summary": "This paper introduces SOPHIA, a novel reinforcement learning framework designed to enhance visual slow-thinking reasoning capabilities in large vision-language models (LVLMs). The work proposes the semi-off-policy sampling and reward evaluation and propagation to address a critical limitation in current LVLM training approaches for complex multimodal reasoning tasks. The method also achieves competitive performance across several benchmarks.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n-\tThe semi-off-policy proposed in the paper cleverly combines the advantages of on policy and off policy, which is a very interesting work.\n-\tThe design of the reward mechanism is relatively novel, avoiding human evaluation and establishing effective end-to-end optimization, with good scalability\n-\tThis method has achieved competitive results on different benchmarks\n\n**Weaknesses**\n\n-\tThe paper lacks a complete description of the process, especially regarding the training and inference processes. It is not easy to understand the complete workflow, for example, does the inference stage require a reasoning language model?\n-\tThe design of the reward mechanism is based on the assumption that high-quality visual descriptions are strongly correlated with final results. Is there clear evidence to support this assumption?\n-\tThe paper only validated InternVL as a VLM, and it is recommended to try other VLMs as this method has been proven to be universal for different VLM models.\n-\tDoes SOPHIA have advantages over direct distillation data? For example, first generate capture data using VLM, and reasoning trajectories with reasoning language model, then build a dataset using reject sampling, and train the model using SFT.\n-\tIt seems that the paper does not analyze the impact of different reasoning models on performance.\n-\tSuggest the author to provide a more detailed explanation of the fundamental difference between the proposed method and knowledge distillation.", "was_revised": true, "final_justification": "The author's response  answers most of my confusion. I will increase my rate to borderline accept.", "rebuttal": "We thank Reviewer F2hY for your comprehensive feedback. We are happy to address the reviewer’s concerns and will incorporate all feedback to improve the paper.\n\n> **Review**: The paper lacks a complete description of the process, especially regarding the training and inference processes. It is not easy to understand the complete workflow, for example, does the inference stage require a reasoning language model?\n\n**Response**: Our paper provides a detailed description of the SOPHIA workflow in both training and inference stages. As shown in Figure 2 and Section 4, SOPHIA combines on-policy visual understanding from the LVLM with off-policy reasoning from a language model during training (line 129-149) , and the reward design, as well as the policy updating, is explained in section 4.2 (line 150-168) and section 4.3 (line 170-176) respectively. During inference and evaluation, only the LVLM is used and no external reasoning language model is required (line 658-667), which is consistent with evaluation methods in other existing works. We will revise our paper to make it more clear.\n\n> **Review**: he design of the reward mechanism is based on the assumption that high-quality visual descriptions are strongly correlated with final results. Is there clear evidence to support this assumption?\n\n**Response**: Thanks for your question. Our reward mechanism is grounded in both recent studies and our own ablation analyses. \nFirst, recent work [1] shows that the adapter layers in LLaVA architecture LVLM act as mappings that project visual embeddings into subspaces spanned by corresponding text embeddings. This indicates that textualized visual information (i.e., high-quality visual descriptions) plays a central role in the following multimodal reasoning. \n\nSecond, we conduct additional manual evaluation on 50 randomly sampled cases from MMMU Pro before and after SOPHIA training, covering diverse disciplines to assess both visual hallucination (including visual grounding error). We report the proportion of responses with visual hallucinations. As shown in the table,  SOPHIA significantly reduces visual hallucinations, consistent with improvements in overall correctness.  This further supports that reward mechanism based on the assumption is a key role in enhancing the model's visual understanding capabilities while reducing hallucinations.\n\n|Model|Visual Hallucinations|\n|---|---|\n|InternVL3.0-38B|38%|\n|InternVL3.0-38B+SOPHIA|16%|\n\nFinally, we empirically validate this assumption through ablation studies presented in  Section 6.3 (Table 5). Specifically, removing the visual description reward from SOPHIA leads to significant performance drops, particularly on challenging reasoning benchmarks such as MathVision and OlympiadBench. For example, on MathVision, removing the caption reward lowers the score from 42.73% to 40.35%. This shows that high-quality visual descriptions are indeed critical for guiding LVLMs toward accurate and robust reasoning.\n\nWe will further clarify this motivation in the revised manuscript, and thank the reviewer for encouraging us to strengthen this discussion.\n\n> **Review**: The paper only validated InternVL as a VLM, and it is recommended to try other VLMs as this method has been proven to be universal for different VLM models.\n\n**Response**: Thanks for this advice. Our method is not tied to a specific architecture and can be applied to other LVLM families with minimal modification. To quickly validate its generality, we further conduct research on Qwen2.5-VL in both 7B and 32B scales with a subset of the training data without warming-up and mixing data. The results on four benchmarks are shown in the following tables.\n\n||MMMU|MathVista|MathVision|OlympiadBench|\n|---|---|---|---|---|\n|Qwen2.5-VL-7B|52.00|68.90|26.38|16.28|\n|Qwen2.5-VL-7B+SOPHIA|57.22|68.50|33.26|26.59|\n|Qwen2.5-VL-32B|65.33|72.80|38.98|38.92|\n|Qwen2.5-VL-32B+SOPHIA|68.22|73.70|43.68|41.99|\n\nWe can observe that SOPHIA is also effective for the QwenVL family. We will present related results after revision.\n\n> **Review**: Does SOPHIA have advantages over direct distillation data? For example, first generate capture data using VLM, and reasoning trajectories with reasoning language model, then build a dataset using reject sampling, and train the model using SFT.\n\n**Response**: We conduct extensive experiments in Table 1 and the ablation studies (Table 5), which show that SOPHIA consistently outperforms distillation data and reject sampling. In Table 1, SOPHIA achieves an average pass@1 accuracy of 55.51% on InternVL3.0-38B across eight benchmarks, compared to 49.51% with the MPO (extension of SFT). In Table 5, removing the caption reward from SOPHIA (SOPHIA w/o CR, i.e., reject sampling) leads to a notable decrease in average pass@1 accuracy on InternVL2.5-38B: from 55.13% (SOPHIA) to 53.54%. This also shows that SOPHIA's advantage over direct distillation data lies in its on-policy enhancement of visual understanding. This is particularly critical for LVLMs, as their visual understanding capabilities are often inherently limited, and direct distillation struggles to effectively improve this aspect. SOPHIA addresses this issue by enhancing the visual understanding of LVLMs on-policy, thereby more effectively improving multi-modal slow-thinking reasoning quality.\n\n> **Review**: It seems that the paper does not analyze the impact of different reasoning models on performance.\n\n**Response**: We present an analysis of the impact of different reasoning models in Appendix C.3 (Table A2). As shown, we vary the reasoning LLM and discuss their influence on SOPHIA’s performance. The results show that SOPHIA maintains robust gains across different reasoning model choices, indicating that SOPHIA can achieve strong performance without an extremely powerful teacher model.\n\n> **Review**: Suggest the author to provide a more detailed explanation of the fundamental difference between the proposed method and knowledge distillation.\n\n**Response**: We clarify the difference between SOPHIA and knowledge distillation in both the Introduction and Related Work sections, and address the strength of SOPHIA compared to distillation as aforementioned. While knowledge distillation passively transfers teacher outputs to a student, SOPHIA integrates on-policy visual understanding with off-policy reasoning and optimizes via reinforcement learning with propagated rewards. Our experiment results further highlight that SOPHIA improves multi-modal reasoning ability rather than mere pattern imitation, distinguishing it fundamentally from distillation-based approaches.\n\n---\n**References**\n\n[1] Jiaqi Liao, et al. LangBridge: Interpreting Image as a Combination of Language Embeddings. In ICCV 2025."}, {"review_id": "tzaQf5BsWg", "reviewer": "Reviewer_WfAw", "summary": "This paper introduces SOPHIA, a novel Semi-Off-Policy Reinforcement Learning framework designed to enhance Large Vision-Language Models (LVLMs) with visual \"slow-thinking\" reasoning. It addresses the limitations of both on-policy RL's restricted exploration and off-policy RL's risk of visual hallucinations. SOPHIA proposes a semi-off-policy behavior model that integrates the LVLM's visual understanding with off-policy reasoning from external Language Models, using outcome-based and propagated visual rewards. Experiments across diverse visual reasoning benchmarks demonstrate SOPHIA's significant performance improvements for LVLMs like InternVL2.5 and InternVL3.0.", "questions": "Please refer to weakness.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n-  The paper clearly identifies a significant problem in LVLMs' slow-thinking reasoning and presents a well-motivated solution. The writing is notably clear, making the proposed methodology easy to understand.\n-  The core design of SOPHIA is innovative and highly sensible, effectively tackling challenges faced by current RL approaches for LVLMs.\n-  The work demonstrates high quality through comprehensive experimental validation and thorough ablation studies, providing robust support for its effectiveness.\n\nWeaknesses\n- The paper could benefit from a more detailed qualitative analysis of typical error modes before and after SOPHIA training, providing deeper insights into how visual hallucinations or logical inconsistencies are specifically mitigated.\n- While SOPHIA is validated using InternVL2.5 and InternVL3.0, demonstrating its effectiveness when applied to other strong base LVLMs, such as Qwen-VL, as the core model for enhancement, would further strengthen the generalizability claims.\n- The paper also lacks a detailed analysis of computational costs, covering both training phase resource consumption and the inference latency associated with multi-step \"slow-thinking\" reasoning.", "was_revised": true, "final_justification": "Thank you for the detailed response. My concerns have been adequately addressed, and I would like to maintain my original positive score.", "rebuttal": "We thank Reviewer WfAw for your positive feedback. We are pleased to answer reviewer’s concerns below and will incorporate all feedback to polish up the paper.\n\n> **Review**: The paper could benefit from a more detailed qualitative analysis of typical error modes before and after SOPHIA training, providing deeper insights into how visual hallucinations or logical inconsistencies are specifically mitigated.\n\n**Response**: Thanks for the suggestion. We conduct additional manual evaluation on 50 randomly sampled cases from MMMU Pro before and after SOPHIA training, covering diverse disciplines to assess both visual hallucination and logical inconsistency. We report the proportion of responses with visual hallucinations and logic inconsistency. As shown in the table,  SOPHIA significantly reduces visual hallucinations and logic error, consistent with improvements in overall correctness.  This further supports our motivation that on-policy visual understanding training can enhance the model's visual understanding capabilities while reducing hallucinations.\n\n|Model|Visual Halluciation|Logic Inconsistancy|\n|---|---|---|\n|InternVL3.0-38B|38%|54%|\n|InternVL3.0-38B+SOPHIA|16%|44%|\n\nDue to space constraints, we are unable to provide specific qualitative examples in the rebuttal. We will present representative cases during the discussion phase and include these examples in the revised paper to better illustrate our motivation and the effectiveness of SOPHIA in reducing visual hallucinations.\n\n> **Review**: While SOPHIA is validated using InternVL2.5 and InternVL3.0, demonstrating its effectiveness when applied to other strong base LVLMs, such as Qwen-VL, as the core model for enhancement, would further strengthen the generalizability claims.\n\n**Response**: Thanks for this advice. Our method is not tied to a specific architecture and can be applied to other LVLM families with minimal modification. To quickly validate its generality, we further conduct research on Qwen2.5-VL in both 7B and 32B scales with a subset of the training data without warming-up and mixing data. The results on four benchmarks are shown in the following tables.\n\n||MMMU|MathVista|MathVision|OlympiadBench|\n|---|---|---|---|---|\n|Qwen2.5-VL-7B|52.00|68.90|26.38|16.28|\n|Qwen2.5-VL-7B+SOPHIA|57.22|68.50|33.26|26.59|\n|Qwen2.5-VL-32B|65.33|72.80|38.98|38.92|\n|Qwen2.5-VL-32B+SOPHIA|68.22|73.70|43.68|41.99|\n\nWe can observe that SOPHIA is also effective for the QwenVL family. We will present related results after revision.\n\n> **Review**: The paper also lacks a detailed analysis of computational costs, covering both training phase resource consumption and the inference latency associated with multi-step \"slow-thinking\" reasoning.\n\n**Response**: The sampling complexity of our method during the training phase is $O(MN)$, where $M$ represents the sampling size for visual understanding, and $N$ represents the sampling size for visual reasoning. During the entire training period, for the InternVL3.0-38B model we use, total GPU hours during the training phase are 2048 hours on A800 GPU. Note that for the same data scale, the GRPO training requires about 1536 hours on A800 GPU.\n\nFor inference, the algorithm of the SOPHIA-trained model is no different from the base LVLM (next token prediction). However, its slow-thinking capability requires more tokens to complete the reasoning process. SOPHIA’s inference time is 1.64 GPU hours on A800 for MathVista benchmarks. For comparison, the inference cost for InternVL3.0-38B is 0.46 hours. Although SOPHIA introduces moderate additional inference latency compared to the base LVLM, this also reflects that SOPHIA is designed to cultivate slow-thinking, step-by-step reasoning, which is an ability absent from standard LVLMs. The increase in inference time is comparable to that observed in other recent reasoning LVLMs and LLMs, and follows the paradigm of test-time scaling in reasoning models. We believe this latency trade-off is reasonable given the significant gains in complex reasoning.\n\nWe will revise our paper to make it more clear."}]

Author rebuttal:

Meta-review: The paper proposes a semi-off-policy RL method for slow-thinking reasoning in vision language models. The main strength is that it tackles a clear gap in current training strategies, and all reviewers agreed it achieves strong empirical improvements on challenging reasoning benchmarks. Reviewers highlighted the novelty of the semi-off-policy design, the combination of visual grounding with reasoning supervision, and thorough ablations. Concerns included the reliance on caption-based visual representations, limited benchmark diversity, lack of qualitative analysis of error modes, dependence on private training data, and unclear generality across mllm architectures. Reviewers also asked about computational cost and inference latency. The authors responded comprehensively, adding results on Qwen-VL, conducting manual hallucination studies, clarifying training/inference pipeline, and explaining how SOPHIA differs from distillation. They also promised to release internal data. This rebuttal convinced most reviewers, with some increasing their scores and all agreeing the issues were addressed satisfactorily.

The final decision is accept. While the paper is not flawless, e.g., stronger evaluation on perception-heavy tasks and more transparent dataset description would further strengthen it, the method is well-motivated, clearly presented, and achieves high impact results with broad interest. Importantly, the authors engaged carefully with reviewer feedback, expanding analyses and demonstrating generality beyond InternVL. The paper advances the field of mllm reasoning in a meaningful way and sets a solid baseline for future slow-thinking approaches.
Decision: Accept (poster)