Vision-SR1: Self-Rewarding Vision-Language Model via Reasoning Decomposition and Multi-Reward Policy Optimization
Authors: Zongxia Li
Wenhao Yu
Chengsong Huang
Zhenwen Liang
Rui Liu
Fuxiao Liu
Jingxi Chen
Dian Yu
Jordan Lee Boyd-Graber
Haitao Mi
Dong Yu
Venue: ICLR 2026 Poster (2026)
Keywords: machine learning
vision-language models
deep learning
reinforceme
Primary area: foundation or frontier models, including LLMs
TL;DR: We present multi reward and multi loss objective reinforcement learning training method to improve visual understanding and reduce hallucination.

Abstract: Vision-Language Models (VLMs) often suffer from visual hallucinations -- generating things that are not consistent with visual inputs -- and language shortcuts, where they skip the visual part and just rely on text priors. These issues arise because most post-training methods for VLMs rely on simple verifiable answer matching and supervise only final outputs, leaving intermediate visual reasoning without explicit guidance. As a result, VLMs receive sparse visual signals and often learn to prioritize language-based reasoning over visual perception. To mitigate this, some existing methods add visual supervision using human annotations or distilled labels from external large models. However, human annotations are labor-intensive and costly, and external signals can introduce high latency cost.

In this paper, we introduce Vision-SR1, a three-stage self-rewarding reinforcement learning method that improves visual reasoning without relying on external visual supervision. Vision-SR1 decomposes VLM reasoning into two components: visual reasoning and language reasoning, where the model is first prompted to produce self-contained visual descriptions sufficient to answer the question without referring back to the input image, before jointly optimizing both visual and language reasoning through our multi-reward loss objective. To validate this self-containment, the same VLM model is re-prompted to perform language reasoning using only the generated visual reasoning as input to compute visual reward.  The final reward is computed through a decoupled reward-advantage framework, where visual reward and language reasoning reward each have their advantages, log probabilities, and KL divergence calculated separately. This decoupling enables more fine-grained reward computation by preventing the entanglement of heterogeneous reward signals. Our experiments show that Vision-SR1 improves visual reasoning, mitigates visual hallucinations, and reduces reliance on language shortcuts across diverse vision-language tasks, while being more efficient than methods that rely on external visual reward models, which require additional GPUs to host. In contrast, Vision-SR1 introduces no extra GPU overhead beyond that of standard training.

Published as a conference paper at ICLR 2026
VISION-SRI: SELF-REWARDING VISION-LANGUAGE MODEL VIA REASONING DECOMPOSITION AND MULTI-REWARD POLICY OPTIMIZATION
Zongxia Li Wenhao Yu-1, Zhenwen Liang', Chengsong Huang".3, Rui Liu Fuxiao Liuê, Jingxi Chen², Dian Yu', Jordan Boyd-Graber', Haitao Mi Dong Yu- Teencett AI Seattle Lab, 2 University of Maryland, College Park,
"Washington University in St. Louis
i Core contributors
c zli12 2321 umd. edu; wehhaooyrwooollllaa.l.. www.eett.l.eee........ com; eorda com
ABSTRACT
Vision-Language Models (VLMs) often suffer from visual hallucinations gen- erating things that are not consistent with visual inputs and language shortcuts, where they skip the visual part and just rely on text priors. These issues arise because most post-training methods for VLMs rely on simple verifiable answer matching and supervise only final outputs, leaving intermediate visual reasoning without explicit guidance. As result, VLMs receive sparse visual signals and often learn to prioritize language-berd based reasoning over visual perception. To mit- igate this, some existing methods add visual supervision using human annotations distilled labels from external large models. However, human annotations labor-intensive and costly, and external signals can introduce high latency cost.
In this paper, we introduce Vision-SRI, three-stage self-rewarding reinforcement learning method that improves visual reasoning without relying on external vi- sual supervision. Vision-SR1 decomposes VLM reasoning into two components: visual reasoning and language reasoning. where the model is first prompted produce self-contained visual descriptions sufficient to answer the question with- referring back to the input image, before jointly optimizing both visual and language reasoning through our multi-reward loss objective. To validate this self- containment, the same VLM model is re-prompted to perform language reason- ing using only the generated visual reasoning as input to compute visual reward. The final reward computed through decoupled eeward-advantage framework, where visual reward and language reasoning reward each have their advantages, log probabilities, and KL divergence calculated separately. This decoupling en- ables more fine-grained reward computation by preventing the entanglement of heterogeneous reward signals. Our experiments show that Vision-SRI improves visual reasoning, mitigates visual hallucinations, and reduces reliance on lan- guage shortcuts across diverse vision-language tasks, while being more efficient than methods that rely on external visual reward models, which require additional GPUs to host. In contrast, Vision-SRI introduces no extra GPU overhead beyond that of standard training.
Code:
SR1.
INTRODUCTION
Recent advances in vision-language models (VLMs) have progressed by integrating pre-trained lan- guage models and vision encoders with instruction tuning (Liu et al., 2023b; et al, 2024; Chen al., 2024: Bai et al., 2025; Li et al., 2025d). Despite these successes, a critical limitation remains their reasoning capabilities: VLMs often produce visual hallucinations descriptions of content that is not actually present the image (Guan et al., 2024: Liu et al., 2024; Li et al., 2025e; Liu et al., 2023a)-or rely language shorteuts, where the model bypasses visual understanding and instead

---

Published as a conference paper at ICLR 2026
depends solely on text priors (Si et al., 2022: Bleeker et al., 2024). RI-style reinforcement learning (RL) methods have recently improved the reasoning abilities of VLMs across diverse tasks (Huang et al., 2025b; Shen et al., 2025; Xia et al., 2025; Zhang et al., 2025). However, these methods often encourage "thinking over seeing", leaning heavily on language reasoning while demoting vi- sual perception (Liu al., 2025; Yao al., 2025). This imbalance makes VLMs susceptible to reward hacking (Fu et al., 2025) and spurious effects (Shao et al., 2025) in RL training. Although VLMs trained with RL often "improve", these improvements often just are probability shifts toward the style of training and test data, leading to language shortcut answers from prior knowledge and overlooking hallucination risks (Li et al., 2025b).
In essence, most existing post-training methods for VLMs rely on one stage, simple, verifiable answer matching and thus lack explicit supervision for visual information. As a result, VI.M's vi- sual signals are sparse, leading them to prioritize anguage-basedl reasoning over visual perception. To mitigate this, some methods introduce intermediate visual supervision through human annota- tions (Thawakar et al., 2025) or distilled labels (e.g.. pre-extracted key steps) from external mod- els (Xu et al., 2024; Zhang et al., 2025; Xiao et al., 2025; Xia et al., 2025; Lu et al., 2025). However, these solutions have significant limitations. Human annotations are labor-intensive, costly, and dif- ficult to scale across multimodal tasks, while distilled signals inherit biases and latency from source models and often fail to generalize across diverse domains. Moreover, distributional shifts between fixed intermediate signals and the continually updated policy can lead to reward hacking (Gao et al., 2023). Most importantly, both approaches remain limited by their reliance on external supervision or simply summing up multiple intermediate rewards, restricting their scalability and applicability.
This paper introduces Vision-SR1, a reinforcement learning framework that encourages VLMs to produce self-contained visual reasoning that can be verified by the VLM itself without external su- pervision. Vision-SR, I explicitly isolates the visual grounding stage from the reasoning stage: visual perception and language reasoning. The visual perception is required to capture all details relevant to answering the query, so that the reasoning stage can proceed without re-accessing the original image. We explicitly compute advantages and rollouts separately for each stage, then calculate indi- vidual Actor policy losses and KL divergence terms for the visual perception and language reasoning stages before combining them into unified training objective.
The training has two rollout passes and one training objective optimization of the same VLM:
First pass (standard rollout): (Image, Query) + (Visual Perception, CoT Reasoning, Answer)
The model generates a structured output that explicitly separates visual perception, chain- of-thought (CoT) reasoning, and the final answer.
An accuracy reward compares the final answer with the ground truth.
Second pass (self-reward rollout): (Query, Visual Perception) - (CoT Geasoning,,nnwerr
The model is re-prompted to reason using only the generated perception (without re- accessing the original image). If the correct answer is derived, the perception is considered faithful, and a self-visual reward is assigned.
Policy Optimization (objective optimization):
The multi-reward policy optimization enables the policy model to receive distinct feedback for visual reasoning quality and answer accuracy through separate advantage computations and rollout-specific loss terms.
The self-rewarding process eliminates the computational overhead of deploying additional reward models on separate GPUs, while the decoupled reward signals are combined through our multi- policy loss objective provide balanced training that strengthens both visual perception and lan- guage reasoning without the entangled learning signals of traditional reward summation. Vision-SRI improves visual reasoning, mitigates hallucinations, and reduces language shortcuts across diverse languaee tasks.
2
METHOD
We build on Group Relative Policy Optimization (Shao et al., 2024, GRPO) for improving VLM reasoning. We first review the key concepts then introduce our method.
2

---

Published as a conference paper at ICLR 2026
First Rollout Fornst Revard
cornec
Image
Arsiver Roward
Reasening
Policy
Visua Reoponin
Col
Arsner
II cornect ortwer -O othereeee
Query
Visad Perceprian Revard
Policy
conver
Autran
Shoge 2: Textted
Re do-proppiig the some Policy VLM to validitte visual perception
Reasoning
+1 correct atsier
-O othernise
Aua Y All bel
A lnstain
B
Policy
Optinvzation
L L Jove (illl bosedfllle))
Figure I: Overall framework of Vision-SR1. During RL training the VLM has rollouts. In the first pass, the model takes an image query pair and generates structured output (visual perception, CoT reasoning, and answer), with answer reward computed against the ground truth. the second pass, the model is re-prompted to answer using only query and its generated visual perception. If the correct answer is derived, a self-visual reward is assigned. We compute the advantages and log probabilities for each rollout for Newai-Reward Policy Optimization.
2.1 PRELIMINARY: REINFORCEMENT LEARNING FOR VLM WITH VERIFIABLE REWARD
We optimize pre-trained VLM policy to be optimized in reinforcement learning. Given a multimodal question (Q) with an image and text question where Q = {i, q). the policy model generates reasoning response s. GRPO optimizes the response s for the policy model. For each multimodal question Q { we sample a group of K candidate responses SQ {st,....sk) 8k (7((1Q), Each response is scored by a scalar reward r(Q,sk) (defined in Sec. 2.2), and we compute a group-relative advantage
K
( sk) = r(Q, - K
Σ s(),s,),
(I)
which centers rewards within the group, removing question-level biases while retaining pairwise preferences. We update the policy by maximizing
(0) EquD Σ APP(Q, log (s Q 8 KK(TTo( (Q) (-e((111))
(2)
where Te, is the frozen, pre-trained reference model and 3 controls the strength of the KL penalty that keeps the updated policy close its original behavior.
The group-centred baseline in equation guarantees Σn ÁMT 0, thereby reducing the estimates without requiring external value critic.
variance of
2.2 STAGE 1 & 2: RE.FFREEWAAIIGG VLM VIA REASONING DECOMPOSITION
Incorporating intermediate visual supervision can strengthen the reasoning ability of VLMs. How- ever, existing methods suffer from key limitations: approaches based on human annotations are labor-intensive and costly (Thawakar et al., 2025), while those that distill supervision from external models introduce additional compute overhead and training latency (Zhang et al., 2025; Xiao et al., 2025; Xia et al., 2025). To overcome these issues, we introduce a self-reward framework that enables the VLM to reward its own visual reasoning. The key idea is decompose the visual reasoning pro- cess into structured components, i.e., the VLM first produces celf-cont visual perception and then assesses whether this perception is sufficient for produce the final answer. This decomposition reduces reliance on external supervision and allows the reward signal to adapt dynamically as the model improves.
3

---

Published as a conference paper at ICLR 2026
Decomposing VLM Reasoning. To encourage the VLM to reason about its visual input, we
quire every response to adhere to a See-Think generation format (Jia et al., 2024; Xia et al., 2025) format. Specifically, for vision-language task, Q = g} where i is the input image and q is the textual query, the model produces the following structured output:
(/ visuallreasoning || (think) (/think) || (answer) a (/answer)
where e is self-contained visual reasoning that captures all visual information necessary solve the task, so that the following language reasoning can proceed without re-accessing the original input image. Besides, is the language reasoning trace, and the final answer.
Relf-Reward for Visual Reasoning. A challenge is judging whether the visual reasoning is self-contained: i.e., whether it encodes all the visual information needed to answer the question Q = fi. q) correctly. Our idea is to treat the visual perception text-only proxy for the image and validate it by re- re-prompting the VLM itself to perform language reasoning using only the generated perception input. If the model can derive the correct answer from (c, ()) alone, we consider to be visually faithful and assign the corresponding visual reward.
à = So (c..g; (Q, e) = =[āa=")
where a" is the ground-truth answer. Instead of using an external reward model, we leverage the policy model's own reasoning ability for self-evaluation. The model itself determines the reward by answering the question using only its generated visual reasoning (Figure 1).
Reward Composition. question Q = {i.q):
The reward combines three aligned components, each conditioned on the
Format reward (s): measures whether the response strictly follows the required layout. This reward is applied to both Visual reward and Accuracy reward.
Answer reward Tans (Q, a): measures the correctness of the final answer (Focc) plus the corre- sponding format reward. Because a is generated after the reasoning trace f, the term implicitly rewards CoT reasoning. This is computed at first rollout with hyper-parameters (0 S 1):
Fans (Q, a) Face (Q, a), + Timt (s)
(4)
Visual reward (Qisma c): measures whether the visual reasoning output is self-contained, i.c., sufficient to answer the question without image (Fvistac)) plus corresponding format reward. A reward of is assigned if, given only the question and the visual reasoning, the VLM can give the correct answer. This computed at second rollout:
Teisuel (Q,a) Tvis.ace
(Q,a), Tvis.ace(Q,a)) (am))+
(5)
2.3 STAGE 3: MULTI-REWARD OPTIMIZATION WITH MULTI-ADVANTAGE LOss COMPUTATION.
Simply summing the visual reasoning reward and the final-answer accuracy reward could produce a sparse and entangled learning signal: the policy has little to tell which rollout was responsible. To disentangle visual reasoning and answer accuracy assignment, we keep the two iifouuts-nsser generation and visual reasoning seeparate throughout the update. Each rollout receives its own log- oog-robabilittes, advantage, and KL term, and the gradients are combined only at the end. This turns the single multi-reward problem into two single-reward sub-problems that share parameters with individually optimized feedback.
Reward-Specific Log-Probability Tracking. During sampling we cache the behavioral log prob- abilities for every token in each rollout:
log Tall (ans.)), log Tald( (Qvisuul.t)
where Oans. is the action at step of the first rollout, and Ovisual, is the action (token) at step of the second rollout. At update time we compute the corresponding log To under the current parameters to compute the policy and KI. losses.
4

---

Published as a conference paper at ICLR 2026
Group-wise z-Score Advantage.
For each reward we follow GRPO to compute the ad vanatge
(6) ams
(
A ans
Mans
visuaa
Viiuua
(6)
ans
E
Visual
visual
+
with means and standard deviations Mans
Σ Pans:
(i)
(rins (ans) where B
ans
the rollout batch size (and analogously for the visual group). Broadcasting Aans to all caption tokens and Avisual to all answer tokens gives two advantage masks that weight the corresponding sig-probabilities during backpropagation for each sub-task.
Actor Loss (Policy Gradient Loss). The actor loss computes weighted policy gradients for the two reward signals (answer and visual), with separate coefficients Aams and Avisunl indicating their contributions.
Lactor
Σ(
(Ali)
log To (a
A visuna, log mo TO (1
2B
where B is the rollout batch size (as in Eq. 6).
KL Divergence Regularization Loss. The KL regularization applies separate penalty co- Buns to prevent excessive policy deviation for each reward component.
efficients
Beap
and
LKL
Sans B
log Toul (nn,,)))) log Ne. (i..e)))
Svisual B
Σ llog Tatal (1) (i - log To (
1=1
Multi-Reward Loss Objective. The total loss combines multi-reward policy gradients with component-specifte regularization to optimize the model across both reward signals.
Littal Lactor LXI.
2.4 THEORETICAL ANALYSIS
We analyze why Multi-Reward Policy Optimization with separate advantage computation could improve VIM RL training compared using only answer rewards. In standard RI. training, the objective depends solely on final answer correctness:
V E Bewnr [rans (a, a a")]
(10)
where (4, contains visual reasoning and language reasoning trace and final a. Since Tans only measures whether matches ground truth a" the intermediate visual reasoning receives no direct supervision signal. For VLMs, the stronger LLM backbone dominates generation of t, and continued RI. training leads to potential reward hacking where the model exploits language priors achieve correct answers without visual grounding Pantazopoulos & Özyigit (2025).
Multi-Reward Loss Decomposition. We decompose the loss computation itself into separate com- ponents as shown Equation 9, where the actor loss handles visual and answer components sepa- rately (with the K regularization term following similar component-wise structure):
Cactor IEnsIE[AAm log To (Aans)] AvisuaalE/ [E[ log we (r(viiiua)))
(11)
Since each advantage is computed from different reward components (visual and answer, Equa- tion 6). this approach creates clear gradient paths from each reward to its corresponding components, enabling independent optimization of visual reasoning and language reasoning capabilities (Zhu et al., 2025; Lyu al., 2025; Liu et al., 2026).
We use 0.5 for Auns and Aviual-
5

---

Published as a conference paper at ICLR 2026
2.4.1 COMPUTATIONAL EFFICIENCY ANALYSIS
A natural question with two-stage rollout training is whether it doubles the computational cost rel- ative to standard one-stage GRPO. This section's analysis shows two-stage rollout is only 10-15% more expensive than standard GRPO, while requiring no extra GPU computation.
In training, both rollout stages reuse the same VLM without loading additional models calling external APIs. In practice, two-stage rollout training adds only a modest overhead over standard GRPO. For example, training for 20 steps (per-device batch size 8, 8 GPUs) on 7B model takes standard GRPO approximately 10.5 hours, while our two-stage training takes approximately 13 hours, overhead of roughly 20%. Additionally, we compare against reward strategies. 1): Using proprietary models (GPT-5 Singh et al. (2025), Claude Anthropic, Gemini Peng et al. (2025b)) as an external judge requires evaluating N B responses per step, where N is the number of rollouts per sample and B is the effective batch size, which can cause API rate limits are hit quickly, pushing training time more than doubling the cost. Using a local open- source judge instead requires dedicating at least GPU to inference, reducing training parallelism from N to N - 1 GPUs and introducing additional inference latency. By contrast, self-reward training avoids both penalties, making two-stage training practical and scalable without incurring the overhead of external judges.
alternative external visual
2.5 DATA PREPARATION
Vision-SR1-474 Our RI. dataset consists of approximately 47K examples collected from 24 open- source VLM benchmarks. It spans three key reasoning domains (Figure 1): mathematical reasoning (30.5%), which strengthens quantitative and logical abilities; commonsense knowledge (30%); and general visual understanding (39.5%) which grounds the model in visual question answering.
3 EXPERIMENTS
To implement our Vision-SRI, we use Qwen2.5-VL-3B and 7B, Mimo-7B- VL Team et al. (2025) as base models. We train the base model with GRPO. The RL phase is trained for 200 steps on the Vision- SRI-47K dataset. During training, the policy model first generates visual reasoning from the input image, then produces language reasoning and final answer. We then compute self-reward for visual reasoning by re- prompting the frozen policy model to answer the question using only its generated visual reasoning, without access to the original image i. Finally, we compute advantages and log probabilities separately for each reward component and combine them in the final loss (Figure 1).?
Table I: Vision-SR1-47K data comprises three do- mains Math, Knowledge, and General Visual Rea- soning providing diverse supervision for VLM generalization and adaptation.
Category
Included Datasets
Size (%)
Math
CLEVR Matt GeoQAs, UniGeo, GBOS. GeomeuyaK.
14K
30.5%
Super CLEVR
Science Knowledge
TQA, Science(A AI2D. PMCC-VQ VQA-RAD. EXAMS V train
14K
30%
General
Visual Rea- ChanQA, DVQA, PlxQA, soning
18K
39.5%
FigmQAA MapQA, TAMMW OKVQA. IcenQA, visual7w. OpenSpaces Spacellaca
3.1 BASELINE METHODS
Vision-R1 (Huang et al., 2025b): The first R1-style reinforcement learning approach, which relies solely answer rewards the training signal. However, since the original Vision-RI was trained only on the math domain and falters on general-domain reasoning, we reproduce it using our 47K dataset to ensure fair comparison.
(Xiao et al., 2025)'s training resembles Vision-RI but incorporates pre-extracted visual annotations as additional reward signal. These visual annotations are derived from a state- of-the-art proprietary multimodal LLM (not specified in the paper).
Perception-RI
"The policy model remains frozen during both rollouts.

---

Published as a conference paper at ICLR 2026
Table 2: Vision-SRI vs. baselines. For Vision-RI, as noted in Section 3.1, the original model checkpoint was trained only on math-domain data. So we also reproduce it using our 47K dataset.
General Visual Understanding
Visual Math Hallucination
MMMU
MMMU
RealWorld
MATH
Hallusion
Avg.
-Pro
QA
Bench
Verve
-Vision
Bench
Vissonary- (3B) by Xia (2025)
27.1
30.6
56.9
10.0
15.0
40.4
26.7
33.9
Percention-Rt RR (78) by Xiao al. (2025)
36.8
40.9
69.4
15.9
52. 1
35.7
65.4
45.2
Vision (7B) by Huang
(2025b)
Backhone mode!: Qwee2.55VV2.33
Zero-shot Inference (before RL.)
30.5
25.5
65.4
15.7
44.3
40.4
27.1
35.5
Vision-R1 47K
comparisoe)
63.0
36.7
42.8
29.9
67.4
47.1
Vision-SRI (ours)
40.8
49.6
66.1
41.9
45.8
29.3
68.
48.8
Racchone mode!: Qwen2.5-V7-77
Zero-shot Inference (betore RL))
33.5
41.5
Visšon-R RI 47K data (fair comparison)
39.8
51.8
66.6
13
53.2
33.8
66.6
50.7
Vision (ours)
Backhone wods!: Muno-VL-78
Zero-shot Inference (before RI.)
38.0
45.6
68.2
30.2
35.5
21.6
71.9
44.4
Vision-RI 47K comparison)
35.3
46.0
Vision-SRI (ours)
39.3
49.5
68.1
44.6
40.0
29.6
75.6
49.5
Visionary-RI (Xia et al., 2025): Trained to produce a caption-reason-answer output format during RL, where the supervision signal comes from an external text-only LLM (not specified in the paper).
For fair comparisons, we only re-train Vision-RI on 47K dataset, since both Perception-R I and Visionary-R, R1 require access to external annotations or supervision signals, which are undisclosed.
3.2 BENCHMARKS AND METRICS
Our evaluation covers three areas to evaluate VLMs abilities. Specifically, the domains include (1) general visual understanding. (2) multimodal math reasoning (3) visual hallucination detection.
General Visual Understanding. We evaluate general visual understanding across five diverse benchmarks. MMMU (Yue et al., 2024) tests cross-modal reasoning and subject knowledge with 11.5K college-level, four-choice questions spanning six disciplines. MMMU-Pro (Yue et al., 2025) increases the difficulty with ten choices per question and adds challenging vision-only setting, where all text embedded within the image to necessitate robust visual parsing. Real- WorldQA (xAI, 2024) features ~000 real-world images from vehicle captures, paired with spatially grounded questions that require verifiable answers. VisNumBench (Weng al., 2025) specifically targets visual number sense through 1..9K questions covering seven numerical attributes and four estimation tasks.
Multimodal Mathematical Reasoning. We assess mathematical reasoning using two specialized benchmarks. Math Verse (Zhang et al., 2024a) consists of 2.6K diagram-centric problems (e.g.. ge- ometry, functions), each rendered in six visual-text variants to disentangle true visual understanding from linguistic shortcuts. Evaluation based on step-by-step Chain-of- of-Thought (CoT) correctness. MATH- Vision (Wang et al., 2024) presents ~3K coapeetiioongrraee problems across 16 disciplines and five difficulty levels, stressing advanced multimodal reasoning.
Hallucination Diagnosis. To diagnose model failures, we use HallusionBench (Guan et al., 2024), benchmark designed to pinpoint specific errors: (i) language-side hallucination. where visual con- text is ignored, and (ii) visual-illusion errors, where the image is misinterpreted. Because the bench- mark uses binary yes/no questions with unambiguous ground truth, each incorrect response can be cleanly attributed to one of these two failure modes.
For our evaluations, we all use Gemini-2.. 2.5-- (Comanici al., 2025) to judge response correct- ness on non-multiple choice format questions, serving as a proxy for human judgment.
3.3
EXPERIMENTAL RESULTS
3.3.1 VISION-SRI v.s. BASELINE METHODS
Table 2 compares Vision-SRI with several baseline methods across diverse language bench- marks. With the Qwen2.5VLL-BB backbone, Vision-SRI reaches 40.7 on MMMU-Pro and 52.2 on
7

---

Published as a conference paper at ICLR 2026
MMMU, outperforming Vision-RI fair comparison runs (34.9 and 42.8, respectively). When aver- aged across all benchmarks, Vision-SRI establishes a clear margin of improvement. With the 72B backbone, it achieves average score of 52.2, compared to 44.5 for Vision-RI. Even with the smaller 3B backbone, Vision-SR1 achieves 48.8 average, outperforming all comparable baselines. For results Mimi-VI-7B, model outside the Qwen-VI. family, we observe similar trend: the average accuracy improves from 44.4 to 49.5. This shows that our method generalizes beyond the Qwen-VL. These results demonstrate that Vision-SRI outperforms prior baseline models across both general-pun purpose and math-specific visual reasoning tasks, validating the effectiveness of our approach.
3.3.2 ABLATION STUDY ON SPATIAL REASONING AND LANGUAGE SHORTCUT DATASETS
In addition to evaluating Vision-SRI on stan- dard visual-reasoning benchmarks, we further evaluate its effectiveness on additional datasets to probe two complementary challenges: spa- tial reasoning and language-shortcut (LS) ro- bustness. MMSI-Bench Yang et al. (2025) and OmniSpatial Jia et al. (2025) target multi-image spatial understanding, requiring models to inte- grate spatial relationships across multiple im- ages. In contrast, ViLP Luo et al. (2025) evalu- ates language shortcuts by pairing each ques- tion with images that can be answered either through textual priors alone or only through
Table 3: Our method also can improve VLMs'
abilities on spatial reasoning and language short- cut (LS) robustness.
Methods
VILP (LS)
MMSI -Bench
Omni Sputial
Backboner Mino VL-78
before R1.
56.4
28.2
40.3
41.6
Vision
58.2
27.7
40.4
42.1
Vision-RRI
Backhnur: Qhem?
5-17-78
before Kl.
45.1
24.0
27.3
32.1
Vision-R1
Vision-RRI
$2.6
27.7
44.2
41.5
pure visual reasoning. Table 3 shows that Vision-SR1 generalizes well to spatial reasoning bench- marks and substantially improves robustness to visual-language shortcuts. In particular, explicitly generating visual descriptions helps the model avoid shortcut behavior and rely more on the actual visual content. Next we propose systematic way to evaluate VLMs' language shortcut frequencies on standard VLM benchmarks.
3.3.3 ANALYSIS ON LANGUAGE SHORTCUT
Table 4: Language Shortcut Rate (LSR) across different benchmarks. Lower values indicate better performance, as a reduced LSR reflects fewer language shortcuts during reasoning. Adding addi- tional reward supervision can reduce the change of visual reasoning reward hacking.
General Visual Understanding
Visual Math & Hallucination
MMMU
RealWorid
VisNum Bench
Math
MATH
Hallusion Bench
Methods
MMMU
Avg.
-Pro
QA
Verse
-Vision
Vision-SRI (3B)
7.5
6,3
10.8
5.4
10.3
8.3
10.1
9.4
t w/o sewfrr
9.0
9.6
11.9
4.2
11.4
9.2
8.5
10.4
Vision-SRI (7B)
6.5
13.4
4.2
11.5
10.7
w/o self-reward
8.7
5.3
10.8
3.9
12.7
10.7
9.1
10.1
We also introduce the Language Shortcut Rate (LSR), metric designed to quantify how often model produces the correct answer with an incorrect visual perception. high LSR suggests the model is leveraging language knowledge prior rather than genuine visual understanding.
Our two-step evaluation uses Gemini-2.5. (nnnini-2.5-faast f-5-flash as judge: (1) Visual Perception Extraction: for each model output, we extracted the generated visual reasoning, denoted as Ć. (2) Self-Containment Check: we then provide the Ò and the original question Q to Demini-2.5-Fllasee evaluator. If the eval- uator can reproduce the correct ground -ruth answer using only this information, Ö' is self-contained. Based on this process, define the Language Shorteut Rate (LSR) as the percentage of instances where the model produces an incorrect (not self-contained) visual reasoning but still gives the cor- rect final answer:
# #iicoorrect visual reasoning & correct answer)
LSR
# (total samples)
8

---

Published as a conference paper at ICLR 2026
A higher LSR indicates that the model is answering correctly while bypassing visual perception, suggesting reliance on language prior shortcuts. An LSR of 0 indicates no shortcutting, i.c., every correct answer is supported by correct, self-contained visual reasoning.
We compute the LSR for 7B model w/ and w/o self rewards on seven selected benchmarks for demo example in Table 4. Visual shortcuts pervade multimodal mathematical reasoning, which raises important questions in previous work R1-VL (Zhang et al., 2025), VLM-R1 (Shen et al., 2025), Vision-RI (Huang et al., 2025b): is multimodal RL training truly improving VLMs' abilities to perform visual reasoning or simply awakening the models' language reasoning ability to guess without actually looking at visual information?
3.3.4 ANALYSIS OF VISUAL ATTENTION CHANGE
Visud Albuntian Change (Bose Nodul Via un-Stt SSL
+32.55
+927%
Male 8--30
Change Maae Alterton
22
Layal frddox
Figure 2: Post-training encourages the VIT to devote more attention to visual tokens at both early feature extraction and late integration stages, while compressing intermediate processing.
We analyze layer-wise shifts in visual attention between the base model and Vision-SRI to under- stand how post-training affects visual processing. We sample 1,000 images from the ViLP dataset and compute, for each image, the per-layer difference in visual attention weights between the post- trained and pretrained models via L.2 distance. Averaging these differences across all 50 samples yields layer-wise breakdown of attention redistribution over visual tokens. As shown in Figure 2, the changes cluster into two distinct patterns. In the early layers (0-7), post-training increases vi- sual attention, with the largest gain Layer 6 (+10.2%), showing the model learns extract richer low-level visual features earlier in the network. The middle layers' (8-13) decrease indicates a re- distribution rather than a uniform increase in attention. In the late layers (14-27), visual attention rises again, peaking at Layer 20 (+9.2%), which shows enhanced visual re-engagement during the visual integration and output generation stages.
RELATED WORK
4.1 POST-TRAINING VISION-LANGUAGE MODELS
Recent vision-language models have increasingly leveraged post-training alignment techniques, in- cluding instruction tuning and reinforcement learning, to enhance general-purpose multimodal per- formance (Liu al., 2023b; Bai al., 2025; Chen al., 2024; et al, 2024; Huang al., 2025b). For example, LLaVA (Liu et al., 2023b) is tuned on GPT-4 generated (image, question, answer) pairs, coupling CLIP encoder with Vicuna to produce visual chat assistant that imitates some GPT-4 vision capabilities. InstructBLIP (Dai et al., 2023) introduces an instruction-awarr aware query trans- former tuned on 26 datasets, which yields a model that substantially outperforms even larger models on zero-shot benchmarks. Beyond standard
citrucctiontuunng methods like LLaVA and Instruct- BLIP, recent work increasingly uses reinforcement learning (RL) to align vision-language models for better reasoning (Huang et al., 2025b; Xia et al., 2025; Xiao et al., 2025). Many of these meth- ods, inspired by techniques from DeepSeek-RL (DeepSeek-Al) et al., 2025), focus on sophisticated reward engineering. Strategies include providing step-wise rewards to supervise the intermediate reasoning (Zhang et al., 2025), adding explicit visual annotations to ground truth for calculating visual rewards (Xiao al., 2025), and applying RL in two-stage curriculum that first strengthens

---

Published as a conference paper at ICLR 2026
text-only reasoning (Peng et al., 2025b). As complementary approach, RL from Al Feedback for VLMs demonstrates that preference-beee based aligmment is also a powerful signal, showing it can substantially reduce object hallucination by learning from Al Al-generated feedback (Yu et al., 2024).
4.2 SELF-REWARDING REINFORCEMENT LEARNING
The existing reinforcement learning with verifiable rewards (RLVR) methods heavily rely on high- quality reward models or human feedback, creating a major bottleneck for scalability (Peng et al., 2025a; Dai et al., 2025; Li et al., 2025c; Luu et al., 2025). To overcome this, recent work explores self-rewarding approaches, where the model itself provides intrinsic reward signals during RL post- training. an idea first pioneered by Yuan et al. (2025). Building on self- reff-rewarding language models, methods replace external reward models with the model's own confidence and uncertainty (logit- based self-certainty) or self- veff-verfiiattin of its solutions, and even elicit a latent endogenous reward already present inside base LLMs (Zhao et al., 2025; Li et al., 2025a; Simonds et al., 2025; Zheng et al., 2025; van Nickerk et al., 2025; Huang et al., 2025a; Zhou et al., 2025). For example, RLIF reward, achieving comparable performance to GRPO while improving
leverages self-certainty out-of-distribution generalization (Zhao et al., 2025). Similarly, RLSC optimizes a self-confidence reward secure large accuracy gains with only a few training samples (Li et al., 2025a).
Although self-generated reward signals have thrived in text-only LLMs, only a few works extend this idea to VLMs (Zhou et al., 2024; Lee al., 2025; Holmes & Chi, 2025), largely due to the complexity of the visual modality and the difficulty of properly defining and evaluating reward signals that capture visual perception. Recent progress includes Calibrated Self-Rewarding, which iteratively generates candidates, self-scores them with step-wise, visually constrained rewards, and fine-tunes via direct preference optimization (DPO) (Zhou et al., 2024). Similarly, RG- VLM uses VLM to directly label rewards for offline trajectories in long-horizon visual tasks, serving as an auxiliary signal that boosts generalization (Lee et al., 2025). Beyond judgment-basd based signals, ARES derives dense shaped rewards from attention weights to accelerate learning under sparse delayed feedback (Holmes reward feedback for VLM alignment without costly supervision, yet the reward is not integrated end-to-end, where the policy receives both visual perception and answer rewards during training.
Chi, 2025). These works show that internal visual signals can provide rich
5 CONCLUSION AND FUTURE WORK
We introduce Vision-SRI, a self-rewarded reinforcement learning framework that decomposes vision-language understanding into visual reasoning and language reasoning components. Our approach uses the VLM itself to generate explicit rewards for visual understanding, then applies Multi-Reward Policy Optimization to provide clear gradient attribution and backpropagation path- ways for each reward component. Vision-SRI strengthens visual perception and reduces language shortcuts, thereby improving VLM performance across several domains of vision-language tasks. Our proposed metric LSR further shows how perception reward lowers the tendency of models to answer via language shortcut rather than genuine visual reasoning.
This work opens up several future research directions. First, future work can focus on improving the efficiency of the visual reasoning then think generation format by treating the visual reasoning com- ponent as latent thinking, thereby reducing the number of decoded tokens while still enabling reward attribution to latent visual processes during the RL phase. It is also important to recognize that some of the observed mathematical gains from RL training in VLMs may come from spurious effects for instance, recalibrating the LLM backbone's 's output distribution can boost multimodal math per- formance without true visual grounding (Shao et al., 2025). This suggests that improvements in accuracy may sometimes reflect better exploitation of language shorteuts rather than genuine per- ception gains. Therefore, future work can also explore more analysis to disentangle visual grounding from shorteut learning.
ACKNOWLEDGMENTS
Boyd-Graber is supported by nsf grant 2229885, the University of Maryland (umd) TRAILS (Trust- worthy ai in Law Society) initiative. Any opinions, findings, and conclusions recommendations
10

---

Published as a conference paper at ICLR 2026
expressed in this material are those of the researchers and do not necessarily reflect the views of the National Science Foundation.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our research, we provide information regarding our prompt tem- plates and experimental setup in the main paper and Appendix. All datasets and code will be released upon conference decision release.
REFERENCES
Anthropic.
The claude 3 model family: Opus, sonnet, haiku. semanticschole pttg/CorpusI::263888
URL nttps://agg
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong. Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL httph:tp////mcc..t /aaaiiiiivv/////a///aaaa///aaavv ieriiiiorre .rgg abs/ 13923.
Maurits Bleeker, Mariya Hendriksen, Andrew Yates, and Maarten de Rijke. Demonstrating and reducing shortcuts in rision-language representation learning, 2024. URL https://arxis:
org/abs/ /00e//44222 2402 17510.
The Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguitii tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 24185 24185-24198, 2024.
Gheorghe Comanici. Eric Bieber, and et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL https: 2207 06261.
Runpeng Dai, Tong Zheng, Run Yang, and Hongtu Zhu. R1-re: Cross-domain relatioshipp extrrcciioooitt with rlvr. arXiv preprint arXiv:2507.04644, 2025.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023, URL https:/ /atp//aiiiggaaaaaaaaaaaaaaaaaa/a www.orgg/abb////o/ /205. 06500.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, F. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu I.,, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu. Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Meng Li. Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Gc, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. Chen, R. I. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun. T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li. Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng
11

---

Published as a conference paper at ICLR 2026
Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li. Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning. 2025. URL https: // aaxiv.v org http:/ab/////2/.. 12948.
OpenAl et al. Gpt-4 technical report, 2024. URL https: / /ww.aa//////.... 08774.
Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in rlhf, 2025. URL 18770.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835- 10866. PMLR, 2023.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang. Lichang Chen. Furong Huang. Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbeneh: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14385, June 2024.
lan Holmes and Min Chi. Attention-based reward shaping for sparse and delayed rewards, 2025.
URL https::///// //tr/aiivvvv/////e///e/////aaava tttp:///g/........////////// 10802.
Chengsong Huang, Wenhao Yu, Xiaoyang Wang. Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning Ilm from zero data. 2025a.
URL https:tt//// http:///axivvvvo
org/ abs/ 05004.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao. Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models, 2025b. URL https::/acxi...o////aaaaaa/// http://////bbb////oe 06749.
Mengdi Jia, Zekun Qi, Shaochen Zhang, Wenyao Zhang, Xinqiang Yu, Jiawei He, He Wang, and Li Yi. Omnispatial: Towards comprehensive spatial reasoning benchmark for vision language models, 2025. URL
wwtp//voorr//oooooooooo abs/ /2w./2506... 03135.
Mengzhao Jia, Zhihan Zhang, Wenhao Yu, Fangkai Jiao, and Meng Jiang, Deseribe-then-reuson Improving multimodal mathematical reasoning through visual comprehension training, 2024. URL
org/ /24/ 1440
Younghwan Lee, Tung M. Luu, Donghoon Lee, and Chang D. Yoo. Reward generation via large vasion-language model in offline reinforcement learning, 2025. URL URUhttps:/acxxvv.o abs/2504. 08772.
Pengyi Li, Matvey Skripkin, Alexander Zubrey, Andrey Kuznetsov, and Ivan Oseledets. Confidence is all you need: Few-shot rl fine-tuning of language models, 2025a. URL https://arxiv. org/ hbs//bb//aa /200..2066... 06395.
Zhimin Li, Haichao Miao, Xinyuan Yan, Valerio Pascucci, Matthew Berger, and Shusen Liu. See or recall: A sanity check for the role of vision in solving visualization question answer tasks with multimodal 1lms, 2025b. URL https: /taxxvivvvv. hbtg//b//25500 09809.
Zongxia Li, Yapei Chang, Yuhang Zhou, Xiyang Wu, Zichao Liang, Yoo Yeon Sung, and Jordan Lee Boyd-Graber. Semantically-aware rewards for open-ended r1 training in free-form generation, 2025c. URL www.::ttts///c...t ptattts://ariivv http:://wwwv/oo///////// abs/ 250 6. 1.6....000 15068.
Zongxia Li, Xiyang Wu, Hongyang Du, Fuxiao Liu, Huy Nghiem, and Guangyao Shi. A survey of state of the art large vision language models: Benchmark evaluations and challenges. In Pro- ceedings of the Computer Vision and Pattern Recognition Conference (CVPR) Workshops, pp. 1587-1606, June 2025d.
12

---

Published as a conference paper at ICLR 2026
Zongxia Li, Xiyang Wu, Guangyao Shi, Yubin Qin, Hongyang Du, Tianyi Zhou, Dinesh Manocha, and Jordan Lee Boyd-Graber. Videohallu: Evaluating and mitigating multi-modee hallucinations synthetic video understanding, 2025e. URL httph: /ttpvaixiivv..////i///ee///aaava org/ abs /2055 01481.
Chengzhi Liu, Zhongxing Xu, Qingyue Wei, Juncheng Wu, James Zou, Xin Eric Wang, Yuyin Zhou, and Sheng Liu. More thinking, less seeing? assessing amplified hallucination in multimodal reasoning models, 2025. URL hitps: atwiv.iiv.... wwwpxiiv.org/.eooo hagg/abs/2550
21523.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigat- ing hallucination in large multi-modal models via robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023a.
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey hallucination in large vision-language models, 2024. /ttp/aaxvvvv..oo. abs/ 2402. 00253.
URL httph:tt////.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b. URL https: //wxvv...vvv//////////e///a/aav org/ alss / 2304. 08485.
Shih- Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjic Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang. Kwang-Ting Cheng, Yejin Choi, Jan Kautz, and Pavlo Molchanov. Gdpo: Group reward-decoupled normalization policy optimization for multi-reward optimization, 2026. URL ht / aariiv...vv..///ivi//aaaaaaaaiiv org/ abs 2601 05242.
Yangxiao Lu, Ruosen Li, Liqiang Jing. Jikai Wang, Xinya Du Yunhui Guo, Nicholas Ruozzi, and Yu Xiang. Multimodal reference visual grounding. arXiv preprint arXiv:2504.02877 2025.
Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, and Honglak Lee. Probing visual language priors in VI.Ms. Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj. Kiri Wagstaff, and Jenry Zhu (eds.), Proceedings of the 42nd In- ternational Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 41 41120-41156 PMLR, 13-19 Jul 2025. URL https: htt.:p:://ppoceeednggeo mmlr. press / / luo25b. html.
Tung Minh Luu, Younghwan Lee, Donghoon Lee, Sunho Kim, Min Jun Kim, and Chang D. Yoo. Enhancing rating-based reinforcement learning to effectively leverage feedback from large vision- language models, 2025. URL https: / /wtpaaawwgg/aaaaaaaaa.a.a abs 12822..
Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li. Qian Zhao, Haian Huang, Weihan Cao, Jiangning Liu, Hongwei Liu, Junnan Liu, Songyang Zhang, Dahua Lin, and Kai Chen. Exploring the limit of outcome reward for learning mathematical reasoning, 2025. URL https: arxiv. www.svivorrr..ooo
org/ / abs /a/aab///000 2002002200.671 06781.
Georgios Pantazopoulos and Eda B. Özyigit. Towards understanding visual grounding in visual language models, 2025. URL https: pttttps://arxiiie /tt///axvvvv..///// /ttp://axx....aaaaaaa/////aaa/aa http://bbb2220...0000. 10345.
Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu. Lei Hou, and Juanzi Li. Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems, 2025a. URL
a25.aa/../2..a.//// 19328.
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang. Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl, 2025b. URL https : /etees//aaiiiii //aaxiv....///////////////aaa org/abs/253333 07536.
abs//b////0 2503.
Neale Ratzlaff, Man Luo, Xin Su, Vasudev Lal, and Phillip Howard. Training mitigation of language reasoning degradation after multimodal instruction tuning. In Proceedings of the AAAI Symposium Series, volume 5, pp. 384-388, 2025.
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang. Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannanch Hajishirzi. Pang Wei Koh, and Luke Zettlemoyer. Spurious rewards: Rethinking training signals rlvr, 2025. URL https: //arxiv wttg
13

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Rumxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL / wttpa/wwwwww.../////a/aa02 03300.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint arXiv:2504.0700
2025.
Qingyi Si, Fandong Meng, Mingyu Zheng. Zheng Lin, Yuanxin Liu, Peng Fu, Yanan Cao, Weiping Wang, and Jie Zhou. Language prior is not the only shortcut: A benchmark for shortcut learning in vqa. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 3698-3712, 2022.
Toby Simonds, Kevin Lopez, Akira Yoshiyama, and Dominique Garmier. Rlsr: Reinforcement learning from self reward, 2025. URL
/wtp:/wwww..////////....... 08827..
Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Hel- yar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker-Whiteone Whitcomb, Alex Beutel, Alex Karpenko, Alex Makelov, Alex Neitz, Alex Wei, Alexandra Barr, Alexandre Kirch- meyer, Alexey Ivanov, Alexi Christakis, Alistair Gillespie, Allison Tam, Ally Bennet, Alvin Wan, Alyssa Huang, Amy McDonald Sandjideh, Amy Yang. Ananya Kumar, Andre Saraiva, An- drea Vallone, Andrei Gheorghe, Andres Garcia Garcia, Andrew Braunstein, Andrew Liu, Andrew Schmidt, Andrey Mereskin, Andrey Mishchenko, Andy Applebaum, Andy Rogerson, Ann Rajan, Annie Wei, Anoop Kotha, Anubha Srivastava, Anushree Agrawal, Arun Vijayvergiya, Ashley Tyra, Ashvin Nair, Avi Nayak, Ben Eggers, Bessie Ji, Beth Hoover, Bill Chen, Blair Chen, Boaz Barak, Borys Minaiev, Botao Hao, Bowen Baker, Brad Lightcap. Brandon McKinzie, Brandon Wang, Brendan Quinn, Brian Fioca, Brian Hsu, Brian Yang. Brian Yu, Brian Zhang, Brittany Brenner, Callie Riggins Zetino, Cameron Raymond, Camillo Lugaresi, Carolina Paz, Cary Hud- son, Cedrie Whitney, Chak Li, Charles Chen, Charlotte Cole, Chelsea Voss, Chen Ding, Chen Shen, Chengdu Huang. Chris Colby, Chris Hallacy, Chris Koch, Chris Lu, Christina Kaplan, Christina Kim. CJ Minott-
Cliff Frey, Cody Yu, Coley Czarnecki, Colin Reid, Colin Wei, Cory Decareaux, Cristina Scheau, Cyril Zhang, Cyrus Forbes, Da Tang. Dakota Goldberg, Dan Roberts, Dana Palmie, Daniel Kappler, Daniel Levine, Daniel Wright, Dave Leo, David Lin, David Robinson, Declan Grabb, Derek Chen, Derek Lim, Derek Salama, Dibya Bhattacharjee, Dimitris Tsipras, Dinghua Li, Dingli Yu, DJ Strouse, Drew Williams, Dylan Hunn, Ed Bayes, Edwin Arbus, Ekin Akyurek, Elaine Ya Le, Elana Widmann, Eli Yani, Elizabeth Proehl, Enis Sert, Enoch Cheung. Eri Schwartz, Eric Han, Eric Jiang, Eric Mitchell, Eric Sigler, Eric Wal- lace. Erik Ritter, Erin Kavanaugh, Evan Mays, Evgenii Nikishin, Fangyuan Li, Felipe Petroski Such, Filipe de Avila Belbute Peres, Filippo Raso, Florent Bekerman, Foivos Tsimpourlas, Fotis Chantzis, Francis Song, Francis Zhang, Gaby Raila, Garrett McGrath, Gary Briggs, Gary Yang, Giambattista Parascandolo, Gildas Chabot, Grace Kim, Grace Zhao, Gregory Valiant, Guillaume Leclere. Hadi Salman, Hanson Wang. Hao Sheng, Haoming Jiang, Haoyu Wang. Haozhun Jin, Harshit Sikchi, Heather Schmidt, Henry Aspegren, Honglin Chen, Huida Qiu, Hunter Lightman, Ian Covert, Ian Kivlichan, Ian Silber, Ian Sohl, Ibrahim Hammoud, Ignasi Clavera, Ikai Lan, Ilge Akkaya, Ilya Kostrikov, Irina Kofman, Isak Etinger, Ishaan Singal. Jackie Hehir, Jacob Huh, Jacqueline Pan, Jake Wilezynski. Jakub Pachocki, James Lee, James Quinn, Jamie Kiros, Janvi Kalra, Jasmyn Samaroo, Jason Wang, Jason Wolfe, Jay Chen, Jay Wang, Jean Harb, Jeffrey Han, Jeffrey Wang, Jennifer Zhao, Jeremy Chen, Jerene Yang, Jerry Tworek, Jesse Chand, Jes- sica Landon, Jessica Liang. Ji Lin, Jiancheng Liu, Jianfeng Wang, Jie Tang, Jihan Yin, Joanne Jang, Joel Morris, Joey Flynn, Johannes Ferstad, Johannes Heidecke, John Fishbein, John Hall- man, Jonah Grant, Jonathan Chien, Jonathan Gordon, Jongsoo Park, Jordan Liss, Jos Kraaijeveld, Joseph Guay, Joseph Mo, Josh Lawson, Josh McGrath, Joshua Vendrow, Joy Jiao, Julian Lee, Julie Steele, Julie Wang, Junhua Mao, Kai Chen, Kai Hayashi, Kai Xiao, Kamyar Salahi, Kan Wu, Karan Sekhri, Karan Sharma, Karan Singhal, Karen Li, Kenny Nguyen, Keren Gu-Lemberg, Kevin King, Kevin Liu, Kevin Stone. Kevin Yu, Kristen Ying, Kristian Georgiev, Kristie Lim, Kushal Tirumala, Kyle Miller, Lama Ahmad, Larry Lv, Laura Clare, Laurance Fauconnet, Lauren Itow, Lauren Yang, Laurentia Romaniuk, Leah Anise, Lee Byron, Leher Pathak, Leon Maksin,
14

---

Published as a conference paper at ICLR 2026
Leyan Lo, Leyton Ho, Li Jing, Liang Wu, Liang Xiong, Lien Mamitsuka, Lin Yang, Lind- say McCallum, Lindsey Held, Liz Bourgeois, Logan Engstrom. Lorenz Kuhn, Louis Feuvrier, Lu Zhang. Lucas Switzer, Lukas Kondraciuk, Lukasz Kaiser, Manas Joglekar, Mandeep Singh, Mandip Shah, Manuka Stratta, Marcus Williams, Mark Chen, Mark Sun, Marselus Cayton, Mar- tin Li, Marvin Zhang. Marwan Aljubeh, Matt Nichols, Matthew Haines, Max Schwarzer, Mayank Gupta, Meghan Shah, Melody Huang, Meng Dong, Mengqing Wang, Mia Glaese, Micah Carroll, Michael Lampe, Michael Malek, Michael Sharman, Michael Zhang, Michele Wang, Michelle Pokrass, Mihai Florian, Mikhail Pavlov, Miles Wang, Ming Chen. Mingxuan Wang. Minnia Feng, Mo Bavarian, Molly Lin, Moose Abdool, Mostafa Rohaninejad, Nacho Soto, Natalie Staudacher, Natan LaFontaine. Nathan Marwell, Nelson Liu, Nick Preston, Nick Turley, Nicklas Ansman, Nicole Blades, Nikil Pancha, Nikita Mikhaylin, Niko Felix, Nikunj Handa, Nishant Rai, Nitish Keskar, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Oona Gleeson, Pamela Mishkin, Patryk Lesiewicz, Paul Baltescu, Pavel Belov, Peter Zhokhov, Philip Pronin, Phillip Guo, Phoebe Thacker, Qi Liu, Qiming Yuan, Qinghua Liu, Rachel Dias, Rachel Puckett, Rahul Arora, Ravi Teja Mullapudi, Raz Gaon, Reah Miyara, Rennie Song, Rishabh Aggarwal, RJ Marsan, Robel Yemiru, Robert Xiong, Rohan Kshirsagar, Rohan Nuttall, Roman Tsiupa, Ro- nen Eldan, Rose Wang, Roshan James, Roy Ziv, Rui Shu, Ruslan Nigmatullin, Saachi Jain, Saam Talaie, Sam Altman, Sam Arnesen, Sam Toizer, Sam Toyer. Samuel Miserendino, Sandhini Agar- wal, Sarah Yoo, Savannah Heon, Scott Ethersmith, Sean Grove, Sean Taylor, Sebastien Bubeck, Sever Banesiu, Shaokyi Amdo, Shengjia Zhao, Sherwin Wu, Shibani Santurkar, Shiyu Zhao, Shraman Ray Chaudhuri. Shreyas Krishnaswamy, Shuaiqi, Xia, Shuyang Cheng, Shyamal Anad- kat, Simón Posada Fishman, Simon Tobin, Siyuan Fu, Somay Jain, Song Mei, Sonya Egoian, Spencer Kim, Spug Golden, SQ Mah, Steph Lin, Stephen Imm, Steve Sharpe, Steve Yadlowsky, Sulman Choudhry, Sungwon Eum, Suvansh Sanjeev, Tabarak Khan, Tal Stramer, Tao Wang. Tao Xin, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Degry, Thomas Shadwell, Tianfu Fu, Tianshi Gao, Timur Garipov. Tina Sriskandarajah, Toki Sherbakov, Tomer Kaftan, Tomo Hiratsuka, Tongzhou Wang, Tony Song, Tony Zhao, Troy Peterson, Val Kharitonov, Victoria Chernova, Vineet Kosaraju, Vishal Kuo, Vitchyr Pong. Vivek Verma, Vlad Petrov, Wan- ning Jiang, Weixing Zhang, Wenda Zhou, Wenlei Xie, Wenting Zhan, Wes McCabe, Will DePue, Will Ellsworth, Wulfie Bain, Wyatt Thompson, Xiangning Chen, Xiangyu Qi, Xin Xiang, Xin- wei Shi, Yann Dubois, Yaodong Yu, Yara Khakbaz, Yifan Wu, Yilei Qian, Yin Tat Lee, Yinbo Chen, Yizhen Zhang, Yizhong Xiong, Yonglong Tian, Young Cha, Yu Bai, Yu Yang, Yuan Yuan, Yuanzhi Li, Yufeng Zhang. Yuguang Yang. Yujia Jin, Yun Jiang, Yunyun Wang, Yushi Wang, Yutian Liu, Zach Stubenvoll, Zehao Dou, Zheng Wu, and Zhigang Wang. Openai gpt-5 system card, 2025. URL https: / /ttxxv.xivvv/////////..///aaa org/ abs 22111 03267.
Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li. Kainan Bao. Hao Tian, Hailin Zhang, Gang Wang. Dawei Zhu, Cici, Chenhong He, Bowen Ye, Bowen Shen, Zihan Zhang, Zihan Jiang, Zhixian Zheng, Zhichao Song. Zhenbo Luo, Yue Yu, Yudong Wang, Yuanyuan Tian, Yu Tu, Yihan Yan, Yi Huang, Xu Wang, Xinzhe Xu, Xingchen Song, Xing Zhang, Xing Yong, Xin Zhang, Xiangwei Deng, Wenyu Yang, Wenhan Ma, Weiwei Lv, Weiji Zhuang, Wei Liu, Sirui Deng, Shuo Liu, Shimao Chen, Shihua Yu, Shaohui Liu, Shande Wang, Rui Ma, Qiantong Wang, Peng Wang, Nuo Chen, Menghang Zhu, Kangyang Zhou, Kang Zhou, Kai Fang, Jun Shi, Jinhao Dong, Jiebao Xiao, Jiaming Xu, Huaqiu Liu, Hongshen Xu, Heng Qu, Haochen Zhao, Hanglong Lv, Guoan Wang, Duo Zhang, Dong Zhang, Di Zhang, Chong Ma, Chang Liu, Can Cai, and Bingquan Xia. Mimo-vl technical report, 2025. URL https: /tte:///rrriiie wawpiv.ivvvv.////v///../////aavv org/ abs / 2506 03569.
Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav-ol: Rethink- ing step-by-step visual reasoning in llms. arXiv preprint arXiv:2501. .01186, 2025.
Carel van Nickerk, Renato Vukovic, Benjamin Matthias Ruppik, Hsien chin Lin, and Milica Gašić. Post-training large language models via reinforcement learning from self-feedback, 2025. URL https: //aa:i/////iv//// wwt//bss/////ooooooooooooo.aasre .1..1....111.............
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multi- modal mathematical reasoning with math-vision dataset, 2024. URL https: / /awxiv.vvvv..//... org/ abs/2402. 11.2..188 14804.
15

---

Published as a conference paper at ICLR 2026
Tengjin Weng. Jingyi Wang, Wenhao Jiang, and Zhong Ming. Visnumbench: Evaluating number sense of multimodal large language models, 2025. URL 14939.
XAI. Realworldga: Real-world spatial understanding benchmark. https:/ http:://x.ai/blll/ grok-1. -ea-and-raallle www.ewawwwrigaa..m. 2024. CC BY-ND 4.0 license. Benchmark dataset released with Grok-1.5 Vision.
Jiaer Xia, Yuhang Zang, Peng Gao, Yixuan Li, and Kaiyang Zhou. Visionary-rl: Mitigating short- cuts in visual reasoning with reinforcement learning, 2025. URL https://arxiv.... abs/2505. 14677.
Tong Xiao, Xin Xu, Zhenya Huang, Hongyu Gao, Quan Liu, Qi Liu, and Enhong Chen. Advancing multimodal reasoning capabilities of multimodal large language models via visual perception reward, 2025. URL https:////// /atxivv..v/../////////a.///aiav/ org/
wtggabs//////ao.eegtggggg/eaaeb/ abs/2506 07218.
Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. arXiv preprint arXiv:2411. 2024.
Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang, Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, Dahua Lin, Tai Wang, and Jiangmiao Pang. Mmsi-bench A benchmark for multi-image spatial intelligence, 2025. URL https: http::////xvvvvveee abs/ 2505. 237 64.
Zijun Yao, Yantao Liu, Yanxu Chen, Jianhui Chen, Junfeng Fang, Lei Hou, Juanzi Li, and Tat-Seng Chua. Are reasoning models more prone to hallucination?, 2025. URL https: arriv.. org/abs/ 2505. 23646.
Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao, Da Chen, Xiaoman Lu, Ganqu Cui, Yunkai Dang, Taiwen He, Xiaocheng Feng, Jun Song, Bo Zheng. Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Rlaif-v: Open-source ai feedback leads to super gpt-4v trustworthiness, 2024.
URL http:: /ttp:///axvvvvii org/ abs 17220.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2025. URL
abs/2401. 10020.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024. URL https: : arxiv. oowooorggao....
/aarg/ab/////ee/ abs/ 22111.11..20 22211.6555 16502.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark, 2025. URL org/abs/2400 02813.
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization, 2025. URL https: /ttxxvixvvvv/////////a////aaav https//wwwg..ab.......o/
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems?, 2024a. URL /attps://arxiv..or 2403.14622
abs/
Yi-Kai Zhang, Shiyin Lu, Yang Li, Yanqing Ma, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, and Han-Jia Ye. Wings: Learning multimodal llms without text-only forgetting. Advances in Neural Information Processing Systems, 37:31828-31853, 2024b.
Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards, 2025. URL https: wtw.xiivvvv.../////a///// org/ abs/ 2505. 19590.
16

---

Published as a conference paper at ICLR 2026
Tong Zheng, Lichang Chen, Simeng Han, R Thomas McCoy, and Heng Huang. Learning to reason via muxture-of-thougee for logical reasoning. arXiv preprint arXiv:2505.15817, 2025.
Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models, 2024. URL https: : / arxiv. org/ abs /2005.405.. 14622.
Yujun Zhou, Zhenwen Liang, Haolin Liu, Wenhao Yu, Kishan Panaganti, Linfeng Song, Dian Yu, Xiangliang Zhang. Haitao Mi, and Dong Yu. Evolving language models without labels: Majority drives selection, novelty promotes variation. arXiv preprint arXiv: 2501:22509.1994, 2025.
Xuekai Zhu, Daixuan Cheng, Dinghuai Zhang, Hengli Li, Kaiyan Zhang, Che Jiang. Youbang Sun, Ermo Hua, Yuxin Zuo, Xingtai Lv, Qizheng Zhang, Lin Chen, Fanghao Shao, Bo Xue, Yun- chong Song, Zhenjie Yang, Ganqu Cui, Ning Ding, Jianfeng Gao, Xiaodong Liu, Bowen Zhou, Hongyuan Mei, and Zhouhan Lin. Flowrl: Matching reward distributions for Ilm reasoning, 2025. URL https: / /arxivv. org/ /ts.//5555 15207.
17

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THE USE OF LARGE LANGUAGE MODELS (LLMS)
We acknowledge the of large language models (LLMs) as assistive tools in this research. Our use of LLMs was limited to refine grammar and improve language clarity. All outputs from these models were meticulously reviewed, revised, and verified by the authors, who retain full responsibility for all content presented in this paper.
B EXPERIMENT DETAILS
B. PROMPT TEMPLATES
This section presents the prompt templates used for constructing the cold start training data and Model Training prompt. The See-Think prompt is used for generating SFT See-Think data and model training. The Caption-Reasoner prompt used to generate text-only caption reasoner SFT data and self-reward during training.
See-Think Prompt Template
{Question}
You are tasked with analyzing image/video to generate detailed description to help you answer the question. First analyze the image/video and produce a self-contained descrip- tion-detailed enough that can lead to the correct answer. Wrap the entire description in < description >< /od<///scrpppp /deccription > tags.
Next, engage in an internal dialogue and include self-reflection or verification in your reasoning process. Provide your detailed, step-by-step reasoning based on the image/video description information and image/video, and enclose this part within think >< /think > tags.
Finally, provide single word or phrase answer the question in {boxed{].
The output format should be: < description image/video description here < /description > < think >reasoning process here /think > \boxed(FINAL ANSWER here).
Note: (Question) is placeholder for the actual question.
Caption-Reasoner (Self-Reee Reward) Prompt Template
Text description:
(Description)
Question: (Question}
You are provided text description of problem and a question. Determine the an- swer to the question based the text description. First provide an internal step-by-step reasoning within < think >< /think tags, then provide a single word or phrase answer in (boxed(].
Note: (Description) is placeholder for the actual text caption. {Question) is placeholder for the actual question.
18

---

Published as a conference paper at ICLR 2026
Table 5: Through self-reward, the model implicitly rewarded for text-only reasoning, leading to improved performance in general reasoning and reduced degradation in math reasoning benchmarks.
Model MMLU-Pro
SuperGPQA
GSM8K
MATH-500
Backbone model: Qwen2.5-V/..
Before RI.
34.3
15.1
78.5
65.2
Vision-RI
47.7
23.1
82.2
66.0
Vision-SRI
48.1
23.2
83.2
68.6
Backbone model: Qwen2.55VI--7B
Before RI.
33.4
17.1
86.0
73.4
Vision-RI R1
53.4
26.7
85.5
68.2
Vision-SR1 56.1
26.3
87.6
70.8
Vision Reasoner (CoT) Prompt Template
Question: {Question}
You FIRST think about the reasoning process as an internal monologue and then provide the final answer. The reasoning process MUST BE enclosed within < think >< /think > tags. The final answer MUST BE put in {boxed{).
Note: (Question) is placeholder for the actual question.
B.2 LIM-AS-AA-JUDGE PROMPT
We use Gemini-2.5-flash as our LLM-as-a-Judge to evaluate
LLM-as-a- -a-dgee Prompt Template
Model: Gemini-2.55flash
Prompt Message:
Question: (Question)
Reference:
[Reference]
Candidate: (Candidate]
You are provided question, gold answer, and candidate answer. Your task is to judge the correctness of the candidate answer. Return your judgment enclosed with < judgment >< /judgment >
Note: {Question) is placeholder for the actual question; (Reference is place- holder for the gold answer; (Candidate) is a placeholder for the model response.
B.2.1 ANALYSIS ON TEXT-ONLY REASONING
interesting question is how different training strategies affect text-only reasoning capabilities of VLMs. In particular, by decoupling visual perception and language reasoning with two separate rewards, we ask whether these abilities can mutually reinforce one another. To examine this, we evaluated the text-only performance of VLMs after RL fine-tuning multimodal data.
Specifically, we tested on four text-only datasets: MMLU-Pro and SuperGPQA maulti-disciplinan general-domain benchmarks), and MATH-500 and GSM8K (mathematical reasoning tasks). Our results (Table 5) compare Vision-RI. 1, our method, and pre-RL. training baselines.
First, observe that GSM8K and MATH-500, multimodal RL training, including both Vision- RI and our method, degrades text-only reasoning performance. This observation aligns with recent findings on "text-only forgetting" in VL Zhang et al. (2024b); Ratzlaff et al. (2025), which show
19

---

Published as a conference paper at ICLR 2026
Table Results of ablation study: Vision-SRI v.s. Vision-SRI w/o visual perception self-reward.
General Visual Understanding
Visual Math & Hallucination
MMMU
RealWorld
VisNum Bench
Math
MATH
Hallusion
Methods
MMMU
Avg.
-Pro
QA
Verse
-Vision
Bench
Vision-SRI (3B)
40.8
49.6
66.1
41.9
45.8
29.3
68.3
48.8
F w/o self-reward
40.0
48.0
62.6
41,6
45.1
30.2
65.8
47.6
Vision-SRI (?B)
40.7
52.2
69.2
43.5
54.5
36.2
68,9
52.2
self-reward
35.7
52.6
that visual instruction tuning can impair language reasoning (particularly in mathematics) depending on the underlying LLM. Second, compared to Vision-R1, I, our method proved more effective at mitigating performance degradation on text-only mathematical benchmarks (MATH-500, GMS8K) and yielded larger gains on general knowledge tasks (MMLU-Pro SuperGPQA). This indicates that separating the optimization signals for visual perception and language reasoning helps preserve text-only competencies, while still enabling improvements from multimodal training.
B.2.2 ABLATION STUDY ON
We train a control version of our model without the visual reasoning self-reward and Multi-Reward Policy Optimization (Vision-SRI w/o self-reward). This ablated model still follows structured output (visual perception, CoТ reasoning, and answer) but is optimized only with answer and for- mat rewards. The self-visual reward for self-evaluating visual reasoning and Multi-Reward Policy Optimization are removed. We note the only difference between Vision-SRI w/o self-reward and Vision-R1 (Huang al., 2025b) lies in the output structure, i.e., using different system prompts, while all supervision signals (answer reward and format rewards) remain the same. Interestingly, our system prompt yields slightly better performance (+1.0 on average). Table 6 reports the ablation results. We find that not including visual reasoning reward and Multi-Reward Policy Optimiza- tion could lead to overall worse VLM task performance compared to including them the training process.
20


Reviews:
[{"review_id": "yVrJmGPKyb", "reviewer": "Reviewer_BBYi", "summary": "This paper proposes Vision-SR1, a self-rewarding reinforcement learning framework that decomposes vision-language reasoning into visual perception (i.e., visual descriptions) and language reasoning stages. The method introduces Multi-Reward Policy Optimization, which separately computes and combines visual perception rewards and answer accuracy rewards, avoiding entangled learning signals. Besides, Vision-SR1 generates self-contained visual perceptions and verifies them using the model itself, improving grounding and reducing hallucination. Experiments on 47K examples show gains across visual reasoning, math, and hallucination benchmarks, outperforming Vision-R1 and related baselines.", "questions": "Could the author consider the experimental questions shown above, particularly I curious how the finetuned model perform in the datasets which directly motivated by VLM overly relying on language priors? I will consider the response and other reviewers' comments to adjust my final score, but plz feel free to skip any experiments you believe is unreasonable. \n\nRegarding the two stage things, I will definitely discuss with other reviewers and AC.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I believe the studied direction is important -- overcoming the VLMs overly relying on learned priors, particularly text priors. Also, the math equation is descried clearly in the paper. \n\nEmpirical results on a couple of comprehensive benchmarks are good, demonstrating improvements. It mostly contains general visual understanding and multimodal mathematical reasoning. Some ablation studies are included as well. It also curated Vision-SR1-47K from some open source VLM benchmarks (table 1), while I am unsure if data may pollute the test set or not, since the performance of finetuning over on this data improve the performance quite a lot --- Vision-R1 47K data (fair comparisons).", "weaknesses": "From my perspective, the proposed two-stage pipeline is a bit ad-hoc due to it explicitly enforces there could be two stages, where the first stage generate some visual descriptions, and then the second stage is doing the reasoning. Such pipeline is expectedly not the ultimate goal for VLMs. It is basically a captioning models + a LLM --> this is not something we want to achieve for VLMs. Such behavior should emerge in VLM inference process, not enforced by constructing a small scale datasets with specific setting ⟨visual reasoning⟩ c ⟨/visual reasoning⟩.\n\nFor evaluation,I would recommend to include some benchmarks directly motivated by language shortcuts and text priors, such as Probing Visual Language Priors in VLMs and Winoground: Probing vision and language models for visio-linguistic compositionality. \n\nAlso, extend to larger scale of models, such as 72B, will make the whole evaluation solid. Extending the experimental results to other base-model beyond Qwen-2.5-VL is also helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dn0OHcxN5Q", "reviewer": "Reviewer_f7j6", "summary": "The authors propose an improvement to vision-R1, decoupling the visual perception description generation part and the answer generation part for reward, to avoid LVLMs overly relying on language reasoning shortcuts. The method is clear, the experiments are solid, and the comparison with vision-R1 is clear.", "questions": "1. Have the authors analyzed how the model's attention changes after this training, and whether it affects its distribution on visual tokens?\n\n2. How do the authors ensure that the generated image description is self-contained? Since the model can see both the question and the image, it could potentially guess the answer and then generate an image description where the description already contains answer-related information.\n\n3. How do the authors guarantee that decoupling the visual perception stage and the answer generation stage can avoid language shortcuts? From a training perspective, in the second stage, after removing the image input, wouldn't the model be more prone to relying on language shortcuts? So why does using this two-stage decoupling method mitigate language shortcuts and visual hallucination? Why can't a single-stage approach achieve this? Is there a theoretical explanation? For example, changes in attention patterns over visual tokens?\n\n4. How do the authors address the infrastructure inefficiency issue for this two-stage training approach?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is clear, the method description is clear and specific, very straightforward, and the experiments are solid.", "weaknesses": "1. The only concern is novelty, as the idea of first generating image description and then generating answer based on the description has been explored before. [1]\n2. Requires inference twice, which would be time-consuming and not infrastructure-friendly.\n\n[1] Multimodal Chain-of-Thought Reasoning in Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iTjKTsx3Wt", "reviewer": "Reviewer_ExG7", "summary": "This paper proposes Vision-SR1, a self-reward reinforcement learning (Self-RL) framework designed to address visual hallucinations and language shortcuts in VLMs. It decomposes reasoning into two stages: visual perception and language reasoning. The model first generates a self-contained perceptual description (c), then re-prompts itself and relies solely on c to answer the question, thereby generating a self-derived visual reward. The method employs a multi-reward strategy for optimization, computing decoupled losses for both visual and answer rewards, effectively enhancing visual reasoning capability while reducing the LSR.", "questions": "The authors mention that self-rewarding may lead to reward hacking. Could the authors quantitatively analyze the intrinsic bias of the reward model in Vision-SR1? Furthermore, would it be possible to introduce lightweight external verification (e.g., a frozen CLIP or BLIP module) to cross-check the true visual consistency of the self-reward, thereby mitigating the circular dependency issue inherent in self-rewarding?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "(1) Vision-SR1 allows VLMs to self-verify whether their visual perception is self-contained and sufficient to answer a given question. This effectively provides dense and adaptive reward signals for intermediate visual reasoning steps, avoiding the high costs, biases, and reward hacking risks associated with external human annotations or large model distillation.\n(2) Advantage functions and loss terms are computed separately for visual and answer rewards, effectively decoupling the training signals. This ensures balanced reinforcement of both visual perception and language reasoning, while also avoiding the signal sparsity and entanglement caused by the traditional approach of summing rewards.", "weaknesses": "(1) Vision-SR1 uses the model itself as a verifier, evaluating the quality of visual perception by re-prompting the same-policy VLM. This may lead to circular dependencies and inherent biases: the model could learn to generate perceptions that \"convince itself\" rather than perceptions that truly align with the image.\n(2) By re-prompting the same-policy VLM to assess visual perception quality, the method relies on two rollouts during training, which may effectively double the training cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SMZej0W8dg", "reviewer": "Reviewer_mWSw", "summary": "The paper proposes Vision-SR1, which decomposes VLM reasoning into visual perception and language reasoning, then self-verifies whether the perception alone is sufficient to answer the question, providing a self-visual reward without external supervision to reduce hallucinations and language shortcuts.", "questions": "- Additional discussion: Whether the generated visual reasoning contains all information needed to answer the question—Is it possible, and is it necessary?\n\n- Additional experiments:\n  - If possible, try assigning the visual reward solely to the visual perception part in the first rollout.\n  - If time permits, train models beyond the Qwen-2.5-VL series.\n  - If time permits, evaluate Vision-SR1 on spatial reasoning tasks such as VSI-Bench [1], MMSI-Bench [2], and OmniSpatial [3].\n\n- Additional human analysis: In the Language Shortcut Rate evaluation, how many cases are there where the evaluator reproduces the wrong answer while the generated visual reasoning is actually self-contained? It would be better to include a more detailed human analysis (a small amount of data is sufficient).\n\n\n[1] Yang, Jihan, et al. \"Thinking in space: How multimodal large language models see, remember, and recall spaces.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[2] Yang, Sihan, et al. \"MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence.\" arXiv preprint arXiv:2505.23764 (2025).\n\n[3] Jia, Mengdi, et al. \"OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models.\" arXiv preprint arXiv:2506.03135 (2025).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear and well-written.\n- Method is effective across many tasks with thorough experimental validation.\n- Provides intuitive theoretical analysis supporting the design choices.\n- Raises several valuable, thought-provoking questions for the community.", "weaknesses": "- Visual perceptions that are sufficient to answer the question without referring back the input image: is it possible (some visual information is hard to express in language)? Is it necessary (visual reasoning might be understood as a carrier of latent space reasoning, used to aggregate and evolve visual information and implicit thinking via attention)?\n- “If the correct answer is derived, a self-visual reward is assigned.” Which part of generation is this visual reward used to encourage in the paper? I am a bit confused. Can this visual reward be assigned solely to the visual perception part in the first rollout? What would the effect be?\n- Only Qwen-2.5-VL is trained; how about other model series? Adding them would be more convincing.\n- Is the Language Shortcut Rate metric reasonable? First, “If the evaluator can reproduce the correct ground-truth answer using only this information, the generated visual reasoning is deemed self-contained.” How many cases are there where the evaluator reproduces the wrong answer but the generated visual reasoning is actually self-contained? It would be better to have a more detailed human analysis (on a small amount of data is fine).\n- Curious about the method’s performance on spatial reasoning tasks, because I think many perceptions and reasoning in 3D space are hard to express in natural language.\n- Typo: an extra comma at line 067.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)