Spotlight on Token Perception for Multimodal Reinforcement Learning
Authors: Siyuan Huang
Xiaoye Qu
Yafu Li
Yun Luo
Zefeng He
Daizong Liu
Yu Cheng
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Reasoning
LVLM
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper introduces VPPO, which integrates token-level visual perception into multimodal RLVR to enhance the reasoning capabilities of Large Vision-Language Models.

Abstract: While Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capabilities of Large Vision-Language Models (LVLMs), most existing methods in multimodal reasoning neglect the critical role of visual perception within the RLVR optimization process. In this paper, we undertake a pioneering exploration of multimodal RLVR through the novel perspective of token perception, which measures the visual dependency of each generated token. With a granular analysis of Chain-of-Thought (CoT) processes, we uncover two key insights: first, token perception in a rollout trajectory is sparsely distributed,  where only a small fraction of tokens have high visual dependency for visually-grounded reasoning; second, different trajectories exhibit significant divergence in their overall visual dependency. Based on these observations, we propose **V**isually-**P**erceptive **P**olicy **O**ptimization (**VPPO**), a novel policy gradient algorithm that explicitly leverages token perception to refine the learning signal. Specifically, VPPO achieves this through a dual mechanism: it reweights a trajectory's advantage by its overall visual dependency, and focuses policy updates exclusively on perceptually pivotal tokens. On a comprehensive suite of eight perception and reasoning benchmarks, VPPO demonstrates substantial gains over leading open-source RL-tuned models, with its effectiveness consistently validated across 7B and 32B model scales. Our findings not only establish a new token-level perceptual perspective for analyzing multimodal RLVR but also present a novel and effective optimization strategy to significantly enhance the multimodal reasoning capabilities of LVLMs.

Published as a conference paper at ICLR 2026
SPOTLIGHT ON TOKEN PERCEPTION FOR MULTI- MODAL REINFORCEMENT LEARNING
Siyuan Huang Xiaoye Qu Yafu Li Yun Luo', Zefeng He*, Daizong Liu Yu Cheng
1 Shanghai AI Laboratory, Shanghaii Jiao Tong University,
"The Chinese University of Hong Kong. *Nanjing University, "Wuhan University (huangsiyuan2, quxiaoye } @pppppp.b...... www.gaa..o.oc...mmmmmm org.cn, chengyuOcse. cuhk. edu. hk
ABSTRACT
While Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capabilities of Large Vision-Language Models (LVLMss, most isting methods in multimodal reasoning neglect the critical role of visual per- ception within the RLVR optimization process. In this paper, pioneering exploration of multimodal RLVR through the novel perspective of token perception, which measures the visual dependency of each generated to- ken. With granular analysis of Chain ofain-of-Tue Thought (CoT) processes, we uncover two key insights: first, token perception in rollout trajectory is sparsely dis- tributed, where only small fraction of tokens have high visual dependency for visually-grounded reasoning; second, different trajectories exhibit significant di- vergence in their overall visual dependency. Based on these observations, we propose Visually- Pereel--ee Policy Optimization (VPPO), novel policy gradi- ent algorithm that explicitly leverages token perception refine the learning sig- nal. Specifically, VPPO achieves this through dual mechanism: it reweights trajectory's advantage by its overall visual dependency, and focuses policy updates exclusively on perceptually pivotal tokens. On a comprehensive suite of eight perception and reasoning benchmarks, VPPO demonstrates substantial gains over leading open-source RL-tuned models, with its effectiveness consis- tently validated across 7B and 32B model scales. Our findings not only estab- lish new token-level perceptual perspective for analyzing multimodal RLVR but also present a novel and effective optimization strategy to significantly en- hance the multimodal reasoning capabilities of LVLMs. Our code is available at httts:://itthut www.cc/...cc...t.. wwwp:/wwwwwwuuiiiiggggggoeeeeeee RL
undertake a
INTRODUCTION
Reinforcement learning from verifiable rewards (RLVR) (Zhang et al. 2025), particularly with on- line algorithms like Group Relative Policy Optimization (GRPO), has dramatically advanced the reasoning capabilities of Large Language Models (LLMs) in text-centric domains, such as math and
code (Shao al.) 2024] Guo al. 2025 OpenAl 2024 Team al.) 2025 Yang et al.] 2025a Anthropic 2025). Recently, many works have attempted to translate this success to Large Vision-
Language Models (LVLMss). These efforts primarily focus on three directions: data-centric en-
hancements (Li et al. 2025; [iang et al. 2025 Liu et al. [223aa] Yao et al. [025 Chen et
2025a 20.5a
Meng et al. 2025] Huang et 2025 Yang et al. (2025b), reward-centric engineering (Shen et al.
2025 Xia et al. [025 Wang et 2005b Xiao et al. 22025 Yu et al. 2025a Wan et al. 2025). ) and
other a algorithmic adjustments (Wang et al. 2025a Zhao et 2025).
However, prevailing RLVR frameworks for LVLMs largely neglect the critical role of visual percep- tion in the optimization process. Effective reasoning is contingent upon accurate perception, which provides the essential grounding for logical deduction (Xiao al. 2025). The geometry problem in Figure e exemplifies this dependency. Given a question: "In circle o0, AC is parallel to OB,
and LBOC 50 What is the measure of 20ABT" To correctly answer this question, critical
CCoresponding authors.

---

Published as a conference paper at ICLR 2026
insight should be derived from the visual diagram, namely segments OA and OB are radi of the circle 30, rendering A40 B isosceles. Therefore, without explicitly integrating perceptual ability into the core learning objectives, models cannot develop genuine multimodal reasoning capabilities 2025a: Xiao et al.) 2025).
Yu et al.
In this paper, we analyze the perceptual mechanisms of multimodal RLVR through innovative lens of token perception, investigating the impact of tokens with varying visual dependency on rea- soning. With a granular analysis, we first point out that in the Chain-of-Thought (CoT) (Wei al.) 2022) processes of multimodal reasoning, the token perception distribution in a rollout trajectory exhibits distinct pattern, where the majority of tokens are generated with low visual dependency, while a critical minority of tokens emerge with high dependency. After aggregating the token per- ception at the trajectory level, we further observe that different reasoning trajectories also exhibit significant divergence in their overall perceptual quality, as only a part of trajectories are genuinely perception-driven paths. Although those paths without significant visual perception may still fortu- itously arrive at the correct answer, the resulting models will exhibit weak multimodal perception capabilities. These observations pinpoint foundational flaw inherited from text-based RLVR, i.e, existing implementations directly train over all tokens with limited understanding of which tokens actually facilitate multimodal perception and reasoning. The indiscriminate broadcasting of single, coarse reward to every trajectory and token hinders further performance gains by failing prioritize critical perception-related trajectories and tokens.
Building upon the above discovery of token per- Policy
Original Vicnal Input
Slupe Gr Grounded Trajectory Advantages
ception, we introduce Optimization (VPPO), novel policy gradient al- gorithm to explicitly integrate the token percep- tion into the policy update of multimodal RL, as illustrated in Figure []] Specifically, our VPPO first quantifies the visual dependency of each to- ken. Based on this visual dependency, we devise two strategies. First, to align the learning objec- tive with purception-groundde reweights each trajectory's advantage using its average dependency. In this way, the learning sig- nal is steered toward robust, perception- grounded reasoning paths over spurious shortcuts. Second, to focus the learning signal on what truly mat- ters, VPPO constructs a sparse gradient mask to concentrate policy updates exclusively on critical esually-grounded reasoning tokens. This directly counters signal dilution, yielding a lower-variance gradient that leads to faster convergence and a stronger final policy. Notably, our VPPO can be seam- lessly plugged into mainstream RLVR algorithms such as GRPO and DAPO.
Trajectory Group
13 74
Atr A(rz)ii A(rs)U A(Ta)Ul
Textual Input
Policy Updale
Masked Visual Input
Filter Pivotal Grodients angle radius "ciftlé centerarc
Visual Dependency
trajectories, VPPO
Figure Our VPPO framework explicitly relies on token visual dependency to shape trajectory advantages and filter token gradients.
To validate the effectiveness of our proposed VPPO, we conduct extensive experiments across a suite of eight challenging multimodal reasoning benchmarks, covering mathematical, geometric, logical, and multi--iscciplie reasoning. Based on Qwen2.5-VI. series models, our 7B variant achieves re- markable 19.2% average accuracy improvement over baseline, also surpassing previous open-source leading methods. This robust performance seamlessly scales to the 32B model, which also brings a 7.6% average accuracy improvement. Crucially, these performance gains are achieved alongside superior training stability and faster convergence, underscoring its efficiency and robustness.
To sum up. our main contributions threefold:
In this paper, we make the first attempt to analyze the perceptual mechanisms of multimodal RLVR through an innovative lens of token perception. We discover that only critical minority of tokens emerge with high visual dependency, while only a part of the trajectories are genuinely perception-driven paths.
We introduce VPPO, novel policy gradient algorithm that explicitly focuses on token perception, leveraging visual dependency to align trajectory-level objectives and focus token-level gradient updates. In this way, the model spotlights perception while reasoning.
Our extensive experiments on eight perception and reasoning benchmarks demonstrate our VPPO's superior performance. We further show its robust scalability across 7B and 32B model scales. We also perform in-depth ablation suuiess to validate the critical designs in our VPPO.
2

---

Published as a conference paper at ICLR 2026
Fine-grained Advantage
Answer Verily
0.0
Rullout
011 043 ... 9gr
Policy Update
A
A
0,1 02
OL
Original Input
Pri P23 Par
Group Fatimate
Policy
0
A
Random Masking
Reuse
Ourpuř Distributions
Macroi Shaping
Microi Masking
Trjectory Dependency
Musk: ...
Fast
OLi 9,3
o.
Mean
Top
Masked
Forward
P. P1
Visual Dependency
Input
Policy
P
DEL(P P')
Plog
p
Figure 2: Overview of our VPPO framework. Given the original and masked image inputs, we first obtain the corresponding output distributions. Then, we compute token-level visual dependency score for each trajectory. Subsequently, these token-level scores are used to generate two hierarchical control signals: at the macro-level, they are averaged into a trajectory-Ievel dependency to shape the advantage, while the micro-level, the top-k% tokens identified to create sparse binary token gradient mask. In this way, the uniform advantage is transformed into a fine-grained, targeted learning signal for the final policy update.
2 RELATED WORK
Multimodal Reasoning. While LLMs have achieved powerful reasoning in text-only do- mains (Guo et al., 2025). their visual counterparts, Large VisioneLanguage Models (LVLMs) (Bai 2024 Team et al. 2024), still exhibit a significant performance gap when tasked with this complex integration (Wang et al. 2024b; Dong et al.) 2025 Fang et al.) 2024b a 2026 Liu et al. 2024b a). Bridging this gap requires adapting the reasoning successes from text- only models to the unique demands of the multimodal space, where foundational algorithms like
ct al. 2025a Hurst et al.
PPO Schhllan et al. 2017) and GRPO (Shao et al. 2024) are being actively explored.
Dominant Strategies in Multimodal RL. Most strategies focus on enhancing components exter- nal the core learning algorithm. These approaches are largely either data-centric, focusing on the curation of essuall-grounder datasets (Bai et al.
2025b Li et al. 2025 Liang et al. (022). distilla- 2025b Huang et al. [2025] Meng et al. 2025). and design 2025); or reward-centric, seeking to engineer
tion of Chain-of-Thoughe data (Chen et al.
of training curricula (Chen et al.) 2025c Wei et al.
more informative, perception-aweee.......... aware signals (Wang et al. [222ee Ma et al. 2025 Fan ct al. 2025 Liu
itt al. 22225b Yang et al. 2025b Xia et al. 2025 Chen et al. 2025d Wan et al. 2025). Other tactics
include modilying rollouts or integrating external vision tools (Liu et al. 2025a
Wang et al. 2025a
Zheng al. 2025b). While modality agnnstic algorithmic advances like Dynamic Sampling Policy
Optimization (DAPO) (Yu et al. 2025b) introduce effective techniques like dynamic sampling and clip-higher, they still broadcast uniform learning signal to all tokens. Our VPPO counters this core limitation by intervening internally. using visual dependency to reweight trajectory advantages and focus gradient updates on pivotal moments of grouudl reasoning.
Pivotal Tokens in Reasoning. Prior works in RL for large language models identify the pivotal tokens via high-entropy "forking points" (Wang et al. 2025c). low-confidence error points targeted for exploration (Vassoyan et al., 2025), or contrastive estimation between models trained on cor- rect vs. incorrect data (Lin et al.) 2024). However, for the multimodal domain, pivotal token is not merely a logical fork but a critical moment of iisually-grounder reasoning. In this paper, we introduce VPPO, the first multimodal RL algorithm designed to formally identify the perceptually pivotal tokens via dependency and then leverage them for targeted optimization.
METHOD
In this paper, as shown in Figure 2 we introduce Visually-Pereee Perceptive Policy Optimization (VPPO) that explicitly focuses token perception by hierarchically shaping trajectory-level advantages and filtering token-level gradients. This targeted signal modulation fosters more stable, efficient, and interpretable learning.
3

---

Published as a conference paper at ICLR 2026
3.1 PRELIMINARY: GROUP RELATIVE POLICY OPTIMIZATION (GRPO)
Given a multimodal prompt (I, ( consisting of a visual input / and a textual query the old policy Te generates group of G responses, {or In the RLVR framework, binary reward R2 E [0, 1] is assigned to each complete response based solely on whether its final extracted answer matches the ground truth. While GRPO mitigates reward sparsity through a group-based advantage estimation, it remains fundamentally reliant on this coarse, outcome- based signal.
The advantage A, for response is its normalized reward:
A R4 mean [Rx
IG
(1)
std R )
The policy Te is then updated maximize a clipped surrogate objective, where this uniform advan- tage Ag is broadeast to every timestep t:
CORPO E
min (iip
(2)
G
Oi
t=1
where rr 5o I c.) Te (ot, (q, is the probability ratio.
While scalable, this outcome-based verification introduces two-tiered limitation follows:
1. Trajectory-Level Ambiguity: It treats correct solutions equally, failing to distinguish a reasoning path that is strongly grounded in visual evidence from one that arrives at the same answer through linguistic priors hallucination.
2. Token-Level Uniformity:
The single, coarse reward is then applied indiscriminately to every token in the sequence, failing to selectively reward the specific, pivotal moments of visually- reasoning that led to the correct outcome.
3.2 VISUALLY-PERCEPTIVE POLICY OPTIMIZATION (VPPO)
To study the perception multimodal reasoning, we first develop a metric to quantify visual depen- dency at each token and analyze the token perception in Section 3.2.1 Subsequently, based on the token perception, we further aggregate them into the trajectory-level dependency and uncover key insights into their non-uniform nature in Section 3.2.2 Based these findings, introduce VPPO in Section 3.2.3 for perception-centric multimodal reasoning.
3.2.1 QUANTIFYING TOKEN VISUAL DEPENDENCY
We define a token's visual dependency the information gain provided by the visual context. This is quantified by computing the Kullback-Leible (KL) divergence between the policy's predictive distribution conditioned the true image versus perturbed version, formally measuring the distri- butional shift attributable to visual input. The choice of KL divergence is validated in Appendix G where it outperforms other metrics like Jensen-Shannon Divergence and simple probability shifts.
Definition 3.1 (Token-level visual dependency). Lei I be the visual input and I' be non- 8t = (ct) the visual dependency S at step
informative, perturbed version. Al given t is the KI. divergence between the policy's outpul distributions conditioned on and
S(st; == DRL. (To(-]s,,)) I)) || no se(-(ss,I')))
(3)
A high S value indicates that the image provides critical information for the token prediction at step marking it as a key moment of visually- groundde reasoning.
With the above metric measuring the visual dependency for each token, we analyze the empirical distribution of token perception. To achieve this, we perform inference with the Qwen2.5-VLL 7B model on the vision-dominant subset of the Math Verse (Zhang et al.] 2024) benchmark. We then compute the token visual dependency for every token across all generated trajectories and demonstrate their frequency distribution in Figure The y-axis is on a logarithmic scale to better visualize the distribution'ss 's long tail, and Kernel Density Estimation (KDE) curve is overlaid for easier visualization the trend. This analysis leads to our first key insight:
4

---

Published as a conference paper at ICLR 2026
Insight 1: Token Visual Dependency is Sparsely Dis- tributed. Within the trajectory, visual reasoning is driven by a sparse set of pivotal tokens. Figure shows the sparse distribution of token-level visual dependency. Plot- ted on logarithmic y-axis, the frequency drops exponen- tially as dependency increases. This highly skewed distri- bution confirms that only a small fraction of tokens are crit- ical for visually-grounded reasoning. Further analysis con- firms their semantic importance, as these high-dependeney tokens predominantly consist of numbers, geometric con- cepts, and logical operators essential for the reasoning pro- cess. Broadcasting a uniform learning signal to all tokens thus dilutes the reward by rewarding many irrelevant, non- perceptual steps.
Distribution Token Visual Dependency
10"
Kit
10%
Dipesding a 12
10%
Prequency (Log
10
104
10
10
Token Visual Dependency
Figure 3: The skewed distribution of token-level visual dependency.
3.2.2 ANALYSIS OF REASONING TRAJECTORIES
After analyzing the token-level dependency, we aggregate this metric to the trajectory level by defin- ing the trajectory dependency S(T) as the mean of the token-level dependency scores over a full trajectory т. This score represents the trajectory's 's overall reliance on visual evidence. To explore its distribution, we use the same experimental setup as before, plotting the frequency of these trajectory dependency scores in Figure This reveals our second key insight:
Insight 2: Trajectories Exhibit Heterogeneous Visual Grounding. Not all correct reasoning paths are created equal. As shown in Figure 4 the distribution of trajectory- level visual dependency is heterogeneous. While loosely Gaussian, the distribution is right-skewed with a long tail, revealing that a distinct subset of high-dependency trajec- tories pulls the mean (0.09) to the right of the distribution's peak. Standard RL frameworks, by assigning a uniform ward, fail to distinguish the high perceptual informative- ness of these trajectories, and thus cannot preferentially learn from genuine
Distribution of Trajectory Visual Dependency
Brequency
6
0.50
0.335
0.10
11:20
Trajectory Visual Dependency
Figure 4: Distribution of trajectory dependency on perception.
reasoning.
3.2.3 VPPO POLICY GRADIENT ALGORITHM
Based on these insights, we introduce VPPO, a novel gradient algorithm that reshapes the learning signal at two levels of granularity to explicitly focus on token perception.
Micro-level: Token-level Gradient Filtering (TGF). Inspired by Insight 1, we focus on the learning signal exclusively on pivotal tokens. For each trajectory t,, we identify the set of indices K,, corresponding to the top-k% of tokens with the highest visual dependency scores. This set defines a binary gradient mask mi,e:
{1
if token t is pivotal riasonngg token otherwise
m,, T(L K.,)
(4)
This mask ensures that policy gradients computed only for the pivotal tokens that bridge vision and language, effectively filtering out noise from generic tokens and combating signal dilution.
Trajectory-level Advantage Shaping (TAS). Inspired by Insight 2, we prioritize learning from superior, high-dependency trajectories. We compute a shaping factor (<(::) for each trajectory ta in a batch B by normalizing its trajectory dependency:
Macro-level:
S(Ti) min,, S S(T;)
((T)) = Buin + (Bmax - Bmin) ma S(1,) - min,, EB S(T,)
(5)
where [8min Smax] is a scaling range. This factor rescales the original GRPO advantage, creating a Shaped Advantage: Á'(r;) = a(T,) ATRPO(TT) This adaptively amplifies updates for trajectories with high visual engagement and dampens those that are less visually grounded.
5

---

Published as a conference paper at ICLR 2026
VPPO Objective. Integrating these two modulations yields the final VPPO objective. It channels the shaped advantage A exclusively the most dependent tokens via the mask my,
lod
VPP (0)
E
max min
(6)
G
0
where A - o(T:) AGRPO.i. The synergy between the shaping factor aaT:)) and the mask mi, provides a structured, interpretable, and efficient solution to the uniform learning signal problem. A detailed, step-by-ster implementation of the entire training procedure provided in Appendix
3.3 THEORETICAL ANALYSIS
We provide theoretical analysis of how VPPO constructs lower varernvariance policy gradient estima- tor. Let V; = V log Te (oy]s; !) be the per-step policy gradient. The standard GRPO estimator for trajectory serves our baseline:
T-1
(G) AGRPO(T)
Σ
V
The VPPO estimator refines this by incorporating shaping factor ((T) and restricting the sum to the set of top-k% visually dependent tokens KG;:
gVPro(T) A(TrAARRO(TT (T)
Σ Vi
tEK+
Theorem 3.1 (Variance Reduction). The variance of the VPPO estimator is approximately related the GRPO estimator by the following expression:
Var(gvrpo) = E[a(r)31 Var(gcRPO)
The full derivation, along with the underlying assumptions, is provided in Appendix D This result reveals a significant variance reduction. By design, the sparsity ratio k is fraction in (0. 1). while the shaping factor (<T) is scaled to a narrow band around 1, ensuring their product [[a is substantially less than 1. Therefore, our VPPO reduces variance by filtering out low-dependency gradients and regularizing update magnitudes for less soualll-grounder trajectories, leading more stable and efficient learning signal.
EXPERIMENTS
Models, Data, and Baselines. To have fair comparison with previous works, following Wang et al. 2025a), we apply VPPO to the Qwen2.5-VL-7B and Qwen2.5- VL-32B base models and train on the ViRL39K. diverse collection of multimodal reasoning problems. We benchmark our models against comprehensive suite of state-of oii--fffthhehe thtthe-rrt, open- sounce reasoning LVLMs across Yu et al.) 2025b), MM- Eureka-7B (Meng et al. 2025). Thinkl.ite Lite-BB (Wang et al. 2025d). VL--Rthinnker-
both model scales. Our 7B comparison includes DAPO (Qwen2.5-VLL-77
(Wang al.) 2025a), R1-Share VL--B (aao et al. (025). NoisyRollout 7B (Liu al., (025a)). and PAPO-D- 7B (Wang et al.) 2025e). while the 32B class includes MM-Eureka-32B (Meng et al. 2025 and NoisyRollout-32. (Lu al. 2025a).
Training Details. Following Wang et al. (202e)), our models are trained for epochs with learning rate of 1e-6 and a rollout batch size of 384. We set the maximum response length to 2048 for 7B models following previous works such as RI-ShareVL, NoisyRollout, and PAPO-D, and 4096
6 for 32B models. To ensure training stability and enable fair comparison, small entropy penalty (coefficient 0. 065 is applied to both VPPO and the baseline. More details are described in Appendix E For VPPO, set the gradient filtering ratio to & 0. 4 and the advantage shaping range to 3min C. 9, with 3max adjusted dynamically per batch. More hyperparameter details are available in Appendix

---

Published as a conference paper at ICLR 2026
Table 1: Main Results (avg@8 acc %).. All benchmarks use exact match on verifiable instances for objective results, avoiding any LLM-as-a-judge. Notably, our results are achieved via direct RL without any supervised fine-tuning. 'Our reproduction uses official author-provided prompts. "NoisyRollout is trained using the training set of Geo3k.
Model
Mathematical & Geometric
Logical
discipline
Avg.
Math
DynaMath
MMK12
Geo3k
MathVision We-Math
Lisie
MMMU-Pro
Open-Source Models (Troined via Pure RI.)
MM-Fureka-7R
67.1
65.4
67.5
40.3
31.1
65.5
46.3
30.3
51.7
ThinkLie-7B
64.2
64.6
62.6
37.6
32.0
66.5
39.4
28.0
49.4
MT-Rethinker-70
40.7
53.5
NoisyRollout-78
67.8
65.5
50.0
$1.8"
22.1
71.0
47.3
34.5
51.3
RI-ShareVL-7D
68.0
65.1
70.9
41.2
30.1
69.9
45.6
35,1
53.2
PAPO-D-7R
30.61
46.7
55.3
Qwen2.5.VT.22
GRPO
66.5
65.8
72.3
40.2
30.7
68.1
45.6
35.2
53.1
DAPO
68.3
66.6
82.1
41.5
30.5
68.0
46.8
35,9
55.0
57.5
Scaling Larger Models
MM Eureka 32B
71.8
72.0
73.4
51.0
43.2
75.0
56.8
43.1
60.8
NoisyRollou--32B
73.0
72.2
60.2
56.6
27.9
75.7
56.2
43.1
58.1
Qwen2.5-VL- 5-VLL32B
47.0
39.3
71.0
52.8
57.0
+ GRRO
74.2
71.6
80.7
51.4
42.8
76.7
58.3
45.4
62.6
DAPO
73.3
72.6
86.4
51.4
42.8
76.2
58.9
46.4
63.5
VPPO
75.1
73.1
86.3
53.4
44.6
77.7
59.2
64.6
Evaluation Benchmarks. We conduct comprehensive evaluation on eight diverse multimodal rea- soning benchmarks. Following Wang et al. (2025e), we use an exact-match scoring methodology, eliminating reliance on LLM-ee-a-judge systems. The benchmarks span mathematical, geometric, logical, and multi-discipline reasoning, including DynaMath (Zou et al. (024)), Geo3k (Lu et al. Math Verse (Zhang et al., 2024 Vision (Wang et al. 2024a), MMK12 (Meng et al, We-Math (Qiao al.] 2024), Visia (Xiao et al. 2024 and MMMU-Pro (Yue al,
2021
2025
2024
(see Appendix for full breakdown). We report average accuracy @ 8 at an inference tem-
perature of 1.0, using single, fixed evaluation pipeline for all models ensure fair comparison.
RESULTS
5.1
MAIN RESULTS
As shown in Table VPPO consistently outperforms the entire field of strong, open-source competitors across both 7B and 32B parameter classes. In the 7B class, our model achieves an average accuracy of 57.5%, significantly outperforming the next-best best model PAPO. This superior performance scales directly the 32B class, where VPPO again leads the field with average accuracy of 64. 6%, surpass- ing the next-best method, DAPO. These results across different model scales demonstrate the effectiveness of our VPPO.
Training Dynamics
Training Accuracy
0.72
0.64
0.56
Melad
0.40
C3P0
These state-of-the-art results are underpinned by superior training dy- namics, illustrated in the training curves against the baselines (Fig- 3). which demonstrates that VPPO exhibits significantly faster ini- tial convergence, achieving higher performance geted, hierarchical learning signal not only leads to a better final model but also acts as potent implicit regularizer, ensuring more efficient and robust path high performance.
Step
Figure 5: Training dynam- ics for VPPO and baselines.
efficiently. This demonstrates that tar-

---

Published as a conference paper at ICLR 2026
5.2 ABLATION STUDIES
Table 2: Ablation Trajectory-level Advantage Shaping (TAS) and Token-level Gradient Filtering (TGF), Their combination yields the best results, confirming the efficacy of our hicrarchical design.
Model Configuration
DynuMath MMK12
We-Math
LogicVista
MMMU-Pru
Avg.
Baseline (DAPO)
68.3
66.6
82.1
41.5
30.5
68.0
46.8
35.9
55.0
+ TAS only
67.5
83.3
43.5
31.3
69.3
47.4
37.3
56.3
TGF only
68.6
80,9
34.7
70.3
48.2
37.3
57.1
VPPO (TAS + TGF)
71.6
68.1
82.8
46.5
33.3
71.5
47.9
37,9
57.5
Ablation Study on VPPO Components. We first analyze the effectiveness of our two primary mechanisms: Trajectory-Ievel Advantage Shaping (TAS) and Token-level Gradient Filtering (TGF). As shown in Table 2 both components individually outperform the baseline. 'GF provides the largest single contribution, highlighting the importance of directing the learning signal to pivotal tokens. However, the combination of both mechanisms in the full VPPO model achieves optimal performance, confirming the synergistic value of our hierarchical design.
Sensitivity to Gradient Filtering Ratio k. We investi- gate how performance varies with the token filtering ra- tio & in TGF. As shown in Figure 6 performance peaks around & 0 4. This highlights crucial trade-off: k that is too low provides insufficient learning signal, while a & that is too high reintroduces noise from non-pivotal tokens, validating our sparse update strategy.
Impact of Gradient Filtering
Score Devistion from Mean
58.2
Average Seare
0.0
57.0
Merios
0.3
Seuth
36.4
Westuc
Log
02
Gradient Filtering
Sensitivity to Advantage Shaping Range. We analyze the sensitivity of model to the TAS scaling range [8min: 3max] Table 3 shows that a conservative lower bound with dynamic upper bound (8min 0.9, 8max Dyn.) performs best. This setting adaptively reweights advantages based on batch-wise dependency distribu- tions, preventing aggressive updates while rewarding essualy-groundee reasoning.
Figure Ablation on the gradient filter- ing ratio (k). The line shows the average score, while bars show per-benchmark deviation from their mean.
Table 3: Ablation study on the scaling range [Smins Suax] for Trajectory-level Advantage Shaping (TAS), including both fixed and dynamic (Dyn.) configurations.
TAS Configuration
MathVerse DynaMath MMK12 Genāk
MathVision We-Math LogicVista MMMU-Pro Avg.
Baseline (DAPO)
68.3
66.6
82.1
41.5
30.5
68.0
46.8
35.9
55.0
Piin = 0.8, 9laax 1.2
68.7
67.5
82.9
43,4
31.9
69.4
36.7
Swin D.8, 8max Dyn.
Puin 0.9, 9max 1.1
69.1
67.6
82.6
43.2
31.5
69.2
46.6
37.2
55.9
9mm = U.9, 9max Dyn.
70.4
67.5
83.3
43.5
31.3
69.3
47.4
37.3
56.3
Validation of the dependency Calculation Method. To further validate the robustness of our core visual dependency metric, we conducted two additional, detailed ablation studies presented in the appendix. The first study (Appendix [T evaluates our choice of image perturbation strategy against several alternatives. The second (Appendix G) compares our KL-divergence metric against other computationally-fessii feasibble calculation heuristics.
Superiority over Entropy-based Token Selection. depicted in Table we compare different methods for selecting pivotal tokens in multimodal reasoning, where the filtering ratio & determines the percentage of tokens retained for gradient computation (via random selection or top-k% rank- ing). For text-only LLMs, high-entropy "forking tokens" is an effective optimization strategy (Wang) et al. 2025c). However, this strategy fails to yield significant gains in multimodal tasks. While high entropy effectively captures logical reasoning steps (e.g., operators, connectors) where the model is uncertain, it overlooks aisuallyggrounle facts. Tokens representing direct observations (c.g.. spe- cific numbers like 25, entities like AAOB) often exhibit low entropy because the model is confident
8

---

Published as a conference paper at ICLR 2026
Performance comparison of Token-level Gradient Filtering (TGF) under three guidance signals: visual dependency (our method), predictive entropy, and random selection. h denotes the ratio of tokens retained for the policy update (e.g., top-k% % random k%)
Table 4:
Guidance Mechanism
MathVerse
DynaMath
MMK12
Gea3k
MathVision
We-Math LogicVista
MMMU-Pro Avg.
Baseline (DAPO)
68.3
66.6
82.1
41.5
30.5
68.0
46.8
35.9
55.0
Random 0.4)
69.3
66.2
76.8
42.0
31.0
69.3
47.5
36.2
54.8
Entropy (k 0.2)
77.9
32.6
Entropy (k
69.3
67.6
80.0
42.8
31.7
69.4
47.4
37.0
55.7
Entropy (k 0.6)
69.9
67.4
81.0
43.4
31.4
69.1
47.1
36.9
55.8
Entropy (k 0.8)
69.6
66.9
81.1
41.6
31.2
69.0
46.6
36.2
55.3
TGF
34.7
37.3
57.1
perceived, yet they possess high visual dependency. Unlike entropy-based methods that miss these foundational premises, VPPO targets both the uncertain reasoning junctions and these confi- dent, indispensable visual facts, thereby building reasoning on a more solid perceptual foundation.
Table 5: Ablation study on the generalizability VPPO by applying to GRPO. The consistent improvement confirms its benefits are independent of the base policy gradient algorithm.
Model
MathVerse DynuMath MMK12
Geo3k
MathVision
We-Math
LogicVista
MMMU-Pro
Avg.
Qwen2.5-VLL-78
39.0
55.7
42,5
37.1
18.4
46,4
42.4
25.1
38.3
+ GRPO
66.5
65.8
72.3
40.2
30.7
68.1
43.6
35.2
53.1
VPPO w/ GRPO
66.4
76.4
31.7
69.5
47.6
35.8
54.8
Generalization to the GRPO algorithm. verify VPPO's generality, we implemented on top of GRPO. As shown in Table 5 VPPO improves GRPO's accuracy by 1. 7% (from 53. 5% to 54.8% 8%). This result is consistent with the 2. 5% improvement observed when applying VPPO to DAPO (Table D confirming that the performance gains attributable to optimization strategy rather than a specific interaction with the base policy gradient algorithm.
Aisuall--perceptin
Table 6: Performance comparison of our binary mask against continuous soft mask for TGF. The binary mask's superior performance validates more decisive filtering of non-pivotal gradients.
Algorithm
Math Verse
DynaMath
MMK12
Geo3k
Math Vision
We-Math
LogicVista
MMMU-Pru
Avg.
Mask
67.2
46.6
36.3
VPPO (Binary Mask)
71.6
68.1
82.8
46.5
33.3
71.5
47,9
37.9
57.5
Binary versus Soft Gradient Filtering. We evaluated our binary mask for Token-level Gradient Filtering (TGF) against continuous soft mask that assigns a calibrated weight to each token's s gradient; the specific implementation detailed in Appendix [1] As shown in Table [ our binary mask is more effective, outperforming the soft mask by 1 % (which itself surpassed the baseline by 1. 2%). We hypothesize the binary mask acts as more decisive noise filter; hard-gating of gradients from non-pivotal tokens creates stronger and more focused learning signal.
Table 7: Performance comparison of advantage shaping versus reward shaping. The superior per- formance of advantage shaping validates its use for more stable policy gradient update.
Algorithm
VathVerse
Mynh
MMK12
Geo3k
MathVision
We-Math
LogieVista
MMMU-Pro
Avg.
DAPO
68.3
66.6
82.1
41.5
30.5
68.0
46.8
35.9
55.0
VPPO w/ Reward Shaping
70.7
68.4
82.6
44.7
33.5
70.1
47.1
37.6
56.8
VPPO (Ads. Shaping)
71.6
68.1
82.8
46.5
33.3
71.5
47.9
37.9
57.5

---

Published as a conference paper at ICLR 2026
Advantage Shaping versus Reward Shaping. We compared our strategy of modulating the ad- vantage term against the alternative of scaling the raw reward. As shown in Table shaping the advantage is more effective, outperforming reward shaping by 0. 7% average. We attribute this to greater stability; directly modulating the final advantage applies a clean scaling to the gradient update, whereas modifying the reward before the advantage calculation can introduce variance and create noisier learning signal.
5.3 GENERALIZATION TO OUT-OF-DOMAIN VQA
To ensure our method does not impair general visual- language capabilities, we evaluated its performance on two unseen, out-of-domain VQA benchmarks: A-OKVQA-
Table 8: Performance on out-of-
domain VQA benchmarks.
val (Schwenk et al., 2022) and SiApleVQA-EN (Cheng
et al. 2025). The results, presented in Table confirm that all evaluated RL fine-tume tine-tuning methods significantly improve upon the Owen2.5-VL-7B base model (approx. +4% av- erage accuracy), indicating positive transfer of reasoning skills to general VQA. Crucially, VPPO achieves the high- est overall score. We attribute this superior generalization its core mechanism of focusing on perceptually pivotal tokens, which enhances the model's fun- damental visual grounding, a core skill that robustly benefits standard VQA tasks.
Model
A-OKVQA
SimpleVQA Avg.
Owen2.5-VL775
84.2
38.6
61.4
GRPO
87.4
43.1
65.3
DAPO
87.9
42.9
65.4
VPPO
87.9
43.8
65.9
5.4 QUALITATIVE ANALYSIS
Visually Dependent Tokens (Top 40% Highlighted)
angle
ciccomference
angle
girverx
dlegrees
are
rudii
träangle OAB inusclles
anple
center subleandd by amc terice the anglle the ctrcumfeewnce no
As shown in figure, then
the dogree angle BAO is
degrooss
En an
Sesecetes trtungte the boxe anglne ane equat Therufore
angle
BAA
Choices:
A:50" B:55 C:60" Dx65*
angle
ABO
180
50
degrees
Aue
think
bused
D
<ttm_eed|>>
<:imm endd
2
Figure 7: The top 40% most visually-dependee tokens are highlighted in purple, forming the core reasoning chain targeted by our gradient filtering mechanism.
To further understand the token perception, we provide qualitative analysis in Figure 7 As shown in this figure, high dependency is assigned to foundational concepts like c ruum ference and the angle value 25. The dependency then correctly propagates intermediate conceptual enti- ties (triangle OAB, arc) and, crucially, to the logical syntax that structures the proof (S (Since, There fore). This demonstrates a sophisticated understanding that captures not only what con- cepts are important but how they are linked to form a coherent proof.
CONCLUSION
In this paper, we identify the uniform learning signal as core bottleneck in multimodal reasoning and introduce Visually-Percepiin Policy Optimization (VPPO) as principled solution. By imple- menting a novel, two-tiered strategy, VPPO first prioritizes iisually-gruunde reward shaping and then focuses policy updates exclusively kens. This hierarchical signal modulation not only establishes a new state-of-the-art across a diverse suite of challenging benchmarks but also fosters greater training stability and efficiency. Our work demonstrates that for complex multimodal tasks, the structure of the learning signal is as important as the reward itself. We believe that this principle of targeted, modality- -aware signal modulation offers a promising and robust path forward for advancing the reasoning capabilities of Large Vision- I Language Models.
trajectories through sparse set of pivotal perception to-
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our research is conducted entirely within the domain of multimodal reasoning. We exclusively use publicly available academic benchmarks, which do not contain any personal, sensitive, or private user data. No new data was collected for this study, and no human subjects were involved. The goal of our work is to enhance reasoning capabilities Al models on mathematical and logical problems. Given this focus abstract problem-solving. we are not aware of any direct, foreseeable negative societal impacts or ethical concerns arising from our methodology or findings.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide anonymous code repository in the sup- plementary materials containing the full implementation of our VPPO algorithm and the complete evaluation pipeline. All datasets used for training and evaluation are publicly available, and com- prehensive breakdown of our experimental setup, including all key hyperparameters, is detailed in Appendix
B and summarized in Table 9 For our theoretical claims, the main results are presented in Section 3.3 while the complete, step-by-step proofs and formal list of our assumptions are pro- vided in Appendix D We believe these resources sufficient for the research community build upon and verify our findings.
REFERENCES
Anthropic. Claude sonnet 4, 2025. URLLattss//ww..attrrpp.....
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025a.
Sule Bai, Mingxing Li, Yong Liu, Jing Tang. Haoji Zhang, Lei Sun, Xiangxiang Chu, and Yansong Tang. Univg-rl: Reasoning guided universal visual grounding with reinforcement learning. arXiv preprint arXiv:2505. 10.55055.12.... 14231, 2025b.
Liang Chen, Hongcheng Gao, Tianyu Liu, Zhiqi Huang, Flood Sung, Xinyu Zhou, Yuxin Wu, and Baobao Chang. G1: Bootstrapping perception and reasoning abilities of vision-language model via reinforcement learning. arXiv preprint arXiv:2505.. 13426, 2025a.
Liang Chen, Lei Li, Haozhe Zhao, and Yifan Song. Vinci. rl-v: Reinforcing super generalization ability in vision-language models with less than $3, 2025b.
Shuang Chen, Yue Guo, Zhaochen Su, Yafu Li, Yulun Wu, Jiacheng Chen, Jiayu Chen, Weijie Wang, Xiaoye Qu, and Yu Cheng. Advancing multimodal reasoning: From optimized cold start to staged reinforcement learning. arXiv preprint arXiv:2506.042000 2025c.
Yi Chen, Yuying Ge, Rui Wang, Yixiao Ge, Junhao Cheng, Ying Shan, and Xihui Liu. Grpo- care: Consistency- reinforcement learning for multimodal reasoning. arXiv preprint arXiv:2506.16144 2025d.
Xianfu Cheng, Wei Zhang, Shiwei Zhang, Jian Yang, Xiangyuan Guan, Xianjie Wu, Xiang Li, Ge Zhang, Jiaheng Liu, Yuying Mai, et al. Simplevqa: Multimodal factuality evaluation for multimodal large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4637-4646. 2025.
Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9062-9072, 2025.
Yue Fan, Xuchai He, Diji Yang, Kaizhi Zheng, Ching Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Narayamaraju, Xinze Guan, and Xin Eric Wang. Grit: Teaching milms to think with images. arXiv preprint arXiv:2505. /5879, 2025.
11

---

Published as a conference paper at ICLR 2026
Xiang Fang. Wanlong Fang, Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Renfu Li, Zichuan Xu, Lixing Chen, Panpan Zheng, et al. Not all inputs are valid: Towards open-set video mo- ment retrieval using language. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 28-37, 2024a.
Xiang Fang, Zeyu Xiong, Wanlong Fang, Xiaoye Qu, Chen Chen, Jianfeng Dong, Keke Tang, Pan Zhou, Yu Cheng, and Daizong Liu. Rethinking weakly-supervised video temporal grounding from game perspective. European Conference on Computer Vision. Springer, 2024b.
Xiang Fang, Wanlong Fang, Changshuo Wang, Xiaoye Qu, and Daizong Liu. Rethinking video- language model from the language input perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint www.v::503.00777 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Shenshen Li, Kaiyuan Deng, Lei Wang. Hao Yang. Chong Peng, Peng Yan, Fumin Shen, Heng Tao Shen, and Xing Xu. Truth in the few: High-value data selection for efficient multi-modal reason- 2025.
ing. arXiv preprint arXiv:2506.0477
Yiqing Liang Jielin Qiu, Wenhao Ding, Zuxin Liu, James Tompkin, Mengdi Xu, Mengzhou Xia, Zhengzhong Tu, Laixi Shi, and Jiacheng Zhu. Modomodo: Multi-domain data mixtures for multimodal Ilm reinforcement learning. arXiv preprint arXiv:2505.2487/ 2025.
Zicheng Lin, Tian Liang Jiahao Xu, Qiuzhi Lin, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang. and Zhaopeng Tu. Critical tokens matter: Token-level contrastive estimation enhances Ilm's 's reasoning capability. arXiv preprint 199993341..19943. 2024.
Daizong Liu, Xiang Fang, Xiaoye Qu, Jianfeng Dong, He Yan, Yang Yang. Pan Zhou, and Yu Cheng. Unsupervised domain adaptative temporal sentence localization with mutual information maxi- mization. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 3567- 3575, 2024a.
Daizong Liu, Xiaoye Qu, Xiang Fang, Jianfeng Dong, Pan Zhou, Guoshun Nan, Keke Tang, Wan- long Fang. and Yu Cheng, Towards robust temporal activity localization learning with noisy labels. In LREC-COLING, 2024b.
Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shich. Noisyrollout: Reinforcing visual reasoning with data augmentation. arXiv preprint 1rXiv:2504.13055 2025a.
Yuqi Liu, Tianyuan Qu, Zhisheng Zhong, Bohao Peng. Shu Liu, Bei Yu, and Jiay Jia. Vision- reasoner: Unified visual perception and reasoning via reinforcement learning. arXiv preprint arXiv:2505. 12081, 2025b.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint urXiv:2105.04160 045,0 2021.
Yan Ma, Linge Du, Xuyang Shen, Shaoxiang Chen, Pengfei Li, Qibing Ren, Lizhuang Ma, Yuchao Dai, Pengfei Liu, and Junjie Yan. One rl to see them all: Visual triple unified reinforcement learning. arXiv preprint 1811 2025.
12

---

Published as a conference paper at ICLR 2026
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang. Junjun He, et al. Mm-eureka: Exploring the frontiers of multi- modal reasoning with rule-based reinforcement learning. arXiv preprint 20t 2025.
OpenAl. Learning to reason with 1lms. 2024. Learning-to-reasen withnwwwthh-eee 11ms/
URL
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multi- modal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang. Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. VIm-rl: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07611 2025.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409,19256, 2024.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens context. arXiv preprint arXiv:2403.05554 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint 15X1v:2501.12599, 2025.
Jean Vassoyan, Nathanaël Beau, and Roman Plaud. Ignore the kl penalty! boosting exploration on critical tokens to enhance fine-tuning. arXiv preprint arXiv:2502.00 06533,, 2025.
Zhongwei Wan, Zhihao Dou, Che Liu, Yu Zhang, Dongfei Cui, Qinjian Zhao, Hui Shen, Jing Xiong, Yi Xin, Yifan Jiang, et al. Srpo: Enhancing multimodal IIm reasoning via reflection-aware rein- forcement learning. arXiv preprint arXiv:2506.0/713, 2025.
Haozhe Wang. Chao Qu, Zuming Huang. Wei Chu, Fangzhen Lin, and Wenhu Chen. VI- rethinker: Incentivizing self-reflection of vasion-language models with reinforcement learning- arXiv preprint arXiv:2504.0883: 2025a.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hong- sheng Li. Measuring multimodal mathematical reasoning with nath-vision dataset. Advances in Neural Information Processing Systems, 37:95095-95169, 2024a.
Peiyu Wang, Yichen Wei, Yi Peng. Xiaokun Wang. Weijie Qiu. Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang, Yunzhuo Hao, et al. Skywork r1v2: Multimodal hybrid reinforcement learn- ing reasoning. arXiv preprint arXiv:2504. 16656, 2025b.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint urXiv:2506.01933 2025c.
13

---

Published as a conference paper at ICLR 2026
Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao. Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411. 2024b.
2024b.
Xiyao Wang. Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung- Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self- self-improvemen. arXiv preprint arXiv:2504. 07934, 2025d.
Zhenhailong Wang, Xuchang Guo, Sofia Stoica, Haiyang Xu, Hongru Wang, Hyeonjeong Ha, Xiusi Chen. Yangyi Chen, Ming Yan, Fei Huang, et al. Perception-awer -warr policy optimization for mul- timodal reasoning. arXiv preprint arXiv:2507.0644: 2025e.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24831 2022.
Yana Wei, Liang Zhao, Jianjian Sun, Kangheng Lin, Jisheng Yin, Jingcheng Hu, Yinmin Zhang, En Yu, Haoran Lv, Zejia Weng, et al. Open vision reasoner: Transferring linguistic cognitive behavior for visual reasoning. arXiv preprint arXiv:2507.05255, 2025.
Jiaer Xia, Yuhang Zang, Peng Gao, Yixuan Li, and Kaiyang Zhou. Visionary-rl: Mitigating short- cuts in visual reasoning with reinforcement learning. arXiv preprint 2rXiv:2505.16677, 2025.
Tong Xiao, Xin Xu, Zhenya Huang. Hongyu Gao, Quan Liu, Qi Liu, and Enhong Chen. Advancing multimodal reasoning capabilities of multimodal large language models via visual perception reward. arXiv preprint MwXiv:2506.072//1 2025.
Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal I1m logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.04993 2024.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.099388, 2025a.
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal rea- soning through cross-modal formalization. arXiv preprint arXiv:2503. 10615, 2025b.
Huanjin Yao. Qixiang Yin, Jingyi Zhang, Min Yang. Yibo Wang. Wenhao Wu, Fei Su, Li Shen, Minghui Qiu, Dacheng Tao, et al. RI-sharevl: Incentivizing reasoning capability of multimodal large language models via share-grpe grpp. arXiv preprint arXiv:2505. 2025.
En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, al. Perception-rl: Pioneering perception policy with reinforcement learning. arXiv preprint arXiv:2504.009555 2025a.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.144: 14476, 2025b.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.0280 02813,, 2024.
Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, et al. A survey of reinforcement learning for large reasoning models. 2025.
arXiv preprint arXiv:2509.0882
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang. Yu Qiao, al. Mathverse: Does your muulii-mddl Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp. 186-186. Springer, 2024.
14

---

Published as a conference paper at ICLR 2026
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang. Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arX1v:2505.03335, 2025.
Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangehi Feng, Dongdong Kuang, and Yuwen Xiong. Easyrl: An efficient, scalable, multi-modality rl training framework. https://githmii com/ 2025a.
tilyoga/EassyR]
Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deepeyes: Incentivizing" thinking with images" via reinforcement learning. arXiv preprint arXiv:2505. 140.2222,.443,,, 2025b.
Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint arXiv:2411.00836 2024.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
APPENDIX CONTENTS
LLM Usage Statement
16
Implementation Details
16
Training Procedure
18
D Proofs for Theoretical Analysis
18
D.I Formal Setup and Assumptions
19
D.2 Proof of Theorem 3.] Variance Reduction)
19
The Role of the Entropy Penalty in Stabilizing Training
21
Ablation Study on Masking Strategy for dependency Calculation
22
Ablation Study Methods for dependency Calculation
23
Ablation Study on Rollout Group Size
24
Soft Mask Calibration for Gradient Filtering
25
J Analysis of Computational Overhead
26
K Limitations
26
L Analysis of the Training Dataset
27
M Analysis of Evaluation Benchmarks
27
Prompt Template
28
Qualitative Case Studies: VPPO vs. Baseline
29
A LLM USAGE STATEMENT
In the preparation of this paper, we used a large language model (LLM) as an assistive tool. Its role was strictly limited to proofreading for grammatical and spelling errors, and rephrasing sentences to enhance readability and clarity. The LLM was not used for generating core ideas, data analysis, or writing the main content of the paper. All intellectual contributions and the final text are the sole responsibility of the authors.
B IMPLEMENTATION DETAILS
Overall Setup. Our implementation is built upon the EasyR1 framework (Zheng al. 2025a Sheng et al., 2024). All experiments were conducted using Torch 2.6.0 with CUDA 12.4. The base models for our experiments are the open-source Qwen2 5 5-VI.7B and Qwen2, 52..
16

---

Published as a conference paper at ICLR 2026
Training Details. We train all models for two epochs the V1RL39K dataset (Wang et al, 2025a). The vision tower is unfrozen during training. For the online RL process, we generate 8 responses per question. Our reward signal is simple binary accuracy score (1 for correct, for incorrect). Our training objective follows the DAPO recipe, incorporating dynamic sampling, clip-higher, and token-level policy gradient loss, without KI. divergence penalty. All key hyper- parameters for the optimizer, RL process, and evaluation are detailed in Table g
Table 9: Key hyperparameters for training and evaluation.
Hyperparameter
Value
General Training
Optimizer
AdamW
Learning Rate
1c-6
LR Schedulc
Constant (no warmup or decay)
Epochs
2
Freeze Vision Tower
False
RL Process
Global Batch Size
128
Rollout Batch Size
384
Rollouts per Prompt
8
Rollout Top-p
0.99
Max Response Length
2048 (7B), 4096 (32B)
Reward Signal
Binary Accuracy (1/0)
DAPO Recipe
Sampling Method
Dynamic Sampling
Clip Ratio Low
0.2
Clip Ratio High
0.28
Loss Averaging Mode
Token-level
KL Penalty
None
VPPO Specific
TAS 3min
0.9
TAS 3max
Dynamical morcc--)rmaalizd)
TGF Ratio (k)
0.4
Evaluation Generation
Temperature
1.0
Top-D
1.0
Max New Tokens
2048 (7B), 4096 (32B)
VPPO Configuration. Our proposed VPPO method introduces two key mechanisms, Trajectory- level Advantage Shaping (TAS) and Token-level Gradient Filtering (TGF), whose specific hyperpa- rameters are detailed in Table The underlying visual dependency metric that guides these mecha- nisms was also carefully selected. As detailed in our ablation studies, the final VPPO configuration uses the following validated components:
Dependency Calculation: Visual dependency is calculated using KL Divergence, which we found to be empirically superior to other heuristics (see Appendix G). This is imple- mented with the efficient "low_ var kl" estimation function provided by the EasyR1 frame- work.
Masking Strategy: We use Random Patch Blackening as the image perturbation method, which was validated as the most effective strategy in Appendix F The image is divided into non-overlapping patches of size 14x14, and each patch is independently set to black with probability of 0.5.
Computational Resources. All models were trained on a cluster of 8 x NVIDIA H800 80GB GPUs.
17

---

Published as a conference paper at ICLR 2026
Algorithm The Visually-Perceppie Policy Optimization (VPPO) Algorithm
Input: Current policy Te, old policy Rlau batch of prompts D [
2: Hyperparameters: Group size G, dependency filtering ratio shaping range Bainn Smax)
3: procedure VPPO PPPO_TRAIINGG.. (_S___STEPTTppp Keay D)
4: Initialize lists for trajectories T < []. original distributions <-
Phase 1: Data Generation (Rollouts)
5: for each prompt (I. in D do
6: for i = 1 to G do
7:
Generate trajectory 7; = (o1...., 0r) using Ta,y (:1 Store original distributions P. ( (188,, l)t==1 Append to T and P. to P
8:
9:
end for
11: end for
b Phase 2: dependency Calculation
12: Initialize list for dependency scores S t
for each trajectory т; and distributions in (T,P) do Let (7, q) be the prompt for тÜ
14:
15:
Create masked image I' + MaskingStrategy(2 (1) Compute masked distributions P. (T St, 1')] ]t=1
16:
17:
Initialize token dependency scores S; < for t T do
18:
19:
Si, f DKI, (Pr,t P(,)) Append S to S₁
20:
21:
end for
Append S: S
23:
end for
Phase Hierarchical Signal Modulation
Compute rewards {1 [ and standard advantages {A,] 177
24:
25:
Initialize lists for shaped advantages A' and masks M +1+ for each trajectory r; and its dependency scores St in (T S) do
26:
27:
Macro-level Advantage Shaping
3, + 7 ΣT= Sp, 0; + Normalize S., within batch, [Smin Smasx) Append a, A, A'
28:
30:
31:
Micro-level Gradient Filtering
32:
K2 + Indices of top values in S₁ Append (I(t K.)) to M
33:
34:
end for
Phase 4: Policy Update
35:
Compute loss (PPO using T, À', and M per Eq. Update policy parameters: & OptimizerStepl C
36:
end procedure
C TRAINING PROCEDURE
For clarity and reproducibility, provide detailed, step-by-step description of our Visually- Perceptive Policy Optimization (VPPO) training procedure in Algorithm []] This pseudocode elab- orates the high-level methodology presented in Section 3.2 of the main text. It details the four core phases of each training step: (1) data generation via rollouts, (2) the calculation of token-level visual dependency, (3) our hierarchical signal modulation, and finally, (4) the policy update using the modulated learning signal.
D PROOFS FOR THEORETICAL ANALYSIS
This section provides the detailed derivations for the theorems presented in Section 3.3
18

---

Published as a conference paper at ICLR 2026
D.1 FORMAL SETUP AND ASSUMPTIONS
Let V, Va log (or]st, [) denote the score function, or the per-step policy gradient, at timestep t. The proofs rely on the following standard assumptions.
Assumption 1 (Uncorrelated Gradients). The per-step gradients within trajectory approxi- mately uncorrelated. Formally, for # j. Elv? v,] s O. This a common assumption in policy gradient analysis, gradients at different timesteps are often driven by different and nearly inde- pendent states.
Assumption (Advantage Independence). The trajectory-level advantage, (C)PO(T), treated random variable that is independent of the per-step gradients, Vr- This is justified as the ad- vantage is a scalar value computed over the entire trajectory's 's outcome, while the gradients are high-dimensional vectors dependent specific states.
Independence), For the purpose of this analysis, we as- sume the trajectory shaping factor a(T) and the advantage (T)PO(T) are uncorrelated. This sim- plification allows to isolate the distinct variance reduction effects of trajectory-Ievel advantage shaping and token-level gradient filtering.
Assumption (dependency-AAvvanta,
Assumption (Second-Moment Dominance). In high-dimensional optimization, the variance of the gradient estimator, Var(g) = |l|lg]|?] [ ? is dominated by the second moment, EE[] gl =] This is because for a well-behaved optimization, the expected gradient |[[[g]]] I| is typically much smaller than the expectation of the squared norm. Therefore, we analyze the variance by comparing the second moments: Var(g) x []]]]]/?]
D.2 PROOF OF THEOREM 3.1 VVARIANCE REDUCTION)
Theorem D.1. Under Assumptions 1-4, the variance of the VPPO gradient estimator reduced by factor of approximately k [] compared to the GRPO estimator.
Proof. We will derive and compare the second moments of the GRPO and VPPO gradient estima- tors.
19

---

Published as a conference paper at ICLR 2026
1. Second Moment of the GRPO Estimator. First, analyze the GRPO estimator, gGRPO(T)
AGRRO Et T- Vt.
T-1
Blllgarro|1 = E
AORPO
Vi
27
E AGRPO(T)²
Σ
T-1 Σ Ve
Assumption
[TAGRPO(7))!
T-1
E[ACRPO(T)71-1
Σvi Vi
Σ
E[AARro(7))]]-E
viell
Σ
v/V1
1
E[(ARPO(7)
Blllvel|]]
vil E [
Assumption
E E[AGRPO
Second Moment of the VPPO Estimator. Next, we perform the same derivation for the VPPO estimator, (TrPo(T) = a(T) (CRF) ΣLEK, Vz.
gvprol |
((()) AGT)ARR Σ V.
BEK.
a{T)² A
Σ V
KEK+
Assumption E[a(T) ((Rr()))ARRO((773
Σ
lee
Assumption 3
[[(a())][[AAcRoo(]))) 1
llvel1 +
E
v
T
tEN,
(JEK,
Assumption
R3
([a(T))][(ARRvV))) (T)²1 Σ
B||lv.119
tEK.
3. Comparison and Conclusion. Assuming the expected norm of the per-step gradients is roughly constant across timesteps, [|] 2o C, the summations for the GRPO and VPPO estimators simplify. The GRPO sum pivotal tokens, K,, where [ k T. This yields:
all T timesteps, while the VPPO sum only over the set of
E EGRPO
[|] T C. [[AGRPO(T)]]
E |v/ro
|| ~ (k: T) C. [lo(r)"]E[AAcRoo())
By taking the ratio and applying Assumption 4, we arrive at the relationship shown in the main text: Var(gvrro) [v[loo
[|] k [B( [[[]gGRro|| [12] k B[a(T))] Var(goRPo) (10)
20

---

Published as a conference paper at ICLR 2026
This demonstrates direct reduction in variance proportional to the sparsity ratio i and the expected squared shaping factor, which leads to more stable training.
E THE ROLE OF THE ENTROPY PENALTY IN STABILIZING TRAINING
In our main experimental setup, small entropy penalty is added to the loss function. This section provides detailed analysis of why this regularization is critical component for achieving stable training with online RL in the context of LVLMs.
Option A: (BC parallll EF)) This gives us angle congruences (V'angle BCA langle EFD:)), but it does not directly help achieving congruence through SSS plus angle congruence necessary congratitude circumstance Existsclude accur/plain. CailainnCCo..n Con iimmatts nen Kim gladly
org>s[level/softwar flown.walk elem установлен recherche cur stity
Figure 8: Catastrophic policy collapse in the DAPO baseline when trained without regularization. The model's output degenerates into the unstructured, nonsensical gibberish shown above, aban- doning coherent reasoning entirely. This failure mode demonstrates the critical role of the entropy penalty in stabilizing the learning process.
The Phenomenon of Policy Collapse. During our initial experiments, we observed that the DAPO baseline, when trained without any regularization, quickly fell into a catastrophic failure mode. After brief period of exploration, its policy would collapse, causing the model to generate incoherent gibberish (Figure (). sequences of tokens that were not only nonsensical but often appeared as random, unformatted strings with no resemblance to valid language. This is severe form of the well-known RL phenomenon known as "policy collapse," where the model forgoes meaningful rea- soning entirely in favor of an exploit, however nonsensical, that it has correlated with positive reward.
Sparse, Coarse-Grained Rewards. This collapse direct consequence of the sparse and coarse-grained nature of the reward signal in the RLVR framework. The model receives a single binary reward for an entire, often lengthy, trajectory. This incentivizes the optimizer to find any "shortcut" or "exploit" that correlates with positive reward, regardless of whether it constitutes genuine reasoning. a random, nonsensical sequence happens to produce the correct final answer by chance, the uniform learning signal of DAPO strongly reinforces every token in that flawed sequence. Without a counteracting force, the optimizer can rapidly converge on this suboptimal, degenerate policy because it's deceptively easy way secure reward.
The Entropy Penalty as a Regularizer. The entropy penalty serves as an essential stabilizing force. We empirically observed that policy collapse in our setup consistently accompanied by a sharp and uncontrolled increase in policy entropy. This pathological state occurs when the sparse reward fails to guide the optimizer, which can then push the policy into a chaotic regime that man- ifests incoherent gibberish. To counteract this and determine the optimal setting, we performed an ablation study on the entropy penalty coefficient. Figure 9 visualizes the direct impact of this penalty the training dynamics, showing how the policy entropy diverges and training accuracy collapses without regularization. The final performance for each setting is presented in Table TÕ The combined results demonstrate that the penalty critical for preventing this failure mode. We found that coefficient of 0.06 strikes the best empirical balance, achieving the highest and most stable training accuracy by keeping exploration within the bounds of coherent language generation.
Implications for VPPO. To ensure fair and controlled comparison, apply the same entropy penalty (with a coefficient of 0.06) to both the DAPO baseline and our VPPO method. This addition primarily stabilize the baseline, allowing for a direct and meaningful performance comparison. Within the standard two-epoch training regime, this penalty successfully prevents the baseline's immediate policy collapse. By focusing updates a sparse, meaningful set of pivotal tokens, VPPO
21

---

Published as a conference paper at ICLR 2026
Training Accuracy Under Entropy Penalty
Policy Entropy Under Entropy Penalty
Bntropy Coet
Training Accuracy
0.72
0.02
0.00
0.64
Entropy
0.08
0.56
2.54
0.48
3.50
30
60
90
120
150
180
210
30
60
510
120
150
180
210
Step
Step
(a) Training Accuracy Dynamies
(b) Policy Entropy Dynamics
Figure 9: Effect of the entropy penalty coefficient (A) on training dynamics. (a) Training accu- racy versus training steps. The unregularized baseline (A = 0.00) suffers from a sharp performance collapse, while our chosen coefficient of 0.06 achieves the highest and most stable accuracy. (b) Policy entropy versus training steps. The accuracy collapse in (a) is shown to be direct result of uncontrolled entropy divergence when no penalty is applied. The penalty successfully regularizes the policy, preventing this failure mode.
Table 10: Ablation study the entropy penalty coefficient for the DAPO baseline. We compare the performance of the baseline under different entropy penalty settings. While training without penalty (0.0) is possible, results in extremely low performance due policy instability. A coefficient of 0.06 is shown to be crucial for achieving stable and effective training.
Entropy Penalty
MathVerse DynaMath MMK12 Geo3k
Math Vision
We-Math LogicVista MMMU-Pro Avg.
0.0 (No Penalty)
60.2
61.3
79.4
33.8
26.0
59.8
38.4
32.8
49.0
0.02
66.2
64.6
80.2
39.9
28.0
65.9
42.8
34.1
52.7
0.04
68.3
64.7
80.9
42.2
29.4
67.9
46.0
35.1
54.3
(Default)
68.3
66.6
82.1
41.5
30.5
68.0
46.8
35.9
55.0
0.08
69.3
66
81.2
42.9
31.1
67.8
46
35.4
55.0
inherently more robust the noisy, uniform rewards that destabilize the baseline, underscoring the profound stability benefits of our hicrarchical signal modulation.
F ABLATION STUDY ON MASKING STRATEGY FOR DEPENDENCY CALCULATION
In our main paper, the calculation of visual dependency, S(s;, == DxL (Ta(-)
I')),
relies on perturbed, non-informative image I'. The choice this perturbation method is a key hyperparameter that can influence which tokens are identified as dependent. To validate choice, we conduct an ablation study comparing our default strategy against several common alternatives.
These different perturbation methods visualized in Figure 10 The specific strategies evaluated are as follows:
Random Patch Blackening (Our Default): This the strategy used for all main results. Following the ViT architecture of our base model, the image is divided into patches of size 14x14. Each patch is then independently dropped (set to black) with a probability of 0.5.
Additive Gaussian Noise: Gaussian noise with a standard deviation of 189 added each pixel value in the image. This value was calibrated such that a pixel has approximately 50% chance of being saturated to its maximum or minimum value, effectively losing its original information.
Gaussian Blur: A Gaussian blur with radius of 6.0 is applied to the entire image, de- grading fine-grained details.
22

---

Published as a conference paper at ICLR 2026
(a) Original
(b) Patch Blacken
(c) Gauss Noise
(d) Gauss Blur
(e) Complete Mask
Figure 10: Visual examples of the masking strategies for dependency calculation. Panel (a)
shows the original, unperturbed image. Panels (b)-(e) illustrate the effect of the different image per- turbation methods evaluated in our ablation study, corresponding to the methods tested in Table II
Complete Masking: The entire image is replaced with solid, neutral grey canvas (RGB value 128, 128, 128), removing all visual information.
For each strategy, trained our model using the same hyperparameters and evaluated its perfor- mance. The results are presented in Table
Ablation study on the masking strategy for visual dependency calculation. We com- pare the impact of different image perturbation methods on final model performance. The results validate our choice of "Random Patch Blackening" as the most effective strategy.
Table 11:
Masking Strategy
MathVerse DynaMath MMK12 Geo3k MathVision We-Math LogieVista MMMU-PPr Avg.
Random Patch Blackening
71.6
68.1
82.8
46.5
33.3
71.5
47.9
37.9
57.5
Additive Gaussian Noise
70.2
67.7
82.3
43.9
32.9
69.8
47.0
38.0
56.5
Gaussian Blur
69.1
68.2
82.4
45.4
32.5
70.0
46.9
37.0
56.4
Complete Masking
71.0
68.1
82.1
43.3
32.8
69.0
47.0
37.9
56.4
Analysis of Results. The results in Table I1 confirm that our default strategy, Random Patch Blackening, achieves the best overall performance with average accuracy of 57.5%. It demon- strates a consistent, albeit modest, advantage over Additive Gaussian Noise (56.5%), Gaussian Blur (56.4%), and Complete Masking (56.4%).
We hypothesize that this strategy's effectiveness stems from its patch-based nature, which aligns with the model's underlying ViT architecture. By removing entire, discrete patches of the image, this method forces the model to perform more robust, localized reasoning from incomplete visual evidence. This is more challenging and informative task than reasoning from globally degraded "gist" of the image. as might be the case with noise or blur. Interestingly, Complete Masking also performs competitively, suggesting that a significant portion of the dependency signal is captured by the stark contrast between the presence and complete absence of visual information. However, the consistent edge of Random Patch Blackening indicates that forcing the model to reason with partial visual context provides a more effective and nuanced signal for identifying pivotal tokens. These findings validate our choice of using Random Patch Blackening as the default perturbation method for all experiments in the main paper.
G ABLATION STUDY ON METHODS FOR DEPENDENCY CALCULATION
Our proposed method relies on quantifying visual dependency by measuring the KL divergence between the policy's full output distributions, no [-[-|s,, and T- I'). While principled, this is not the only way to measure the influence of a visual input. To validate our choice, we conduct an ablation study comparing default method against other computationally feasible, alternative token-scoring heuristics.
The methods evaluated as follows:
23

---

Published as a conference paper at ICLR 2026
KL Divergence (Our Default): This is the strategy used for all main results. It mea- sures the total change across the entire vocabulary distribution. Our implementation uses a memore-efficient estimation of the true KL value.
(t I) = DKL (Te(-Ise, 1) mo(-1se,1())
Jensen-Shannon Divergence (JSD): This method is symmetrized and smoothed version of KL divergence. It is implemented using the same memory-eflicient estimation tech- nique, testing whether symmetric distance metric more effective than the asymmetric information gain measured by KL.
(St, 7) = Dis (M(-)se, ((-(s,,
Top-1 Probability Drop: This simple heuristic measures only the change in probability for the token 0 that was actually sampled, testing how much the image boosts the confidence the final choice.
Stop (st, I) To(ou]se; I) To(oo)ss,
For each strategy, we trained our model using the same hyperparameters and evaluated its perfor- mance. The results are presented in Table
Table 12: Ablation study on the method for dependency calculation. We compare the impact of different computationally-feeer feasible token-scoring heuristics final model performance. The results validate our choice of using KI. Divergence as the most effective method for quantifying visual dependency.
Guidance Metric
Varheerse DynaMath MMK12 Gea3k MathVision
We-Math LogieVista MMMU-Pro
Avg.
KL Divergence (Default)
71.6
68.1
82.8
46.5
33.3
71.5
47.9
37,9
57.5
JS Divergence
71.8
67.6
82.7
45.1
32.6
70.8
47.8
36.9
56.9
Top-1 Probability Drop
61.5
64.4
74.9
31.5
30.1
62.3
44.7
33.3
50.3
The most significant finding the substantial underperformance of the Top-1 Probability Drop heuristic, which lags behind our default method by 7.2% in average accuracy. This demonstrates that a simple heuristic focused only on the single sampled token is an insufficient proxy for visual reliance. It captures only a fraction of the total change and is blind significant shifts happening elsewhere in the output distribution, such as when the visual input dramatically alters the ranking of the next most likely candidates.
In contrast, Jensen-Shannon Divergence (JSD) performs very competitively, achieving a result only 0.6% below our default. This is expected, both KI. and JS Divergence principled, full- distribution metrics that measure the overall change between the two output distributions. However, the slight but consistent advantage of KL Divergence is theoretically significant. KL Divergence is an asymmetric measure of information gain, while JSD is symmetric distance metric. The core motivation our work is specifically measure the information gain provided by the visual input to guide the policy. Therefore, KL Divergence is the more theoretically aligned choice. The em- pirical results, validating that this principled selection also yields the best performance, confirm its superiority for this task.
H ABLATION STUDY ON ROLLOUT GROUP SIZE
The number of rollouts per prompt, the group size (G), is critical hyperparameter in online RL algorithms like VPPO. It directly influences the trade-off between the quality of the advantage estimation and the computational cost of data generation. A larger group size provides more stable and accurate estimate of the expected reward, but at the cost of increased computation.
To validate choice and explore this trade-off, we conduct size. Our main experiments use a default setting of G = 8. We evaluate this against a smaller
ablation study the rollout group
group size of G 5 and larger group sizes of G 12 and G 16 to assess potential performance
24

---

Published as a conference paper at ICLR 2026
Table 13: Ablation study on the number of rollouts per prompt (G). We compare model perfor- mance across different group sizes. The results validate our choice of G 8 as providing a strong balance between advantage estimation quality and computational efficiency.
Rollout Group Size (G) Varhe DynaMath MMK12
Geo3k
MatoV
We-Math LogieVista
MMMU-Pro
Avg.
70.7
68.2
80.7
44.8
32,9
69.5
48.4
36.8
56.5
G 8 (Default)
71.6
68.1
82.8
46.5
33,3
71,5
47.9
37.9
57.5
12
71.3
68.1
83.5
46.9
32,9
70.2
48.3
37.8
57.4
G 16
72.2
68.4
84.2
46.5
33.2
71.1
48.7
37.0
57.7
gains from more extensive sampling. The results, presented in Table 13 show the impact of this hyperparameter on final model performance.
The results in Table 13 reveal a clear trend of diminishing returns as the group size increases. Increasing the group size from G 5 to our default of = 8 yields a substantial performance gain of 1.0% on average, demonstrating the value of a more stable advantage estimate.
However, further increases in group size offer minimal additional benefit. Increasing the rollouts by 50% to G 12 results in 0.1% decrease in average performance, while doubling the rollouts to = 16 provides only a marginal 0.2% improvement over our default setting. Given that the com- putational cost of the rollout phase scales linearly with the group size, doubling the work for such a small gain is not an efficient trade-off. This analysis confirms that our default setting of = strikes optimal balance between the quality of the advantage estimation and computational effi- ciency, capturing the vast majority of the potential performance gains without incurring unnecessary computational expense.
I SOFT MASK CALIBRATION FOR GRADIENT FILTERING
For the ablation study comparing binary and soft masks, we designed a continuous soft mask to ensure fair comparison with our main approach. The method is carefully calibrated so that the average weight assigned to tokens in a trajectory matches our target filtering ratio (A: = 0.4). This prevents the soft mask from simply applying universally higher lower learning signal and instead tests the effect of graded vs. a discrete update. The process involves three steps for each trajectory:
1. 7--ccore Normalization: We first normalize the raw visual dependency scores (S,) within the trajectory to have mean of 0 and a standard deviation of 1. This converts them into Z.- scores, making the subsequent calibration step robust to varying score distributions across different trajectories.
Z, = S. -14
05 +
2. Offset Calibration: We then find a unique offset, which, when subtracted from the Z- scores before applying sigmoid function, results in the desired average weight. This offset is solved numerically satisfy the constraint:
Σ sigmoid (Z,
Muarget
where N is the number of tokens in the trajectory and Mrarget is our target average weight (0.4).
3. Weight Generation: Finally, the calibrated weight wy for each token's gradient is calcu- lated using the determined offset:
Wi sigmoid Z. c)
25

---

Published as a conference paper at ICLR 2026
J ANALYSIS OF COMPUTATIONAL OVERHEAD
To address the computational cost of the second forward pass required for token perception, we conducted detailed empirical analysis. First, we quantified the overhead against the DAPO base- line. As shown in Table [4| this introduces modest and consistent overhead of approximately 0% across both 7B and 32B model scales. The low cost is attributable to calculating token probabilities in single, parallel forward pass.
Table 14: Comparison of total training time, training throughput, and computational overhead be- tween the DAPO baseline and our VPPO method. The overhead introduced by VPPO's second forward pass is consistent -10% across both 7B and 32B model scales.
Model Scale
Method
Total Training Time Training Throughput Overhead (%) (samples/sc)
(hours)
7B
DAPO
15.5
1..9
(8x H800)
VPPO
17.0
~=.27
+9.7% %
32B
DAPO
91.2
~0224
(32x H800)
VPPO
100.3
~022
+10.0 %
While the overhead is minor, conducted more rigorous evaluation under fixed time budget to confirm that VPPO's performance gains stem from improved learning efficiency. To this end, we trained the 7B DAPO baseline for an extended 17.0 hours, matching the exact training time of our VPPO-7B model.
Table 15: Performance comparison under equal time budget (17.0 hours) for 7B models. When given the same computational resources, VPPO significantly outperforms the DAPO baseline, indi- cating superior learning efficiency.
Method (7 Model)
Time
MathVerse
DynaMath
MMK12
Gea3k
MathVision
We-Math
LogicVista
MMMUU-Po
Avg.
DAPO (Baseline)
15.5h
68.3
66.6
82.1
41.5
30.5
68.0
46.8
35.9
55.0
DAPO (Rqual-Budge)
17.0h
68.6
67.0
81.9
42.1
30.6
67.6
46.2
36.3
55.0
VPPO (Ours)
17.0h
71.6
68.1
82.8
46.5
33.3
71.5
47.9
37.9
57.5
The results of this equal-budget comparison, presented in Table 15 are definitive. The baseline's performance stagnates even with the additional training time, whereas VPPO achieves 5-ppint average gain. This demonstrates that by shaping the learning signal at both the trajectory and token levels, VPPO acquires complex reasoning skills more effectively within the same time budget. These findings validate that the minor computational cost is highly effective trade-off for the substantial and broad-based improvements in multimodal reasoning.
K LIMITATIONS
While our results demonstrate the effectiveness of VPPO, it is important to acknowledge its current limitations and outline avenues for future research.
Computational Overhead. Our method introduces modest and fully manageable computational overhead. To compute the KL divergence, VPPO requires second forward pass through the model using a perturbed (masked) visual input during the rollout phase. Empirically, we found this re- sulted in only minor increase in total training time (approximately 10% increase, from 15.5 to 17 hours on our 7B setup). Given the significant gains in final performance and training stability, we believe this minor additional cost represents a highly favorable and practical trade-off. However, ploring even more efficient, single-pass approximations visual dependency remains an interesting direction for future research.
26

---

Published as a conference paper at ICLR 2026
Scope of Generalization. Our experiments have demonstrated the effectiveness of VPPO mod- els up to the 32B parameter scale. While the strong results on both 7B and 32B models suggest a positive scaling trend, the efficacy of our method on extremely large-scale models (c.g., 72B+ pa- rameters) has not yet been verified. Such models may exhibit different emergent properties, and further research is needed confirm if our hierarchical modulation remains optimal at that scale. Furthermore, the benefits of VPPO were demonstrated on reasoning-intensive benchmarks (c.g., math, geometry, logic). Its applicability more subjective or creative tasks, such as detailed image captioning or visual storytelling, where the notion of single "visually-grounded" reasoning chain less clear, remains an open question.
Methodological Assumptions and Hyperparameters. The dependency calculation at the core of VPPO contingent on the choice of image perturbation method. Our ablation study (Appendix (F) validates our choice of Random Patch Blackening, but it is plausible that the optimal masking strat- egy is task- or dependdde Similarly, while our ablations (Subsection (32)) identified optimal values for the key hyperparameters, i.e. the filtering ratio & and the shaping range [8min [max] these values were determined on our specific training dataset and may require re-tuning when applying VPPO new datasets or model scales to achieve maximum performance.
L ANALYSIS OF THE TRAINING DATASET
This section provides further details on the ViRL3 9K dataset (Wang et al. 2025a), which serves as the foundation for our reinforcement learning experiments. The choice of this dataset deliberate, as its core properties align perfectly with the requirements for training robust multimodal reasoning model.
Topical Diversity and Reasoning Depth. A primary strength of ViRL39K is its broad topical diversity. The dataset is not confined to a single domain but instead contains approximately 39,000 queries spanning a wide range of challenging subjects, including mathematics, physics, chemistry, biology, and chart interpretation. This diversity is crucial for training general-purpose reasoning model, as it prevents overfitting narrow task distribution and encourages the development of more fundamental, transferable reasoning skills.
Suitability for Reinforcement Learning. The most critical feature of ViRL39K for our study is its verifiability. Every instance in the dataset is programmatically generated and comes with defini- tive, unambiguous ground-truth answer. This property indispensable for any RLVR framework, as it allows for the implementation of clean, reliable, and automated reward function. By enabling simple binary accuracy signal, it removes any need for subjective, model-based judges and en- sures that the learning process is guided by objective correctness. For comprehensive overview of the dataset's construction process and statistical breakdown, we refer the reader to the original publication.
M ANALYSIS OF EVALUATION BENCHMARKS
This section provides brief analysis of the eight benchmarks used in our main evaluation. We deliberately selected this suite to cover wide spectrum of challenges, from domain-spectin mathe- matical skills to general logical cognition, ensuring a holistic assessment of our model's capabilities.
Mathematical and Geometric Reasoning. deep. domain-specific skills.
This category forms the core of our evaluation, testing
DynaMath ZZou et al. 2024 is a unique benchmark designed to test the robustness of vi- sual mathematical reasoning. Instead of using a static set of questions, it employs program- based generation to create numerous variants of seed problems, systematically altering merical values and function graphs to challenge a model's ability to generalize rather than memorize.
27

---

Published as a conference paper at ICLR 2026
Geo3k (Lu et al. 2021 is large-scale benchmark focused on high-school level geome- try. Its key feature is the dense annotation of problems in a formal language, making particularly well-suited for evaluating interpretable, symbolic reasoning approaches.
Math Zhang al. 2024) is specifically designed to answer the question: "Do MLLMs truly see the diagrams?" tackles the problem of textual redundancy by pro- viding six distinct versions of each problem, systematically shifting information from the text to the diagram. This allows for fine-grained analysis of a model's 's reliance on visual versus textual cues.
MATH-Vision (Wang et al. 2024a) elevates the difficulty by sourcing its problems from real math competitions (e.g.. AMC, Math Kangaroo). Spanning 16 mathematical disci- plines and difficulty levels, it provides challenging testbed for evaluating advanced, competition-level multimodal reasoning.
MMK12 (eng et al. 2025) is a benchmark focused on K-12 level multimodal mathe- matical problems. It provides strong test of foundational math reasoning skills that are essential for more advanced applications.
We-Math (Qiao et al. 2024) introduces a novel, human-centric evaluation paradigm. assesses reasoning by decomposing composite problems into sub-problems based on a hi- erarchy of 67 knowledge concepts. This allows for a fine-grained diagnosis model's specific strengths and weaknesses, distinguishing insufficient knowledge from failures in generalization.
Logical Reasoning. To assess more general cognitive abilities, we include a dedicated logical reasoning benchmark.
Logic (Xiao al. 2024) designed fill critical gap by evaluating general logical cognition beyond the mathematical domain. It covers five core reasoning skills (inductive, deductive, numerical, spatial, and mechanical) across variety of visual formats, testing the fundamental reasoning capabilities that underlie many complex tasks.
Multi-discipline Reasoning. Finally, to test performance on challenging, college-level problems that require true multimodal integration, we use robust version of a well-known benchmark.
MMMU-Pro Yuu et al. 2024) is a hardened version the popular MMMU benchmark. specifically created to be unsolvable by text-only models by filtering out questions with textual shortcuts, augmenting the number of choices to reduce guessing, and introducing vision-only format. It serves strong of model's ability seamlessly integrate visual and textual information in high-stakes, academic context.
N PROMPT TEMPLATE
For all training and evaluation experiments, we used the single, standardized prompt template shown below. Its structured format is designed to elicit consistent Chain-of-Thought (CoTT) response, which is crucial for the automated parsing of final answers.
Reasoning Template
SYSTEM:
You are helpful assistant.
USER:
[question]
You first think through the reasoning process internal monologue, enclosed within <think> </think> tags. Then, provide your final answer enclosed within \boxed!).
28

---

Published as a conference paper at ICLR 2026
O QUALITATIVE CASE STUDIES: VPPO VS. BASELINE
To provide a more intuitive understanding how VPPO improves reasoning performance, this section presents qualitative analysis of three representative examples. For each case, contrast the failure mode of the baseline with the correct reasoning process of our VPPO-7B model on the exact same problem. Notably, for each of these examples, our VPPO-7B model produced the correct answer on all eight of its generation passes, demonstrating the stability and robustness of its learned reasoning process.
These case studies are specifically chosen to highlight the practical impact of our hierarchical signal modulation. They illustrate how the baseline's uniform learning signal can lead critical errors in visual perception or logical deduction, while VPPO's targeted approach successfully and consis- tently guides the model to the correct solution by focusing on pivotal, assuall-groundde steps.
29

---

Published as a conference paper at ICLR 2026
Generated Sample by ThinkLite- VL-7B
D
M
E
Jyeo oo
B G CHFC
As shown in the figure, a square DEFG model should be cut on a piece of triangle ABC paper. Among them, G and F are BC, D and E are AB and AC respectively, AH perpendicular BC and intersects DE M, if BC 12.0. AH 8.0, then the edge length of the square DEFG is O
Question:
Choices: (A) 24 em (B) 4cm
(C) 24
7 cm (D) 5cm
Ground truth: (A) cm <think>
To solve for the edge length of the square DEFG, we need to leverage the properties of similar triangles and the given information.
Identify the similarity of triangles: Since DEFG square inscribed in the triangle ABC, AADE and AABC' are similar by similarity (Angle-Angle similarity) criterion. This because LADE = LABC and LDAE LBAC.
2. Use the similarity ratio: The ratio of the corresponding sides of similar triangles is equal. Therefore, have:
DE
AM
BC
AH
3. Express the given and unknown values: Here, DE is the side length of the square we need to find, BC 12 cm, and AH = 8 cm. We need to find the length of AM (the height from A to DE).
BC, and E and that the height from A to also the height from A to BC divided by the ratio the side lengths the triangles. The height from A to DE (AM) is the height from A to BC (AH) minus the height from M to BC.
4. Since G and
on AB and AC respectively,
the fact
5. Let's denote the side length of the square as Then, the height from to DE (AM) Se
8 12 8 2x (Incorrect expression for AM; should be AM = AH M 2). Using
the similarity ratio, we have:
æ
8
12
6. Solve the equation:
12
12
1
12
12
2r
12
=
24
(Incorrect calculation; from the previous step
1, the result should be 6.)
Therefore, the edge length of the square DEFG is 24 cm. </think> The correct is
30

---

Published as a conference paper at ICLR 2026
Generated Sample by VPPO-7B
D
M
E
Tyeo
B F
As shown in the ligure. square DEFG model should be a piece of triangle ABC paper. Among them, and F are on BC, and E AB and respectively, All perpendicular BC and intersects DE at M, if BC= 12.0, All 8.0, then the edge length of the square DEFG () Choices: (A) 24
Question:
5 cm (B) 4cm
24 (C)
7 cm (D) 5em
Ground truth: (A) 24 cm
<think>
To determine the edge length of the square DEFG, need to the properties of similar triangles and the given dimensions the triangle and the square. Here is the step-by-step reasoning:
1. Identify the similar triangles:
Since DEFG is a square, DE is parallel to BC, and therefore AADE ~ AABC by the AA similarity criterion AAngle-Angle similarity). This gives us the ratio of the corresponding sides of the similar triangles.
2. Set the proportion: Since AADE AABC, we have:
DE
AM
BC
AH
Let the side length of the square DEFG be Then, DE and AM AH MH 8 Since
DE AM
BC AH we can write:
4
3. Solve the equation:
Cross-multplying gives:
8x = 12(8 x)
Expanding and simplifying:
96 12z
8z + 12x 96
20r 96
96 24
20 5
24 cm. Therefore, the edge length of the square DEFG </think> The correct answer is A
31

---

Published as a conference paper at ICLR 2026
Generated Sample 2 by
As in the figure. then angle CDB 8:64* D:37"
A
B
00.com
126
C
Ground truth: (C) 27 <think>
solve for measure of angle /CDB, we need to use the properties of circles and relationships between the angles.
Identify the type of angle given: The angle formed by the two radii at the center of the cir- cle 126".
2. Use the fact that the angle subtended by are at the center is twice the angle subtended by the same any point the cireumference.
3. The angle
center is 126", so the angle subtended by same are at point (which is an inscribed angle) would be half of 126", which 63 (Misapplication of the inscribed angle theorem; C is an endpoint of the arc, not vertex viewing the arc.)
is diameter, angle LACB is right angle (90").
5. Triangle ABCD is right triangle with angle LACB as 90" (Incorrect statement; LACB in AABC, and it is not angle of BCCD). The angle LCDB is the angle need to find.
angle LCDB an inscribed angle that intercepts the same are as 63 angle the center. Therefore, angle LCDB is half of the supplementary angle of 126", which 54". (Multiple logical errors: refers to non non-existent 63 central angle, incorrectly relates ares, and the conclusion contradicts the premise.)
So, angle /CDB = 54" </think>
54
32

---

Published as a conference paper at ICLR 2026
Generated Sample by VPPO-7B
As shown in the figure. then angle CDB
Choices: A:54" 0:64" C:27 D:37
D
A
B
126'
Ground truth: (C) 27
<think>
determine the measure angle CDB, we need use some properties of circles and relationships between angles in the circle. Here the step-by-step reasoning:
1. Identify the given information and the relevant theorems:
We given that the measure of the central angle LAOC' is 126". According the central angle theorem, the measure of the central angle is twice the measure same arc.
inscribed angle that subtends the
The measure of the inscribed angle LABC that subtends the same are as the central angle ZAOC is half of the central angle. Therefore, LABC 1/4 x 126 63%.
that the sum of the interior angles in a quadrilateral 360": Since AB is the diameter of the circle, LACB = 90 (angle in a semi-circle). Therefore, AABC, we have:
LBAC 180 LABC LACB = 180 63 90 27%.
3. Relate the angles in the circle: Since LBAC and LCDB angles that subtend the BC, they equal. Therefore. /CDB ACAC 27
So, the measure of angle CDB is C
</think>
The correct
C
33

---

Published as a conference paper at ICLR 2026
Generated Sample by VL--Rethinkr-77B
60
y
Question: In the triangle with hypotenuse length 20, find y.
Choices: (A) 4 (B)
(C) (D) 10
Ground truth: (D) 10
<think>
find the value y in the given right triangle, we can use the trigonometric relationship for sine function (Incorrect function; cosine should be used as is the adjacent side). The sine of an angle in a right triangle is defined as the length of the opposite side divided by the length of the hypotenuse.
Given:
The hypotenuse I = 20
The angle 0 = 60
The opposite side the angle is (Incorrect identification; is the side adjacent the 60' angle, not opposite.)
The sine of 60 is given by:
sin(
20
(Incorrect formula based on the wrong side identification.) We know that sin(60") v3 Therefore:
v3
y
To solve for y. multiply both sides of the equation by 20:
v3
20 x
y 10v3
(This is the value for x, the opposite side, not y.) Next, simplify the expression. Using the approximate value of /3 z 1.732:
x 10 x 1.732 17.32
However, we can now check the answer choices. The value 10/3 17.32 matches closest the provided options. (Illogical conclusion; 17.32 not close to the integer options, and the chosen answer 'C' (8) does not follow from this value.) </think>
34

---

Published as a conference paper at ICLR 2026
Generated Sample by VPPO-7B
60
y
Question: the triangle with hypotenuse length 20, find y.
Choices: (A) 4 (B)
(C)8 (D) 10
Ground truth: (D) 10
<think>
To find the length of the side y the right triangle where the hypotenuse is and the angle is 60 degrees, we can use the properties a 30-60-90 triangle. In a 30-60-90 triangle, the sides are in the ratio 1 v3= where the hypotenuse is the longest side and twice the length of the side opposite the 30-degree angle.
Here, the hypotenuse is 20, which is twice the length of the side opposite the 30-degree angle. Therefore, the side opposite the 30- 30-degrce angle is half of the hypotenuse. Let's go through the steps:
Identify the given information: 2. In right triangle with 60-degree angle, the side opposite the 30-degree angle is half the hypotenuse. Here, the side y is the side opposite the 30-degree angle in the 30-60-90 60-90 triangle formed the given angle.
hypotenuse is 20 the angle degrees.
The relationship in 30-60-90 triangle is:
y 2 hypotenuse
1
x 20 y 2
y 10
Therefore, the length of y is 10. </thinks
The correct answer is D
35


Reviews:
[{"review_id": "jbChy6SaW4", "reviewer": "Reviewer_paED", "summary": "The paper explores how tokens generated during visual reasoning influence the post-training of large language models. The core motivation is to understand which reasoning tokens are most affected when noise is introduced into the image. Through empirical analysis, the authors find that a small subset of tokens plays a disproportionately important role in the reasoning process, and accurately predicting these tokens is essential. Building on this insight, they propose a method for visual reasoning that is validated through experiments and ablation studies.", "questions": "1. The sensitivity of the method to the entropy loss hyperparameter appears to stem from VPPO being built on top of DAPO. A crucial experiment would be to substitute DAPO with GRPO within VPPO to observe how the results change. This would help disentangle the specific contributions of VPPO from those of the underlying policy optimization algorithm.\n\n2. Additionally, following [1], are the tokens with the highest KL divergence also the ones with the highest entropy? Clarifying this connection would help align the findings with recent literature and enhance the interpretability of the pivotal token analysis.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written and formatted nicely; it is easy to understand the core message of the paper.\n2. The authors present a well-informed set of experiments and ablations that help show the effectiveness of their method.\n3. The paper's motivation is well rooted in recent literature evaluating the effectiveness of reasoning traces during post-training of LLMs [1,2].\n\n[1] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning\n\n[2] TreeRL: LLM Reinforcement Learning with On-Policy Tree Search", "weaknesses": "1. A key concern is the sensitivity of the proposed method to the Entropy Loss parameter introduced for DAPO. This parameter requires careful tuning to achieve optimal performance.\n\n2. While the authors cite [1] and acknowledge related work in LLM reasoning, the claim that “in the multimodal domain, a pivotal token is not just a logical fork but a critical moment of visually grounded reasoning” lacks sufficient support, as no direct comparison is made with those prior methods.\n\n3. It would have been valuable to include results comparing VPPO, GRPO, and DAPO on the 32B model, as this could provide deeper insight into their relative effectiveness at scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "42Wl39QVdP", "reviewer": "Reviewer_iQRs", "summary": "This paper introduces VPPO, a variant of GRPO that encourages rollouts with higher visual dependence. It adopts a framework similar to PAPO, contributing several additional insights. It gives a proper measurement of visual dependence for each rollout and reveals two valuable insights with empirical evidence. Building upon these, it proposes two tailored adaptations to GRPO that cover both micro and macro hierarchy. Finally, comprehensive and detailed experiments validate the proposed modules, forming the paper as a coherent and insightful work.", "questions": "1. As in weakness 2, why do you adopt a discrete binary mask operation instead of a continuous soft mask based on the dependent value?\n2. What are the intuitive benefits of the amplification term on the advantages? Why do you choose not to reflect the overall visual dependency on the reward term?\n3. Can you further clarify the formulation difference between VPPO and PAPO?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. **Well-presented Paper** The paper conducts a step-by-step and coherent exploration, from intuitive motivation, empirical evidence, tailored solutions, and comprehensive experiments. Each part is presented clearly and soundly.\n2. **Insightful Empirical Findings**. The paper introduces a measurement for visual dependence. Building on this metric, the paper demonstrates that only a small proportion of tokens are highly visually dependent, and the overall visual dependency is varied across rollouts. This metric and findings might inspire further research.\n3. **Theoretical Guarantee**. The paper proves that VPPO constructs a lower-variance policy gradient estimator, guaranteeing overall training stability.\n4. **Comprehensive Experiments**. There are very comprehensive experiments studying various aspects of the proposed methods. They collectively validate the effectiveness and robustness of VPPO.", "weaknesses": "1. **Solid yet Expected Contributions**. Despite the overall high quality, VPPO inherits the framework from PAPO and makes incremental improvements. This leads to the paper's contributions being sufficient but not groundbreaking.\n2. **Minor Gaps between Insights and Solutions**. For the first insight, the paper demonstrates that highly visually dependent tokens are sparsely distributed across rollouts. VPPO then computes gradients exclusively yet evenly on top-ranked tokens. The paper does not discuss the reason behind adopting the discrete binary mask instead of a continuous soft mask based on the dependent value. For the second insight, the paper shows that the overall visual dependency is varied across rollouts. However, it lacks an empirical link between visual dependency and reasoning quality. Additionally, the amplification term is a bit confusing: in cases where a rollout exhibits a high visual dependency but yields incorrect results, the term would amplify discouragement, which somewhat misaligns with motivation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BpsbPi5FiF", "reviewer": "Reviewer_CS1Y", "summary": "This paper introduces Visually-Perceptive Policy Optimization (VPPO), a novel reinforcement learning algorithm for multimodal reasoning that addresses the inefficiency of uniform learning signals in existing approaches. The authors identify that current multimodal RLVR methods fail to distinguish between tokens with different visual dependencies, leading to suboptimal learning. VPPO introduces two key mechanisms: (1) Token-level Gradient Filtering (TGF) that focuses updates on visually-dependent tokens, and (2) Trajectory-level Advantage Shaping (TAS) that reweights advantages based on overall visual dependency. Evaluated on eight benchmarks using Qwen2.5-VL models, VPPO achieves 19.2% and 7.6% average accuracy improvements for 7B and 32B models respectively.", "questions": "I am not an expert in this area, and I am willing to discuss with other reviewers.\n- After applying the proposed method, will the model diminish its capability on general visual language tasks, such as VQA?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Thorough analysis: The paper provides excellent insights into token-level visual dependency distributions and their impact on RL learning.\n\n- Strong empirical results: Consistent improvements across 8 benchmarks and 2 model scales demonstrate robustness.\n\n- Faster convergence: Figure 5 shows VPPO not only achieves better final performance but converges faster", "weaknesses": "I see no major weakness. Thanks for the authors' hard work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tItT0rfDWc", "reviewer": "Reviewer_4piz", "summary": "This paper presents VPPO, a novel policy gradient algorithm for multimodal RL that improves LVLM reasoning by explicitly incorporating token-level visual perception. It identifies visually dependent tokens and trajectories to refine learning signals through advantage shaping and gradient filtering. VPPO demonstrates significant performance gains on diverse benchmarks, backed by theoretical analysis and extensive ablations.", "questions": "Please refer to the Weaknesses section above for details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.The paper introduces a fresh and critical perspective by analyzing and leveraging token-level visual perception in multimodal RL, addressing a clear gap in existing \"modality-agnostic\" methods. The insights into sparse token dependency and heterogeneous trajectory grounding are well-motivated.\n2. VPPO consistently achieves new state-of-the-art results on a comprehensive suite of challenging multimodal reasoning benchmarks, showing substantial accuracy gains over leading baselines across different model scales (7B and 32B). This robust performance is a key highlight.", "weaknesses": "1.the additional computational cost of a second forward pass for token perception calculation needs more detailed quantification across different model scales (e.g., specific training times or throughput comparisons for 32B models). \n2. Evaluation is restricted to reasoning-intensive benchmarks. The method's generalizability to other multimodal tasks, larger model scales, and different architectures remains unexplored, particularly given its reliance on a specific image perturbation strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)