Perception-Aware Policy Optimization for Multimodal Reasoning
Authors: Zhenhailong Wang
Xuehang Guo
Sofia Stoica
Haiyang Xu
Hongru WANG
Hyeonjeong Ha
Xiusi Chen
Yangyi Chen
Ming Yan
Fei Huang
Heng Ji
Venue: ICLR 2026 Poster (2026)
Keywords: multimodal reasoning
reinforcement learning
policy optimization
large language models
visual perception
GRPO
DAPO
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: PAPO enhances multimodal reasoning through perception-aware reinforcement learning, reducing perception errors significantly with overall improvement across multiple benchmarks.

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has proven to be a highly effective strategy for empowering Large Language Models (LLMs) with long chain-of-thought reasoning abilities. However, its design and optimizations remain tailored to purely textual domains, resulting in suboptimal performance when applied to multimodal reasoning tasks. In particular, we observe that a major source of error (67%) in current multimodal reasoning lies in the perception of visual inputs. To address this bottleneck, we propose PAPO, a novel policy gradient algorithm that encourages the model to generate visually grounded reasoning without external supervision. Specifically, we introduce the Implicit Perception Loss in the form of a KL divergence term, which maximizes the difference between two probability distributions over the same rollout sequence, conditioned on either the original or corrupted visual input. Notably, PAPO does not rely on any additional data annotation, reward models, or stronger teacher models, and can therefore be seamlessly integrated into mainstream RLVR algorithms such as GRPO and DAPO. To further enhance the training stability of PAPO, we introduce the Double Entropy Loss, which effectively regularizes the new KL objective without compromising performance. Despite its simplicity, PAPO yields significant overall improvements of 4.4%-17.5% on diverse multimodal benchmarks. The improvements are more pronounced, approaching 8.0%-19.1%, on tasks with high vision dependency. We also observe a substantial reduction of 30.5% in perception errors, indicating improved perceptual capabilities with PAPO. Overall, PAPO offers a new perspective on advancing multimodal RLVR via the optimization objective, moving beyond rollout or reward design and pointing toward deeper integration of perception and reasoning.

Published as a conference paper at ICLR 2026
PERCEPTION-AWARE POLICY OPTIMIZATION FOR MUL- TIMODAL REASONING
Zhenhailong Wang"*, Xuehang Guo",, Sofia Stoica', Haiyang Xu Hongru Wang', Hyeonjeong Ha', Xiusi Chen', Yangyi Chen', Ming Yan², Fei Huang", Heng Ji 'University of Illinois Trbana-Champpain "Equal contribution *Corresponding author w0ngz30i 11 wiliniis.ee. edu, shuofeng. xhy wbwaabbncccom hengj 1eigli@il edu
Alibaba Group
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) has proven be highly effective strategy for empowering Large Language Models (LLMs) with long chain-of-thought reasoning abilities. However, its design and optimizations remain tailored to purely textual domains, resulting in suboptimal performance when applied to multimodal reasoning tasks. In particular, we observe that a major source of eπror (67%) in current multimodal reasoning lies in the percep- tion of visual inputs. To address this bottleneck, we propose Perception-Awar Policy Optimized (PAPO), novel policy gradient algorithm that encourages the model to learn to perceive while learning to reason. Specifically, we introduce the Implicit Perception Loss in the form of a KL divergence term, which can be seamlessly plugged into mainstream RLVR algorithms such as GRPO and DAPO. Notably, PAPO does not rely any additional data annotation, reward models, stronger teacher models. Despite its simplicity, PAPO yields significant overall improvements of 4.4% 17.5% on eight multimodal reasoning benchmarks. The improvements are more pronounced, approaching 8.0%-19.1 %, on tasks with high vision dependency. We also observe substantial reduction of 30.5% in perception errors, indicating improved perceptual capabilities with PAPO. Overall, PAPO offers new perspective on advancing multimodal RLVR via the optimization objective, moving beyond rollout or reward design and pointing toward deeper integration of perception and reasoning.
INTRODUCTION
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key driver of recent progress in large language models (LLMs), particularly in enhancing their reasoning capabilities. By optimizing models using verifiable signals, such as structured thinking formats and final answer accuracy, RLVR has demonstrated strong empirical success in models like DeepSeek-RI (Guo et al., 2025), as well as through algorithmic innovations like Group Relative Policy Optimization (GRPO) (Shao et al., 2024). Large Multimodal Models (LMMss,, however, continue struggle with complex multimodal reasoning tasks that require both fine-grained perception and multi-step reasoning (Yue al., 2024; Zhang et al., 2024; Wang et al., 2025e). This limitation stands in contrast to the strong reasoning performance of LLMs in textual domains.
Aiming address this gap. a growing body of work (Chen et al., 2025; Shen et al., 2025a; Meng et al., 2025b; Huang et al.. 2025; Yang et al., 2025; Liu et al., 2025a; Wang et al., 2025d; Xiao et al., 2025a; Zhu al., 2025b; Wang et al., 2025b; Liang et al., 2025; Xia al., 2025; Xiao et al., 2025b; Shen et al., 2025c; Wan et al., 2025; Yao et al., 2025) has explored applying RLVR to LMMs in hopes of similarly improving their multimodal reasoning abilities. Initial successes have been reported, particularly in terms of generalization ability when using GRPO compared to supervised finetuning (Chen et al., 2025; Shen et al., 2025a; Huang et al., 2025). However, most prior efforts have primarily focused on improving data and rollout quality (Li et al., 2025a; Liang et al., 2025; Wang et al., 2025b; Li et al., 2025b; Liu et al., 2025a; Yao et al., 2025) or reward design (Xiao et al., 2025a; Xia et al., 2025; Li et al., 2025c) leaving the core optimization objective largely unchanged

---

Published as a conference paper at ICLR 2026
from its application in textual domains. This raises two fundamental research questions: (1) Are there unique challenges in multimodal reasoning that do not arise in text-only settings and cannot be addressed solely through data- reward- level modifications? (2) If so, how designing a new RIVR optimization objective that is better grounded in multimodal domains?
address this by
To investigate the first question,
Frror Analysis
Inference Example
ducted a comprehensive error analysis on multimodal reasoning model trained us- ing the standard GRPO pipeline. We man- ually examined 200 error cases across four benchmarks and categorized the types of er- rors. Surprisingly, as shown in Figure 1, we found that 67% % of the errors stemmed from perception (see $ 2.2 for more details). We attribute this bottleneck to the fact that exist- ing RLVR objectives do not explicitly incen- tivize models generate visually grounded responses. Recent approaches (Xia et al., 2025; Xiao al., 2025a; Li et al., 2025c) have also recognized the importance of per- ception, introducing additional rewards that either directly assess perception quality or require the model to explicitly perform cap- tioning before reasoning. While promising, these strategies often impose rigid sep- aration between perception and reasoning, rather than enabling joint learning of both. They also rely additional large neural- based reward models, resulting in significant computational overhead and limitations im- posed by the reward model's capacity.
Perception
Question: Find
134 (67.009))
GREO: Percepetin firrer Example
Fha Total Errur
<tiikk fand side
GRPO
59.15%
60dgarez unll
bypormee
Thereter,
Perception -30.50%
(bavell 9y
PAPO: Corrected Perception
13
<thinke In 3040-00 biangl,, the
teporerss the length the
log (the esto appoite Mi-terre
61
Hrm
typtimeet IS the cake
balfofis
PAPO
Thenefenc
geithinic Uessed
Figure 1: Comprehensive error-type breakdown and inference example between GRPÒ and PAPO. We ob- serve that perception errors account for the majority (67%) of failures in current multimodal reasoning mod- els trained with GRPO. PAPO significantly reduces the dominant perception -irception-driven errors by 30.5%, with the reduced portion indicated in gray. On the right, we present an inference example illustrating how en- hanced perception enables better reasoning.
In this work, we challenge the prevailing view that multimodal reasoning in RLVR can be addressed solely through data, rollout, or reward modifications. Instead, we investigate a deeper and more efficient integration of perceptual incentives into the core optimization objectives. To this end, we propose Perception-Aware Policy Optimized (PAPO), a novel policy gradient algorithm that enhances multimodal reasoning through visually grounded optimization. Notably, PAPO can serve as a direct drop-in replacement for GRPO (Shao et al., 2024) or DAPO (Yu et al., 2025).
The key idea behind PAPO is to encourage the model learn to perceive while learning reason. Intuitively, well-learned multimomdal model should rely on informative visual context while performing reasoning. Based on this intuition, we introduce a Kullback-Leii Leibler divergence (KL) objective, Implicit Perception Loss (KL pprp)) which we maximize within an RLVR framework. As illustrated in Figure 2, this "reverse KL." loss is computed between two probability distributions over the same rollout token sequence, conditioned on either the original or the corrupted visual inputs. From an information gain (Shannon, 1948) perspective, this loss encourages the model to assign higher probability to the response when more informative visual input is provided. As result, we are able to incentivize the generation of visually grounded responses without requiring any external supervision. To better regularize the unbounded KLyuu objective, we further introduce Double Entropy Loss which effectively enhances training stability without compromising performance.
Despite its simplicity, PAPO delivers consistent improvements over GRPO and DAPO across eight multimodal reasoning benchmarks with an average gain of 4.4%--1.55%. The improvement particularly pronounced (8.0%-12 19.1%) in tasks with higher vision-dependenccy where the input question provides limited visual clues. Furthermore, we observe a significant 30.5% reduction in perception-related errors with PAPO, as evidenced by the manual analysis shown in Figure 1. Finally, PAPO also shows faster convergence with early-stage gains starting around 25 steps.
To summarize, our main contributions are threefold: (1) We present PAPO, new policy optimization algorithm that encourages the model to generate visually grounded responses. To our knowledge, this the first work to explore deeper integration of perception-awar supervision signals beyond reward-
2

---

Published as a conference paper at ICLR 2026
level modifications. (2) Comprehensive evaluations across varying levels of vision dependency show consistent improvements PAPO over GRPO and DAPO, using identical training data and reward functions. (3) We conduct extensive analyses of PAPO and identify potential training instabilities, which we mitigate through antropy-based losses.
2
PRELIMINARY
2.1 GROUP RELATIVE POLICY OPTIMIZATION (GRPO)
GRPO (Shao et al., 2024) is variant of the Proximal Policy Optimization (PPO) (Schulman et al., 2017) algorithm that removes the value model and estimates advantages via group-based computation. In the context of multimodal reasoning, consider dataset D containing datapoints consisting of visual inputs I, questions q. and ground truth answers a. The GRPO learning objective with respect to the policy Te be written follows, where 9 represents the parameters in a large multilmodal model:
lod
(0) EE No (.)) a.)
G
10:
min (0)A clip (r., at 1 en) A.s] - BDK [mal|Ther]]
with (0) no(on,lg, I, c1,
(I)
(or g, (.,)))
G denotes the size of the group which contains multiple responses O sampled from the rollout policy Teas. corresponding to one input instance (q, I). 61, c₁ E R are hyperparameters for clipping too large updates. The original GRPO (Shao et al., 2024) sets e, ea. while recent work (Yu al., 2025) shows benefits of clip-higher, i.e., eΛ > 6t- We follow the clip-higher configuration in all of our experiments. The token-level advantage A., is defined as the sequence-level reward R; normalized across the group. Given a reward verifier eq (). which checks whether a response equivalent to the ground truth, the advantage is computed as follows:
R. me)n(R)
1.0, if ofq((a,o o.),
Å., R4
where R;
(R)
0.0, otherwise.
where R [R1, R Rc} is the rewards for the current group.
Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) (Yu al., 2025) is representa- tive follow-up GRPO, introducing several modifications such as Clip-Higher, Dynamic Sampling, and Token-Level Policy Gradient Loss. We refer readers to the original paper for detailed descriptions. In this work, we investigate the application of PAPO to both GRPO and DAPO.
2.2 ERROR ANALYSIS OF MULTIMODAT. REASONING
We first investigate the question: Are there unique challenges in multimodal reasoning that do arise in text-only settings? We follow a typical GRPO pipeline to train Qwen2.5-Ve VLL-3B (Qwen Team, 2024a) on ViRL39K (Wang et al., 2025b) (experimental details can be found in $4) and manually examine and categorize eтror types based on 200 error instances sampled from four benchmarks: GKometry3K (Lu et al., 2021), MMK12 (Meng al., 2025b), Logic (Xiao al., 2024), and Math Verse (Zhang et al., 2024). We identify four dominant error types: (1) Perception Error: Inaccurate interpretation of the visual content. For example, in Figure 1, the model associates x with the wrong side; (2) Reasoning Error: Mistakes in the logical inference process, such as applying incorrect rules or theorems; (3) Calculation Error: Mistakes performing arithmetic operations; (4) Inconsistency Error: Discrepancies between intermediate reasoning steps and the final answer.
We show the error distribution Figure 1. stem from poor perception. In many cases, the model performed well in logical or algebraic reasoning but failed to accurately interpret visual inputs, such as spatial relationships or label associations. We attribute this bottleneck in perception to the GRPO objective not providing any incentive for the model to generate visually grounded responses. This leads us to a key question: can we jointly improve perception and reasoning in multimodal RLVR? We present approach in the next section.
To our surprise, we find that the majority of errors, 67.0%,
3

---

Published as a conference paper at ICLR 2026
GRPO
Minimize
Klrrd
Reference Model
Group
Policy Model
Rollout
Og
Answer Verifier
Original Image
Question
OG
TO
Random Masking
Copy
K Khya
Maximize Difference
Regularize Entropy
Policy
Forward
munk
Model
Corrupted Image
Question
Response
PAPO
Figure 2: Illustration of the PAPO objective, which extends GRPO by adding the Implicit Perception Loss (KLp.) Additional Double Entropy Loss regularization (Ta], [= can be added for enhancing training stabilities. The KLprop is formulated as maximizing the difference between the original policy and corrupted policy T.ussk computed with masked visual input. Intuitively, PAPO encourages the model produce visually grounded responses while still achieving high rewards.
3 METHOD
3.1 TASK FORMULATION
We follow a typical RLVR setting (Shao et al., 2024; Yu et al., 2025), where the training dataset D contains visual inputs questions q, and short ground truth answers A simple rule-based verifier (hiyouga, 2025) is used to assign rewards for each rollout during training. We do not rely on any existing ofiin-of-though data and initiate RL training directly without supervised fine-tume tne-tuning. In the following method sections, we use GRPO as example to elaborate on the PAPO algorithm.
3.2 PAPO
To address the aforementioned unique challenges in multimodal RLVR, we propose Perception- Aware Policy Optimized (PAPO). The key idea behind PAPO is to encourage the policy to prefer visually grounded responses that can achieve high rewards. PAPO requires no additional annotations, no reliance on stronger teacher models, and no expensive neural reward models. We formally describe the key components of the PAPO algorithm as follows. Figure 2 shows an illustrative overview of the algorithm.
Implicit Perception Loss (KLpre)). To indicate whether generated response depends meaning- [o(olg.))
ful visual information, we define the following ratio: y (f) Tolola-Tum where o is a generated sequence of tokens, is the question and I is the original visual input. And Imask is defined as corrupted visual input, which constructed by masking out sufficiently large portion of the original input. Figure 2 shows example of I muskk where 60% of the patches are masked.
From an information gain (Shannon, 1948) perspective, this ratio quantifies the degree to which the model's 's output distribution changes when meaningful visual information is removed. A higher ratio indicates that the model assigns significantly lower probability to the correct output when deprived of full visual context, suggesting that the visual input contributes substantial information to the decision. Conversely, a low ratio implies that the model's prediction remains largely unaffected by masking, indicating that it may rely primarily the textual input rather than truly grounded visual understanding. Thus, intuitively, for well-behaved multimodal policy model 6, we want perep( (0) to be high. Based on this intuition, introduce additional loss to the GRPO objective, the Implicit
4

---

Published as a conference paper at ICLR 2026
Original
Prob
Prob 0.8
Gaussian Noise
std 0.6
Random
Masking
std 0..8
Semantic-
Aware
Masking
Figure 3: Visualization of different masking strategies. Random masking selects patches uniformly, whereas semantic-aware masking prioritizes patches containing salient objects. Gaussian noise is less effective obscuring informative semantics, even when applied with high noise factor.
Perception Loss (Kep))), which implemented by maximizing the following Kullback-Leible (KL) divergence: WKL [Te]] |=g mask) = DKL [To 1) || Ie [olo, [mask]]
Entropy Regularization. Since maximize KL divergence that is theoretically unbounded, the model may "hack" KI prep. eventually leading to performance collapse. To further enhance the training stability of PAPO, we introduce Double Entropy Loss, an effective regularizer that prevents collapse while preserving performance. This idea stems from our observation that rising rollout entropy in both ma and mmakk representative sign of collapse. Double Entropy Loss encourages Rg
the model to keep both entropy values low.
Combining the Implicit Perception Loss and Double Entropy Loss with the GRPO objective yields the complete PAPO objective:
o
JAAPo (0) PKo (0la.))
1m1
min (f) Are; clip (0),
E1,
c Acc]
AD [Tol]
DDL [Tol|TT m 71 [To]
H mask]
(2)
where the KL KLppp is implemented as DK [Tol |mmask] = (() log r (0) following Schulman (2020). i indexes the i-th rollout response. The entropy values for the Double Entropy Loss are implemented as H[mo] = log no (olg, !), H [mmaakk = log To To(olg. mask). And 7. Yh and n22 hyperparameters used for loss weighting. Similarly, we derive the DAPO-version objective of PAPO (PAPOp). as shown in Appendix B.
Masking Strategy. We investigate two strategies for creating the corrupted visual input mmakk for the KLyrep loss: (1) random masking and (2) semantic-aware masking. For both strategies, we first set a target masking ratio (e.g., 60 %). which determines the fraction of patches to be masked. We adopt patch-based masking rather than pixel-level noise (e.g., adding Gaussian noise) because patch-based based masking more effectively removes informative semantic content, whereas pixel-level noise typically preserves semantics even at high noise levels (see Figure 3 for a comparison).
In random masking, patches are selected uniformly. In semantic-awan masking, we leverage DINOv2 (Oquab al., 2023), self-supervised, pre-trained vision encoder, to identify salient patches: we aggregate its patch-level self-attention scores and select the top-scoring patches (see Appendix G details). Empirically, we find that random masking yields better performance with negligible computational overhead (detailed in $5.2).
4 EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
We train all models on ViRL39K (Wang et al., 2025b) for epochs using a learning rate of 1e-6. We perform direct RL training from Owen2.5-VL-3BB, 7B and VVL22- comparing the standard
5

---

Published as a conference paper at ICLR 2026
Table 1: Performance (av ace %) comparison of Qwen2.5-VL and Qwen3-VL models be- tween GRPO, DAPO and PAPO on general and more vision-dependent multimodal reasoning tasks. Math Versey refers the vision-centric subset of Math MathVerse Verse (Zhang et al., 2024). A indicates the averaged relative gain over the baseline for each task. We observe consistent improvements against both GRPO and DAPO, with gains approaching 8%-19%, especially on tasks with high vision-dependency. Training dynamics for these models are compared in Figure 4.
Method
General Multimodal Reasoming
Viddon- Dependent Multimodal Reasoning
Overall
Geodk Math We-Math MMKI2 MathVerse AVG
A
Logic Counting MMMU-Prr Math Versey AVG A
AVG
A
Qweel.5-VL
GR9D-38
28.72
29.31
58.30
57.24
55.25
51.89
38.14
55.81
25.06
02.26
42.97
PAPOp-38
30,95
61.38
60.09
57.39
57.14
53.39
1.38
38.67
62.56
27.31
53.95
45.57
5.00
49.92
4.36
GRPO 7B
00.15
05.05
68.12
72.26
06.51
02.51
45.62
73.94
35.17
61.71
54.11
58.78
7B
40.25
69.53
66.79
72.52
68.43
63.50
1 1.53
46.07
89.81
36.63
64.97
59.37
61.66
1.39
DAPO-3B
31.20
60,89
59.05
66,83
56.25
55.60
60.60
74.25
25.12
53.00
09.11
52.40
PAP0,-5
35.65
DAPO-7B
35.52
61.91
58.01
75.93
55.84
57.58
37.05
90.05
29.02
55.01
PAPOg-7B
44.11
67.53
68.30
50.61
68.58
65.83
46.70
91.38
36.34
64.87
59,82
19.00
63.16
17.54
Qwen3-VL (chinking)
GRPO-28
47.71
$7.96
40.13
20.51
43:97
PAPO,-28
41.08
56.08
59.17
48.57
$1.89
51.36
4.52
52.83
90.63
23.42
50.05
46.73
6.27
49.30
1,255
Training Acc Reward on Vit39K
0.55
0:00
0.30
0.65
0.60
065
0.45
0.90
0.35
0.55
0.50
0.5
PAPO, 3B
GRPO 3B
0.45
GRPO PAPO,
0.35
PAPO, 3B
DAPO 3B
0.90
DAPO 7B PAPO,
50
100
150
200
30
:00
150
2003
0
50
100
150
200 0
50
100
150
200
Stegs
GRPO
Steps
Stepns
DAPO
Steps
Figure 4: Comparison of the training dynamics on the accuracy reward. Solid lines indicate running averages with a stepping window size of 20. PAPO demonstrates consistently faster learning from the early stages on both GRPO and DAPO. Notably, DAPO-7B suffers from model collapse in the later stages, whereas PAPO achieves continued improvements without collapse, highlighting the effectiveness of the proposed Double Entropy regularization. Further analysis on regularizing the DAPO baseline is presented in Appendix H.
GRPO and DAPO baselines with our proposed variants, PAPOg and PAPOp. Note that GRPO uses a reference KL penalty, while DAPO removes it and employs dynamic sampling. Additional details on the hyperparameter configurations are provided in Appendix C.
4.2 EVALUATION
To systematically evaluate the effectiveness of PAPO, we conduct experiments and ablation studies on eight benchmarks that cover diverse multimodal reasoning problems, including: (1) Math and Geometric Reasoning: Geometry3K (Lu et al., 2021), Math (Lu et al., 2023), Math (Zhang et 2024), and We-Math (Qiao et al., 2024): (2) Multi-discipline Multimodal Reasoning: MMMU- Pro (Yue et al., 2024); (3) Logical Reasoning: Logic (Xiao et al., 2024); (4) Counting: SuperClevr Counting (I.i et al., 2023). All evaluation metrics are based on exact match against ground truth answer. We report average accurarcy @ 8 for all benchmarks with a inference temperature of 1.0. We omit datasets or instances with free-form answers that require LLM-as-a-judge evaluation.
Analysis on Vision Dependency. As also discussed in (Zhang et al., 2024; Yue et al., 2024), we observe that not all mainstream multimodal benchmarks are guaranteed to have visually dependent problems. That is, some reasoning tasks may rely heavily on textual content and do not require the visual content for deriving the answer. For example, Figure 6 exhibits VQA problems of different deiendeney levels, highlighting the varying degrees of reliance on visual information in multi- modal question answering. To this end, we conduct a manual screening of the included benchmarks and identify the following two categories: (1) Vision-Dependent Multimodal Reasoning: Bench- marks in which instances explicitly require proper interpretation of the visual input; (2) General

---

Published as a conference paper at ICLR 2026
Table Results on integrating PAPO with modifications from other perspectives (e.g., Noisy- Rollout). The base model is Qwen2.5-VL-33. We find that NoisyRollout yields substantial gains on some datasets but shows inconsistent improvements across different tasks (reductions on 4/9 benchmarks). PAPO is compatible with NoisyRollout and delivers additional gains when combined.
General Multimodal Reasoning
Vision- Deion-Dependent Multimodal Reasoning
Overall
Method
Gea3k
Math We-Math MMK12 MathVerse
I LogieVista Counting MMMU-Pro Varhe
AVG
GRPO
28.72
59.34
58.90
57.24
55.25
38.14
55.81
25.66
52:26
47.
GRPO RolsoRollont
28.40
61.34
58.58
06.78
61.34
37.47
61.44
28.79
51.30
50.0
PAPO,
30.95
61.38
60.09
57.39
57.14
38.67
62.56
27.11
53.95
49. 19.92
PAPO: Rollotl
30.66
58.83
60.60
66.95
56.21
39,51
71.38
29.47
53.36
51.89
Multimodal Reasoning: Benchmarks where instances may place weaker requirements on attending to the visual input when answering the question. More details are presented in Appendix D.
5 RESULTS
5.1 MAIN RESULTS
PAPO consistently outperforms GRPO and DAPO for multimodal reasoning. We present the main results on both 3B and 7B base models in Table 1. The A%: denotes the average relative gain against the GRPO/DAPO baseline across all tasks. PAPO shows consistent overall improvements (4.4%-17.5%) with identical training dataset. rollout space and reward design, compared to the baselines. In Figure 4, we further present comparison of training dynamics based on the accuracy rewards on ViRI.39K. PAPO showcases faster learning even from the early steps. While DAPO-7B encounters model collapse in later stages, PAPO continues to improve steadily, demonstrating the efficacy of the proposed Double Entropy regularization. In Appendix H, we further explore regularizing the DAPO baseline with our proposed entropy-based -based loss and demonstrate improvements over the stronger, regularized DAPO baseline. In Appendix E, we provide additional qualitative analysis of the attention patterns, illustrating how PAPO encourages stronger and more accurate attention to the image patches.
More significant improvements on vision-dependent reasoning. The performance gains are more pronounced the iision-dependen subset, leading to relative gain of 8.0%-19.1% This aligns with our expectation regarding the impact of Implicit Perception Loss, as encourages visually dependent responses.
Significantly reduced perception errors. We further conduct a comprehensive qualitative study on the error distribution, following the setup detailed in $2.2. Figure 1 shows andore-ndd-fter comparison of errors with GRPO and PAPO. We observe a significant reduction in perception errors, demonstrating the effectiveness of addressing the perception bottleneck in GRPO for multimodal reasoning. In Appendix K, we provide additional evaluations on DCR-Bench-v2 (Fu et al., 2024), which demonstrate the benefits of PAPO on fine-grained perception tasks. In Appendix N. we further analyze the perception errors and their outcomes under PAPO, investigating whether these errors are corrected or shift other categories.
Compatibility with other algorithmic modifications. As discussed in $1, prior work has focused on modifying the RLVR framework from the data, rollout, and reward perspectives. Since PAPO modifies only the optimization objective, it is in theory complementary to modifications from these other perspectives. To further examine this, we integrate PAPO with NoisyRollout (Liu et al., 2025a). The results are presented in Table 2. We summarize our findings as follows: (1) NoisyRollout yields inconsistent improvements across tasks, achieving large gains on some tasks but reductions on out of 9 benchmarks. (2) Combining PAPO with NoisyRollout yields complementary gains, demonstrating strong compatibility with advances from other perspectives, such as the rollout stage.
Robustness in GRPO with removed KL penalty. To conduct more controlled investigation of the robustness of PAPO with KI Keppep under the removal of the original KL penalty, we consider an additional GRPO variant where the reference KL is removed without introducing other modifications like in DAPO. Table 8 (in Appendix F) shows that PAPO achieves overall improvements of 1 1..2% and 4.0% on the 3B and 7B models, respectively, outperforming the GRPO + Removed KL baselines.
7

---

Published as a conference paper at ICLR 2026
Robustness in low vision-dependent tasks. Currently, PAPO applies the Implicit Perception Loss loss to all instances and all tokens, encouraging perceiving the visual inputs. Although this minimalist design works empirically well, we further investigate the model's 's robustness strictly vision-independent scenarios. Specifically, we consider a text-only general reasoning benchmark MMLUUprr (Wang et al., 2024), and insert dummy visual inputs into the context (images containing pure random noise). If the model were to attend to these noisy visual tokens indiscriminately. we would expect degradation in performance. The results are presented in Table 13 (Appendix). We find that PAPO still achieves competitive or stronger performance under this extreme setting, indicating good robustness in -oneiision-deeendeecy scenarios.
5.2 ABLATION ON KEY DESIGN CHOICES
Impact of Masking Ratio and Strategy. We investigate the most effective way to cor- rupt the original visual input to maximize the benefit of PAPO. In Table 3, we compare both masking strategies PAPOc. i... random masking vs. semantic-aware masking, and masking ratios, which control the percentage of patches to be masked. Implementation de- tails for the masking strategies provided in Appendix G. We find:
Table 3: Impact of masking strategy and ratio. Performance comparison of PA PAPOG using different approaches for constructing maskk The base model is Qwen2.5-VL. Despite its simplicity, random mask- ing empirically outperforms semantic-aware mask- ing. A sufficiently large masking ratio (e.g., 0.6) yields stronger performance. See details in $5.2.
Model
General
Vision
Overall
Size
Method
AVG
A%
AVG
4%
AVG
A
Random masking empirically outperforms semantic-aware masking. We hypothe- size that semantic-aware masking underper- forms because, as illustrated in Figure 3, it tends to obscure entire salient regions, caus- ing the model to attend to all objects indis- criminately instead of focusing on the most informative parts.
GRPO Baselines
3B 7B
GRPO GRPO
51.89
42.97
47.92
62.51
54.11
58.78
Impact of Masking Strategy on PAPO
random @0.6
52.53
1.73
45.17
1 4.52
49.26
1 2.97
3B
semantic @0.6
52.13
0.34
43.78
1.88
48.42
1 1.02
random @0.6
63.56
1.91
$7.49
5.37
60.86
3.55
7B
semantic 0.0.
63.39
1.48
56.83
3.89
60.47
2.55
Masking a sufficiently large portion of the image, c.g., 0.6 to 0.8, results in best per- formances. However, using a completely blackened image is not favorable, as it en- courages the model to attend to the image regardless of its content. We also observe that complete blackening is more likely to cause KL prop Hacking (detailed in $5.3).
Impact of Masking Ratio on PAPO
random @0.4
52.51
1.55
41.12
2.29
48.78
1.88
random 6906
52.53
1.73
45.17
4.52
49.26
2.97
random @0.8
52.57
1.49
4121
2.69
48.87
2.02
random 1.0
52.13
0.71
43.98
2.31
48.51
1.42
Impact of Implicit Perception Loss weight- ing. We ablate on the choice of 7, which is the weighting coefficient of KL pppp as shown in Equation 2. Table 4 presents the per- formance comparison based on PAPOg-3B when varying 7 from 0.005 to 0.04. We sum- marize our findings as follows:
Table 4: Impact of KLprep loss weighting. Per- formance comparison on PAPOG with Qwen2.5- VL-3B using different values of y. Increasing y up to 0.02 generally improves performance, while an excessively large 7, such as 0.04, leads model collapse (see detailed discussion in $5.3). Larger models are also more sensitive high y as shown in Figure 12.
A larger y under 0.02 tends result in more pronounced improvements, especially on more
Genernl
tasks. Without additional regularization, setting 7 0.02 for PAPOc-3B models and 7 0.01 for 7B models serves as good de- fault.
Method
AVG
4%
AVG
4%
AVG
A%
GRPO
51.89
42.97
47.92
PAPO R4,005
52.40
f 1.19
43.73
1.92
48.55
1.51
PAPO 60.00
52.53
1 1.73
45.17
4.52
49.26
f
2.97
PAPO @9.02
53,39
33.38
45.57
5.60
49.92
4.36
y should not be set too large (e.g., 0.04), it causes severe model collapse that cannot be regularized even with Double Entropy Loss. We also observe that larger models more
PAPO @1.04 (collapsed)
31.21
38.31
8

---

Published as a conference paper at ICLR 2026
sensitive to higher Y values and require earlier regularization shown in Figure 12 in the Appendix. We further discuss the impact of y on pppp Hacking in $5.3.
In settings without reference KL penalty (including PAPOp), needs be set more conserva- tively (e.g., 0.01), and Double Entropy Loss is indispensable (see Figure 10 in the Appendix).
5.3 DEEP DIVE ON TRAINING STABILITY
In this section, we aim gain deeper under- standing of Kpppp Hacking, a unique failure mode where the model over-optimizes the Im- plicit Perception Loss. We present our find-
Training Acc Reward Vir13K
ings on: (1) the model's generation behavior after collapsing; (2) early signs indicating model collapse; (3) the most influential fac- tors contributing to the hacking; and (4) reg- ularization approaches to prevent or delay its occurrence.
PAPO; (No Regularization Collappee
PAPO; Increase Reference KL.
PAPO; Single Original kntropy
PAPO; Single Masked Entropy Collapse
PAPO, Double Entropy
Collapsing behavior. We first examine how the model behaves after collapsing in terms of its generation. We manually examine the gen- erated tokens of a collapsed model, a PAPO 7B with = 0.02 (without regularization), and a non-collapsing model, GRPO-7B. observe notably abnormal generation behav- ior, namely, a tendency to generate entirely unrelated tokens during reasoning (see Fig- ure 13 in the Appendix). We quantitatively verify this by leveraging GPT-4.1-min (Ope- nAI, 2025) as a judge to score the relatedness of the model's response. As presented in the bottom left of Figure 13, the collapsed model shows significantly lower relatedness. See Appendix M for more details on this experiment.
50
100
125
150
175
200
Steps
Figure 5: Comparison of different regularization strategies. All strategies are applied to the same collapsing baseline, PAPOc (? 0.02, no regu- larization). Among the four methods described in the main text, three successfully prevent the collapse entirely, while adding Single Masked Entropy only delays it. The proposed Double Entropy Loss demon- strates the best training dynamics and prevents the collapse. Detailed results are compared in Table 5.
Early signs of KI Lprrp Hacking. When the hacking occurs, the model exhibits drastic decrease in KL KLprp, accompanied by collapsing training rewards. Meanwhile, the Clipping Ratio-High increases, indicating a growing proportion of tokens undergoing policy gradient updates beyond the clipping threshold, which serves as an early sign of collapse. We also observe interesting pattern: the entropy loss for both the original my and the corrupted policy pap increases as the collapse unfolds. This observation inspires regularization strategies, which are detailed later in this section. See Figure 11 in the Appendix for the metric dynamics.
Influential factors towards KI KLppep Hacking. We summarize three main factors that make the model more prone to KLprpp Hacking: (1) Model size: Larger models tend to be more sensitive to hacking under the same configu- ration. For example, setting y 0.02 causes collapse on 7B models but not on 3B mod- els. (See Figure! 12 a.) (2) Loss weighting: A higher KI poop weighting, such as 0.04, is more likely to lead to collapse. (See Figure 12 b.) (3) Masking ratio: Using an extreme mask- ing ratio, e.g., 1.0, leads to faster collapse. (See Figure 12 c.)
Table Performance comparison between the three regularization methods that successfully pre- vent model collapse, as shown Figure 5. The base model is Qwen2.5-VL-7B. Among these methods, Double Entropy Loss achieves the best overall im- provement of 4.4%
General
Vision
Overall
Method
AVG
A7
AVG
4
AVG
A
GRPO
62.51
54.11
58.78
PAPOg KL
63.14
6 1.12
57.03
f
00.42
2.40
PAPOg w/ Single Ent
63.34
6 1.53
58.
1
61.12
3.50
PAPOg w/ Double Ent
63.50 f 1.53
$9.37
7.96
61.66
1.39
Regularization approaches for preventing KLprep Hacking. Inspired by the aforementioned
findings, we investigate four different approaches prevent the collapse: (1) Increasing the KL penalty against the reference model; (2) Adding a single entropy loss on the original policy sequence no: (3) Adding single entropy loss the corrupted policy sequence mmaskk (4) Adding Double

---

Published as a conference paper at ICLR 2026
Entropy Loss both Re and rguask See Figure 5 in the Appendix for the training dynamics of the four approaches. Among them, (1))2)(4) successfully prevent the collapse from happening.
We further examine their evaluation performance, as shown in Table 5. We find that adding entropy- based loss on the original sequence is essential for regularizing training stability. Overall, Double Entropy Loss achieves the most significant performance gain while also successfully avoiding KI prup Hacking. In Figure in the Appendix, we also show that adding entropy loss alone fails to prevent DAPO from collapsing, whereas pAPOp stabilizes training and yields significant improvements.
RELATED WORK
Prior work on multimodal RLVR have primarily focused on enhancing three components of the original GRPO framework: Data, Rollout, and Reward, while leaving the core optimization objectives largely untouched. We offer a comprehensive categorization and comparison in Table 15 in the Appendix.
Data-Centric Approaches. Earlier efforts such as R1-V (Chen et al., 2025; Huang et al., 2025; Meng et al., 2025a) distill chain-of-thought (CoT) data from strong textual reasoning models like Deepseek-RI (Guo et al., 2025) and explore directly applying R1 R1-style training pipelines to multi- modal tasks, demonstrating initial promise in generalization. Recent work such as MoDoMoDo (Liang et al., 2025; Li et al., 2025a) explores more sophisticated sample selection mechanisms improve data quality.
Rollout Improvements. NoisyRollout (Liu et al., 2025a) and VL (Yao et al., 2025) show the benefits of diversifying the rollout space using responses generated from moderately augmented visual inputs. VL-Rethinker (Wang et al., 2025b) and Skywork V2 (Wang et al., 2025d) adopt a Selective Sample Replay mechanism to mitigate the prevalent issue of vanishing advantages.
Reward Enhancements. Several approaches (Ma et al., 2025; Liu et al., 2025b; Fan et al., 2025) incorporate grounding-related metrics, such as IoU for bounding boxes. Visionary-RI (Xia al., 2025) introduces captioning-ban baste rewards, prompting the model to generate detailed textual descrip- tions of visual input before reasoning. While initially promising. this approach enforces a separation between perception and reasoning, which can be suboptimal for capturing low-level visual details.
Perception as Tool-Using. Another line of emerging work takes a different view on improving perception in multimodal reasoning, relying tool use for perception. Recent efforts such as DeepEyes (Zheng et al., 2025), ACTIVE-OS (Zhu et al., 2025b), and Pixel Reasoner (Su et al., 2025) enhance perception by incentivizing the LMM perform visual operations, such zooming in. However, these methods do not directly improve the native perception capabilities of the multimodal models.
Consequently, we find that the prevailing assumption: multimodal reasoning can be effectively addressed solely through data- and reward-level modifications text-based RL, is inherently limiting. Our work challenges this paradigm by demonstrating that incentivizing visually grounded reasoning requires deeper integration into the core optimization objective, rather than treating vision as a secondary modality addressed through auxiliary adjustments.
7 CONCLUSION AND LIMITATIONS
In this paper, we present PAPO, novel policy gradient algorithm that encourages the reasoning steps in Large Multimodal Models (LMMs) to be internally grounded in visual inputs. PAPO requires no additional annotations, no reliance on stronger teacher models, and no expensive neural reward models, making it direct drop-in replacement to GRPO or DAPO. Despite its simplicity, PAPO significantly improves complex visual reasoning. One limitation of our current work that we have not yet explored scaling to larger model sizes or evaluating compatibility with other model families, such as the Intern (Zhu et al., 2025a) series. Additionally, while PAPO introduces only moderate computational overhead (see Appendix O), we have not focused on optimizing training efficiency, which remains an important direction for future research. Another promising avenue for future work investigating a finer- meained selection of tokens which to apply the PAPO objective, rather than applying it uniformly to all tokens.
10

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY STATEMENT
We include an anonymous source code in the supplementary material, containing training and evaluation instructions for reproducing the results in this paper. Implementation details of the datasets and models are provided in SC. The prompt used for the jude evaluation in Figure 13 is presented in Figure 14.
ETHICS STATEMENT
This work focuses on fundamental research in reinforcement learning and multimodal reasoning. Our methods developed and evaluated entirely publicly available benchmarks without involving human subjects, sensitive personal data, or private information. The proposed algorithm, PAPO, is designed general optimization technique and does not raise concerns regarding faimess, bias, discrimination, privacy, or security. We believe that our study poses no foreseeable ethical risks and fully complies with research integrity standards.
REFERENCES
Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. RI-v; Reinforcing super generalization ability in vision-language models with less than $3. https://githii: com/ Aaeeept-Ag////e R1-V, 2025. Accessed: 2025-07-03. 1, 10, 29
Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Narayanaraju Jyothi, Sravana Guan, Xinze Guan, and Xin Eric Wang. Grit: Teaching milms think with images, 2025. 10, 29
Ling Fu, Zhebin Kuang, Jiajun Song. Mingxin Huang, Biao Yang, Yuzhe Li, Linghao Zhu, Qidi Luo. Xinyu Wang. Hao Lu, et al. Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning. arXiv preprint arXiv:2501 000021,, 2024. 7, 25
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang. Lijuan Wang, and Zicheng Liu. An empirical study of end-to-end video-language transformers with masked visual modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
22898 22909, 2023. 22
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 2025. 10
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022. 22
hiyouga. Mathruler. h Mattaaaaaaatttteetaraaaaa 2025. 4
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao. Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint orXiv:2503.065. 067799 2025. 1, 10, 29
Xin Lai, Junyi Li, Wei Li, Tao Liu, Tianjian Li, and Hengshuang Zhao. Mini-o3: Scaling up reasoning patterns and interaction turns for visual search. arXiv preprint arXiv:2509.0700 2025. 29
Shenshen Li, Kaiyuan Deng, Lei Wang, Hao Yang, Chong Peng. Peng Yan, Fumin Shen, Heng Tao Shen, and Xing Xu. Truth in the few: High-value data selection for efficient multi-modal reasoning. arXiv preprint arXiv:2506.04755 2025a. 1, 10
Yuting Li, Lai Wei, Kaipeng Zheng, Jingyuan Huang, Linghe Kong, Lichao Sun, and Weiran Huang. Vision matters: Simple visual perturbations can boost multimodal math reasoning. arXiv preprint arXiv:2506.007336 2025b.
11

---

Published as a conference paper at ICLR 2026
Zhuowan Li, Xingrui Wang. Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van-Durme, and Alan-L. Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14963-14973. IEEE, 2023. doi: 10.1 N..1099CCVR5577222222.0
6, 18
Zongxia Li, Wenhao Yu, Chengsong Huang, Rui Liu, Zhenwen Liang, Fuxiao Liu, Jingxi Che, Dian Yu, Jordan Boyd-Graber, Haitao Mi, al. Self-rewarding vision-language model via reasoning decomposition. arXiv preprint arXiv:2508. / 19652, 2025c. 2, 29
Tian Liang, Xiaoyuan Liu, Pinjia He, Haitao Mi, Zhaopeng Tu, and Dong Yu. Modomodo: Learn- ing mixturee-ff-datase with reinforcement learning for multimodal reasoning. arXiv preprint
arXiv:2505.24871, 2025. 1, 10
Xiangyan Liu, Jinjie Ni. Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh. Noisyrollout: Reinforcing visual reasoning with data augmentation. arXiv preprint arXiv:2504. 13055, 2025a. 1,7, 10, 29
Yuqi Liu, Tianyuan Qu, Zhisheng Zhong, Bohao Peng, Shu Liu, Bei Yu, and Jiaya Jia. Vision- reasoner: Unified visual perception and reasoning via reinforcement learning. arXiv preprint arXiv:2505. /2081, 2025b. 10, 29
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint S5KI::2500301785, 2025c. 29
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song- Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing, ACI/IJCNIP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pp. 6774-6786. Association for Computational Linguistics, 2021. doi: 10.18653/V AAL-LLLNG.LLL URL https: 1ttg/1..1863//1122222222 acl- Soll-log.com 528. 3, 6, 18, 25
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li. Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista Evaluating Math Reasoning in Visual Contexts with GPT-4V, Bard, and Other Large Multimodal Models. arXiv prepring arXiv:2310.00255, 2023. URL https::/doot orgg (85554855//a// 2310. 02255. 6, 18
Yan Ma, Linge Du, Xuyang Shen, Shaoxiang Chen, Pengfei Li, Qibing Ren, Lizhuang Ma, Yuchao Dai, Pengfei Liu, and Junjie Yan. One rl to see them all: Visual triple unified reinforcement learning. arXiv preprint arXiv:2505.18129, 2025. 10, 29
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang. Ping Luo. Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint 2r.iv:2.50.077. 2025a. 10
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao. Qiaosheng Zhang, and Wenqi Shao. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. arXiv preprini arXiv:2503.073:0 2025b. Submitted March 10, 2025. 3, 18, 29
OpenAl. Introducing gpt-4.1 in the api, 2025. URL https: / openai. com/ index/
4-t- 9, 25,
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaekdin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. CoRR, abs/2304.07193 2023. URL https: // /arxivv... or / abs /2004. 07193. Published in *Transactions Machine Learning Research* (TMLR), Jan. 2024. 5, 21
12

---

Published as a conference paper at ICLR 2026
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang. Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xim Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07533 2025. 29
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like like mathematical reasoning? arXiv preprint arXtv:2407.01284 2024. 6, 18
Alibaba Group Qwen Team. Qwenn.5-vl--bbiinstuuct
htttp::/huuggiggace.co//// /wen//
Qwen2. S-VL-3B 3B- Instruct, 2024a. 3, 17
Alibaba Group Qwen Team. Qwen2.5-vl-77-instruee
httpss:/hhugginnffaee.c///e
Qwen2. 5 5-VL--VV----- Inst 2024b. 17
John Schulman. Approximating kl divergence. httpss//hagggingfaceccc////
Alibaba Group Qwen Team. Qwen2--1-2b-llinnknng.
Swen3--V--2-Thhnkkie 2024c. 17
html, 2020. 5
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (XXi::177706347,, 2017, 3
Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27 (3):379-423, 1948. 2, 4
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024. 1, 2, 3, 4, 17
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable rl rl-style large vision-language model. arXiv preprint arXiv:2504. 07611,, 2025a. 1
Junhao Shen, Haiteng Zhao, Yuzhe Gu, Songyang Gao, Kuikun Liu, Haian Huang, Jianfei Gao, Dahua Lin, Wenwei Zhang, and Kai Chen. Semi-off-policc reinforcement learning for vision-language slow-thinking reasoning. arXiv preprini arXiv:2507.16814 2025b. 29
Liangcheng Shen, Yulin Li, Haitao Mi, Wenxuan Wang, Zhaopeng Tu, and Dong Yu. Satori-rl: Spatially anchored training with verifiable rewards for lasion-language reasoning. arXiv preprint arXiv:2505. 19094, 2025c.
Alex Su, Haozhe Wang, Weiming Ren, Fangzhen Lin, and Wenhu Chen. Pixel reasoner: In- centivizing pixel-space reasoning with curiosity-driven reinforcement learning, arXiv preprint arXiv:2505. 2025. 10
Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.20752. 2025. 29
Zhongwei Wan, Zhihao Dou, Che Liu, Yu Zhang, Dongfei Cui, Qinjian Zhao, Hui Shen, Jing Xiong, Yi Xin, Yifan Jiang, et al. Srpo: Enhancing multimodal Ilm reasoning via reflection-aware reinforcement learning. arXiv preprint arXiv:2506.01713. 2025.
Haochen Wang, Xiangtai Li, Zilong Huang, Anran Wang, Jiacong Wang, Tao Zhang, Jiani Zheng, Sule Bai, Zijian Kang, Jiashi Feng, et al. Traceable evidence enhanced visual grounded reasoning: Evaluation and methodology. arXiv preprint arXiv:2507.07999, 2025a. 29
Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen I Lin, and Wenhu Chen. VI-rethinker:
Incentivizing self-reflection of lanion-language models with reinforcement learning. arXiv preprint 0883:77 2025b. Published April 10, 2025. 1, 3, 5, 10, 17, 18, 29
Jiaqi Wang, Kevin Qinghong Lin, James Cheng, and Mike Zheng Shou. Think not? selective reasoning via reinforcement learning for vision- language models. arXiv preprint arXiv:2505. 16854, 2025c. 29
13

---

Published as a conference paper at ICLR 2026
Peiyu Wang, Yichen Wei, Yi Peng, Xiaokun Wang, Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang. Yunzhuo Hao, Xuchen Song, Yang Liu, and Yahui Zhou. Skywork r1v2: Multimodal hybrid reinforeement learning for reasoning. arXiv preprint arXiv:2504.16656, 2025d.
1, 10, 29
Yubo Wang, Xueguang Ma, Ge Zhang. Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37: 95266-95290, 2024. 8
Zhenhailong Wang. Joy Hsu, Xingyao Wang, Kuan-Hao Huang, Manling Li, Jiajun Wu, and Heng Ji. Visually descriptive language model for vector graphics reasoning. Transactions Machine Learning Research, 2025e.
Yana Wei, Liang Zhao, Jianjian Sun, Kangheng Lin, Jisheng Yin, Jingcheng Hu, Yinmin Zhang, En Yu. Haoran Lv, Zejia Weng, et al. Open vision reasoner: Transferring linguistic cognitive behavior for visual reasoning. arXiv preprint arXivv2507.05255 2025. 29
Jiaer Xia, Yuhang Zang, Peng Gao, Yixuan Li, and Kaiyang Zhou. Visionary-rl: Mitigating shortcuts in visual reasoning with reinforcement learning. arXiv preprint arXiv:2505. 2025. URL
https: //wxiv org/ abs 20os/2505..0 /ttp//555......7 2, 10, 29
Tong Xiao, Xin Xu, Zhenya Huang.
Hongyu Gao, Quan Liu, Qi Liu, and Enhong Chen. Advancing multimodal reasoning capabilities of multimodal large language models via visual perception reward, arXiv preprint arXiv:2506.07218 2025a. URL pettps://arxiee wttp:/iwwrg////////////e 07218. 1,2
Wen Xiao, Weijie Zhang, Jie Hu, Rui Chen, and Jie Yang. Perception-rl: A reinforcement learning framework for multimodal perception with verifiable rewards. arXiv preprint arXiv:2506.0721 08,1, 2025b. 1.29
Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal Ilm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.044 04977. 2024. 3, 6, 18
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9653-9663, 2022. 22
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. Rl-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint
2025. 1
Huanjin Yao, Qixiang Yin, Jingyi Zhang, Min Yang, Yibo Wang, Wenhao Wu, Fei Su, Li Shen, Minghui Qiu, Dacheng Tao, and Jiaxing Huang. R -shhare:::: Incentivizing reasoning capability of multimodal large language models via share-grpo, 2025. 1, 10. 29
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025. 2, 3, 4,
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun. al. Mmmu-pre A more robust multi-discipline multimodal understanding benchmark. arXiv preprint 24Xi:22409.02811 2024. 1, 6, 18
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang. Peng Gao, and Hongsheng Li. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? CoRR, abs/2403. 005/2403..1624, 14624, 2024. URL https:/ /ttp://axiiv. //taxii.org/////e/////./../oooat wwtrg//a////// /200/24033. 14624. Also published in the ECCV 2024 proceedings. 1,3, 6, 18
Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang. Chao Shen, and Xing Yu. Deepeyes: Incentivizing "thinking with images" via reinforcement learning. arXiv preprint arXiv:2505. 14362, 2025. 10, 29
14

---

Published as a conference paper at ICLR 2026
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXi::250..1047, 2025a. 10
Muzhi Zhu, Hao Zhong, Canyu Zhao, Zongze Du Zheng Huang, Mingyu Liu, Hao Chen, Cheng Zou, Jingdong Chen, Ming Yang, and Chunhua Shen. Active-o3: Empowering multimodal large language models with active perception via grpo. arXiv preprint arXiv:2505.2145/7 2025b. URL https: / / arxiv. wwt.aa.wwww..aa//ooooooooaa 01125555...117.
21457. 1, 10
15

---

Published as a conference paper at ICLR 2026
APPENDIX
Use of Large Language Models Statement
17
PAPOD Objective
17
C Implementation Details
17
D Vision-Dependency Analysis
18
Additional Analysis on Attention Pattern
F Controlled Experiments Reference KL Removal with GRPO
20
G Masking Strategies
21
G.1 Random Masking
G.2 Semantic-Aware Masking..
21
................................
G.3 Further Analysis of Masking Strategies
22
H Additional Experiments on Regularizing DAPO Baseline
22
I Additional Results on Ablation Studies
24
Additional Results on prep Hacking Analysis
24
K Additional Results on OCR Tasks
25
Robustness in Low Vision-Dependent Tasks
25
M Exploration On Collapsing Behaviors
25
N Additional Analysis on Reduced Perception Error
26
Computational Overhead Analysis
27
Additional Analysis on Related Work
27
16

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS STATEMENT
Large language models, such as ChatGPT, are used exclusively for grammar checking during the writing process. They are not used for research ideation. addition, LLM-based coding assistants, such as Copilot, are employed to support code implementation.
B PAPO, OBJECTIVE
Similar to PAPO we extend DAPO (Yu et al., 2025) with our perception-aware supervision signal implement pA The complete pAPOp objective is shown as follows:
1 ΣΣ{
JEAPOD (0) ( - (0lg.0) DE lii
sml 1ml
min (O),(O) clip (r,, a, ten) Ana
+yDKe (Tal|=00 (r00") 1/ mT[[TT]]
mask)
with Ho: vis.quiivleetdeee
val (a, < G
where y is the weighting coefficient for KL-pep- 1/1 and f/2 are the weighting coefficients for the Double Entropy Loss terms, and (0)) /m<a)]a./ma) quantifies the model's reliance visual information. We implement Lerep using the same approximation method in PAPO Note that in PAPOp. the reference KL penalty (Dxx [re|))re/)) is removed as in DAPO. In addition, the dynamic sampling strategy is enabled prevent sampling instances with either all-correct or all-wrong rollouts. The maximum number of retries for dynamic sampling is set to 20 for both the DAPO baseline and PAPO D-
mo(0:19.1)
C IMPLEMENTATION DETAILS
Dataset. We use ViRL39K (Wang et al., 2025b), a diverse collection 38.8K multimodal reasoning QA pairs covering problems in math, STEM, charts, and social topics. Note that the training data includes only input queries and final answers, without intermediate chain-of-thought (CoT) annotations.
Models. We employ the Qwen2.5-VL-3B (Qwen Team, 2024a), Qwen2.5-VL--BB (Qwen Team, 2024b) and Owen3-VL-2B-thhnninn (Qwen Team, 2024c) as our base models. By default, the base model refers to Qwen2.5-VL. We consider the following main model variants and default configurations:
GRPO: GRPO (Shao et al., 2024) baseline with clipping factors set to 0.2, 9Λ 0.3, and reference KL penalty coefficient set to 8 == 0.01.
DAPO: DAPO (Yu al., 2025) baseline with clipping factors to q 0.2, Cд 0.28, reference KL removed, token-level loss averaging enabled, and dynamic sampling with maximum of 20 retries.
p: PAPO instantiated from GRPO.
PAPO PAPO instantiated from DAPO.
Table 6 summarizes the hyperparameter configurations for the best performing model variants as shown in Table 1, Detailed ablations and analysis are presented $5.2 and $5.3. For PAPO models. we use random masking with default masking ratio of 0.6. Double Entropy Loss is essential for settings with higher 7 on models, and for configurations without the reference KI penalty (i.e., PAPOp), due to the inherently weaker regularization against deviation from the base model.
We conduct RLVR on all model variants using the following typical response format, where reasoning steps enclosed in <think>< <think></thinks> </think> and the final answer is enclosed in [boxed(]. Each model is trained for 2 epochs on ViRL39K (Wang et al., 2025b) with a learning rate of le-6 and weight decay of 1e-2. We use and 4 NVIDIA H100 80G GPUs for 3B and 7B
Training.
17

---

Published as a conference paper at ICLR 2026
Model
M) 7/2
Mask Ratio
E1y 6h
Qwen2.5-VL GRPO-33
0.01
0.2, 0.3
Qwen2.5-VL GRPO-7B
0.01
0.2, 0.3
Qwen3-VL GRPO-2B
0.01
0.2, 0.3
Qwen2.5VVL PAPO
0.02
0.6
0.01
0.2, 0.3
Qwen2.5-VL pAPO
0.02
0.05
0.6
0.01
0.2, 0.3
Qwen3-VL. PAPOc-2B
0.01
0.6
0.01
0.2, 0.3
Qwen2.5-VL DAPO-3B
0.2, 0.28
Qwen2.5-VL. DAPO-7B
0.2, 0.28
Qwen2.5VVI. PAPO
0.03
0.6
0.2, 0.28
Qwen2.5-VL PAPO
0.01
0.03
0.6
0.2, 0.28
Table 6: Hyperparameter configurations for models in Table 1.
experiments respectively. We use rollout batchsize of 384, and generate n = 5 responses per prompt. More details on training configuration can be found in the code supplementary.
D VISION-DEPENDENG ANALYSIS
We observe that current multimodal rea- soning benchmarks exhibit varying degrees of reliance on visual information, ranging from questions answerable through textual cues alone to those requiring comprehensive visual analysis. To systematically charac- terize this spectrum, propose taxon- omy of vision dependency levels and an- alyze their distribution across mainstream multimodal reasoning datasets, including ViRL39K (Wang al., 2025b), Geome- try3K (Lu et al., 2021), Math Vista (Lu et al., 2023), Verse (Zhang et al., 2024), Log- it'ista (Xiao et al., 2024), MMK12 (Meng al., 2025b), We-Math (Qiao al., 2024), SuperClevv-Counting (Li et al., 2023), and MMMU-Pro (Yue et al., 2024). Specifically, we categorize vision dependency into three levels based on the significance distribution of critical information between textual ques- tions and visual contents:
Table 7: Analysis of vision-dependency across nine multimodal reasoning datasets.
Dataset
Low
Medium
High
Training Dataset
ViRL39K (Wang et al., 2025b)
x
General Reasoning
Geo3K (Lu et al., 2021)
LogicVista (Xiao al., 2024)
X
Math Verse (Zhang al., 2024)
Math (Lu et al., 2023)
7
MMK12 (Meng et al., 2025b)
We-Math (Qiao et al., 2024)
Vision-Dependent Reasoning
Counting (I.i et 2023)
x
x
Varseersey (Zhang et al., 2024)
X
/
MMMU-Pro (Yue al., 2024)
X X
Low: In low vision-dependenccy tasks, the questions typically embed substantial visual information within the textual input itself, such as specifying the length of an important triangle side, thereby reducing the model's reliance on visual processing.
tasks provide partial contextual information textually while requiring the model to perceive and extract complementary visual features from the image.
Medium: Medium-level
High: High vision iisoondeppendency tasks contain minimal or no visual information in the textual input, requiring the model derive answers entirely through visual reasoning.
We manually examine the data instances and dataset construction pipelines for each benchmark and summarize our findings in Table 7. In Figure 6, we further illustrate representative examples from each category, demonstrating how the practical manifestation of these dependency levels directly correlates with the perception challenges we observe in current multimodal reasoning models. Notably, experiments show that PAPO's improvements most pronounced (8.0%) high vision-dependency tasks, unveiling that learning perception-awee policies is essential for robust multimodal reasoning.
18

---

Published as a conference paper at ICLR 2026
Low Vision-Dependee Dependency
Medium Vision- Denen-Depyndeecy
Vision- Dependency
Question
Question
Question
Plane d-fffrrnet types of crops the eapeiiacatal lides stons the figure tall
epilateen trisngls right trisngls, and
the program fowcharr domm in the figer csccutad, then the value the oalput
Searco Calogery: Geomsrry
sn isoedes trinagle are ploced as shavea in the fixues, Gaven that the lone angle of
Source MAIKT
Categery: Mach
oypecal mar salectels, with
Sarree MMKKE Categry: Lagie
plasting type crup. axar planting
beoedes trinngle
Answer
Answer
120
04-71
Answer
Question
Question
Question
equidistatt
many image??
Source Greck
Caleueey: Geunetry
Answer
Calegry:: Gonmerry
Sourcer Countiez
Category 31 Count
Answer
Answer
Question
Question
Question
As slorm the figere, if engle C- 350, thin the degive at smple ADI
As shows in the flgam, O the ounter the dtelk, then the degree of anglo ADB (s
(No Question Contcol)
Soutee: Methlene Carrgey:: Cemmetry
Choires
Colcuees: Goomary
Saurze Marklteree
Categey: Genmerr
Answer
Calee D:150"
Saurce
Answer
Question
Question
Use sodte paper sheet with central angle ot 1210 and radius of aJl in roll inte
In the Eqgare above, the ratio the leagth of leegth
(No Qucstion Contectt
MathV'au Calepoy: Geumerr
Suuvec MMMMU Pro Caregrr:: Physecs
Saurce: Mor Cangory: Namn
ceuicel bettonicos paper cap (is showe
the picturi), then buttem parimuter
teagth of timw
Answer
paper cap
Answer
Question
Question
Question
Moxenm figann trienihe AMC
shawea the fleams, triangle ABC
Soutee Ne-haa Caergry Geemetrs
lsocedes trieggle and tie siles the
Issurlls inkangle, (13-(1 (V" Choicoc Mit 60; C. 120; D. 150;
Source: MINENU Pre Coteger:: Plynees
Choilen: A. S0, de C: 130, D. 1900 correct atanet
side of ack. What
(?"
anewver
Catrggrs:
Answer
Answer
Answer
Figure 6: Illustrative examples of different levels of vision-dependencyy
19

---

Published as a conference paper at ICLR 2026
GRPO Response Tokens Image Tokens uttention (avg) Attention Distribution: Image: 5.5%, Text: 94.5%
PAPO Response Tokens Image Tokens attention (avg)
Attention Distribution: Image: 8.4%, Text: 91.6%
Figure 7: Qualitative analysis of attention patterns comparing the GRPO baseline and PAPO We find that (1) PAPO encourages more attention on image tokens versus text tokens, and (2) PAPO places stronger attention on salient objects while reducing attention on empty regions (as highlighted in the circles).
Table 8: Controlled experiments on reference KL removal. For PAPOg. we add a Double Entropy Loss with coefficient of 0.03 for both 3B and 7B models. We find that PAPOg highly compatible with this setting, achieving further improvements with an average relative gain of 11.2% and 4.0%. Improvements against the GRPO No KL,ef more pronounced on more vision-dependent tasks, highlighting stronger perception capabilities for reasoning.
General
Vision
Overall
Method
AVG
4%
AVG
%
AVG
%
3B
GRPO
51.89
42.97
47.92
GRPO + No Kl
53.96
1 4.75
45.46
1 5.37
50.18
1 5.03
PAPOG + No KL
56.21
9.26
49.33
13.60
53.15
11.19
7B
GRPO
62.51
54.11
58.78
GRPO + No KL,,r
63,99
t 2.05
57.94
1 5.36
61.30
1 3.53
PAPOg No KL.
63.31
f
59.18
7.54
61.47
1 3.99
E ADDITIONAL ANALYSIS ON ATTENTION PATTERN
We visualize the attention heatmap from the response tokens the input image tokens. In Figure we present the attention patterns of GRPO and PAPO, along with the overall attention distribution across image and text tokens in the inputs. We observe that (1) PAPO encourages be allocated to the input image tokens overall, and (2) PAPO directs attention more precisely toward salient objects while suppressing attention to empty uninformative regions.
attention to
F CONTROLLED EXPERIMENTS ON REFERENCE KL REMOVAL WITH GRPO
To have further controlled investigation the robustness of PAPO with KLprop under the removal of the original KL penalty, we consider an additional GRPO variant in which we remove the reference KL without introducing other modifications in DAPO. Under this setting, remove the DKL Freyl term (referred to as K L from the GRPO and PAPOg training objective (Eq 2).
In Table 8, compare GRPO No KL with PAPOg No KL using 7 0.01 and Double Entropy Loss with 77, = m/ = 0.03. We observe that PAPO performs well in this setting, achieving overall improvements of 11.2% and 4.0% for the 3B and 7B models, respectively. Its superiority
20

---

Published as a conference paper at ICLR 2026
over the GRPO No K baseline is particularly evident rision-dependens tasks, with gains of 13.6% and 7.5%, indicating enhanced perception capabilities for reasoning.
G MASKING STRATEGIES
We extend our elaboration on our two masking strategies for creating corrupted visual inputs maskk used in the Implicit Perception Loss KLpppp- We show an illustration of different masking strategies Figure As visualized in Figure we observe that patch-based masking removes informative semantic contents more effectively while pixel-level noises typically preserve semantics even at high noise levels. We detail the two patch-level masking strategies as follows.
G.1 RANDOM MASKING
Random masking is simple and computationally efficient approach for generating 4mask Given an input image and a blackening probability E [0, 1], we traverse the image in a grid pattern with 14 by default). For each patch location, generate independent
patch size X pixels random variable r ~ Uni form(0, 1) and mask the patch if <
This random masking implementation ensures that each patch has the probability p of being masked, independent of other patches. The expected fraction of masked patches is with minimal computa- tional overhead.
G.2 SEMANTIC-AWAN MASKING
Semantic- aware masking aims to preferentially mask patches that contain more semantically important visual information. This approach leverages a pre-trained vision encoder identify salient regions before applying masking. Our implementation uses DINOv2 (Oquab et al., 2023) as the vision encoder for strong self-supervised representation learning capabilities.
Iitention-Based Saliency Computation. Given an input image I, we first process the image to obtain patch-level attention maps. For a model with layers and attention heads, this yields attention matrices A (!) "xxNN for each layer / 2. L, where N the total number of patches. As middle layers often capture more meaningful semantic relationships, we employ 6, 7, layers to aggregate patch-level self-attention scores for saliency computation.
Specifically, for each selected layer 1, aggregate attention across heads using mean pooling:
A
(
A
(I)
The saliency score computed for each patch by summing the attention receives from all other patches:
N
(5)
1.+1
1=2
where the +1 offset accounts for the first CLS token.
Saliency scores are averaged across selected layers:
!
8:
(6)
iEL
where I. 6, 7, 8,9 is is the set of selected layers.
Saliency-Based Patch Selection. With computed saliency scores for all image patches, we select patches for masking through:
Ranking, Sort patches in descending order of saliency scores to identify the most semanti- cally important regions.
21

---

Published as a conference paper at ICLR 2026
Table Normalized entropy of the attention distribution (from the response tokens to the input image tokens), comparing PAPO trained with random masking and semantic masking.
Model (Qwen2.5-VL)
Mean Geo3K
Median Geo3K
Mean Counting Median Counting
3B PAPO-G w/ Random Masking
0.9520
0.9522
0.9351
0.9362
3B PAPO-G w/ Semantic Masking
0,9522
0,9517
0,9331
0.9342
7B PAPO-G w/ Random Masking
0.9457
0.9472
0.9525
0.9530
7B PAPO-G w/ Semantic Masking
0,9401
0.9403
0,9513
0.9512
Top-k Selection. Given masking ratio p. select the top k [p (N
-1) patches with
highest saliency scores for masking.
Patch Masking, Apply the same zero-out operation as in random masking to the selected high-saliency patches.
G.3 FURTHER ANALYSIS MASKING STRATEGIES
In Table 3, we empirically observed that random masking performs better than semantic-aware masking. We believe the reason is that random masking naturally strikes a balance between masking out enough informative content while avoiding the removal of an entire "block" of salient information. similar phenomenon has been observed in prior work on masked autoencoders (He et al., 2022) and masked visual modeling (Xie et al., 2022; Fu et al., 2023), where random masking is found to be competitive with, or even superior to, block-wise and attention-based masking strategies.
To further investigate what makes random masking performant, we conduct two additional analysis: (1) analyzing the attention patterns PAPO models trained with random masking versus semantic masking, and (2) examining the error distribution in cases where the random- mandom-makkin model succeeds but the semantic-masking model fails.
Attention Pattern Analysis. The goal of this experiment to investigate whether the masking strategy leads to different attention behaviors after training. We extract the average attention weight from response tokens to input image tokens, and then compute the normalized entropy of the attention distribution across the image tokens. The normalized entropy is defined Ent(P) (/ /Ent Puni), where Puns is the uniform distribution over image patches; this normalization accounts for different numbers of image tokens. A higher entropy indicates more scattered attention distribution, whereas lower entropy indicates a more centralized one. We conduct this experiment on two datasets, Geo3K and Counting, sampling 100 instances per dataset.
As shown in Table 9, we find that the overall attention patterns do not differ significantly between the two masking strategies. However, the random-masking model generally exhibits slightly higher entropy, indicating more scattered attention distribution. This observation aligns with our hypothesis that semantic masking encourages the model to focus more heavily on centralized salient region.
Error Analysis. We also investigate what types of errors cause the remantic-masking model to underperform relative to the random-masking model. We sample 30 instances per dataset from Geo3K, Counting, and Verse where the masking model fails, and manually categorize the error types. As presented in Figure 8, we find that the predominant error source is still perception-related errors, suggesting that the primary advantage of randong over semantic-masking lies in improved perception ability.
model is correct but the semantic-
H ADDITIONAL EXPERIMENTE ON REGULARIZING DAPO BASELINE
As shown in Figure 4, we observe model collapsing on the DAPO-7B baseline. In this section, we investigate further regularizing this baseline with an entropy loss and compare with PAPO
Unlike PAPO that uses both Te and Trmask, DAPO baseline operates with a single policy ma. Accord- ingly, we explore the effects of adding an entropy loss term to the DAPO objective:
JDAPO (0) (0) m)H{{TT))
(7)
22

---

Published as a conference paper at ICLR 2026
Correct
Purseption Error
Reasoning Erroe
Inconslistncey Error
Calculation Error
Random Masidng Successful
30 (100%)
Goe3k
Semantic Masking Fallure
(73%)
(7%)
(17%))
Random Masking Successful
30 (100%)
Counting
Semantic Masking Failure
(93%)
(7%)
Random Masking Successful
30 (100%)
MallVorrs
Semantic Masking Fallure
(77%)
(7%) (7%) (10%)
Figure 8: Error categorization for cases where the maaanni model fails but the random- masking model succeeds. We find that the most common reason for underperformance is poorer perception.
Training Ace Reward on Vir139K
0.75
0.70
0.60
0.55
DAPO 7B
DAPO 7B + Ent Loss
PAPO, 7B (Double Ent Loss)
0.45
0
25
50
75
100
125
150
175
200
Steps
Figure 9: Training dynamics of DAPO baseline with entropy loss. Adding entropy loss to the DAPO-7B baseline effectively delays the collapse. PAPO, with Double Entropy Loss maintains more stable training and achieves superior performance. The comparison of benchmark performance presented in Table 10.
where H is computed as the log probabilities of the response tokens, and r is set to the same as 71 0.03 as in PAPOp. Figure 9 and Table 10 present the training dynamics and evaluation performance the regularized DAPO baseline, compared with the original DAPO and PAPO Our main findings as follows:
Adding single entropy loss to DAPO successfully delays collapse and improves end-task performance, but the regularization does not fully prevent collapse.
PAPO with Double Entropy Loss consistently outperforms both DAPO variants and completely prevents collapse throughout training.
23

---

Published as a conference paper at ICLR 2026
Table 10: Performance of DAPO baseline with entropy loss. While adding entropy loss effectively reduces collapsing on DAPO-7B, PAPOp with Double Entropy Loss consistently outperforms this regularized DAPO baseline.
General
Vision
Overall
Method
AVG
% rel
AVG
3%
AVG
A% ral
vel
DAPO-7B
57.58
51.79
55.01
DAPO-7B Ent Loss
64.77
13.52
59.14
17.73
62.27
15.86
PAPO (w/ Double Ent)
65.83
15.61
59.82
19.09
63.16
17.54
Training Acc Reward on Vir139K
0.55
0.50
0.45
PAPO (y 0.01, No ke
(No Reg) - Collapse
0.40
PAPO, (y 0.01, No kL
(w/ Double Ent)
0.35
pAPO 3B (y 0.02, No kL
(w/ Double Ent) - Collapse
0.30
0
25
50
75
100
125
150
175
200
Steps
Figure 10: Impact of KL weighting (7) under settings without reference KL. Double Entropy Loss indispensable for stabilizing training in this setting. Due inherently weaker regularization, should be set to smaller value. When set higher (e.g., 0.02), model collapse still occurs, even with Double Entropy Loss.
I ADDITIONAL RESULTS ON ABLATION STUDIES
We provide additional results for the ablation studies discussed in $ 5.2. Figure 10 presents the impact of varying the KLprop weighting on PAPOG under settings with the reference KL removed. Due inherently weaker regularization such settings, higher y (c.g.. 0.02) can lead irreversible collapse, even with Double Entropy Loss applied. Empirically, good default in this setting, including PAPO is 0.01. Figure 10 also highlights the importance of Double Entropy Loss in enhancing training stability, even when is low.
In Table we further investigate the impact of the Double Entropy loss coeficients (n). Here, smaller coefficient means less regularization strength. At 0.01, we observe model collapse in the later stages training, whereas 0.03 and 0.05 the entropy loss successfully prevents hacking behavior. Empirically, 0.05 performs the best across 9 evaluation benchmarks.
ADDITIONAL RESULTS ON KLpRcP HACKING ANALYSIS
We provide additional results for the deep dive analysis on the KL hacking problem as discussed in 5.2. Figure 11 presents the early signs of model collapsing due to KL Hacking. Figure 12 presents the key factors influencing the likelihood of model collapse. Figure presents a comparison of the effectiveness of different regularization strategies.
24

---

Published as a conference paper at ICLR 2026
Table 11: Impact of Double Entropy Loss coeflicient. "DE" is short for Double Entropy Loss, "@ k" indicates the coefficient value for 7/ and 172, as introduced in $3.2.
General Multimodal Reasoning
Vision- Deien-Dependent Multimodal Reasoning
Overall
Method
Geo3k
Vita We-Math MMK12 Verse
LogieViste
Counting MMMU-Pro MathVersey
AVG
GRPO
40.18
65.48
68.12
72.26
66.51
45.62
73.94
35.17
61.71
58.78
PAPO: DE
@ 0.01 (collapsed)
39:95
62.03
65.19
69.71
62.72
14.30
92.50
38,01
39.77
59.35
0.03
40.00
68.09
67.59
72.43
68.28
46.46
59.91
35.85
61.17
61.09
0.05
40.25
69.53
66.79
72.52
68.43
46.017
89.81
36.66
64.97
61.66
a.
b.
C.
d.
e.
Collape Collapied
No Ciilipac Collipwert
Cullapec
Training Ase Roward Vit3iK
Implicit Perception Loos (XL
6:30
Colliwedd
+0.2
0.001
8.005
Originel Eintopy:
Masked Entropy
-0.4
8.00
0:005
1.
0.55
-0.6
Clipping
6635
06
2:50
-1.0
0:25
No Colligee Callgnnol
No Collage Collazood
-13
0.000
150
Steps
Steps
Steps
Steps
Figure 11: Early signs of model collapsing due to KI prep Hacking. The "No Collapse" and "Collapsed" models refer to PAPOG-7B (? 0.01) and PAPOc-7B (? = 0.02 without double entropy regularization), respectively. When collapsing occurs, we notice (a-b) the Implicit Perception Loss drops drastically, accompanied by a collapsing training reward, (c) the clipping ratio-high continuously increases, which indicates the proportion of tokens undergoing policy gradient updates beyond the clipped threshold and (d-e) the entropy loss increases in both the masked policy "g and the original policy me.
mask
K ADDITIONAL RESULTS ON OCR TASKS
To further investigate the model's perception-centric ability, we include additional evaluation on OCR- bench-v2 (Fu et al., 2024). Table 12 shows that PAPO also brings benefits on such information tasks which require fine-graines perception on the visual inputs.
L ROBUSTNESS IN LOW VISION-EEPENDENT TASKS
Table 13 presents the results MMLU-Pro with dummy visual inputs, which guarantees that the image is irrelevant to the query. We find that PAPO achieves competitive or better performance in this extreme setting, indicating PAPO robustness to noisy or uninformative visual inputs.
M EXPLORATION ON COLLAPSING BEHAVIORS
In addition notable decline in performance (Table 4), collapsed models also generate responses containing entirely unrelated tokens. To explore the extent of this abnormal generation behavior, we extend our analysis to token relevance in model outputs after collapsing occurs. Qualitative examples and quantitative results can be found in Figure 13.
Experimental Setup. We evaluated both a collapsed 7B PAPO model (? = 0.02, without regular- ization) and a non-collapsed 7B GRPO baseline on Geo3K (Lu et al., 2021). To assess the coherence (OpenAI, 2025) as a judge to
and relevance of generated responses, we employed GPT-4. evaluate how well each model's response relates to and addresses the input question to 10. Our evaluation prompt below specifically instructs the judge to focus on whether the response attempts solve the given problem rather than correctness, with scoring guidelines ranging from (completely unrelated/gin (iibber//gbbbrrshh)) to 10 (perfectly related reasoning, even if the final answer is incorrect). The complete evaluation prompt provided in Figure 14.
scale from 0
25

---

Published as a conference paper at ICLR 2026
Ingact Klyy
Model
of Masking Raioo
Trining Ane Reward mn VirtKK.
0.20
Training Virtlgg
Thiming Rewwrrd on VinUSK
0.50
2.M0
3.35
12.00
2.50
0.35
PAPA, Cillapee PAPPA 75
281
0.50
2.30
3.30
Ce0c 1.8 Lulape
100
125
190
185
25
150 155
300
0
25
190
125
150
115
Steps
Steps
Figure 12: Influential factors towards KI Krpp Hacking. We identify three main factors: (a) KL-rrep weighting (higher values lead greater likelihood of collapse); (b) size (the larger the model, the more likely it is to collapse); (c) an extreme masking ratio (e.g., 1.0) results in a faster collapse.
prep
Table 12: Results on OCR-bench-
Table 13: Results on MMLU-Pro with dummy visual inputs. The base model is Qwen2.5-VL.
v2. The base model is Qwen2.5-VL
Size
Method
OCR-bench-r2 (%)
Size
Method
MMLU-pro Dummy Visual (%)
3B
Base
20.14
3B
Basc
23.23
3B
GRPO
26.32
3B
GRPO
39.34
3B
PAPOc
27.22
3B
PAPOg
40.74
7B
Base
27.75
7B
Base
31.57
7B
GRPO
31.17
7B
GRPO
50.76
7B
PAPOG
32.11
7B
PAPOG
50.55
Quantitative Analysis. Our relatedness evaluation revealed that the collapsed model demonstrated significantly degraded coherence, with an average relatedness score approximately 18% lower than the baseline model. This substantial drop in relevance scores reflects the model's tendency to generate tokens that bear little semantic relationship to the input context or fail to attempt solving the given problem. Additionally, by computing per-token KL divergence between responses generated from original images versus randomly patch-blackened versions of the same images. The collapsed model showed approximately 8.4 times higher variance in KL divergence compared to the baseline, indicating that the model has learned to exploit the KLpcp by generating highly unpredictable token sequences.
measured the variance in the KI KL loss across response tokens
Qualitative Observations. Based on the grading of GPT-4. 1-min and manual inspection, find that collapsed models frequently generate responses that may begin with some problem-relevant content but contain substantial portions of irrelevant text, numbers, apparent meaningless contents. An example can be found in Figure 13.
N ADDITIONAL ANALYSIS ON REDUCED PERCEPTION ERROR
In Figure 1, we show that PAPO significantly reduces perception-related errors. We further analyze how these instances behave: do they all become correct, or do they still struggle with other types of errors? To investigate this, we sample 60 instances that originally resulted in perception errors under the GRPO baseline and examine their outcomes under PAPO. As shown in Figure 15 41% of the original perception errors are corrected, 24% remain perception errors, and 25% shift to other error categories. This indicates that eliminating perception errors alone may not be sufficient for some challenging instances. However, overall, PAPO achieves a substantial 40% complete error removal, demonstrating strong effectiveness in improving the overall performance.
26

---

Published as a conference paper at ICLR 2026
Example Generating Unrelated Tokens
Question: Find x.
Ground Truth: 221
<think> In right triangle, the Pythagoream theorem can be used to find length the hypotenuse, [...]
boxed (@ifundefined (F (1
SICLASSOPTIONeupl Cappiiooapoooooolorr collorr colorbex (blue! wh!whi
['normalfont %
elsel@@ifundefined (color) fommm2a 'nnormaffont2999
peop Variance
Relatedness
4
2
GRPO
Collapsed PAPO
GRPO
Collapsed PAPO
Figure 13: Collapsing behavior. A distinctive generation pattern in collapsed models is the produc- tion of irrelevant tokens. We verify this quantitatively by prompting GPT-4. 1-mini OpenAI (2025) to provide relatedness scores of the responses from 0 to 10 for GRPO and collapsed PAPO (y 0.02, regularization) model. We further compare the variance of KLppp the response tokens. As illustrated, the collapsed PAPO model exhibits significantly lower relatedness scores and higher variance across tokens in KI prep- See $5.3 for detailed discussion.
Table 14: Analysis of computational overhead. We report the average time per training step (in seconds) and the time spent the additional forward pass in PAPO. The experiments conducted using 2 and NVIDIA H100 GPUs for the 3B and 7B models, respectively. While we observe moderate increase training time per step (67.2 seconds for 3B and 108.6 seconds for 7B) we leave further optimization of training efficiency to future work.
Model
Method
Per Step (s) Additional Forward Pass (8)
GRPO
360,9
3B
PAPOG
428.1
48.8
GRPO
258,5
7B
PAPOc
367.1
49.7
O COMPUTATIONAL OVERHEAD ANALYSIS
The main computational overhead stems from the additional forward pass on the rollout sequences with corrupted visual input. In Table 14, report the averaged wall-clock time for each training step and the additional forward pass comparing GRPO and PAPOg. 3B and 7B experiments are conducted on 2 and 4 Nvidia H100 GPUs respectively.
P ADDITIONAL ANALYSIS ON RELATED WORK
In Table 15, additionally offer comprehensive categorization and comparison of recent literature within the broader area of advancing multimodal reasoning. We highlight our novelty as the first work to focus on improving the optimization objectives in multimodal RL.
27

---

Published as a conference paper at ICLR 2026
Prompt Scoring Relatedness
You evaluatoe for mathematical visual ressoning tacks, Your job evaluste how well a model's ressoning and answer relites given question.
provided
An image containing problem
The question with specific response format instructions
The ground truth anower
The model's generated answer
Your task nie the model's answer on a spale 0-10 based on how well relotes to and addrresees the question:
"*Sooring Guidelines:
the strongly the model's ""g--**; Completely unrelated (talks about other inclovant things topics)
general, the higher
and addressex the questioo.
**2-3**; Minimally related (meniions some relevant terms but most centents are not attempting to solve the problom)
Somewfstt
(stlempts sddres
mujor unrelated contents)
**g-744; Moderately relatad (addresses the problem with some correct reasoning but significant unrelated sentenoes words)
+eg.gee: Highly related (good attempe solving the problem with mostly pertaining reasoning. minor unrelated sentences words)
**10=*; Perfectly related (correctly addresses the problem with totally related reasoning. even if final answer differs from ground truth)
Please differentiate between scores of each level (e.g., is more unrelaned than 3)
** "Importani **
Focus whether
ATTEMPTS
given problem,
-A mathemetically incomeet answer can still acore high if shows proger undenstanding and reasoning
(bxsdd of following instructions The ground tuth provided for comtext, but your score should be based on relevance and reasoning quality, not exact matching
Consider the (ormatting roquirements (like using sthink?
**Questoon [fommatedd question)
**Ground Truth Answer: ** (ground truth)
*MMode'ss Generated *n
(generated answer)
Based on the image, question, and model's response, provide score from 0-10 indicating how well the model's answer relates to and addresses the question.
Respond with ONLY numerical (0-10), nothing elsc.
Figure 14: Prompt to GPT-4.I-ming for scoring the relatedness between the -gnerregeeeeted response and the input query.
Perception
Reasenlng
Inconsistency Error
Calculation
Original Frvor Examples
(100%))
(01%)
(24%)
6100%)
Figure 15: Detailed analysis of 60 GRPO perception-erron cases and their outcomes under PAPO.
28

---

Published as a conference paper at ICLR 2026
Table 15: Comprehensive categorization and comparison of different approaches. We highlight our novelty as the first work to focus on improving the optimization objectives in multimodal RL.
Method
Improvement Perspective
Highlight
Think or Not? (Wang Training Frame- et al., 2025c)
Introduces Cold-Start SFT stage with "thought dropout," allowing model skip unnecessary thinking on simple tasks improve efficiency.
work
Semi-off-Polic (Shen
Training Frame- work
Separates perception and reasoning into two stages/mooell:: generates captions first, then performs reasoning on them, and propagates rewards back to improve the multimodal model.
al., 2025b), Vision- SRI (Li al, 2025c)
Open Vision Rea-
Data
Synthesizes reasoning trajectories by leveraging strong text-only reasoning models such DeepSeek-RI.
soner (Wei et al., 2025),
RI-V (Chen et al., 2025),
LMM-RI (Peng al.,
2025)
Reason-RFT al., 2025), MM-
(Tan Data
Constructs diverse multimodal reasoning dataset covering counting, structural perception, spatial transformation.
Eureka (Meng al., 2025b), VL-
rethinker (Wang al., 2025h)
NoisyRollout (Liu et al., 2025a), RI-Share VL (Yao al, 2025)
Rollout
Introduces cemaantic-prsservve data augmentation during rollout, such Gaussian noise, to diversify exploration and improve robust-
VL-rethinker (Wang et al., 2025b), Skywork RIV2 (Wang et al.,
Rollout
Introduces Selective Sumple Replay to mitigute the issue of vamishing advantages.
2025d)
Visual-RFT et al., 2025c), Vision- RI (Huang et al., 2025),
(Liu
Reward
Introduces grounding related rewards such as IoU and Dynamic IoU to enhance detection and localization performance.
V-Triune (Ma et al,, 2025).
Perception- RI (Xiao et al., 2025b),
VisionReasoner (Liu et al., 2025h)
Visionary-RI RI (Xia et al., Reward 2025)
Introduces model-based rewards computed directly captions gen- erated by the policy model.
GRIT (Fan et al., 2025), Reward
Introduces multi-turn rewards using "thinking-withe with-images"" wiihgn-wwth-bbbbinngggg puradigms.
TreeVGR (Wang al.. 2025a), DeepEyes (Zheng al., 2025), Mini-o3 (Lai
al., 2025)
PAPO (ours)
Optimization Objective
Distinct from previous work, PAPO explores improvements from the core optimization objective tailored to the multimodal domain, and shows compatibility with advances from other perspectives.
29


Reviews:
[{"review_id": "e3SOeF38FH", "reviewer": "Reviewer_Tryh", "summary": "- This paper introduces PAPO for multimodal reasoning to encourage visually grounded generation by modifying the core optimization objective, addressing the finding that perception errors on models trained with standard RLVR account for 67% of failures.\n- PAPO incorporates an Implicit Perception Loss, a KL divergence term that maximizes the difference between output distributions conditioned on original versus masked visual inputs, combined with a Double Entropy Loss for regularization.\n- The algorithm, which requires no additional data or reward models, is integrated into GRPO and DAPO frameworks and evaluated on several benchmarks, demonstrating consistent performance improvements and reduction in perception errors.", "questions": "Please refer to the weaknesses section for detail. Relatively minor questions:\n- Considering the benchmarks are largely skewed toward math and geometry, it would be interesting to see how an OCR-based masking strategy (low masking ratio but information-wise critical) would perform with PAPO.\n- While the Double Entropy Loss plays a crucial role in stabilizing PAPO, its own sensitivity to hyperparameter choice is not explored.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This work addresses an important problem of perception error in multimodal reasoning, starting from a clear, evidence-based motivation to achieve practical performance gains.\n- The method effectively combines known techniques, such as visual input masking and entropy regularization, with existing RLVR frameworks (GRPO/DAPO) in a simple yet potent way, yielding significant benefits without making the framework overly complex.\n- The proposed methodology is validated by extensive analyses, covering hyperparameter impacts, masking strategies, and training dynamics, including a transparent investigation of the failure mode.", "weaknesses": "- My main concern is the incomplete experimental comparison to establish its state-of-the-art significance. The paper references a number of existing works that modify GRPO/DAPO for multimodal reasoning, yet the experiments in Table 1 only compare PAPO against the original GRPO and DAPO baselines. Without a comparative analysis against other contemporary SOTA variants under identical training conditions, experimental significance seems limited.\n- The framework's generalizability is undermined by its exclusive reliance on the Qwen2.5 model family. While this limitation is acknowledged in Section 7, it is difficult to determine if this is a broadly effective approach or an architectural artifact of Qwen.\n- The paper's readability and logical flow are hampered by its over-reliance on the appendix. Critical analyses that form the core of the method's justification are deferred from the main text. This forces the reader to constantly switch contexts to verify the paper's central claims on stability and robustness. For the self-containedness of the main paper, consider aggregating key experimental highlights from the Appendix as a concise figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xVn8yWumrq", "reviewer": "Reviewer_XvCB", "summary": "This paper proposes Perception-Aware Policy Optimization (PAPO), an RLVR algorithm. It addresses dominant visual perception errors in multimodal LLMs by integrating an Implicit Perception Loss (KL divergence with masked visual inputs) into GRPO, fostering visually grounded reasoning. A Double Entropy Loss ensures training stability against \"loss hacking.\" PAPO achieves significant performance boosts on multimodal benchmarks, notably without external data or reward models.", "questions": "Please refer to the Weaknesses section above for details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper convincingly identifies and empirically quantifies visual perception errors as a primary source of failure in multimodal reasoning, providing a strong foundation for its contribution.\n2. PAPO's core idea of using an Implicit Perception Loss to intrinsically encourage visually grounded reasoning, alongside a robust Double Entropy Loss for stability, is novel, intuitive, and empirically proven to yield substantial performance gains.\n3. PAPO achieves state-of-the-art improvements without relying on additional data curation, external reward models, or stronger teacher models, making it a highly practical and accessible method for enhancing multimodal LLMs.", "weaknesses": "1.I am concerned about the result where random masking empirically outperforms DINOv2-based semantic-aware masking. The authors' hypothesis that semantic-aware masking obscures entire salient regions, leading to indiscriminate attention, requires more robust empirical validation. This outcome, coupled with the observation that pixel-level noise is less effective at obscuring informative semantics, prompts a critical need for deeper analysis into the optimal balance between information retention and strategic perturbation. \n2.While the paper commendably demonstrates PAPO's effectiveness in reducing perception errors, particularly on high vision-dependency tasks, I am concerned about the potential implications for tasks with low visual dependency. Explicitly enforcing visual grounding via the Implicit Perception Loss in such scenarios might introduce unnecessary computational overhead or divert the model's attention away from pertinent textual information. I recommend that the authors conduct dedicated sub-dataset analyses on low-dependency tasks and/or perform a thorough bad-case analysis to understand PAPO's behavior in these contexts. Furthermore, while Figure 1 indicates a significant reduction in perception errors, it is imperative to analyze whether this reduction leads to a compensatory increase in other error categories (e.g., reasoning or calculation errors).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B5Z9PZy3Qc", "reviewer": "Reviewer_zdmV", "summary": "The paper proposes Perception-Aware Policy Optimization (PAPO), a new reinforcement learning framework for large multimodal models that integrates perceptual awareness directly into policy optimization. Unlike prior works, PAPO jointly learns both perception and reasoing within the core RL objective. The key idea is an implicit perception loss which encourages visually grounded reasoning without external supervision. A double entropy regularizer is introduced to stabilize training. PAPO serves as a plug-in-play replacement for GRPO/DAPO and achieves 4.4-17.5% average improvement across eight multimodal reasoning benchmarks, with up to 19.1% gain on vision-intensive tasks and 30.5% fewer perception-related errors. The method is simple, annotation-free and computationally efficient, showing faster convergence and more robust multimodal understanding", "questions": "Could the authors clarify why maximizing the reverse KL divergence between rollouts with original and masked visual inputs improves perception–reasoning coupling? A brief theoretical or deeper empirical explanation would help justify this design choice.\n\nApart from this question, I have no further concerns.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel idea of integrating perception awareness directly into the policy optimization objective, moving beyond traditional data- or reward-level modifications in multimodal RL.\n\n2. The methodology is well-motivated, theoretically sound, and empirically validated. The authors provide thorough experiments across eight multimodal reasoning benchmarks, showing consistent and interpretable gains over strong baselines such as GRPO and DAPO. The analysis includes detailed ablations and error breakdowns, supporting the claimed contributions.\n\n3. The paper is clearly written and logically organized. The motivation, derivation of the objective, and implementation details are easy to follow. Figures and examples effectively illustrate both intuition and quantitative results.", "weaknesses": "1. The experiments are conducted only on the Qwen2.5‑VL model (3B/7B) and do not include other pretrained models (e.g., InternVL).\n\n2. The paper lacks a discussion of scalability. Specifically, whether the proposed method remains effective (and efficient) when applied to much larger models (e.g., ~30B-70B parameters).\n\n3. The paper could porvide a deeper discussion of how the proposed method differs (in objective, reward design, computational cost, etc.) from those works would improve the clarity of novelty:\n\n  [1] Wang, Jiaqi; Lin, Kevin Qinghong; Cheng, James; Shou, Mike Zheng. Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models.\n\n  [2] Shen, Junhao; Zhao, Haiteng; Gu, Yuzhe; Gao, Songyang; Liu, Kuikun; Huang, Haian; Gao, Jianfei; Lin, Dahua; Zhang, Wenwei; Chen, Kai. Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning.\n\n  [3] Wei, Yana; Zhao, Liang; Sun, Jianjian; Lin, Kangheng; Yin, Jisheng; Hu, Jingcheng; Zhang, Yingmin; Weng, Zejia; Wang, Jia; Han, Chunrui; Peng, Yuang; Han, Qi; Ge, Zheng; Zhang, Xiangyu; Jiang, Daxin; Patel, Vishal M. Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning.\n\n  [4] Liu, Ziyu; Sun, Zeyi; Zang, Yuhang; Dong, Xiaoyi; Cao, Yuhang; Duan, Haodong; Lin, Dahua; Wang, Jiaqi. Visual Reinforcement Fine-Tuning.\n\n  [5] Tan, Huajie; Ji, Yuheng; Hao, Xiaoshuai; Lin, Minglan; Wang, Pengwei; Wang, Zhongyuan; Zhang, Shanghang. Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fp1IwQp3PW", "reviewer": "Reviewer_BkD7", "summary": "This paper identifies a critical bottleneck in large multimodal models (LMMs): most failures in complex reasoning tasks arise from visual perception errors rather than logical flaws. To address this, the authors propose Perception-Aware Policy Optimization (PAPO), a novel algorithm that extends Reinforcement Learning with Verifiable Rewards (RLVR) frameworks such as GRPO and DAPO.\nPAPO introduces two key components:\nImplicit Perception Loss, which encourages visually grounded responses by maximizing the KL divergence between output distributions conditioned on original versus corrupted visual inputs.\nDouble Entropy Loss, which regularizes the objective and improves training stability.\nExperiments on eight multimodal reasoning benchmarks show a 4.4%–17.5% overall improvement and a 30.5% reduction in perception errors.", "questions": "Please finish the concerns in the weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Clear and Important Motivation: The work is well-grounded in a detailed error analysis showing that perception errors account for the majority of LMM failures. This provides strong motivation for the proposed approach.\n- Novel and Efficient Design: PAPO’s use of an implicit “reverse KL” objective to enforce visual grounding is elegant and requires no extra annotations, reward models, or teacher models, making it an efficient drop-in replacement for existing RLVR methods.\n- Comprehensive Experiments: The paper evaluates multiple benchmarks and model sizes, includes ablations on masking strategies and loss weighting, and provides in-depth analysis of failure modes and stability mechanisms.", "weaknesses": "- Flawed Core Assumption: The hypothesis that model confidence should always decrease when visual information is removed is not universally valid. Robust models should ignore irrelevant visual cues, meaning PAPO may incorrectly penalize robustness in cluttered or partially relevant visual scenes (Although their mask strategies seem good).\n- High Computational Overhead: PAPO increases per-step training time by over 40% for the 7B model, while the accuracy gains are often modest, making the cost–benefit ratio questionable.\n- Metric-Method Collusion Risk: The primary evaluation metric (accuracy@8) can be artificially inflated by low-entropy policies. PAPO’s entropy regularizer may bias results toward deterministic outputs rather than genuine reasoning improvements.\n- Counterintuitive Ablation Results: The finding that random masking outperforms semantic masking suggests the method may be learning brittle correlations rather than true visual grounding, can authors make more comparisons on this problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)