Spotlight on Token Perception for Multimodal Reinforcement Learning
Authors: name: Siyuan Huang; name: Xiaoye Qu; name: Yafu Li; name: Yun Luo; name: Zefeng He; name: Daizong Liu; name: Yu Cheng
Published: 2025-10-10 11:25:33

Abstract: While Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the
reasoning capabilities of Large Vision-Language Models (LVLMs), most existing
methods in multimodal reasoning neglect the critical role of visual perception
within the RLVR optimization process. In this paper, we undertake a pioneering
exploration of multimodal RLVR through the novel perspective of token
perception, which measures the visual dependency of each generated token. With
a granular analysis of Chain-of-Thought (CoT) processes, we uncover two key
insights: first, token perception in a rollout trajectory is sparsely
distributed, where only a small fraction of tokens have high visual dependency
for visually-grounded reasoning; second, different trajectories exhibit
significant divergence in their overall visual dependency. Based on these
observations, we propose Visually-Perceptive Policy Optimization (VPPO), a
novel policy gradient algorithm that explicitly leverages token perception to
refine the learning signal. Specifically, VPPO achieves this through a dual
mechanism: it reweights a trajectory's advantage by its overall visual
dependency, and focuses policy updates exclusively on perceptually pivotal
tokens. On a comprehensive suite of eight perception and reasoning benchmarks,
VPPO demonstrates substantial gains over leading open-source RL-tuned models,
with its effectiveness consistently validated across 7B and 32B model scales.
Our findings not only establish a new token-level perceptual perspective for
analyzing multimodal RLVR but also present a novel and effective optimization
strategy to significantly enhance the multimodal reasoning capabilities of
LVLMs.