R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large
  Language Models via Share-GRPO
Authors: name: Huanjin Yao, name: Qixiang Yin, name: Jingyi Zhang, name: Min Yang, name: Yibo Wang, name: Wenhao Wu, name: Fei Su, name: Li Shen, name: Minghui Qiu, name: Dacheng Tao, name: Jiaxing Huang

In this work, we aim to incentivize the reasoning ability of Multimodal Large
Language Models (MLLMs) via reinforcement learning (RL) and develop an
effective approach that mitigates the sparse reward and advantage vanishing
issues during RL. To this end, we propose Share-GRPO, a novel RL approach that
tackle these issues by exploring and sharing diverse reasoning trajectories
over expanded question space. Specifically, Share-GRPO first expands the
question space for a given question via data transformation techniques, and
then encourages MLLM to effectively explore diverse reasoning trajectories over
the expanded question space and shares the discovered reasoning trajectories
across the expanded questions during RL. In addition, Share-GRPO also shares
reward information during advantage computation, which estimates solution
advantages hierarchically across and within question variants, allowing more
accurate estimation of relative advantages and improving the stability of
policy training. Extensive evaluations over six widely-used reasoning
benchmarks showcase the superior performance of our method. Code will be
available at https://github.com/HJYao00/R1-ShareVL.