R1-ShareVL: Incentivizing Reasoning Capabilities of Multimodal Large Language Models via Share-GRPO
Authors: Huanjin Yao
Qixiang Yin
Jingyi Zhang
Min Yang
Yibo Wang
Wenhao Wu
Fei Su
Li Shen
Minghui Qiu
Dacheng Tao
Jiaxing Huang
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Modal Large Language Models
Reinforcement Learning
Reasoning
Primary area: applications

Abstract: In this work, we aim to incentivize the reasoning ability of Multimodal Large Language Models (MLLMs) via reinforcement learning (RL) and develop an effective approach that mitigates the sparse reward and advantage vanishing issues during RL.  To this end, we propose Share-GRPO, a novel RL approach that tackle these issues by exploring and sharing diverse reasoning trajectories over expanded question space. Specifically, Share-GRPO first expands the question space for a given question via data transformation techniques, and then encourages MLLM to effectively explore diverse reasoning trajectories over the expanded question space and shares the discovered reasoning trajectories across the expanded questions during RL. In addition, Share-GRPO also shares reward information during advantage computation, which estimates solution advantages hierarchically across and within question variants, allowing more accurate estimation of relative advantages and improving the stability of policy training. Extensive evaluations over 6 widely-used reasoning benchmarks showcase the superior performance of our method. Code is available at https://github.com/HJYao00/R1-ShareVL.

R1 R1-Share VL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO
Huanjin Y Qixiang Yin"", Jingyi Zhang', Min Yang", Yibo Wang', Wenhao Wu"
Fei Su*, Li Shen", Minghui Qiu?, Dacheng Tao 1s .Jiaxing Huang 1s Nanyang Technological University "ByteDance TTsinghua University
*Beijing University of Posts and Telecommunications
6The University of Sydney Sun Yat-sen University
Equal Contribution
Corresponding Author
Abstract
In this work, we aim to incentivize the reasoning ability of Multimodal Large Language Models (MLLMs) via reinforcement learning (RL) and develop an ef- fective approach that mitigates the sparse reward and advantage vanishing issues during RL. To this end, we propose Share-GRPO, a novel RL approach that tackle these issues by exploring and sharing diverse reasoning trajectories over expanded question space. Specifically, Share-GRPO first expands the question space for a given question via data transformation techniques, and then encourages MLLM to effectively explore diverse reasoning trajectories over the expanded question space and shares the discovered reasoning trajectories across the expanded questions dur- ing RL. In addition, Share-GRPO also shares reward information during advantage computation, which estimates solution advantages hierarchically across and within question variants, allowing more accurate estimation of relative advantages and improving the stability of policy training. Extensive evaluations over 6 widely-used reasoning benchmarks showcase the superior performance of our method. Code is available https: :// thub. com Haaom//JYaoOO//0 R1 R1-ShareVL.
Introduction
The recent success of Reinforcement Learning (RL) in Large Language Models (LLMs), such as Kimi-K1.5 [ and DeepSeek-RI [2), shows its promise in incentivizing model's long-chain reasoning capability, enabling LLMs tackle complex tasks such as mathematical and scientific reasoning. The core design of these advances (e.g., GRPO (3] in Deepseek-RI) lies in online reinforcement learning without the need reward models, which encourages LLM generate group of reasoning paths and iteratively refine its reasoning process with a group relative advantage estimation mechanism based on rule-based reward functions. Typically, a simple reward strategy is adopted: reasoning paths leading to correct answers receive higher rewards, while those leading to incorrect answers receive lower ones, where the model is optimized via the group relative advantages estimated from the rewards.
Inspired by these advancements, we aim develop simple and effective reinforcement learning method for Multimodal LLMs (MLLMs) to incentivize their long-chain reasoning ability. A simple way directly apply these LLM online reinforcement learning methods like GRPO on MLLMs. However, we empirically observe that directly applying GRPO on MLLMs suffers from sparse reward and advantage vanishing issues, leading to degraded performance in enhancing MILM's 's reasoning capability 4 5 6]:
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Senorticaly Contomunt
Reward
Volid Advantage Ratio
Use peepeelllogom//AA (KLL to fo-ndd 3 andjie 3o
Repeeux Shorin
Shore-6RR
4. Selttie Space
0.65
680
0.59
Shore GAPD
GRFO
0.45
Use lolog (XLLI to fint IX - + Jid - 30
20
60
8C Step
Exganshel Sclation
(b) Reward Curve (Left) and Valid Advantage Ratio Curve (Right)
0,-0
Sop
Qaj
0,
20
Quent AL-76
Openst.7hntte-17
NMM-Erddee-7
RL R1-Shore-73
70
663
67
(NTA, if messure icto 4 and tect, calculote the volue of -171
X
This image hos been motted by 50 degnnes
$2.8
Please mema by rouase it back and so the
40.2
penblem: For paraleingnnm (M mexsure of So.
mexsum
ioth 3 the volue ofO.
0, 01.77
MatriViato
#MAStur (4) Comparison with other Roasoning MILMMs.
MuthWorse
(a) Explore and share diverse reasoning trajectories over expanded question space
Figure 1: (a) Share-GRPO expands the question space via semantically consistent transformations, and then explores diverse reasoning trajectories from different question variants and shares the discovered trajectories among them. (b) Share-GRPO provides denser rewards and higher valid advantage ratios compared to GRPO, demonstrating its effectiveness in mitigating sparse reward and advantage vanishing issues. (c) Share-GRPO outperforms the baseline and other SOTA RL-based reasoning MLLMs on both mathematical and general reasoning benchmarks.
(1) Sparse reward: Most current MLLMs, especially smaller ones, exhibit very limited long-chain reasoning capability. As result, only few generated reasoning paths receive positive rewards, especially on challenging questions and particularly during the early stage of training. This leads to sparse rewarding, inefficient exploration and instable training in GRPO-like methods.
(2) Advantage vanishing: GRPO-liKe methods compute relative advantages by comparing the rewards of group of responses sampled from given question, leading to advantage vanishing when receiving homogeneous responses. Specifically, along reinforcement learning process, the model tends to gradually predict similar and all correct responses for well-learned questions, and similar and all incorrect responses for poor-lcarned questions. In this way, the relative ad advantages tend to approach zero when the group of responses become more homogeneous, and collapse to zero when all responses receive identical rewards (e.g., all correct or all incorrect), resulting ineffective reinforcement learning.
Motivated by these observations, we propose Share-GRPO, a novel approach that introduces the concept of information sharing into MILM reinforcement learning to mitigate sparse reward and advantage vanishing issues. The core idea of Share-GRPO lies in exploring and sharing diverse reasoning trajectories over expanded question space shown in Fig. (a). Specifically, Share- GRPO first expands the question space for a given question via data transformation techniques, and then encourages MLLM to effectively explore diverse reasoning trajectories over the expanded question space and shares the discovered reasoning trajectories across the expanded questions during reinforcement learning. In this way, each expanded question variant can both contribute and benefit from the reasoning trajectories generated by others in the expand question space, allowing the model to jointly explore and learn from shared solution space across expanded questions.
In addition, Share-GRPO also shares reward information during advantage computation, which estimates solution advantages hierarchically across and within question variant, allowing more accurate estimation of relative advantages and improving the stability of policy training. Specifically, we estimate advantages at two levels: a local level. which consists of responses generated from each individual question variant, and a global level, which aggregates responses across all variants of the same seed question. This hierarchical advantage estimation enables more robust and fine-grained relative advantage computation, where the local level captures intra-variam structure and variance while the global level exploits eross-variant diversity and complementarity and stabilizes reward signals.
In this way, Share-GRPO effectively mitigates the sparse reward and advantage vanishing issues: (1) GRPee expands the question space and enables more diverse solution space for each given question, which effectivvly increases the likelihood of generating successful reasoning response and

---

thus mitigates the sparse rewarding issue as illustrated in the left curve of Fig. (b). (2) Share-GRPO allows the model to explore diverse reasoning trajectories from the expanded question space and shares the discovered reasoning trajectories, ultimately mitigating the advantage vanishing issue effectively as illustrated in the right curve in Fig. (b). (3) Share-GiRPO estimates solution advantages hierarchically across and within question variant, which enables more accurate estimation of relative advantages and stable reinforcement learning process.
In summary, the main contributions of this work are summarized as follows: First, we introduce the concept of information sharing into MLLM reinforcement learning, and propose Share-GRPO
which explores and shares diverse reasoning trajectories over expanded question space, effectively mitigating the sparse reward and advantage vanishing issues. To the best of our knowledge, this is the first work that explores information sharing for MLLM reasoning reinforcement learning. Second, we design hierarchical advantage estimation method by sharing reward information, which estimates solution advantages hierarchically across and within question variant, allowing accurate and robust advantage estimation. Third, extensive experiments on MI.LM reasoning benchmarks demonstrate the superiority of our proposed methods as illustrated in Fig. (c).
2 Related Work
2.1 Multimodal Large Language Model
Multimodal Large Language Models (MLLMs) 7 8 I0 14 16 18 demonstrate
outstanding performance in semantic understanding of cross-domain visual content and multimodal reasoning. Early research on MLLMs primarily focused on text-image alignment and the integration
of multiple modalities [99 20,
21 22 23). Subsequently, models like GPT-4V (24] achieved
breakthroughs in cross-modal understanding through multimodal instruction ane-ttnnng. enabling them to support simple tasks such image captioning, visual question answering and OCR. More complex tasks, such as mathematical reasoning, document understanding. etc., require MI.LMss to be able to perform complex logical deductions. For MILMM reasoning, models such Multimodal- CoT 25 and LLaVA-CoT (26) employ -f-thuuhttt (CoT) reasoning, breaking down the multimodal reasoning process into step-by-step inference steps while leveraging multimodal data to improve the model's 's reasoning capabilities. Additionally, Mulberry (27) proposes CoMCTS to generate effective reasoning paths through multi-model collaboration. Different from these studies, this work focuses on reinforcement learning to improve MLLM reasoning capability.
2.2 Reinforcement Learning for Multimodal Large Language Model Reasoning
Reinforcement learning has become an essential technology for enhancing the capabilitiess of MILMs.
Early research primarily focused on Reinforcement Learning from Human (R)H) 128 29
30 31, which aligns the outputs of multimodal models with human preferences by incorporating
human feedback signals. Recently, DeepSeek-RI 1 (2) utilizes a simple rule-based reward function to provide effective and reliable reward signals during the RL process. This indicates that the Group Relative Policy Optimization (GRPO) with result-level rewards effectively enhances the reasoning ability of LLMs [32 B3). In the multimodal domain, researchers have begun exploring the use of RL to enhance the visual reasoning capabilities of ML.Mss. Recent works, such Vision-R1 (34 and MM-Eureka S have open-sourced large-scale SFT cold start data and RL data. RI-V 35
Reason- RFT 36. R1-VL [4] and other methods [37 38 39 40 have designed various rule-based
reward functions to enhance the reasoning abilities of MLLMs, such as geometric understanding and spatial perception. Unlike these methods, our Share-GRPO explores information sharing for MILM reasoning reinforcement learning to mitigate sparse reward and advantage vanishing issues.
2.3 Information Sharing in Deep Learning
Information sharing is a key strategy in deep learning, enabling more effective learning through the exchange of signals across modalities, tasks, or hierarchical model components. In multi- modal learning, models such as ViL.ERTT [4] and LXMERT [42] employ cross-modal attention to achieve fine-grained information fusion between vision and language streams. In contrastive learning (e.g., SimCLR 4 MoCo (4)) ) shared representations across augmented views enhance
feature robustness. This concept extends to reinforcement learning, especially in multi-task and
3

---

multi- settings, where information sharing improves sample efficiency and mitigates sparse rewards. Methods like Distral [45] and PopArt [46] promote shared policy structures, while agents in multi-agent RI. benefit from shared value functions or communication protocols 44 48 49
further demonstrate that shared representations enhance generalization multi-task RI.. Unlike prior work, we introduce information sharing into MLLM reasoning reinforcement learning to mitigate sparse rewards and advantage vanishing for more effective reasoning learning.
Method
This section first provides the preliminary of Group Relative Policy Optimization (GRPO), and then presents the proposed Share-GRPO that introduces the concept of information sharing into MLLM reinforcement learning. Further details are elaborated in the subsequent subsections.
3.1 Preliminary
Group Relative Policy Optimization (GRPO). GRPO 3 variant of Proximal Policy Optimiza- tion (PPO) [50], designed to enhance the performance of LLMs on complex reasoning tasks. such as mathematical and scientific reasoning. Starting with a pretrained MLLM be optimized, GRPO first uses it to initialize policy model xo and reference model Tola. For given image-text pair (I, the reference policy model Te.. generates a set of responses f01, 02, ..., ).). A group-based reward function then computes the corresponding rewards (R1. R2.... RG which are subsequently used to estimate the advantage Ai for each response relative to the group:
A, R₁ mean ((R))
(1)
std (113)
Similar to PPO, GRPO employs clipped objective with a KL penalty term:
(O)
B(I,T)--p,onna ((11T)
To(o; I,?)
To(o; I,T)
min
A, clip
1+)) To (0; I,T)²
1 1 1 + A. A. -DDLL (Fol (tal()ret))
n
To (0; I,T)
(2)
Sparse Reward and Advantage Vanishing Issues. Despite the effectiveness of GRPO, it generally faces two challenges when applied to MLLMs: the sparse reward issue and the advantage vanishing issue. Sparse rewarding arises due to the limited reasoning ability of current MLLMs, where only few reasoning paths receive positive rewards, leading to inefficient exploration and instable training. To alleviate this, prior work such as RI-VL [4] introduces step-wise reward signals to provide dense rewards throughout the reasoning process. Advantage vanishing occurs when MI.LMs generate homogeneous responses for the same question and receive identical rewards, causing the relative advantages to collapse to zero and resulting ineffective reinforcement learning. To tackle this issue, VL-Rethinker [6] and Skywork R1 [51) select the samples with large magnitudes of advantages and reuse them in RL process, while MM-Eureka (5) employs an online filtering strategy to remove the samples with zero advantage. Different from the prior works, our Share-GRPO effectively addresses both these two challenges by exploring and sharing diverse reasoning trajectories over expanded question space, therefore encouraging reward diversity and stable policy optimization.
3.2 Share-GRPO
We propose Share-GRPO a novel online MLLM reinforcement learning framework that mitigates the sparse reward and advantage vanishing issues via exploring and sharing diverse reasoning trajectories over expanded question space. Specifically, for given question, Share-GRPP GRPO first applies semantically consistent transformation to generate a set of varied but semantically equivalent questions, thereby expanding the question space. It then encourages the MILM to explore diverse reasoning paths over the expanded question space and facilitates the sharing of discovered reasoning trajectories and their rewards across the expanded questions during the reinforcement learning process, as illustrated in Fig.

---

Shared Policy Optimization
A
Augi As
%
malo
Ryco Rea
Q^ (T. T)
9. Fm
Policy Medal
Rewerd Medel
Shannng rn Reavard
Sharing Respunse
o
Aotta
KaCe 1o.a
ne(oI-10u
Exponded
Response
Reward
Shored Adventage Estimation
Shored Policy Optimization Computation
Question Space
Figure 2: Overview the proposed Share-GRPO. For a given question, Share-GRPO first applies se- mantically consistent transformation to generate a set of varied but semantically equivalent questions, thereby expanding the question space. then encourages the MLLM to explore diverse reasoning paths over the expanded question space and facilitates the sharing of discovered reasoning trajectories and their rewards across the expanded questions during the reinforcement learning process.
3.2.1 Reasoning Space Expansion
Question Space Expansion. To expand the question space for a given question, we introduce Semantically Consistent Transformation (SCT) which generates group of question variant Q {Q1, Q2. Qm } for each given question Qori = (Tori, Jors). Specifically, we propose two types of transformation techniques, iee,, offline textual SCT and online multimodal SCT, for more diverse, comprehensive and flexible question space expansion.
(1) Offline Textual Semantically Consistent Transformation. Prior to online reinforce learning, we first employ offline textual SCT o(.) to rewrite the textual prompt T orss for each give question. Specifically, prompt GPT-4o to generate semantically consistent variants, resulting in expanded question set. The textual prompts of the generated variants differ from that of the original question Tors in syntactic structure and lexical expressions, while preserving the original intent and the corresponding correct answer:
Qottiie (TToor)) Tori).
(2) Online Multimodal Semantically Consistent Transformation. During online reinforcement learning, we introduce a multimodal SCT strategy to further expand the question space on the fly. Given an image Tori in the input question, we apply visual transformations v)(-) alter its visual content. Specifically, we carefully select transformations (e.g., rotation, noise injection) that preserve critical visual cues necessary for reasoning, and avoid transformations (e.g., cropping, color distortion) that may disrupt key information. Each image undergoes one randomly selected transformation with probability p. In addition, to mitigate the potential semantic inconsistencies between the visual and textual inputs after visual changes, we perform manual textual transformation that appends a transformation speccifi prompt to the corresponding textual prompt, providing contextual guidance aligned with the visual modification:
gonline = (T(((TTr))) (Tors))
Solution Space Expansion. With the expanded question space Q = (Q1. Qz.....qmi Share-GRPO enables to explore diverse reasoning trajectories in enlarged solution space for each given question. Specifically, for each question Q; E Q the policy model To generates n candidate reasoning Q1 responses, resulting in an expanded response set: O = [(o ..... of {of Q
,..00m }}.
3.2.2 Shared Advantage Estimation
With the expanded reasoning space, Share-GRPO shares reward information during advantage computation, which estimates reasoning trajectory advantages hierarchically across and within question variant.
5

---

Following GRPO [], we adopt rule-based reward functions to compute the reward for each gener- ated reasoning trajectory, Le., R {(r] ..... ) (rfmm ..... } outcome-level accuracy reward, which assigns higher rewards to reasoning paths that lead to correct answers and lower rewards to those leading to incorrect ones, In addition, we employ a format reward that encourages the reasoning trajectory to follow detailed step-by-step process before providing the final answer.
Specifically, we adopt an
With the computed rewards R, we propose hierarchical advantage estimation approach that computes advantage at two levels: global level, which aggregates responses across all variants of the same original question; and a local level, which considers responses generated from each individual question variant.
(1) Global-level Advantage Estimation. We first estimate the advantage from global perspective, where the relative advantage is computed using the rewards obtained from all question variants Q =QQ,, Q2,..., Qm.:
R. - mean (11r) ..... ,.., 1 " (rim (... std (11 rn...
-man
Aglobal
(5)
21) ",..... (1 ;.. Qm
(2) Local-level Advantage Estimation. We also estimate the advantage local level, where the relative advantage is computed within the responses generated from each individual question variant Qj E Q. Specifically, for each question variant Qj. the local advantage is estimated follows:
R -een (r9 ....TA Q, std ((rf).... ......)))
A Aocaal
With the global-level advantage and local-level advantage estimated via Eqs. 5 the final advantage as follow:
and 6
we can obtain
Agglohal Azlobal k.j,b
+ A.j.k
j= k 8 k
A.j.k
where the local advantage Alocal is only computed when the responses generated from the same question variant, i.e., when j k. By incorporating hierarchical advantage estimation, Share-GRPO achieves more accurate relative advantage computation, leading to more stable and effective policy training.
3.2.3 Shared Policy Optimization
With the expanded reasoning space and the shared advantage estimation, Share-GRPo enables to explore and share diverse reasoning trajectories and allows more accurate advantage estimation for each given question. Then, we optimize policy model Ty by sharing diverse reasoning trajectories
O (1o
Q1 J..., {o
Q... ....0"" across question variants Q = {Q1, Q2, Qm
(0)) B(Q) (111
To(o,)) Q [kk)
To(O,0) Qx)
1- 1 1 ) Wiik ]
min
Q) Abier clip
qbier
m
Taaa (0, [2k)
Q: [Qx) TH (0
(8)
4 Experiments
In this section, we first provide implementation details in Sec. and then present main results in conduct comprehensive
Sec. 4.2 that demonstrate the effectiveness of Share-GRPO, In Sec. 4.3

---

Table Main Results. To examine the effectiveness of Share-GRPOI we compare our RI-ShareVL which is trained by Share-GRPO without cold-start supervised fine-tuning against SOTAs across multiple reasoning tasks, including both domain-specific and general-purpose tasks. denotes
evaluation on official weights using VLMEEal [44
Model
Math Vista
MMStar
MMMU
Math Math
A12D)
Avg.
GPT-4o 55
63.8
65.1
70.7
50.8
30.4
84.9
60.9
Claude3,7-Sonnn 56
66.8
71.8
52.0
41.3
Kimil.5
70.1
68.0
31.0
ALaVA-Reasoner-B8
50.6
54.0
40.0
78.5
LLaVA-CoT-IIBB
54.8
57.6
78.7
Mulberry- 7B
63.1
55.0
Qwen2.5-V1 58 (Base Model)
68.2
63.9
58.6
49.2
25.1
83.9
58.1
X-REASONER-7H [59
69.0
56.4
29.6
RI R1-OOnvision-7BB
64.1
47.1
29.9
Vision-R1-7B[34
73.5
64.3*
54.2*
52.4
29.4"
84.2"
59.7
OpenV13Thinker-78
70.2
63.2
51.9
47.9
29.6
82.7
57.6
FMMFuurka-77BSS
73.0
65.1"
55.3"
50.3
26.9
84.1"
59.1
ThinkL.te-7B 60
74.3
63.7
53.1
52.2
29.9
83.0
59.3
RI-ShareVL--7B
75.4
67.0
58.1
52.8
29.5
84.5
61.2
Scaling to Larger Models
Qwen2.5-VLL-222 [5
(Base Model)
74.7
69.5
70.0
49.9
38.4
84.6*
64.5
MM-Eureka-32BB
74.8
67.3"
64.6*
56.5
34.4
85.4"
63.8
RI-ShareVL-325
77.6
70.2
70.1
59.0
40.3
86.2
67.2
ablation studies examine the impact of each design in Share-GRPO. Sec. 4.4 provides more
discussion and analysis of Share-GRPO, More details are elaborated the subsequent subsections.
4.1 Implementation Details
In this work, we adopt Qwen2.5-VL-7B and Qwen2.5-VL-32B 52 as our base models. For training data, we randomly sample 52K multimodal data from MM-Eureka (5]. Model optimization is carried out using EasyR1 [53] codebase, with training conducted NVIDIA H100 GPUs for the 7B model and 32 H100 GPUs for the 32B model. For RL-related hyperparameters, use global batch size of 128, a rollout batch size of 512, a rollout temperature of 0.7, and a learning rate le-6.
4.2 Main Results
To comprehensively examine the effectiveness of our proposed Share-GRPO we conduct experiments on models of different sizes (i.e., 7B and 32B). Notably, unlike prior studies [4] 37 34]. we do involve an additional cold-start stage with supervised fine-tuning. As shown Table [ we provide an extensive comparison against state-of-the-ar models across 6 widely used and challenging benchmarks, covering diverse range of reasoning tasks from specialized domains to general purposs reasoning. A detailed description of the benchmarks can be found in the appendix.
Comparison with baselines. We first compare our RI-Share 7B and RI R1-ShareVi VL 32B trained by Share-GRPO with the corresponding base models, i.e., Qwen2.5-VL-77 and Qwen2.5-VL. V.L-32B As presented in Table T Share-GRPO effectively improves the long-chain reasoning capabilities of MLLMMs by large margins. For example, on the challenging mathematical benchmarks like Math Vista and MathVerse. RI-ShareVL--BB achieves improvements of +7.2% and +3.6%, respectively. is worth noting that, based on previous studies, RL can enhance MLLMs long-chain reasoning ability mathematical tasks, but it often comes at the cost of degraded performance on multi- discipline and general benchmarks. For instance, ThinkLite-7B drops -0.2% and -5.5% on MMStar and MMMU, respectively. In contrast, our RI-Share VL-7B model achieves +3.1% improvement on MMStar and comparable accuracy on MMMU, demonstrating Thare-GRPO' generalization capability enhancing reasoning across diverse tasks. When scaling our method larger models (i.e., Qwen2.5-Vi VL-332 with stronger foundational capabilities, our method remains robust and consistently improves performance. particular, --Shhre achieves +9.1 improvement over the baseline model Verseerse, along with average performance gain of +2.7%.

---

Comparison with MLLMs trained via RL. We then compare RI-Share VL with other state-of-the art MLLMs trained by reinforcement learning approaches. Our RI RI-ShareVL-7B using the same base model and fewer training data outperforms MM-Fureka-7B with an average performance gain of +2.1% %, especially a notable improvement of + +1.4% on Math Vista. Notably, beyond its capability in long-chain mathematical reasoning, R1 VL. also exhibits stronger reasoning generalization to nulti-discipline and general reasoning tasks. Specifically, compared to ThinkLite-7B which also excels in mathematical reasoning, RI-ShareVIL VL achieves better performance on the multi-disciplin benchmark MMMU and the general benchmark MMStar, outperforming it by +5.0% and +3.3%, respectively. Besides, similar conclusion can be observed on larger models: our R1- VL 32B further improves overall performance compared with JIM-Eureka3355 -3BB by +3.4%, demonstrating the effectiveness and generalization of Share-GRPO
4.3
Ablation Study
Table 2: Ablation study of Share-GRPO.
Ablation Study of Share-GRPO. As shown in Table 2 we conduct abla- tion studies to examine the individual contribution of each design in Share- GRPO, including shared policy opti- mization (i.e., offline and online se- mantically consistent transformation) and shared advantage estimation (i.e.,
Method
Shared Policy
Shared Advantage
MathVista
Offine Online
Global
Local
Quen2.5-VL-T7 (Baseline) Qwea2.5-VL. Booa2.5-VL-7B-.. GRPO
68.2
72.8
73.9
74.8
Share-GRPO (Ours)
75.4
global and local advantage estimation). Compared to the GRPO baseline, incorporating the informa- tion sharing among only offline question variants with global shared advantage estimation yields a performance boost of +1.1%. Further including the information sharing among online multimodal semantically consistent transformations results in exploring and sharing more diverse reasoning paths and a +0.9% performance improvement. Finally, enabling both global and local advantage estima- tion achieves the best result of 75.4% on Vista, highlighting the effectiveness of hicrarchical advantage computation. These results demonstrate that both policy sharing and advantage sharing contribute significantly to the final performance of Share-GRPO.
4.4 Discussion
Complementarity Between Share-GRPO and Dynamic Sampling. We compare Share-GRPO with dynamic sampling [61 and further discuss their complementarity. as shown in Table Dynamic sampling performs multiple rollouts generate candidate responses, from which subset with valid rewards is selected to update the policy. Distinct from it, Share-GRPO enhances the diversity of reasoning responses and shares them during policy optimization, yielding richer reward signals. On average, GRPO with dynamic sampling achieves a performance of 65.1 1%, while Share-GRPO yields higher score of 66.7%, demonstrating the potential of leveraging diverse reasoning paths enrich reward feedback and enhance policy learning. Moreover, combining Stare-GRPO with dynamic sampling achieves a higher accuracy of 67.2% on average, highlighting the great complementarity of Share- GRRPO
Table 3: The complementarity between Share-GRPP and Dynamic Sampling.
Method
Masta
MMStar
MMMU
Avg.
GRPO + Dynamic Sampling
73.6
65.7
56.1
65.1
Share-GRPO
75.4
67.0
58.1
66.8
Share-GRPO + Dynamic Sampling
75.9
66.7
59.0
67.2
Hyperparameter studies of Share GRPO. We study the impact of the number of question variant m in Semantically Consistent Transformation (SCT) in Table 4 with the number of responses per question fixed at 6. When m = 2, Share-Gra -rrre achieves an accuracy of 75.4% on Math Vista, significantly outperforming the baseline GRPO. When m is increased to 4, the performance of the model trained with Share-GRPO further improves, demonstrating the benefit of scaling up the number of question variants.
The Impact of the Number of Sampling N. Table 5 we compare the performance of GRPO with different sampling numbers N against Share-GRPO. We observe that with N = 6, the model trained
8

---

Rt-Shore-7a: chhrii these (CS)aEE)) 4)) ((9))
To find the aqustion the torgent liss the groph AQ) ((FF x reed follow psint largency
2.Find MogeTo find the slope of the tongent line, nead compute N() usig the choin rulect((t) f((++ Tarmt
((3-))
243+1
From groph, see ('(2) -3
Quastion: Lest/ be twice differentiable function on the interval -1 is i i with 70) 0 and (C2) 3 the graph f the derivative of f is shown above. The graph of creeses
5.1.8 K() "F'"" TR
J.Equation of the tongent line exaattion of the forgent at (S-) with stope
y-3 (x
4. Let be the function gleen by +T) which the equation for the line
Simplifying
y "2"" 15+
tongent to the groph of ot =
Convert fraction with denominator
Choicee
15 36
Av 5x/12 7/4
12
12"""1
Simplify the froction:
Sa/12 5/4
Cy 7a/12 e 7/4
Mbored()) </thinko
Correct Answer (A]
\bosed(A)
Figure 3: Qualitative Results of reasoning capability of RI-ShareVL. on mathematical problems.
Table Hyperparameter study of Share- GRPO. We study the impact of the number of question variants in Share-GRPO.
Table 5: The study on the Number of Sam- pling N. We study the impact of the number of sampling in SRare-GRPO.
Method
Question
Math Vista
Method
Sampling
Math Masta
Share-GRPO
2
75.4
GRPO
6
72.3
Share-GRPO
3
75.6
GRPO
12
72.8
Share-GRPO
75.9
GRPO
24
73.0
Share-GRPO
(3+3)
74.7
Share-GRIP
(6+6)
75.4
using GRPO achieves a score of 72.3% on Math As the number of sampling increases, the performance improves to 72.8 N 12. However, further increasing the sampling number N to 24 yields only marginal gains of 0.2%, while introducing additional computational overhead. Therefore, increasing the number of sampling reaches a performance ceiling, making an ineffective way to further improve reasoning reinforcement learning. Instead of simply increasing N, Share-GRPO enhances the diversity of reasoning paths and leverages the concept of information sharing amplify reward signals and enhance training stability. By sharing responses and incorporating hierarchical advantage estimation, our RI-Share VL 7B achieves a score of 75.4% with only generated responses per question, surpassing the performance of GRPO even with 24 sampled responses.
4.5 Qualitative Results
Fig. 3 iillsstaats that Share-GRPO effectively enhances the model's reasoning ability on complex mathematical problems. In this example, the model accurately interprets the question and arrives at the correct answer, showing strong performance in symbolic reasoning and function analysis. This highlights the capability of Share-GRPO to guide the model toward precise and coherent solutions in mathematically demanding tasks.
Conclusion
In this paper, we propose Share-GRPO, novel reinforcement learning framework for MLLMs, which introduces the concept of information sharing to effectively mitigate the challenges of sparse rewards and advantage vanishing. Share-GRPO expands the question space by generating semantically consistent variants, and encourages MLLMs to explore and share responses across a more diverse solution space. Furthermore, Share-GRPO estimates advantages hierarchically within and across question variants at both global and local levels to effectively guide optimization. We conduct extensive experiments, ablation studies and discussion, which demonstrate the superiority of our proposed methods on various reasoning benchmarks.

---

References
[1] Kimi Team, Angang Du, Bofei Gao, Bowei Xing. Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du Chonghua Liao, et al. Kimi kl. S: Scaling reinforcement learning with Ilms. arXiv preprint 2iv:2500. 12599, 2025.
[2] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2944,, 2025.
[3] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402. 03300, 2024.
[4] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. RI-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503. 129377777,7 2025.
[5] Fanqing Meng, Lingxia Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, ct al. Mm-eureka: Exploring visual aha moment with rulc-based large-scale reinforcement learning. arXiv preprint arXiv:2503.07366 2025.
[6] Haozhe Wang, Chao Qu Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504 08837, 2025.
[7] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diego Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 23030 2023.
[8] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, ct al. Qwen technical report. arXiv preprint arXiv:2309 16609, 2023.
[9] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 24/.:24/2..11 2024.
[10] Peter Tong, Ellis Brown Penghao Wa, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang. Jihan Yang. Manoj Middepogu. Ziteng Wang, al. Cambrian-1: A fully open, vision-centric exploration of multimodal Ilms. Advances in Neural Information Processing Systems, 37:87310-887 87356, 2024.
[11] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, el al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 24198- 2024.
[12] Dongchen Lu, Yuyao Sun, Zilu Zhang. Leping Huang. Jianliang Zeng. Mao Shu, and Huo Cao. Internvl- x: Advancing and accelerating internyl series with efficient visual token compression. arXiv preprint arXiv:2503.213307 2025.
[13] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac. Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 18055 2023.
[14] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: A gpt-4v level mllm your phone. arXiv preprint 240: 0180, 2024.
[15] Ziheng Wu, Zhenghao Chen, Ruipu Luo, Can Zhang, Yuan Gao, Zhentao He, Xian Wang. Haoran Lin, and Minghui Qiu. Valley2: Exploring multimodal models with scalable vision-language design. arXiv preprint SxXiv:5001.0590/ 2025.
[16] Huanjin Yao, Wenhao Wu, Taojiannan Yang, YuXin Song, Mengxi Zhang, Haocheng Feng, Yifan Sun, Zhiheng Li, Wanli Ouyang, and Jingdong Wang. Dense connector for milms. Advances in Neural Information Processing Systems, 37:33108-3342 331400, 2024.
[17] Jingyi Zhang, Jiaxing Huang, Sheng Jin and Shijian Lu. Vision- models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
10

---

[18] Huanjin Yao, Ruifei Zhang, Jiaxing Huang, Jingyi Zhang, Yibo Wang, Bo Fang, Ruolin Zhu, Yongcheng Jing. Shunyu Liu, Guanbin Li, et al. A survey agentic multimodal large language models. arXiv preprint arXiv:2510. 10991, 2025.
[19] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, (1134899-349188 2023.
[20] bean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for 2022.
few-shot learning. Advances in neural information processing systems,
[21] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22396, 2024.
[22] Xiang Lan, Feng Wu, Kai Hc, Qinghao Zhao, Shenda Hong, and Mengling Feng. Gem: Empowering milm for grounded ecg understanding with time series and images, arXiv preprint arXiv:2503, 25X0::2503,0677,,, 2025.
[23] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llavanext: Improved reasoning, oer, and world knowledge, 2024.
[24] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309. 9(1x1 2023.
[25] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.0092. 2023.
[26] Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step, 2024. URL http://arxiv.. 1tg/abs/2411.0444
[27] Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang. Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering milm with ol like reasoning and reflection via collective monte carlo tree search. arXiv preprint axXv:24112.18319 2024.
[28] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 35:27730-27744, 2022.
[29] Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023.
[30] Tianyu Yu. Yuan Yao, Haoye Zhang, Taiwen He. Yifeng Han. Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tar Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milmss via behavior alignment from fine-grained correctional human feedback. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18807-1386, 2024.
[31] Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open-source ai feedback for super gp trustworthiness. arXiv preprint arXiv:2405. /7220, 2024.
[32] Haotian Luo, Li Shen, Haiying He. Yibo Wang. Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-pruner: harngtiiarmmnizzig fine-tuning for ol-like reasoning pruning. arXiv preprint arXiv:2501. 12570, 2025.
[33] Haotian Luo, Haiying He, Yibo Wang. Jinluan Yang. Rui Liu, Naiqiang Tan, Xiaochun Cao, Dacheng Tao, and Li Shen. Ada-rl: Hybrid cot via bi-level adaptive reasoning optimization. arXiv preprint arXiv:2504: 1659, 2025.
[34] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao. Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.0674/ 2025.
[35] Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in dision-language models with less than $3. https:///iithb.. wowpe//eee//eeeeeee//eeeeeeeee V. 2025. Accessed: 2025-02-02.

---

[36] Huajie Tan, Yuheng Ji. Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang. and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.2073 2025.
[37] Yang. Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. Rl-onevision: Advancing generalized multimodal reasoning through modsl formalization. arXiv preprint arXiv:2503, 10615, 2025.
[38] Hardy Chen, Haoqin Tu, Fali Wang. Hui Liu, Xianfeng Tang. Xinya Du, Yuyin Zhou, and Cihang Xie. Sit rI? early investigation into training rl-like reasoning large vision-language models. arXiv preprint arXiv:2504.1 1468, 2025.
[39] Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-We Chang. Openvithinker: An early exploration to complex vision-language reasoning via iterative improvenrt arXiv preprint arXiv:2503.77352, 2025.
[40] Yi Peng, Xiaokun Wang, Yichen Wei, Jiangbo Pei, Weijie Qiu, Ai Jian, Yunzhuo Hao, Jiachun Pan, Tianyidan Xie, Li Ge, et al. Skywork rlv: pioneering multimodal reasoning with chain-of-thought. arXiv preprint arXiv:2504.05599 2025.
[41] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and- and-language tasks. Advances in neural information processing systems, 32, 2019.
[42] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:/908.000 07490, 2019.
[43] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PmLR, 2020.
[44] Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pages 9729-9738, 2020.
[45] Yee Teh, Victor Bapst, Wojeiech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. Advances neural information processing systems, 30, 2017.
[46] Mattco Hessel, Hubert Soyer, Lasse Espeholt, Wojeicch Czarnecki, Simon Schmitt, and Hado Van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3796-3803, 2019.
[47] Jakoh Foerster, Toannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016.
[48] Ryan Lowe, Yi I Wu, Aviv Tamar, Jcan Harb, OpenAl Pieter Abbcel, and Igor Mordatch. Multi-agent actor-critic for mixed cosperative-compettiive environments. Advances in neural information processing systems, 30, 2017.
[49] Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowledge in multi-task deep reinforcement learning. arXiv preprint arXiv:2401. 2024.
[50] John Schulman, Filip Wolski, Prafulla Dhariwal, Alee Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 170iv:17770063 06347, 2017.
[51] Yichen Wei, Yi Peng. Xiaokun Wang. Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang, Yunzhuo Hao, Xuchen Song, et al. Skywork r1v2: Multimodal hybrid reinforcement learning for reasoning. arXiv preprint 2r54v:200 16656, 2025.
[52] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijic Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[53] Zheng Yaowei, Lu Junting, Wang Shenzhi, Feng Zhangchi, Kuang Dongdong, and Xiong Yuwen. Easyrl: An efficient, scalable, multi modality rl training framework. https:///iithb. com www.///wwwga//aaaaaaaeeeo Easy11 2025.
12

---

[54] Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 11198-11201, 2024.
[55] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.2127. 2024.
[56] Anthropic. Claude 3.5 sonnet, 2024.
[57] Ruchong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang. Zhiqing Sun, Zhe Gan, Yinfei Yang. Ruoming Pang, and Yiming Yang. Improve vision language model chain-of-thought reasoning. arXiv preprint orXiv:2410.16/1
2024.
[58] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Weabin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502. 2025.
[59] Qianchu Liu, Sheng Zhang. Guanghui Qin, Timothy Ossowski, Yu Gu, Ying Jin, Sid Kiblawi, Sam Preston, Mu Wei, Paul Vozila, et al. X-reasoner: Towards generalizable reasoning across modalities and domains. arXiv preprint arXiv:2505.0398), 2025.
[60] Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung- -hing Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mcts- Mets-guided sample selection for data-efficient visual reasoning self-improvement. arXiv preprint rxXv:25044007334 2025.
[61] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open- IIm reinforcement learning system at scale, arXiv preprint or Xiv:2500. 14476, 2025.
[62] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai-Wei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
[63] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint Xiv: 240:3 2403. 20330, 2024.
[64] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv: 16502, 2023.
[65] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang. Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169-186. Springer, 2024.
[66] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hong- sheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances Neural Information Processing Systems, 37:95095-95169, 2024.
[67] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. diagram is worth a dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1-14, 2016, Proceedings, Part IV 14, pages 235-251. Springer, 2016.
13

---

Appendix
A
More Discussions
Comparison with Data Augmentation. We conduct a comparison between Share-GRPO and GRPO combined with data augmentation to evaluate the effectiveness our method versus simple input- level augmentation, as shown in Table We observe that applying question variants and image transformations for GRPO yields some performance improvement, but these gains remain limited due to the persistent sparse reward problem. In contrast, Share-GRPO further leverages response sharing and shared advantage estimation to better enrich reward signals and achieve better improvements.
Table 6: Comparison with Data Augmentation. Trans. denotes transformation.
GRPO GRPO Image Trans. GRPO Language Trans. GRPO Image & Language Trans. ShareGRPO
Vist 72.8
73.4
73.1
73.6
75.4
Analyzing the Relationship Between Training Steps and Performance in Share-GRPO. As shown in Table the performance of Share- GRPre-GRPP on Math Vista gradually improves during training and reaches its peak at Step 100, achieving a score of 75.4. After this point, the performance shows minimal fluctuation, with scores of 74.7 and 74.5 at Steps 125 and 150, respectively, This demonstrates the stability and effectiveness of the Share-GRPO training process.
Table 7: The Relationship Between Training Steps and Performance in Share-GRPO
Step 25
Step 50
Step 75
Step 100
Step 125
Step 150
Vista
72.4
73.8
73.1
75.4
74.7
74.5
The Validity of Generated Questions. To ensure the validity of the generated questions (i.e. Textual Semantically Consistent Transformation), we conducted series quality assurance exper- iments. Specifically, we tested multiple prompt designs to obtain stable and high-quality outputs, developed rule-based scripts to automatically filter out incomplete or inconsistent generations, and manually reviewed numerous samples to further ensure the quality of the results. Moreover, we performed GRPO training using only one newly generated question variant per sample. The train- ing process remained stable and yielded slight performance improvement as shown in Table g empirically demonstrating the quality of the generated questions.
Table 8: Performance comparison of GRPO model using original and newly generated questions.
One original question
One new question
GRPO
72.8
72.9
Extended performance comparison between GRPO and ShareGRPO. We provided more ex- perimental results comparing GRPO and ShareGRPO across all benchmarks using Qwen 2.5-VL. The results, shown in Table 9 demonstrate that ShareGRPO achieves consistent improvements over GRPO, with an average performance gain 1.9% across benchmarks.
Table 9: Extended performance comparison between GRPO and ShareGRPO.
Methods
Math Vista
MMStar
MMMU
MathVerse
Math Vision
A12D
AVG
GRPO
72.8
65.4
56.4
50.7
26.7 29.5
84,0
59.3
ShareGRPO
75.4
67.0
58.1
52.8
84.5
61.2
B Explanation of Equation 8.
We clarify the meaning of no( | Q) in the GRPO framework. For given question Q, we first roll out responses and compute both its advantage A and the conditional probability na(: | Q) for each
14

---

token in the response. Here, mo(- Q) denotes the parameterized policy (with parameters 6) that outputs probability distribution over possible tokens conditioned on Q. If the advantage is positive, the probability of the corresponding tokens is increased; if negative, the probability is suppressed.
In ShareGRPO, {Qi, Q.... Qm.. For each variant Qj, we generate corresponding set of responses: O (oo 0n ..... "ofm"...off } Each response Q: is explicitly generated by the policy model conditioned on its corresponding question variant Q2--
extend this formulation by introducing multiple question variants
The notation To (0 2) Qx) in Eq. represents the probability (under the current policy) of generating
the response o Qa but conditioned on a possibly different question variant Qk- This formulation naturally leads to two cases:
1. When k, this corresponds to the self-updating case, where the model updates its parameters using the response under its own generating context.
2. When j * k. this corresponds to the cross-updating case, where response generated from one question variant is reused to update the probability under different variant.
This design allows ShareGRPO to perform cross-variant updates, enabling information sharing and enhancing robustness across semantically consistent yet syntactically distinct variants.
C Benchmarks
We evaluate our trained models against the following authoritative benchmarks.
Math Vista 622 is used to evaluate the mathematical problem-solee solving ability of MLLMs, containing 6141 questions covering areas such as arithmetic, geometry, algebra, and statis- tics.
MMStar is an innovative multimodal assessment benchmark that includes 1500 care- fully selected visual key samples, addressing issues of visual redundancy and data leakage existing assessments.
MMMU 64 is large-scale interdisciplinary multimodal understanding and reasoning benchmark that collects 11.5 multimodal questions from university exams, quizzes, and textbooks.
Math Verse 65 includes 2612 multimodal mathematics problems and has manually anno- tated 1.5672 test samples, comprising 3 main types of questions and 12 subcategories, such as plane geometry, solid geometry, and functions.
Math Vision 66 is collection of 3,040 high-quality mathematics problems, all accompa- nied by visual contexts, sourced from real mathematics competitions.
A12D [67 a dataset that contains over 5000 scientific charts, which can used tasks such as image classification and visual question answering.
D Online Transformation Prompt
In Fig we show the prompts for online multimodal semantically consistent transformation, used for image transformation and modification of the corresponding questions.
E Offline Transformation Prompt
We show the prompts for offline textual semantically consistent transformation.
Offline Textual Semantically Consistent Transformation Prompt
I have a question and its answer. Please generate 4 alternative questions that are semantically similar but use different wording or phrasing, Each
15

---

Original question: For parallelogram JKLM, if the measure of is 2b and the measure of JM is 3a, calculate the value of b.
Rotate 180' (counterclockwise)) "The image is rotated 180 degrees
counterclockwise. Please rotate it back and help me solve the question based the corrected image. For parallelogram JKLM, if the measure of JK 2b + and the measure of is 3a, calculate the value of b.
Add gaussian noise: "This image has been corrupted with noise. Please solve the question based this image. For parallelogram JKLM, if the measure of K is 2b + and the measure of JM 3a, calculate the volue of b.
Horizontally flipped:
"The image is horizontally flipped. Please flip it back to its original orientation and help me solve the question based on the corrected image. For parallelogram JKLM, if the measure of JK is 2h + and the measure of JM is 3a, calculate the value of b."
Figure 4: The Specific Prompt of Online multimodal semantically consistent transformation.
alternative question should complete, preserving necessary information such as given premises, context, or constraints. All questions must lead to the same final answer.
Dutput requirements
Generate 4 different but equivalent questions based on the original. Each question must be self-contained and include any relevant context, premises, or options.
If the original question includes options, make each alternative preserves them.
The final answer to each new question must be identical to the original. Return the output as a list, with one question per element.
Do not include any explanations.
Example
Question: Consider the $2015$ integers $n$, from $1$ to $2015$. Determine for how many values of $n$ it is verified that the number $ 3 is multiple of $5$.
Answer: 403
Output: [
"Consider the 2015 integers $n$ from 1 to 2015. For how many of them is $n 3'n divisible by 5?", "Among the integers $n$ from 2015, how many satisfy that $n 3 is a multiple of 5?",
"Out of the 2015 integers between and 2015, how many make 3 3 divisible by 5?"
"Consider all integers $nš from 2015. many values of $n$ make the expression $n"3 3"n divisible by 5?"
]
Input
Question: {question)
Answer: (answer)
Output:
16

---

Rollout Case
Share-GRPO allows MLLMs to explore diverse reasoning trajectories from the expanded question space and shares the explored reasoning trajectories. Below are two cases from the Share-GRPO algorithm rollout process.
For Rollout Case 1-1, I, it can be observed that the original question is answered incorrectly. If MLLMs consistently fails on this question, the advantages in GRPO vanish, preventing effective learning. However, through semantic-consistett transformations in Share-GRPO, new variants such as 1o.1-2 1-2 and No.1-3 yield correct reasoning paths. These successful trajectories can then be shared back with the original question, allowing the model to recover meaningful optimization signals.
Case No.2-1, can be observed that the original question is answered correctly. Assuming all responses its rewritten variants are also correct, GRPO will encounter only successful trajectories, leading to sparse rewards and advantage vanishing, By introducing input transformations through Share-GRPO, variants of Case No.2-1 (i.e., No.2-2 and No.2-3) result in incorrect answers. This facilitates learning from both correct and incorrect reasoning trajectories, enhancing the model's generalization and robustness question shifts.
For Rollout
Rollout case No.1-1 1-1
Original Question with Original Image shown figure. the radius of circle 60 5.0, triangle ABC the inseribed triangle &0 passing through point with CD AB. ICD u, AC 6.0, and the length of B is 5.0, what is the value of the unknown variable ui! Choices:
2.2.0
B: 21.0
C: 14.0
D: 3.0
<think>
solve for the unknown variable CD, need properties of triangle and the circle. Specifically, the fact that the altitude from the right angle to the hypotemuse in right triangle creates two smaller triangles that are similar to the original triangle and other.
Identily given
in problem:
O is circumeircle with radius
ABC inseribed right triangle CD
CD u. AC 6, and BC =
Since CD altitde C' AB, AABC right triangle right angle
Use the Pythagorean thearem find the length of AB:
AC BC
V6I
The length of the allitude CD in a right triangle can be found using the urea the triangle two ways:
Area using and height:
Area
z x AC BC
x0xxx===xxxx
using hypotenuse altitude:
AB
/61
2
Equate the expressions the
30
30
15 66
66
consider possibility simpler solution. Notice the problem might be simplified if we consider the geometric properties of the circle and the triangle. but the problem sctup suggosts that the answer should be simpler rational number. Given choices, correct value u is likely one the provided options. By chocking the options, find that the correct value 2. This be verified by checking the geometric properties and the area calculation. Therefore, </think>>
Since answer given choices,
unknown variable
[Incorrect]
(boxed[A]
17

---

Rollout case No.
Rewrite Question with Original
the figure, circle @Q has radius 5.0, and triangle A BC is insorihed in the cirele, passing through point C: with the altitude CD AB. Given that CD u, AC 6.0, and length BC what the value u.? Choices:
2.0
C: 14.0
D: 3.0
<think> We are given a circle with radius = 5.0, and un inseribed triangle AABC with the following properties:
perpendiclaar CD longth altitue from AB).
AC BC
We need find the value
The area of AABC be calculated in two ways:
1. Using the base A B and the altitude C.D:
Area x X XDD AB
Using AC, BC, included angle LACB:
Area x AC x BC sin Lin(L)
Since both expressions represent the of the triangle, equate them:
2 AB AC BC LACB).
Simplifying. get:
AB x v AC x BC x sin(/ACR)
Since AABC inseribed circle with radius R 5.0, we can use extended of sines:
AAB sin (iC(ZACB) 2R.
Thus,
AB = 2ft sinsL = 100si(<ACCC))
Substitute LAsin(LACB) into area equation:
(IAsin(ZACB))
LCCB)
Since sin( LACBB) o divide both sides siní LACBB:
10a AC BC.
Substitute AC 6.0 and BC = 5.0:
5.0
Solving
10 3.0.
To ensure the solution is correct, check the consistency with the given information. The value 3.0 satisties the relationship between the area the triangle and the given side lengths. Therefore, calculation consistent. </thinic>
(boxed[D)
[Correct]
Rollout case No.1-3
Rewrite Question with Rewrite Image
image rotated by degrees. mentally rolate of circlo OO 5.0. and triangle 2 BC is the inscribed trianglo of GO passing through point C' with C.D 1. A.B. If C.D AC G.O, and the length BC 5.0, what the value the unknown variable u?
problem. figure,
18

---

Choiees:
A: 2.0
B: 21.0
C: 14.0
<think>
Understand the Geometry
The circle has radius of 5, so circamradius Rt 5.
AABC inscribed cirele, meaning all vertices and lie on the circumference.
CD 1. ABB,
perpendicular
means altitude from AB.
Use the Propemies of the Circumeirele
The area of ACC can be calculated in two ways:
(I) Using the formula for the area of triangle with base and height:
Area
x AB AB
Using formula
triangle inscribed in circle:
abc
Area
1R
a, Identify the Sides the Triangle From the problem, know:
triangle, circumradius.
AC b 6.0, BC
5.0,
Let AB
4. Use the Area Formula for the Inseribed Triangle
abc
5-6.0
30c
3e
Area
4R
4.5
20
2
5. Express the Area Using the Altitude
Area
AB CD
xxx
2
6. Fiuuate the Two Expressions for the Area
3c
x c x
2
7. Solve for Multiply both sides 2 to eliminate the fraction:
3c c.
Divide both sides by (assuming of O):
</thinks>
(hoxed(D)
[Correct]
Rollout case No.2-1
Original Question with Original Image
As shown in the ligure, ABC'D cyclic quadrilateral, and E is point on the extension of ADD. Ir LB = 60% then LEDC equals.
<think>
cyclic quadrilateral, the opposite angles are supplomentary, le.., they up to 180", Given 60%, to
ZEDC.
19

---

Since ABCD eyclic quadrilatraal 2B and LD are suppeementary.
LD LB
LD + 60" 180"
LD 120"
the quadrilateral AEDC, 2D and LEDC supplementary with LAED (which LAEB because it's exterior angle LAED);
/EDC /D 180"
LEDC 120" 180
LEDC 60
Therefore, LEDC = 60".
</think> 1boxed (60))
[Correct]
Rollout case No.2-2
Rewrite Question with Original Image Given that ABCD eyclic quadrilatral und point lies on exlension of AD, if ZB 60"". determine the value /EDC.
<think>
solve the problem, the properties cyclic quadrilaterals and angles around point. Let's break down step by step. Identify given
BCD eyclic quadrilateral.
L BB 60"
E point on the extension of ADD.
We need to find LEDC.
Use the property eyclic quadrilaterals: cyelic quadrilateral, the opposite angles up 180" Therefore,
LB + LD 180
3. Substitute the given value of LB:
LB = 60 = 60
LD = 180 = LD = 180" -0 1200
Determine LEDC:
Since E the sum of the two opposite interior angles A BDC.
extension AD. LEDC external ABDC The exteral LE LEDC' is
Identify interior angles of 2 ADDC:
/ BCD internl
the ryclic quadrilatrral ABCD
Since LBCD is part the cyclic quadrilateral, it is supplementary to LB,
LBCD = 180 LB 180 - 60" 120"
Caleulate LEDC:
ZEDC /B BCD = 60 + 60 120
Therefore, the value of LEDO 120". </think> sboxed 120%)
[Incorrect]
Rollout case No.2-3
Question with Original Image
This image has been rotated by 90 dogrees. Please mentally rotate back and solve the problem. Given that ABCD is eyclic quadrilateral and point lies on the extension of ADD <B 60%, determine the value LEDC.
20

---

<think>
the problem, to use properties of eyclic quadrilaterals the angles in eyclic quadrilateral, Let's go through the reasoning step step.
1. Identify the given information:
ABCD eyelic quadrilateral
LBB 60".
E'
extension of AD.
2. Use the property cyclic quadrilaterals:
cyclic qundrilateral, the opposite angles sum to 180". Therefore,
+
Calculate LD:
LD = 180 60 120
4. Find LEDC:
LEDC angle ABC.. angle triangle equal
non adjaceni angles.
Therefore,
LEDC =
Determine LCBD:
In ABCD, LBCD the / because ABCD cyclic quadrilateral and
LBCD LB 60"
Therefore,
LCBD 180 LB LBCD 180" 60"
60" 60"
Calculate LEDC:
LEDC /B /CBD 60 60
120
Therefore, the measure of LEDC 120".
</think> (boxed 120")
[[ncorrect]
G Limitations
Share- Haire-GRPO is an etrly-stage exploratory work that introduces the concept of information sharing into MILMM reinforcement learning to mitigate sparse reward and advantage vanishing issues, when effectively stimulating the reasoning capabilities of multimodal large language models. Despite the improvements in reasoning performance brought by Share-GRPO, our model, like other reasoning- oriented MILMM, still exhibits hallucinations, particularly in intermediate reasoning steps. In addition, our offline textual semantic transformations may occasionally generate low-quality questions by GPT-4o, such incomplete outputs. To address this, we need to use rules to filter out such low-quality samples during preprocessing.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide our contributions and scope both in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Yes, we provide all necessary information to reproduce the experimental
results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification; We will provide the code in supplementary material, and our code and data will be publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e-g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips //nips.cc/pil wwcppbblccgguiicuceccccccoe.oopp
http///oooSuSiiss..ee.eee Stt.Subbiiiioooiiciii...o.i for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details are described in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[NA]
Justification:
The paper does not include these experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g ge-.neggtive error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this information in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research follows the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes. All assets are properly credited and used under their respective licenses.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will provide it in supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
27

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs any important, original, non- standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
28


Reviews:
[{"review_id": "QTz8zIlMMa", "reviewer": "Reviewer_Amky", "summary": "This work proposes Share-GRPO, a RL methodology for VLLM that expands the input question space by (1) use LLM to paraphrase the textual questions (2) transform the input image by pre-defined operations (e.g., rotation, noise injection). The author argues that the proposed transformation techniques introduce diversity and elicit diversified outcome rewards, eventually addressing the sparsified reward and advantage vanishing issues in GRPO. Experimental results on multimodal math and understanding tasks show the effectiveness of Share-GRPO over GRPO.", "questions": "See also weakness section for full details.\nQ1. In principle, these transformations should preserve task difficulty, so it is unclear why reward signals vary across variants. Can you add more deeper analyses (e.g., probing activation patterns or qualitative error cases) to strengthen the claim. This will impact my final scores significantly. \n\nQ2. Can you also report the full experiment results with Qwen 2.5-VL + GRPO in Table1? This is very important to justify the effectiveness of the proposed Share-GRPO.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Clarity and readability – The manuscript is well-structured and easy to follow, allowing readers to quickly grasp the core ideas and experimental setup.\n2. Conceptual simplicity with practical impact – The proposed synthetic-variant expansion is a straightforward yet effective alternative to prior sampling- or selection-based approaches.\n\nWeaknesses & Suggestions: \n\n1. Insufficient analysis of the transformation’s effect on reasoning\nThe paper lacks a theoretical argument—or at least a more extensive empirical study—showing that the proposed transformations genuinely elicit distinct reasoning behaviors and mitigate GRPO’s sparsified-reward and advantage-vanishing issues.\nIn principle, these transformations should preserve task difficulty, so it is unclear why reward signals vary across variants. Adding deeper analyses (e.g., probing activation patterns or qualitative error cases) would strengthen the claim.\nTable 4 shows nearly identical scores for 2–4 question variants, further calling into question the diversity effect.\n\n2. Expansion-technique details and validation:\nSince paraphrasing relies on GPT-4o prompts, a validation step is needed to guarantee semantic equivalence; the limitation is acknowledged but not addressed experimentally.\nThe chosen image transformations are only listed, not motivated. Please justify each operation (e.g., rotation, noise) and discuss how it differs from standard data-augmentation baselines.\n\n3. Baseline coverage: \nA direct comparison between vanilla GRPO and Share-GRPO on the identical backbone (e.g., Qwen 2.5-VL + GRPO vs. Qwen 2.5-VL + Share-GRPO) is essential and should appear in Table 1.\nTables 2–3 cover only a subset of benchmarks. Report GRPO results on all datasets to provide a complete effectiveness picture.", "was_revised": true, "final_justification": "Share-GRPO is an interesting work as it mitigates the sparse reward issue by paraphrasing input questions and elicit different rewards for question variants. The results are positive and are better than GRPO. \n\nThe origin paper doesn't include direct baseline to justify improvements but the authors provided during rebuttal and hence I am willing to raise my score to 4. \n\nThe biggest limitations of this paper to me are \n(1) lack of in-depth discussion of the transformation techniques and their impacts on reasoning: \nThe operations authors selected to transform the input questions are not well motivated. ShareGRPO is using this transformation for reasoning reward diversification while similar techniques were primarily used for data augmentation during training. Although the authors should specify how they verify the transformation results. They include some descriptions in rebuttal but I expect more from a really strong paper. \n\n(2) no improvements between question variants N=2 to 4. \nThis raises the question of whether the transformation techniques authors selected truly introduce enough variance. How do they verify the transformed inputs and how do they perturb when increasing the variant numbers. \nOther reviewers also share the same concern. The author tried to address this by saying the reward spasification issue is mitigated when N=2. But more evidence should be provided. \n\nOverall, this paper is positive but still have some limitations.", "rebuttal": "Dear reviewer Amky,\n\nWe appreciate the reviewer's professional, detailed, and valuable comments and suggestions.\nWe sincerely thank you for recognizing our contributions regarding the clarity and readability of the manuscript, the conceptual simplicity and practical impact of our approach, and the effectiveness of Share-GRPO over GRPO.\nBelow, we are going to respond to your concerns one by one.\n\n---\n\n**W1: Analysis of the transformation’s effect on reasoning**\n\nFor textual transformations, the overall difficulty of the problem does not change significantly.\nHowever, it is important to highlight that, based on our experiments and supported by previous studies [1,2], current MLLMs have data leakage and memorization issues.\nSpecifically, MLLMs may memorize the correct answers, which allows them to produce the correct final answer even when their intermediate reasoning steps are incorrect, leading to reward hacking issues in GRPO (i.e., cases where models answer all correctly in rollout without genuine reasoning).\nTherefore, by applying textual transformations to introduce new question variants, ShareGRPO can alleviate the sparse reward issue caused by memorization, as responses are generated for the questions the model has not memorized.\nThis enables MLLMs to better learn to reason during the ShareGRPO training.\n\nAdditionally, MLLMs are sensitive to prompts [3,4]. Even semantically equivalent questions with different formulations can elicit different responses from the model.\nWe leverage this property by generating semantically consistent question variants, which encourages more diverse reasoning responses.\nThese diverse responses are then shared and used to update all question variants.\nThis diversity (which is more likely to include both positive and negative samples) helps to address the advantage vanishing problem.\nAt the same time, training with ShareGRPO enhances the model’s robustness to prompt variations.\n\nFor multi-modal image transformation, we believe that adding noise or rotating the image increases the difficulty of the problem. This helps reduce the proportion of all-correct cases during training and enhances the model’s robustness to visual variations.\n\n\n> Table 4 shows nearly identical scores for 2–4 question variants, further calling into question the diversity effect.\n\nBased on Table 3 and Table 4, we observe that increasing the number of questions from 1 to 2 raises the accuracy from 72.8% to 75.4%, demonstrating the effectiveness of ShareGRPO compared to GRPO.\nThe improvement is limited when increasing from 2 to 4 questions, as with two variants, the MLLM is already able to generate both positive and negative rollouts, effectively mitigating the sparse reward issue. As a result, further gains are less pronounced. Therefore, to balance efficiency and performance, we use two question variants as the default setting.\nIn summary, the significant performance improvement when increasing the number of question variants from 1 to 2 demonstrates that our method effectively mitigates the advantage vanishing problem, highlighting the effectiveness of ShareGRPO.\n\n[1] Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination\n\n[2] MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI\n\n[3] How Susceptible are LLMs to Influence in Prompts?\n\n[4] Large Language Models Sensitivity to The Order of Options in\nMultiple-Choice Questions\n\n---\n\n**Q1: Qualitative error cases about transformations**\n\nThis is a good question.\nWe agree that providing deeper analyses of how question transformations affect reward signals and model reasoning would strengthen the claim.\nWe would like to kindly clarify that we provide some qualitative cases in Appendix E. Due to the technical restrictions from NeurIPS, we used existing images rather than new ones to provide qualitative explanations.\nFor Rollout Case No.1-1 in Appendix E, it can be observed that the original question is answered incorrectly. If MLLMs consistently fails on this question, the advantages in GRPO vanish, preventing effective learning.\nHowever, through semantic-consistent transformations in ShareGRPO, new variants such as No.1-2 and No.1-3 yield correct reasoning paths. These successful trajectories can then be shared back with the original question, allowing the model to recover meaningful optimization signals.\nFor Rollout Case No.2-1, it can be observed that the original question is answered correctly. Assuming all responses to its rewritten variants are also correct, GRPO will encounter only successful trajectories, leading to sparse rewards and advantage vanishing.\nBy introducing input transformations through ShareGRPO, variants of Case No.2-1 (i.e., No.2-2 and No.2-3) result in incorrect answers. This facilitates learning from both correct and incorrect reasoning trajectories, enhancing the model’s generalization and robustness to question shifts.\n\nWe hope these analyses and explanations could address your concerns. We will include them in the revised manuscript. Thank you!\n\n---\n\n**W2: Expansion-technique details and validation of transformations**\n\n> Since paraphrasing relies on GPT-4o prompts, a validation step is needed to guarantee semantic equivalence;\n\nThank you for raising this important concern.\nIn fact, we conducted extensive experiments to validate the quality of the generated questions.\n1) We experimented with various prompts to ensure stable and high-quality outputs.\n2) We developed rule-based scripts to automatically filter out incomplete generations, ensuring that the retained sentences are complete and that the options remain consistent with the original questions.\n3) We manually reviewed a large number of cases to further ensure the quality and correctness of the outputs.\n4) We also conducted GRPO training using only one newly generated question variant per sample. The training process was stable and achieved a slight performance improvement as shown below, which empirically demonstrates the quality of the generated questions.\n\n| Models    | one original question | one new question  |\n|-----------|-----------|--------|\n| GRPO      | 72.8      | 72.9   | \n\n> Please justify each operation (e.g., rotation, noise)\n\nwe carefully select transformations (e.g., rotation, noise injection) that preserve critical visual cues necessary for reasoning, and avoid transformations (e.g., cropping, color distortion) that may disrupt key information.\nFirst, we intentionally avoided transformations that could affect critical information in the images. For example, cropping might remove a key edge in a math problem, and color transformations could impact questions that rely on color-related information.\nNext, we selected transformations such as rotation and noise injection, which preserve the critical visual cues necessary for reasoning. Together with our transformation-specific prompts, these changes do not affect the correctness of the problem’s answer.\n\n> discuss how it differs from standard data-augmentation baselines.\n\nI believe there is some misunderstanding here.\nThe innovation of this paper does not lie in introducing data augmentation. The main contribution is using carefully designed semantic transformations to generate shared and diverse responses, and then self/cross-updating the question variants with these shared responses.\nSpecifically, GRPO calculates the conditional probability of each generated response with respect to the original question and updates the token probabilities accordingly, i.e., $\\pi_{\\theta}(o_i^{Q} \\mid Q)$. The GRPO + data augmentation approach works similarly, except it replaces the original question with the augmented one.\nIn contrast, ShareGRPO generates a set of shared responses for the original question and its semantically consistent variants. These responses are then used for both self-updating ($\\pi_{\\theta}(o_i^{Q_j} \\mid Q_k)$, where $j = k$) and cross-updating ($\\pi_{\\theta}(o_i^{Q_j} \\mid Q_k)$, where $j \\neq k$), based on their respective conditional probabilities.\n\nAdditionally, we compare the results of ShareGRPO and GRPO with data augmentation, as shown below. The results indicate that the performance gains of ShareGRPO are not due to data augmentation, but rather from mitigating advantage vanishing through shared responses.\n\n| Methods    | GRPO | GRPO + image transformations | GRPO + language transformations | GRPO + image/language transformations | ShareGRPO |\n|-----------|-----------|--------|-------|------|------|\n| MathVista | 72.8      | 73.4   | 73.1  | 73.6 | 75.4 |\n\n---\n\n**W3 & Q2: Compare ShareGRPO with full experiment results with Qwen 2.5-VL + GRPO**\n\nThank you for your suggestion.\nWe have now provided the full experimental results comparing Qwen 2.5-VL + GRPO and Qwen 2.5-VL + ShareGRPO across all benchmarks. The results, shown in Table below, demonstrate that ShareGRPO achieves consistent improvements over GRPO, with an average performance gain of 1.9\\% across all benchmarks.\n\n| Models    | MathVista | MMStar | MMMU  | MathVerse | MathVision | AI2D | AVG  |\n|-----------|-----------|--------|-------|------------|-------------|------|------|\n| GRPO      | 72.8      | 65.4   | 56.4  | 50.7       | 26.7        | 84.0 | 59.3 |\n| ShareGRPO | 75.4      | 67.0   | 58.1  | 52.8       | 29.5        | 84.5 | 61.2 |\n\n---\n\nMany thanks for your professional, detailed, and valuable reviews!\nWe have done our best to address each of your concerns and hope our response can resolve them. Please let us know if you have any other questions. If you feel all questions have been addressed, we would greatly appreciate it if you could kindly consider re-rating our work. We are looking forward to hearing from you!"}, {"review_id": "fdKUwLxXFq", "reviewer": "Reviewer_v7hu", "summary": "This paper proposes a Share-GRPO mechanism to train multimodal LLMs. Share-GRPO first augments the original problem into multiple variants, then employs the policy model to generate multiple candidate reasoning responses for all augmented variants. When updating the policy, Share-GRPO introduces an advantage-sharing mechanism to provide a rich reward signal to the policy model, alleviating the reward sparsity issues in standard GRPO. Extensive experiments demonstrate the effectiveness of the proposed Share-GRPO.", "questions": "1. In Equation.(7), what does $\\pi_\\theta(o_i^{Q_j}|Q_k)$ mean? The notation $o_i^{Q_j}$ is already defined as \"response $o_i$ generated by the policy model conditioned on problem variant $Q_j$.\" However, the conditional notation $(\\cdot | Q_k)$ is confusing to me. I would appreciate it if the authors could clarify this further.\n2. The proposed Share-GRPO framework also appears to be seamlessly applicable to the text-only reasoning domain. I wonder whether the authors have conducted experiments using text-only LLMs on challenging mathematical benchmarks such as AIME 2024, GPQA, etc.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea of Share-GRPO, which augmenting the original problem to provide rich reward signal to alleviate reward sparsing issue, is convincing and reasonable.\n2. The experiments presented in the paper are comprehensive and solid.\n3. The paper is well-structured and mostly clearly written.\n\nWeaknesses:\n1. As presented in Limitations, the data augmentation done by generative models such as GPT-4o may introduce noise into augmented problems.\n2. In Figure.1(b), although the valid advantage ratio curve of Share-GRPO is slightly higher than the standard GRPO's, their trends are almost the same, and their values are only slightly different from about step 20, finally becoming almost indistinguishable by step 70. I don't think Share-GRPO truly solve the advantage vanishing issue based on this figure.\n3. Ablation studies and experiments in Tables 4 and 5 are only conducted on one benchmark \"MathVista\". It would be more convincing if the authors provide more results such as the average performance in Table.1.\n4. In Equation.(5) and (6), the subscript \"k\" does not appear in the RHS of the equation. Its meaning here is unclear to me.\n5. In Table.4, Share-GRPO achieves suprior performance only when setting $m=2$, which means only 1 augmentation is conducted for each problem, with the other being the original problem in my understanding. I have 2 concerns about this observation:\n\n(a). In Section 3.2.1, the authors introduce several augmentation approaches. I wonder which one was ultimately chosen for the experiments. If it was selected randomly, could the reproducibility of Share-GRPO be weakened, since different augmentations may heavily affect the data samples?\n\n(b). Why does increasing *m* from 1 to 2 lead to a significant performance gain (72.8 → 75.4 on MathVista), while increasing *m* from 2 to 4 yields only a minimal improvement (75.4 → 75.9 on MathVista)? I’m interested in this discrepancy, but the authors do not discuss it.", "was_revised": true, "final_justification": "The authors’ rebuttal has effectively addressed my concerns regarding training robustness and the advantage vanishing issue in Share-GRPO, so I have decided to raise my score. However, due to some intrinsic limitations of Share-GRPO, such as the additional computational cost introduced by augmentation, I prefer to give a score of 4.5, which corresponds to a weak accept by NeurIPS 2024 standards.", "rebuttal": "Dear reviewer v7hu,\n\nMany thanks for your valuable comments and questions, which help us a lot to improve our work.\nWe are glad that you find ShareGRPO’s richer reward signals convincing, and that you consider our experiments comprehensive and the paper well-structured.\nWe are going to try our best to address your questions one by one as follows.\n\n---\n\n**W1: Data quality concerns regarding GPT-4o-based problem augmentation**\n\nThank you for raising this concern.\nWe have taken several measures to ensure the quality of the augmented problems, as detailed below, and we will include these in the revised paper:\n1) We experimented with various prompts to ensure stable and high-quality outputs.\n2) We developed rule-based scripts to automatically filter out incomplete generations, ensuring that the retained sentences are complete and that the options remain consistent with the original questions.\n3) We manually reviewed a large number of cases to further ensure the quality and correctness of the outputs.\n4) We also conducted GRPO training using only one newly generated question variant. The training process was stable and achieved a slight performance improvement as shown below, which empirically demonstrates the quality of the generated questions.\n\n| Models    | one original question | one new question  |\n|-----------|-----------|--------|\n| GRPO      | 72.8      | 72.9   | \n\nWe will include these discussion in the revision.\n\n---\n\n**W2: Discussion about Figure.1(b) and advantage vanishing issue**\n\nThank you for raising this concern.\nWe would like to kindly clarify that there are many factors that lead to the advantage vanishing problem. For example, (1) Memorization and reward hacking: MLLMs may memorize the final answer due to data leakage and can produce the correct final answer even when intermediate reasoning steps are incorrect. Please refer to our response to the reviewer Amky's W1 for further details about memorization issues; (2) Trapped in low quality questions: For certain low-quality questions, the MLLM is unable to produce correct answers, leading to all rollouts being incorrect; and (3) Genuine reasoning ability: The model may have truly learned to solve the problem through reasoning, leading to consistently correct answers.\n\nShareGRPO is primarily designed to address issues (1) and (2) by generating shared and diverse responses through semantically consistent transformations. These responses are then used for both self-updating ($\\pi_{\\theta}(o_i^{Q_j} \\mid Q_k)$, where $j = k$) and cross-updating ($\\pi_{\\theta}(o_i^{Q_j} \\mid Q_k)$, where $j \\neq k$), based on their respective conditional probabilities.\n\nAs training progresses, the reasoning ability of MLLMs steadily improves, enabling them to genuinely solve problems, even when presented with semantically varied questions, and to develop true reasoning skills.\nThis inevitably leads to more cases where the model answers all variants correctly, thus resulting in training convergence and a decrease in the valid advantage ratio.\nTo address issue (3), dynamic sampling from DAPO [1] is specifically designed for such scenarios. As shown in Table 3, our results demonstrate the complementarity and compatibility of ShareGRPO with dynamic sampling.\n\n[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale\n\n---\n\n**W3: Average performance in Tab.4 and Tab.5**\n\nThis is a good question!\nWe have provided the average performance results below. We believe including these results makes our findings more convincing. Thank you!\n\n| Models    | MathVista | MMStar | MMMU  | MathVerse | MathVision | AI2D | AVG  |\n|-----------|-----------|--------|-------|------------|-------------|------|------|\n| GRPO      | 72.8      | 65.4   | 56.4  | 50.7       | 26.7        | 84.0 | 59.3 |\n| ShareGRPO | 75.4      | 67.0   | 58.1  | 52.8       | 29.5        | 84.5 | 61.2 |\n\n\n| Method     | Question | average |\n|------------|----------|---------|\n| GRPO       | 1        | 59.3    |\n| ShareGRPO  | 2        | 61.2    |\n| ShareGRPO  | 3        | 61.4    |\n| ShareGRPO  | 4        | 61.8    |\n\n\n| Method     | Sampling | MathVista |\n|------------|----------|-----------|\n| GRPO       | 6        | 58.7      |\n| GRPO       | 12       | 59.3      |\n| GRPO       | 24       | 59.7      |\n| ShareGRPO  | (3+3)    | 60.7      |\n| ShareGRPO  | (6+6)    | 61.2      |\n\n---\n\n**W4: Clarification on the notation in Eq.5 and Eq.6**\n\nThank you for pointing out the confusion regarding the subscript \"k\" in Eq. (5) and (6).\nWe acknowledge that the notation could be more clearly defined, and we will revise the equations and provide additional explanations in the revised version to ensure clarity.\n**Due to character limitations**, we have provided a detailed explanation in our response to **Reviewer EzA1’s W2**. Please kindly refer to that section for further details.\n\n---\n\n**W5 (a): Clarification on the reproducibility of ShareGRPO**\n\nThank you for raising this concern.\nWe fixed the random seed during training and conducted five independent runs.\nThe results show that the performance fluctuation is minor. We believe this level of variation is acceptable, as similar fluctuations are also observed in GRPO, due to the inherent randomness in reinforcement learning.\n\n| Methods                   | GRPO           | ShareGRPO       |\n|---------------------------|----------------|------------------|\n| Error bars on MathVista   | 72.8 ± 0.3     | 75.4 ± 0.39      |\n\n\n---\n\n**W5 (b): Explanations on the Performance Gains with Increasing the Number of Question variants $m$**\n\nThank you for raising this concern!\nBased on Table 3 and Table 4, we observe that increasing the number of question variants from $m=1$ to $m=2$ leads to a significant performance improvement (72.8\\% → 75.4\\% on MathVista).\nThis improvement is primarily due to our use of semantically consistent transformations and shared response optimizations, which effectively address the advantage vanishing issue.\nHowever, the gain is smaller when increasing $m$ from 2 to 4. This is because, with two variants, the MLLM is already able to generate both positive and negative rollouts, which sufficiently addresses the sparse reward issue. As a result, further increasing $m$ provides diminishing returns. Therefore, to balance efficiency and performance, we use two question variants as the default setting.\n\n---\n\n**Q1: Explanation of Eq.7**\n\nThank you for your comment!\nWe first clarify the meaning of $\\pi_{\\theta}(\\cdot \\mid Q)$ in GRPO.\nIn GRPO, for a given question $Q$, we first roll out $n$ responses and, for each response, compute both its advantage $A$ and the conditional probability $\\pi_{\\theta}(\\cdot \\mid Q)$ for each token in the response. Here, $\\pi_{\\theta}(\\cdot \\mid Q)$ denotes the parameterized policy (with parameters $\\theta$) that outputs a probability distribution over possible tokens conditioned on $Q$.\nIf the advantage is positive, the probability of the corresponding tokens is increased; if the advantage is negative, the probability of these tokens is suppressed.\n\n> In Equation.(7), what does $\\pi_{\\theta}({o_i^{Q_j}}  \\mid {Q_k})$ mean?\n\nIn ShareGRPO, we extend this framework by introducing multiple question variants [$\\{Q_1, Q_2, \\dots, Q_m\\}$]. For each variant $Q_j$, we generate a set of responses $\\mathbf{O}=[[\\{\\{o_{1}^{Q_1}, ..., o_{n}^{Q_1}\\}], ..., [\\{o_{1}^{Q_m}, ... o_{n}^{Q_m}\\}\\}]]$. Each response $o_i^{Q_j}$, where j belongs to m, is thus explicitly generated by the policy model conditioned on $Q_j$.\n\nThe notation $\\pi_{\\theta}(o_i^{Q_j} \\mid Q_k)$ in Eq. (7) represents the probability (under the current policy) of generating the entire response $o_i^{Q_j}$, but conditioned on a possibly different question variant $Q_k$.\n\n1) When $j = k$, this is the “self-updating” case, where we update the parameters using the response under its own generating context.\n\n2) When $j \\neq k$, this is the “cross-updating” case, where we reuse a response generated from one question variant and update its probability under a different variant.\n\nThis design allows ShareGRPO to perform cross-variant updates, promoting information sharing and robustness across semantically consistent but syntactically different variants.\n\nWe appreciate your feedback and will further clarify this notation in the revised manuscript.\n\n---\n\n**Q2: Apply ShareGRPO to text-only LLM**\n\nThank you! This is certainly worth exploring further.\nWe didn't conduct experiments with text-only LLMs before, as Share-GRPO was originally designed for MLLMs, incorporating features such as multi-modal image transformations.\nHowever, we agree that extending Share-GRPO to text-only reasoning tasks is a promising direction for future work.\nHence, we conducted a preliminary exploration by applying ShareGRPO to pure text tasks, and observed consistent performance improvements in this setting as well.\nWe trained the Qwen2.5-7B-Instruct model using both ShareGRPO (w/o multi-modal transformations) and GRPO methods, with 5K samples from the Light-R1 dataset.\nThe results show that ShareGRPO achieves better results than GRPO.\n\n\n| Methods                          | AIME'24 | GPQA |\n|----------------------------------|---------|------|\n| Qwen2.5-7B-Instruct (Baseline)   | 12.0    | 36.4 |\n| w/ GRPO                          | 37.1    | 40.3 |\n| w/ ShareGRPO                     | 40.3    | 42.9 |\n\n\n--- \n\nLastly, thank you very much for your constructive feedback and suggestions. We will incorporate these points into the revised paper. If you have any further questions, please feel free to let us know. We will be available throughout the rebuttal period."}, {"review_id": "PMphE4nG1Q", "reviewer": "Reviewer_EzA1", "summary": "The paper proposes a variant of GRPO applied to multimodal LLMs. The core of the method is to augment each question with different transformations to enrich the training data. The GRPO algorithm is then modified to incorporate two levels of normalization: one across all variants of the same question and another within each individual variant. The stated motivation is that this two-tiered normalization scheme allows for more precise credit assignment.", "questions": "- I would be interested to know the performance without the global shared advantage and with only the local shared advantage (but with the same data augmentation). In such a case, the method would revert to the normal GRPO, where the advantage is only normalized across the same question (specifically, the same variant of the same question). I suspect that simply increasing the number of samples per question variant would be sufficient for enhancing credit assignment, making the global advantage normalization unnecessary.\n- Ideally, a learning curve should also be shown in the paper to understand whether the optimization has converged, especially for ablation runs.", "limitations": "The authors addressed several limitations common in multimodal LLMs.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The experiments show promising results, with the proposed model performing better than the baselines. The demonstrated scalability to a 32B model is also a notable strength.\n- The method is straightforward and appears easy to implement.\n\nWeaknesses:\n\n- The major weakness is the limited novelty. In essence, it is a data augmentation technique. Both the \"offline textual SCT\" and \"online multimodal SCT\" could be performed in a purely offline manner; one could simply apply the online transformations prior to training to generate a large number of questions. The shared advantage computation is a straightforward combination resulting from two different group definitions and is closer to an engineering detail than a separate algorithm justifying a new name.\n- The notation could be clearer and more rigorous. For example, the definitions of i, j, and k in A_{i,j,k} of equation (5) are not provided (though they can be inferred). The std function over a set of sets in equation (5) is not formally defined. In equation (6), the right-hand side should include a notation for k to clarify which original question variant Q_j belongs to. This leads to the more confusing equation in (7), which could be written more compactly without using cases.", "was_revised": true, "final_justification": "The author addressed my concerns regarding computational cost and the similarities to GRPO, particularly clarifying the pairing of answers to all variants of the question during policy updates. As such, I have slightly increased my score.\n\nNonetheless, some issues remain:\n\n1. Statistical significance: The author claims that their method is significant, but no statistical significance test is provided. Their main argument is that a common method A gives +x%, method B gives +y%, and y% is n times x%, implying that method B’s improvement is significant. However, this is not a valid statistical measure. Standard error and a t-test are required to establish statistical significance.\n2. Limited novelty: The method is essentially a combination of data augmentation, a variant of group definition, and pairing of off-policy responses. In my opinion, none of these individual components are novel. A deeper analysis of how those off-policy pairs contribute to performance could yield novel insights, but the current submission falls short in this regard.", "rebuttal": "Dear reviewer EzA1,\n\nWe appreciate your valuable comments and suggestions.\nWe are glad to hear that you find our experimental results promising and our method straightforward and easy to implement.\nBelow, we are going to address your concerns one by one to the best of our ability.\n\n---\n\n**W1: Clarification on our motivation and contribution**\n\nMany thanks for raising this concern.\n\n> The major weakness is the limited novelty. In essence, it is a data augmentation technique. Both the \"offline textual SCT\" and \"online multimodal SCT\" could be performed in a purely offline manner; one could simply apply the online transformations prior to training to generate a large number of questions.\n\nWe would like to kindly clarify that the key innovation of ShareGRPO is not simply the use of data transformation. We only use transformations to generate semantically consistent question variants and create shared responses for subsequent joint optimization.\nThe core idea lies in leveraging shared and diverse responses for both self-updating ($\\pi_{\\theta}(o_i^{Q_j} \\mid Q_k)$, where $j = k$) and cross-updating ($\\pi_{\\theta}(o_i^{Q_j} \\mid Q_k)$, where $j \\neq k$), each based on their respective conditional probabilities.\n\nIn addition, we also conducted experiments to compare the results of data augmentation and ShareGRPO. The results indicate that the performance gains of ShareGRPO are not due to data augmentation, but rather to the mitigation of advantage vanishing through shared responses.\n\n| Methods    | GRPO | GRPO + image transformations | GRPO + language transformations | GRPO + image/language transformations | ShareGRPO |\n|-----------|-----------|--------|-------|------|------|\n| MathVista | 72.8      | 73.4   | 73.1  | 73.6 | 75.4 |\n\n> The shared advantage computation is a straightforward combination resulting from two different group definitions and is closer to an engineering detail than a separate algorithm justifying a new name.\n\nWe respectfully clarify that the global shared advantage is also an innovation of this paper, specifically designed to facilitate shared advantage updates. It further enhances both the direction and magnitude of credit assignment to $\\pi_{\\theta}(o_i^{Q_j} \\mid Q_k)$.\nUnlike GRPO, which computes the advantage within the set of responses generated for each individual question, ShareGRPO extends this to a hierarchical advantage calculation that incorporates both global and local advantages. The local advantage calculation differs from that in standard GRPO, while the global advantage refers to computing the advantage across all responses generated from all question variants. This specialized advantage calculation in ShareGRPO leads to improved performance, as shown in the table below.\n\n| Methods    | Local advantage | Global advantage | MathVista\n|-----------|:-----------:|:--------:|:-------:|\n| ShareGRPO | ✔️    |    |  74.1 |\n| ShareGRPO |     | ✔️   | 74.8  |\n| ShareGRPO | ✔️    | ✔️   | 75.4  |\n\nLastly, we summarize the contributions of this work as follows to highlight its uniqueness: We propose Share-GRPO, the first method to introduce information sharing into MLLM reinforcement learning. It shares diverse reasoning trajectories across an expanded question space to alleviate sparse reward and advantage vanishing issues. We further design a hierarchical advantage estimation mechanism that shares reward information both across and within question variants, enabling more accurate and robust estimation. Finally, extensive experiments on six MLLM reasoning benchmarks validate the effectiveness of our approach.\n\n---\n\n**W2: Refine Equations (5), (6), and (7)**\n\nThank you for taking the time to carefully review our work and point out these issues!\nWe agree that the notation could be clearer and more rigorous. We originally added the index $k$ in Equations (5) and (6) because, in Equation (8), we need to distinguish the calculations of $A$ for different policies $\\pi_{\\theta}(\\cdot \\mid Q_k)$. Below, we adopt a different formulation of the equations and provide a detailed explanation of the formulas.\n\nThe indices $k$ and $j$ are primarily used in Equation (8) for the conditional probability $\\pi_{\\theta}(o_i^{Q_j} \\mid Q_k)$. Their purpose is to distinguish between the response $o_i$ generated from $Q_j$ and the question variant $Q_k$ on which the conditional probability is evaluated. This allows us to differentiate between self-updating (when $j = k$) and cross-updating (when $j \\neq k$) scenarios in the computation.\n\nFor $\\pi_{\\theta}(o_i^{Q_j} \\mid Q_k)$, when $k = j$, all responses $o_i$ are generated from $Q_j$ and are used to update the same question $Q_k$. In this case, we apply both global and local advantage estimation, as these responses are directly associated with their original variant. \n\n$$\nA_{i, j}^{\\text{hier}} = A_{i, j}^{\\text{global}} + A_{i, j}^{\\text{local}}\n$$\n\nFor $\\pi_{\\theta}(o_i^{Q_j} \\mid Q_k)$, when $k \\neq j$, the response $o_i$ is generated from $Q_j$ but used to update a different variant $Q_k$. Here, we adopt only the global advantage estimation, since the response is being shared across variants.\n\n$$\nA_{i, j}^{\\text{hier}} = A_{i, j}^{\\text{global}}\n$$\n\n\nFor global advantage computation, we estimate the advantage from a global perspective, where the relative advantage is computed using the rewards obtained from all question variants.\n\n$$\nA_{i, j}^{\\text{global}} = \\frac{R_i^{Q_j} -\\text{mean}\\left(\\{{[\\{r_{1}^{Q_1}, ..., r_{n}^{Q_1}}\\}], ..., [\\{r_{1}^{Q_m}, ..., r_{n}^{Q_m}\\}\\}]\\right)}{\\text{std}\\left(\\{[\\{r_{1}^{Q_1}, ..., r_{n}^{Q_1}\\}], ..., [\\{r_{1}^{Q_m}, ..., r_{n}^{Q_m}\\}\\}]\\right)}.\n$$\n\nFor local advantage computation, we also estimate the advantage at a local level, where the relative advantage is computed within the responses generated from each individual question variant Qj ∈ Q. Specifically, for each question variant Qj, the local advantage is estimated as follows:\n\n$$\nA_{i, j}^{\\text{local}} =\n\\frac{\n    R_i^{Q_j} - \\mathrm{mean}\\left(\\{ r_1^{Q_j}, ..., r_n^{Q_j} \\}\\right)\n}{\n    \\mathrm{std}\\left(\\{ r_1^{Q_j}, ..., r_n^{Q_j} \\}\\right)\n}\n$$\n\nThank you for you suggestions. We have revised and clarified these expressions in the updated manuscript.\n\n----\n\n**Q1: The results without global shared advantage and the clarification of advantage computation**\nThank you for your question.\n\n> I would be interested to know the performance without the global shared advantage and with only the local shared advantage (but with the same data augmentation).\n\nWe provide results without the global shared advantage, which show a slight decrease in performance. This is because the global shared advantage offers higher confidence in credit assignment.\n\n| Methods    | Local advantage | Global advantage | MathVista\n|-----------|:-----------:|:--------:|:-------:|\n| ShareGRPO | ✔️    |    |  74.1 |\n| ShareGRPO |     | ✔️   | 74.8  |\n| ShareGRPO | ✔️    | ✔️   | 75.4  |\n\n> In such a case, the method would revert to the normal GRPO, where the advantage is only normalized across the same question (specifically, the same variant of the same question).\n\nWe believe that there may be some misunderstanding. \nShareGRPO, even when using only local shared advantage and without the global shared advantage, **does not revert to normal GRPO**.\nWe would like to kindly highlight that the essential innovation of ShareGRPO is not simply the introduction of transformation or the use of a global shared advantage.\nThe core idea lies in leveraging shared and diverse responses for both self-updating ($\\pi_{\\theta}(o_i^{Q_j} \\mid Q_k)$, where $j = k$) and cross-updating ($\\pi_{\\theta}(o_i^{Q_j} \\mid Q_k)$, where $j \\neq k$), each based on their respective conditional probabilities.\nThe semantically consistent transformation is primarily a means to create a shared and diverse response pool, while the global shared advantage further enhances both the direction and magnitude of credit assignment to $\\pi_{\\theta}(o_i^{Q_j} \\mid Q_k)$.\n\n\n> I suspect that simply increasing the number of samples per question variant would be sufficient for enhancing credit assignment\n\nWe agree that simply increasing the number of samples per question would enhance credit assignment.\nHowever, as shown in Table 5, simply increasing the number of rollouts leads to more homogeneous responses and quickly reaches a performance ceiling. For example, when the GRPO rollout number increases from 12 to 24, the accuracy on MathVista improves by only 0.2\\%, while significantly increasing computational burden.\nTherefore, it is more effective and efficient to increase the number of question variants and adopt shared response updating as used in ShareGRPO.\n\n**Q2: Inclusion of learning curve**\n\nThank you for your suggestion!\nWe agree that providing learning curves is valuable for analyzing optimization and convergence.\nAccordingly, we have included learning curves in Figure 1(b) of the main paper.\nWhile we would like to provide additional learning curves in the rebuttal, technical restrictions from NeurIPS prevent us from submitting figures at this stage.\nWe will include more learning curves in the revised version, such as reward curve, and response length curve.\n\n---\n\nThank you again for your insightful and professional comment, which made our work more complete and solid!\nIf there are any further questions, please let us know.\nIf you feel all questions have been addressed, we would be grateful if you could kindly consider re-rating our work. Looking forward to hearing back from you!"}, {"review_id": "WLdUkmDhfL", "reviewer": "Reviewer_MP3K", "summary": "This article addresses the problems of reward sparsity and advantage vanishing encountered when applying GRPO to the MLLM domain. According to the article, reward sparsity refers to the difficulty MLLMs have in generating correct answers, which makes it hard for the model to obtain positive training data with high rewards. Advantage vanishing means that the model often produces similar answers to the same question; since these similar answers receive comparable rewards, their relative advantage becomes very small. The article proposes expanding the questions and augmenting the images to enable the model to generate a more diverse set of answers, thereby achieving a better distribution of rewards and advantages, which facilitates model training.", "questions": "Please refer to the weakness section. This is a good paper, and I recommend acceptance. If the authors can address my concerns mentioned in the weakness section, I would be happy to raise my score.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The advantages of this article are obvious.\n\n[+] The article clearly identifies the drawbacks of GRPO, and these issues are indeed important to address.\n\n[+] The proposed method is very effective and shows significant improvement compared to other approaches applying GRPO.\n\nThe shortcomings of the article are minor; I only have a few small questions:\n\n[-] Why is the improvement not obvious on certain benchmarks, e.g., MMMU and AI2D ?\n\n[-] The drawbacks of GRPO identified in the article do not seem to be limited to the multimodal setting. Could the shared-GRPO approach also be applied successfully in the NLP domain?\n\n[-] The approach of augmenting images to enable the model to produce diverse answers is quite similar to the method in the paper [MM2024, oral] \"Self-Supervised Visual Preference Alignment.\" Methods like SeVa, which use DPO for self-training, have shown significant performance gains. Would it be possible to consider comparing GRPO with DPO?", "was_revised": true, "final_justification": "This is a well-written article with a clear problem statement and a clear methodology. After our discussion, I realized that the author's experiments mainly focus on the mathematical domain. In fact, there should be more applications in other domains as well. I suggest conducting further research in more areas to provide a more solid conclusion.", "rebuttal": "Dear reviewer MP3K,\n\nThank you for your professional and valuable feedback.\nWe appreciate your recognition that our paper clearly identifies the drawbacks of GRPO and addresses these important issues. We are also encouraged by your positive assessment of our proposed method’s effectiveness and its significant improvements over other GRPO-based approaches.\nBelow, we address your concerns and questions individually:\n\n---\n\n**W1: Analysis of model improvements on certain benchmark, e.g., MMMU and AI2D**\n\nThank you for your comments.\nOur training data, as well as other open-sourced R1-related datasets, are primarily focused on the mathematics domain.\nTo further evaluate the generalization ability of R1-Share, we considered challenging out-of-domain (OOD) benchmarks like MMMU and AI2D. \nSpecifically, MMMU covers multiple disciplines, and AI2D focuses on science diagrams. Both benchmarks differ significantly from the training data.\nWe compared R1-Share with MM-Eureka, since our training data is a subset of that used for MM-Eureka. On MMMU, existing reasoning methods like MM-Eureka achieve an accuracy of 55.3\\%, whereas R1-Share achieves 58.1\\%, marking a 2.8-point improvement.\nFor AI2D, R1-Share also achieves state-of-the-art performance, with R1-Share-32B reaching 86.2\\% accuracy.\nThese results demonstrate that R1-Share offers superior OOD generalization capabilities compared to GRPO.\n\n| Methods         | MMMU | AI2D |\n|-----------------|------|------|\n| MM-Eureka-7B    | 55.3 | 40.3 |\n| R1-Share-7B     | 58.1 | 42.9 |\n| MM-Eureka-32B   | 64.6 | 85.4 |\n| R1-Share-32B    | 70.1 | 86.2 |\n\n---\n\n**W2: Apply ShareGRPO in NLP domain**\n\nThis is a question well worth exploring.\nWe conducted a preliminary exploration by applying ShareGRPO to LLM, and observed consistent performance improvements in this setting as well.\nWe trained the Qwen2.5-7B-Instruct model using both ShareGRPO (w/o multi-modal transformations) and GRPO methods, with 5K samples from the Light-R1 dataset.\nThe results show that ShareGRPO can also be applied to the NLP domain and achieves better performance than GRPO.\n\n\n| Methods                        | AIME'24 | GPQA |\n|--------------------------------|---------|------|\n| Qwen2.5-7B-Instruct (Baseline) | 12.0    | 36.4 |\n| w/ GRPO                        | 37.1    | 40.3 |\n| w/ ShareGRPO                   | 40.3    | 42.9 |\n\n---\n\n**W3: Comparison between GRPO and DPO**\n\nThank you for your comment.\nSeVa [1] is an insightful and relevant work which uses designed augmentation to the image input to generate negative data for DPO training, and we will cite it in the revised manuscript. GRPO is designed to incentivize long-chain reasoning capabilities without pre-generated reasoning annotations, while DPO training requires reasoning data to be generated in advance, making the training results highly dependent on the quality of the dataset.\n\nTo delve deeper into this question, we include a comparison to further examine the reasoning capabilities incentivized by each method, by constructing DPO data based on the same underlying dataset.\nUsing our 52K dataset, we use Qwen2.5-VL-72B generates long-chain steps to serve as the chosen data for DPO.\nAnd we prompt GPT-4o to generate reject responses using the method from SeVa [1] and STE [2].\nThe results show that GRPO is more effective at incentivizing the model’s reasoning ability.\n\n| Methods | MathVista |\n|---------|-----------|\n|Baseline | 68.2      |\n| DPO     | 70.1      |\n| GRPO    | 72.8      |\n| ShareGRPO   | 75.4  |\n\n\n[1] Self-Supervised Visual Preference Alignment\n\n[2] Self-Taught Evaluators\n\n----\n\nThank you again for your review and for recognizing our work. We are encouraged by your feedback and will do our best to further refine this manuscript.\nIf you feel your questions have been addressed, we would be grateful if you could kindly consider raising your score. Thank you!"}]

Author rebuttal:

Meta-review: The paper proposes a new RL method for MLLMs that mitigates the sparsity reward and advantage vanishing problems. This is achieved by generating semantically consistent variants of a given question and then sharing the diverse reasoning trajectories discovered across all variants during policy updates.

There is a consensus to accept the paper after the rebuttal and discussion period. Reviewers recognize the importance of the problem, the novelty of the method, and good presentation and empirical studies. 

Overall, this is a solid work to the field of MLLM training. Please incorporate the new results and discussions in the final version.
Decision: Accept (poster)