Title: R1-ShareVL: Incentivizing Reasoning Capabilities of Multimodal Large Language Models via Share-GRPO
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Modal Large Language Models

Reinforcement Learning

Reasoning
Abstract: In this work, we aim to incentivize the reasoning ability of Multimodal Large Language Models (MLLMs) via reinforcement learning (RL) and develop an effective approach that mitigates the sparse reward and advantage vanishing issues during RL.  To this end, we propose Share-GRPO, a novel RL approach that tackle these issues by exploring and sharing diverse reasoning trajectories over expanded question space. Specifically, Share-GRPO first expands the question space for a given question via data transformation techniques, and then encourages MLLM to effectively explore diverse reasoning trajectories over the expanded question space and shares the discovered reasoning trajectories across the expanded questions during RL. In addition, Share-GRPO also shares reward information during advantage computation, which estimates solution advantages hierarchically across and within question variants, allowing more accurate estimation of relative advantages and improving the stability of policy training. Extensive evaluations over 6 widely-used reasoning benchmarks showcase the superior performance of our method. Code is available at https://github.com/HJYao00/R1-ShareVL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article addresses the problems of reward sparsity and advantage vanishing encountered when applying GRPO to the MLLM domain. According to the article, reward sparsity refers to the difficulty MLLMs have in generating correct answers, which makes it hard for the model to obtain positive training data with high rewards. Advantage vanishing means that the model often produces similar answers to the same question; since these similar answers receive comparable rewards, their relative advantage becomes very small. The article proposes expanding the questions and augmenting the images to enable the model to generate a more diverse set of answers, thereby achieving a better distribution of rewards and advantages, which facilitates model training.\n\nSTRENGTHS AND WEAKNESSES:\nThe advantages of this article are obvious.\n\n[+] The article clearly identifies the drawbacks of GRPO, and these issues are indeed important to address.\n\n[+] The proposed method is very effective and shows significant improvement compared to other approaches applying GRPO.\n\nThe shortcomings of the article are minor; I only have a few small questions:\n\n[-] Why is the improvement not obvious on certain benchmarks, e.g., MMMU and AI2D ?\n\n[-] The drawbacks of GRPO identified in the article do not seem to be limited to the multimodal setting. Could the shared-GRPO approach also be applied successfully in the NLP domain?\n\n[-] The approach of augmenting images to enable the model to produce diverse answers is quite similar to the method in the paper [MM2024, oral] \"Self-Supervised Visual Preference Alignment.\" Methods like SeVa, which use DPO for self-training, have shown significant performance gains. Would it be possible to consider comparing GRPO with DPO?\n\nQUESTIONS:\nPlease refer to the weakness section. This is a good paper, and I recommend acceptance. If the authors can address my concerns mentioned in the weakness section, I would be happy to raise my score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a variant of GRPO applied to multimodal LLMs. The core of the method is to augment each question with different transformations to enrich the training data. The GRPO algorithm is then modified to incorporate two levels of normalization: one across all variants of the same question and another within each individual variant. The stated motivation is that this two-tiered normalization scheme allows for more precise credit assignment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The experiments show promising results, with the proposed model performing better than the baselines. The demonstrated scalability to a 32B model is also a notable strength.\n- The method is straightforward and appears easy to implement.\n\nWeaknesses:\n\n- The major weakness is the limited novelty. In essence, it is a data augmentation technique. Both the \"offline textual SCT\" and \"online multimodal SCT\" could be performed in a purely offline manner; one could simply apply the online transformations prior to training to generate a large number of questions. The shared advantage computation is a straightforward combination resulting from two different group definitions and is closer to an engineering detail than a separate algorithm justifying a new name.\n- The notation could be clearer and more rigorous. For example, the definitions of i, j, and k in A_{i,j,k} of equation (5) are not provided (though they can be inferred). The std function over a set of sets in equation (5) is not formally defined. In equation (6), the right-hand side should include a notation for k to clarify which original question variant Q_j belongs to. This leads to the more confusing equation in (7), which could be written more compactly without using cases.\n\nQUESTIONS:\n- I would be interested to know the performance without the global shared advantage and with only the local shared advantage (but with the same data augmentation). In such a case, the method would revert to the normal GRPO, where the advantage is only normalized across the same question (specifically, the same variant of the same question). I suspect that simply increasing the number of samples per question variant would be sufficient for enhancing credit assignment, making the global advantage normalization unnecessary.\n- Ideally, a learning curve should also be shown in the paper to understand whether the optimization has converged, especially for ablation runs.\n\nLIMITATIONS:\nThe authors addressed several limitations common in multimodal LLMs.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Share-GRPO mechanism to train multimodal LLMs. Share-GRPO first augments the original problem into multiple variants, then employs the policy model to generate multiple candidate reasoning responses for all augmented variants. When updating the policy, Share-GRPO introduces an advantage-sharing mechanism to provide a rich reward signal to the policy model, alleviating the reward sparsity issues in standard GRPO. Extensive experiments demonstrate the effectiveness of the proposed Share-GRPO.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea of Share-GRPO, which augmenting the original problem to provide rich reward signal to alleviate reward sparsing issue, is convincing and reasonable.\n2. The experiments presented in the paper are comprehensive and solid.\n3. The paper is well-structured and mostly clearly written.\n\nWeaknesses:\n1. As presented in Limitations, the data augmentation done by generative models such as GPT-4o may introduce noise into augmented problems.\n2. In Figure.1(b), although the valid advantage ratio curve of Share-GRPO is slightly higher than the standard GRPO's, their trends are almost the same, and their values are only slightly different from about step 20, finally becoming almost indistinguishable by step 70. I don't think Share-GRPO truly solve the advantage vanishing issue based on this figure.\n3. Ablation studies and experiments in Tables 4 and 5 are only conducted on one benchmark \"MathVista\". It would be more convincing if the authors provide more results such as the average performance in Table.1.\n4. In Equation.(5) and (6), the subscript \"k\" does not appear in the RHS of the equation. Its meaning here is unclear to me.\n5. In Table.4, Share-GRPO achieves suprior performance only when setting $m=2$, which means only 1 augmentation is conducted for each problem, with the other being the original problem in my understanding. I have 2 concerns about this observation:\n\n(a). In Section 3.2.1, the authors introduce several augmentation approaches. I wonder which one was ultimately chosen for the experiments. If it was selected randomly, could the reproducibility of Share-GRPO be weakened, since different augmentations may heavily affect the data samples?\n\n(b). Why does increasing *m* from 1 to 2 lead to a significant performance gain (72.8 → 75.4 on MathVista), while increasing *m* from 2 to 4 yields only a minimal improvement (75.4 → 75.9 on MathVista)? I’m interested in this discrepancy, but the authors do not discuss it.\n\nQUESTIONS:\n1. In Equation.(7), what does $\\pi_\\theta(o_i^{Q_j}|Q_k)$ mean? The notation $o_i^{Q_j}$ is already defined as \"response $o_i$ generated by the policy model conditioned on problem variant $Q_j$.\" However, the conditional notation $(\\cdot | Q_k)$ is confusing to me. I would appreciate it if the authors could clarify this further.\n2. The proposed Share-GRPO framework also appears to be seamlessly applicable to the text-only reasoning domain. I wonder whether the authors have conducted experiments using text-only LLMs on challenging mathematical benchmarks such as AIME 2024, GPQA, etc.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes Share-GRPO, a RL methodology for VLLM that expands the input question space by (1) use LLM to paraphrase the textual questions (2) transform the input image by pre-defined operations (e.g., rotation, noise injection). The author argues that the proposed transformation techniques introduce diversity and elicit diversified outcome rewards, eventually addressing the sparsified reward and advantage vanishing issues in GRPO. Experimental results on multimodal math and understanding tasks show the effectiveness of Share-GRPO over GRPO.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Clarity and readability – The manuscript is well-structured and easy to follow, allowing readers to quickly grasp the core ideas and experimental setup.\n2. Conceptual simplicity with practical impact – The proposed synthetic-variant expansion is a straightforward yet effective alternative to prior sampling- or selection-based approaches.\n\nWeaknesses & Suggestions: \n\n1. Insufficient analysis of the transformation’s effect on reasoning\nThe paper lacks a theoretical argument—or at least a more extensive empirical study—showing that the proposed transformations genuinely elicit distinct reasoning behaviors and mitigate GRPO’s sparsified-reward and advantage-vanishing issues.\nIn principle, these transformations should preserve task difficulty, so it is unclear why reward signals vary across variants. Adding deeper analyses (e.g., probing activation patterns or qualitative error cases) would strengthen the claim.\nTable 4 shows nearly identical scores for 2–4 question variants, further calling into question the diversity effect.\n\n2. Expansion-technique details and validation:\nSince paraphrasing relies on GPT-4o prompts, a validation step is needed to guarantee semantic equivalence; the limitation is acknowledged but not addressed experimentally.\nThe chosen image transformations are only listed, not motivated. Please justify each operation (e.g., rotation, noise) and discuss how it differs from standard data-augmentation baselines.\n\n3. Baseline coverage: \nA direct comparison between vanilla GRPO and Share-GRPO on the identical backbone (e.g., Qwen 2.5-VL + GRPO vs. Qwen 2.5-VL + Share-GRPO) is essential and should appear in Table 1.\nTables 2–3 cover only a subset of benchmarks. Report GRPO results on all datasets to provide a complete effectiveness picture.\n\nQUESTIONS:\nSee also weakness section for full details.\nQ1. In principle, these transformations should preserve task difficulty, so it is unclear why reward signals vary across variants. Can you add more deeper analyses (e.g., probing activation patterns or qualitative error cases) to strengthen the claim. This will impact my final scores significantly. \n\nQ2. Can you also report the full experiment results with Qwen 2.5-VL + GRPO in Table1? This is very important to justify the effectiveness of the proposed Share-GRPO.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}