Title: EvolvedGRPO: Unlocking Reasoning in LVLMs via Progressive Instruction Evolution
Venue: NeurIPS 2025
Accepted: yes
Keywords: multi-modal reasoning

reinforcement learning

self-improvement
Abstract: Recent advances in reinforcement learning (RL) methods such as Grouped Relative Policy Optimization (GRPO) have strengthened the reasoning capabilities of Large Vision-Language Models (LVLMs). However, due to the inherent entanglement between visual and textual modalities, applying GRPO to LVLMs often leads to reward convergence across different responses to the same sample as training progresses, hindering effective gradient updates and causing the enhancement of chain-of-thought reasoning to stagnate or even collapse.
To address this issue, we propose a progressive instruction evolution framework, EvolvedGRPO, to gradually generate more complex questions via editing instructions in an adversarial way, progressively aligned with the model’s evolving capabilities. Specifically, we design two instruction editing strategies across modalities, incorporating incrementally increasing editing instructions and RL-based adversarial data augmentation to improve the effectiveness of model training. To address GRPO's limitations on overly difficult problems, we first train on basic subproblem versions of complex multi-modal questions in both the visual and textual modalities, progressively increasing difficulty to enable prefix-style process rewards, effectively combining the strengths of both process rewards and group-wise relative rewards. Finally, EvolvedGRPO achieves state-of-the-art performance among open-source RL models on multi-modal reasoning tasks, even approaching the closed-source GPT-4o in reasoning capabilities, and demonstrates better performance on unseen LVLM general benchmarks. The Code for EvolvedGRPO is available at https://github.com/SHENZHEBEI/EvolvedGRPO.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the problem of training stagnation caused by reward convergence when fine-tuning LVLMs with RL methods like GRPO. To tackle this challenge, the authors propose EvolvedGRPO, a progressive instruction evolution framework. This framework utilizes an adversarial setup where a question generation model (Q) dynamically creates training samples of increasing difficulty for an answer model (A). Specifically, model Q generates image and text editing instructions to increase the complexity of problems, thereby matching the evolving capabilities of model A. Ultimately, experimental results show that this method achieves state-of-the-art performance on several multi-modal reasoning benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Impressive Performance: The paper achieves excellent results on several challenging multi-modal reasoning benchmarks, outperforming other open-source models and even approaching the performance of the closed-source GPT-4O on certain metrics.\n2. Compelling Motivation: The paper's premise is highly valuable. Exploring self-evolving reinforcement learning algorithms to dynamically adjust training data difficulty is an important and promising research direction for pushing model capabilities beyond their current limits and achieving a higher level of intelligence.\n\nWeaknesses:\n\n1. Lack of Clarity and Missing Key Details: For a paper whose core contribution is a new algorithm, the description of the method is surprisingly scattered and vague. It dedicates considerable space to well-known background concepts like SFT and GRPO, which hinders the understanding and reproducibility of its core contribution.\n\n    - In the core reward function, Equation (9), the symbol format' appears without any definition or explanation provided anywhere in the paper. Additionally, the explanation for the test variable is also same problem. \n\n    - The \"Text Editing Instruction\" is a central innovation of the method, yet the main text lacks a detailed description of its specific operations. Readers are forced to infer its meaning from the example in Figure 1, but such a key mechanism should be explicitly and thoroughly described in the text.\n    - The paper's description of the adversarial training process is not systematic. A clear algorithm flowchart or pseudocode block is crucial for understanding the alternating training of the Q-A models and key steps like instruction generation and replacement, but this is absent from the main body of the paper.\n    - The presentation of results in Table 1 could be misleading. Specifically, the 'Avg.' column is calculated and presented even for baseline models where scores for the GSM8K and MATH500 datasets are absent (e.g., OpenVLThinker-7B, NoisyRollout, MM-EUREKA). This creates an inconsistent basis for comparison and could cause readers to misinterpret the overall performance of these baseline models\n\n\n    - Typo: A typo, \"generation generation,\" appears on line 195, which affects the reading experience.\n\n\n2. Mismatch Between Motivation and Method's Effect: The paper's initial motivation is to solve the RL training bottleneck in multi-modal reasoning caused by the entanglement of visual and textual information. However, its core \"text editing\" method—for instance, transforming an answer z into a problem of solving for D = 3*z/4 + 9/4—essentially appends a separate, unimodal mathematical calculation task at the end of the original multi-modal task. This operation does not increase the intrinsic complexity of the visual-textual reasoning itself, but rather appears to train the model's arithmetic capabilities.\n\n3. The paper lacks the necessary analysis to prove that this method actually solves the problem it set out to address: reward convergence and training stagnation. For example, the authors do not analyze whether the reward variance for different responses to the same sample actually increases after using EvolvedGRPO. The logical link between the method's effect and the initial motivation is not established.\nInsufficient Experimental Analysis and Comparison:\n\n4. The paper's image manipulation method (e.g., adding noise or filters) is conceptually very similar to that of NoisyRollout[1]. However, the paper merely cites this work without providing any in-depth comparative analysis or discussing the differences in concept or effectiveness.\nThe paper lacks a dedicated ablation study for the image editing component. From the experiments, we cannot determine the actual performance contribution of adding noise or editing the images. This calls into question the effectiveness of the \"multi-modal\" aspect of the instruction evolution.\n\n[1] NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces EvolvedGRPO, a reinforcement learning framework aimed at enhancing the reasoning capabilities of Large Vision-Language Models (LVLMs). Built upon the Grouped Relative Policy Optimization (GRPO) method, the key contribution of EvolvedGRPO lies in its progressive instruction evolution mechanism that dynamically generates increasingly difficult multi-modal reasoning tasks. It achieves this by iteratively editing both textual and visual instructions in a self-supervised, adversarial way. Two models—a question generator and an answer model—are jointly trained, where the former adversarially challenges the latter with more difficult tasks. Empirical results show that EvolvedGRPO outperforms other open-source baselines and approaches GPT-4o on several reasoning benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: Introduces an effective curriculum-style RL method tailored for LVLMs. Combines adversarial data generation with reward shaping to break reward convergence, a common issue in GRPO.\nWeaknesses: The instruction evolution mechanism still relies heavily on implicit assumptions about controllability and reward fidelity. The reward design assumes that small transformations of inputs yield proportionally meaningful reward gradients—this may not always hold in multimodal contexts. The generalization to truly open-ended tasks is unclear.\n\nQUESTIONS:\nHave you examined whether small perturbations in instructions (e.g., minor edits in visual modality) produce meaningful reward signal changes? Could the reward collapse still persist in late training?\nWhile hallucination reduction is mentioned, how robust is the filtering pipeline?\nCan the framework actively regulate difficulty based on performance plateaus? If so, how is this determined?\nCould you elaborate on the application potential in real-world settings such as visual question answering for medical or industrial imagery?\n\nLIMITATIONS:\nMore detail is needed on how instruction hallucination or adversarial drift is avoided in later stages of evolution.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOne of the main problems with GRPO is that as a model becomes better, its answers become equally as good/accurate. This paper proposes to resolve the 'reward saturation' problem presented by GRPO by presenting a framework that: \n(a) Generates progressively more difficult multimodal problems by applying image/text edits using a question generator. The question generator is trained adversarially to propose image/text edits that (i) do not modify the original answer and (ii) are hard for an answer model to solve, but are still solvable.\n(b) Utilizes curriculum learning to apply increasing numbers of edits (k) so that the model's reasoning chain lengthens.\n\nThe authors test their method on five multimodal reasoning benchmarks (MathVista-mini, MathVision-full, MathVerse-mini, GSM8K, MATH500) and three broad multimodal benchmarks (MMMU, MMStar, AI2D). Their method, which fine-tunes Qwen2.5-VL-7B-Instruct, shows gains over baselines. In ablations, these gains are mostly attributed to the adversarial curriculum.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The authors clearly outline the problem that their approach is trying to solve: reward variance collapse and its impact on GRPO training.\n2. The authors introduce a novel approach that integrates adversarial instruction generation and curriculum scaling.\n3. Their method shows improvement across 8 benchmarks.\n4. Ablations that demonstrate complementary effect of both adversarial instruction generation and curriculum scaling.\n\nWeaknesses\n1. The authors do not provide detailed information on their image editing pipeline (aside from stating \"external tools such as Diffusion models\" and similar high level descriptions). It would be advisable to add more details to the appendix.\n2. Results are reported from single runs. No standard deviations or statistical significance tests are provided, and confidence intervals are not present. It would be ideal to report results for >= 3 random seeds with confidence intervals and statistical tests against the strongest baseline.\n3. They use GPT-4o as a judge and source of reward. This can add biases prevalent in GPT-4o, potentially rewarding responses that look most similar to its own, while penalizing those that are different from its own. The authors may wish to perform a robustness study, using a different model (e.g., Claude) or humans as the judge.\n4. Ablations are high-level only. The authors did not isolate lower-level components, such as image/text edits.\n\nQUESTIONS:\n1. Address the weaknesses.\n2. I really like your method. Have you attempted to use it with real-world vision language tasks?\n\nLIMITATIONS:\nSort of. In appendix F, there is a brief mention of using more advanced image editing techniques in the future. However, this may not meet this requirement.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces EvolvedGRPO, a new framework designed to enhance the reasoning capabilities of VLM. The authors identify a key problem with existing reinforcement learning methods like GRPO: as LVMs improve, the reward scores for different answers to the same multi-modal question tend to converge. This lack of reward variance hinders effective learning. To solve this, EvolvedGRPO progressively generates more complex problems that are matched to the model's current abilities through an adversarial process involving a \"question generation model\" (Q) and an \"answer model\" (A).  The two models are trained adversarially, with Q rewarded for making challenging yet solvable problems and A rewarded for correct answers, while a curriculum learning strategy gradually increases the number of editing instructions to guide the model toward more complex reasoning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper is well-written and easy to follow.\n- The paper identifies and addresses a specific, critical bottleneck in training LVMs.  The proposed EvolvedGRPO framework is an innovative solution that tackles this by progressively increasing the difficulty of training data to match the model's growing capabilities.\n- The authors demonstrate its EvolvedGRPO effectiveness across a wide range of benchmarks.\n\n**Weaknesses**\n- Limited Scope of Demonstrated Instruction Evolution: The text editing instructions are primarily described as performing \"mathematical transformations\" to increase reasoning steps. This approach is well-suited for the mathematical and logical reasoning benchmarks tested, but its applicability to more abstract, qualitative, or common-sense reasoning tasks is less clear. \n- Concern over Quality Control and Cumulative Error: The framework's success is highly dependent on the quality and fidelity of the instructions produced by the question generation model (Q). Because the method increases problem difficulty through multi-step iteration , small errors or logical deviations introduced in early steps can be amplified in subsequent ones, potentially creating samples that are complex in appearance but logically flawed or unfaithful to the original intent. While the paper mentions using a judge model and rule-based checkers for filtering, it does not detail how robust this filtering mechanism is against subtle, accumulating errors that might not violate a simple rule but still corrupt the problem.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}