EvolvedGRPO: Unlocking Reasoning in LVLMs via Progressive Instruction Evolution
Authors: Zhebei Shen
Qifan Yu
Juncheng Li
Wei Ji
Qizhi Chen
Siliang Tang
Yueting Zhuang
Venue: NeurIPS 2025 poster ()
Keywords: multi-modal reasoning
reinforcement learning
self-improvement
Primary area: deep_learning
TL;DR: A progressive instruction evolution framework that iteratively unlock LVLMs' reasoning ability

Abstract: Recent advances in reinforcement learning (RL) methods such as Grouped Relative Policy Optimization (GRPO) have strengthened the reasoning capabilities of Large Vision-Language Models (LVLMs). However, due to the inherent entanglement between visual and textual modalities, applying GRPO to LVLMs often leads to reward convergence across different responses to the same sample as training progresses, hindering effective gradient updates and causing the enhancement of chain-of-thought reasoning to stagnate or even collapse.
To address this issue, we propose a progressive instruction evolution framework, EvolvedGRPO, to gradually generate more complex questions via editing instructions in an adversarial way, progressively aligned with the model’s evolving capabilities. Specifically, we design two instruction editing strategies across modalities, incorporating incrementally increasing editing instructions and RL-based adversarial data augmentation to improve the effectiveness of model training. To address GRPO's limitations on overly difficult problems, we first train on basic subproblem versions of complex multi-modal questions in both the visual and textual modalities, progressively increasing difficulty to enable prefix-style process rewards, effectively combining the strengths of both process rewards and group-wise relative rewards. Finally, EvolvedGRPO achieves state-of-the-art performance among open-source RL models on multi-modal reasoning tasks, even approaching the closed-source GPT-4o in reasoning capabilities, and demonstrates better performance on unseen LVLM general benchmarks. The Code for EvolvedGRPO is available at https://github.com/SHENZHEBEI/EvolvedGRPO.

EvolvedGRPO: Unlocking Reasoning in LVLMs via Progressive Instruction Evolution
Zhebei Shen Qifan Yu!-
Juncheng Lift Wei Ji²
Qizhi Chen
Siliang Tang'
Yucting Zhuang!
!Zhejiang University Nanjing University Peeking University (shenzhebei, yuqifan, junchengli, siliang)@zju. edu cn
Abstract
Recent advances in reinforcement learning (RL) methods such as Grouped Rel- ative Policy Optimization (GRPO) have strengthened the reasoning capabilities Models (LVLMss). However, due to the inherent en- tanglement between visual and textual modalities, applying GRPO to LVLMS often leads to reward convergence across different responses to the same sample as training progresses, hindering effective gradient updates and causing the en- hancement of chain-of-thought reasoning to stagnate even collapse. To address this issue, we propose a progressive instruction evolution framework, Evolved- GRPO, to gradually generate more complex questions via editing instructions in an adversarial way, progressively aligned with the model's ' evolving capabilities. Specifically, design two instruction editing strategies across modalities, incorpo- rating incrementally increasing editing instructions and RL-based adversarial data augmentation improve the effectiveness of model training. To address GRPO's limitations on overly difficult problems, we first train on hasic subproblem versions complex multi-modal questions in both the visual and textual modalities, pro- gressively increasing difficulty to enable prefix-style process rewards, effectively combining the strengths of both process rewards and group-wise relative rewards. Finally, EvolvedGRPO achieves state-of-the performance among open-source RL models on multi-modal reasoning tasks, even approaching the closed-source GPT-4o in reasoning capabilities, and demonstrates better performance on un- seen LVLM general benchmarks. The Code for EvolvedGRPO available
of Large
1 INTRODUCTION
Large Language Models (LLMs) have achieved remarkable progress in multi-step reasoning tasks, most notably through models such as OpenAl's GPT-4o [I] Building on this progress, recent work DeepSeek-R1 (2) highlights that reinforcement learning (RL) with verifiable rewards is particularly effective eliciting nuanced self-verification and self-correetion behaviors in LLMs, substantially enhancing the reliability of mathematical and logical reasoning chains.
Going a step further, with the emergence of advanced large vision- languae models (LVLMs), there has been further RL exploration into LVLMs [3] However, general RI. methods such as Grouped Relative Policy Optimization (GRPO) necessitate assigning varying reward scores to different responses from the same prompt to generate meaningful
4
5 6 7 to enhance their reasoning abilities.
"Equal contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

gradients. Yet in multi-modal tasks, unlike purely textual reasoning tasks that benefit from clear formats, making stepwise reward design feasible, the entanglement between visual and textual modalities poses fundamental challenges for assigning such differentiated rewards.
Specifically, the LVLM's visual reasoning ability improves over multiple rounds of training, the model gradually memorizes both visual and textual information from the training dataset, enabling it recall answers solely from the reasoning processes it has internalized during previous training, thereby reducing the need for genuine perception and reasoning. This results in low intra-group reward variance under the GRPO strategy, making effective gradient updates difficult and causing the reasoning capability to stagnate or even collapse 4 As result, the GRPO strategy in multi-modal tasks struggles to maintain sufficient group reward variance as training progresses, and two problems are notably amplified. (a) For relatively simple samples, different model responses tend to yield uniformly high rewards despite varying reasoning paths, due to the absence of a mechanism to reward deeper or more faithful visual grounding. This leads to premature reward convergence and provides limited learning signal for improving multi-moda reasoning capabilities. (b) For overly complex samples, the model often fails to construct valid long- interpret visual and textual cues. Since only final outputs are evaluated, all failed attempts may receive equally low rewards, making it difficult for the policy to learn incremental improvements.
reasoning chains that jointly
To tackle these challenges faced by GRPO in multimodal reasoning, previous works mainly focus on either distilling knowledge from stronger external models [3, 6 g challenging multi-modal perceptual datasets LVLMs. However, such adaptation paradigms either depend on the performance upper bound of external teacher models or incur substantial human labor costs for data construction. Moreover, the inherent disconnect between external teacher models and the internal answering model limits the potential to fully exploit the model's reasoning capabilities, resulting in training data that fails to keep pace with the model's s evolving abilities.
9 [0] or collecting more [4 to adapt to the evolving capabilities of
In this paper, we propose a progressive instruction evolution framework (EvolvedGRPO) that iteratively increases the difficulty of samples through self-evolved editing instructions with low overhead to overcome the challenges of reward saturation and adapt to the model's ' continually advancing reasoning capabilities. Based on this evolutionary framework, we observe that the model can gradually improve its performance through adversarial interactions within the framework. Formally, we use one model as the question generator Q, which is used to generate two types of multi-modal data editing instructions aimed at augmenting the dataset with learnable multi-step reasoning tasks that remain within the model's capability range: (1) Image editing instructions Ivs using external tools such as Diffusion Models [15 are employed to edit the training image corresponding to the instruction I,,, thereby accurately recognizing and analyzing visual information; (2) Text editing instructions I, which are designed to alter the original question-answer pair, forcing transformation of the original answer and introducing multi-step reasoning into the resulting multi- modal question, thereby increasing the number of reasoning steps required. Meanwhile, we employ another model the answer model A, which is trained to solve multi-modal reasoning problems via chain-of of-thought reasoning. In each iteration, we generate an instruction pool and randomly sample i mixed instructions to synthesize each multi-modal question based on an original question. Benefiting from the Markov property, the trustworthy answer is derived in a step-by-step manner using a TTRL (16] applied to the outputs of the trained model.
Moreover, we further effectively enhance the instruction evolution to activate LVLMs' deeper reasoning ability with two progressive training strategies: (1) train the two models adversarially through reinforcement learning, using the accuracy of answer model A as the critic of question generator Q to improve the quality of each generated instruction; (2) gradually increase the number of sampled instructions to progressively guide the model toward longer k+1-step reasoning chains in curriculum learning manner. Under such progressive promotion, extensive experiments on challenging multimodal reasoning benchmarks and general LVLMs benchmarks show that EvolvedGRPO effectively enables the model to explore the most suitable chain-of-thouge paths, thereby unlocking continual growth and qualitative improvements in reasoning trajectories. Overall, our main contributions are three-fold:
* We propose EvolvedGRPO, a progressive instruction evolution framework leverages GRPO-trained question generator to apply multi-modal editing instructions, constructing increasingly challenging datasets tailored to the model's evolving reasoning capabilities.

---

Self-improved Reward
Question:
Question:
As shown the figure, rectangle is value divided into three parts: central rectangle, and two seriic ircles on either side. The area of the shaded part is
shown figure, rectangle into three parts: central rectangle, and semicircles either side. The area of the shaded part
Progressive cnttruction Evolution
model
the sap answer obtained in be a pa Sl. Assume DI = 3 2/4 974, now find
instruction
Apply wave
Answer:
Answer:
Image:
Textediting instuction
Image:
[(>)-== (1er11)
Text editing Incrrution
70O 3/4/4 9/4 (Iter)
Seed Data
P Progressive GRPO Reward
self-ewolved model A
Figure 1: Overview of our proposed EvolvedGRPO for enhancing the model's 's reasoning capability by progressively using instruction evolution during GRPO-baser training in an adversarial way.
We adversarially train the question generator and answer model with curriculum-style instruction scaling, enabling continuous improvement of LVLMs' reasoning capabilities.
Our method ensures that, across diverse tasks and reasoning scenarios, LVLMs can consis- tently unlock their underlying reasoning capabilities a self-driven manner with minimal overhead.
2 RELATED WORK
Equipping LVLMs with Reasoning Ability. With the remarkable generalizability of LLMs in
zero-shot setting, Large Vision-Langun Language Models (LVLMs) [17
18
19
20 21
22
23
24
25
26
27 28 29
integrating LLMs with visual modality have demonstrated impressive visual- language
33 Motivated by the significant advances in reasoning demonstrated by LLMs, substantial efforts have been directed toward enhancing the
understanding ability across diverse scenarios (30
31
32
reasoning capabilities of LVLMs supervised fine-tuning (SFT) [3 35 36
37 38 and chain-of-
40 [1]. More recently, Reinforcement Learning (RL) approaches such as GRPO 22 have emerged as an effective strategy to unlock underlying reasoning abilities in LVLMs 3 4 S 6 7 Despite these advancements, applying RL to multi-moda tasks remains challenging due to the difficulty of defining fine-grained rewards across entangled modalities, which limits the full exploitation of reasoning capabilities in LVLMs. This is essential for establishing such granular reward signals unlock reasoning ability in LVLMs.
thought (CoT) prompting [38 39
Multi-model Reasoning Instruction Dataset. Although reasoning LVLMs have demonstrated remarkable performances in various VI reasoning tasks, they still require efficient utilization of high-quality data to enable robust and transferable reasoning capabilities. Recent methods rely heavily on external powerful models (e.g., GPT-4o, DeepSeck-RI) to synthesize multi-step questions [10] or long chain-of-thoup distillation, which consequently limits the upper bound of the model's reasoning ability. Furthermore,
reasoning instructions [3] 6 8 9. essentially performing pre-trained knowledge
several works [1] [2 13
43 focus on collecting more challenging multi-mode perceptual
48 49 for reasoning instruction construction, but require substantial human labor costs for data collection. Generally, these methods suffer from limited generalization across modalities and tasks, failing to establish tight coupling between visual and textual semantics. In contrast, we propose an instruction evolution method that progressively expands the reasoning capabilities by leveraging an adversarial instruction synthesis strategy based solely the LVLM itself, naturally adapting to the evolving needs of the model.
datasets via tool-invoking methods
44 45 46 47
3

---

PRELIMINARIES
Large Vision- LaiionLangguage Models (LVLMS) LVLMs represent class of multi-modal large language models that integrate both visual and textual information as input, generating textual outputs. The model architecture of LVLMs commonly consists of three components: large language model, vision encoder and MLP-based vision-language merger [3 The LVLMs use vision encoder to transform inputs visual information into visual embeddings, and MI.P-based vision-language merger align and integrate the visual embeddings and textual embeddings into a unified feature space. Finally, the large language model autoregressively generates the outpur.
Supervised Fine Tuning (SFT). As the most commonly used technique for LLMs and multi-modal LLMs, Supervised Fine Tuning (SFT) is simple and efficient method to align pre-trained models with downstream tasks. Formally, given a multi-modal dataset 'D including question g, answer a and image the training loss of SFT is:
C
ΣΣ
I(a log no(clg, u, a<i))
(I)
where is the parameter of model, L the token length the answer a, C' is the number of candidate token classes determined by sampling methods such as top-p top-k, I(a; c) is the indicator function that equals the condition a, is true and equals otherwise, and ne (clg, e, a<)) represents the model's predicted probability of the i-th token equals to given the question image v and the sequence of preceding tokens açi-
Group Relative Policy Optimization (GRPO) As in LLMs, often the last token typically receives reward score, Group Relative Policy Optimization (GRPO) reduces the memory and computational burden compared to PPO 50). which eliminates the need for an additional value function approxima- tion. GRPO uses the average reward multiple outputs, sampled in response to the same question as the baseline. For each question q, GRPO samples a group of outputs {o), o2,..., o from the old policy nuw and optimizes the policy model by maximizing the following objective:
T: mcan(fryl
Wi ().....G)
std(tr, )1=1))
where A is the advantage calculated based on relative rewards of the outputs within each group.
(8) = q P(Q), fo To.. (O
(3)
G
ln. {min ( (0)A clip (l (,, Ar.) 3Dxx. ( [rr])
where
To To(o ci. and D [moe [Tedl
Tref(Orala. log Te o12()
Trer(Oralq. oice) 1. (o Osc()) 0s<c)))
(()
To. (ot, o1)
Here, Ty and Ne. represent the current and old policy models, respectively, and q and are the questions and outputs sampled from the question dataset and the old policy Tew.. The is clipping- related hyperparameter to stabilize training.
This method leverages the comparative nature of reward models, which are typically trained on datasets containing comparisons between outputs for the same question. Instead of adding KL penalty directly to the reward, GRPO regularizes by adding the KL divergence between the trained policy and the reference policy to the loss function, simplifying the calculation of Á

---

METHOD
In this section, we first introduce a formulation for evolved editing instructions to dynamically provide evolution intents ($4.1). Based on these, we develop progressive instruction evolution framework within LVLMs generate increasingly challenging problems that require longer chains of reasoning ($4.2) Finally, we further enhance the instruction evolution progressive training strategies, The overall framework shown in Figure [
unlocking the full reasoning LVLMs $4.3)
4.1 Formulation for Evolved Editing Instructions
During the multi-round GRPO training with fixed data, the increase in accuracy and the training on fixed problems cause the standard deviation to decrease to zero, leading to reward convergence and, ultimately, training stagnation and collapse. To address this, we introduce EvolvedGRPO, which incorporates adversarial data augmentation to introduce additional information during training.
In general. the paradigm for reasoning instruction involves generating multi-step questions in single step. We innovatively propose a progressive instruction evolution that incrementally refines instructions for both visual and textual modalities, In this section, we will demonstrate that both image editing instructions and text editing instructions are designed for Progressive Instruction Evolution in LVLMs to unlock their reasoning abilitty.
Given an initial sample (go, do, vo) Q x A V. where Q, A. V represent the question, answer, and visual space, respectively. Consider total of k steps in sample edition, the editing process is recursively defined
Instruction J1, (Q1,01,0))
Instruction J2.
Instruction
Jn. (gts abe, Uk:
(go, Co, vo)
Image Editing Instruction We employed image editing on training dataset to enhance the difficulty of inferring multi-modal samples. We maintain the original answer unchanged during the image editing process. We train a question generator @ produce precise image edition instruction L, to control external image editing tools(e,g.. (ontex)) for image editing. As we want to minimize the hallucination introduced in the process of question generation, we use judge model [5]] ensures that the newly generated samples maintain consistency with the original ones, e.g. preserving the correctness of the answer after bootstrapping. we should maximize sim Uk. Uo) in semantics, where sim denotes consistency in the content of the problem investigation.
Text Editing Instruction Editing the original question and answer at the same time will also introduce a large number of hallucinations. We construct text-editing instruction chain, where the instructions primarily involve performing text editing operations generated using external knowledge (e.g., mathematical concepts) the answer from the previous step, thereby encouraging the inclusion of additional reasoning steps required to infer the final answer. We employ connective templates to seamlessly integrate the original multi-modal problem (g, v) with the instruction St-
For ground truth answer, we use the TTRL [16] calculate the final answer. Specifically, for each function fi corresponding to the text editing instruction 4; we can get ag:
N Tim=
aj. arg DE Wi Mork
B Te fa (( ...)))
(4)
T z)) denotes the model's output distribution conditioned on the input æ;; y4 TT(- (x) indicates that the i-th output is sampled from the answer model's distribution conditioned on input æ. To facilitate rapid convergence with minimal sampling while accounting for the unbounded nature of the solution space and the Markov property of action selection, we adopt the approximate formulation:
0; arg
max UE JNN
[yg
v], Mj - (((a)-))); Vi (1....,k)
(5)
Reducing time complexity from O(NN) to O(kN), significantly accelerating large-scale sampling.
5

---

4.2 Progressive Instruction Evolution within LVLMs
In this section, we perform reinforcement learning training in both the question generator and the answer model A. The final trained model A will serve as the model for reasoning. We alternate between training two models through Group Relative Policy Optimization growth in model capabilities.
to achieve continuous
Answer Model Training For the i-th iteration(i = 1,...), we first train the answer model AA. We will train on data X = (21...., Inn) E DD while using the question generator Q to generate the instruction Z. We use the answer model A step by step vote and generate the new answer ag. Then we use the answer model A to generate a list of outputs {oi, ....... oG). We check whether answer correct by using the rule-based mathruler
[53] to directly judge whether the answer response
is same as ag functionally. As we want to maximize the probability the answer model get the true answer Ty(a (a q, v), we set reward as:
R _ equivalent(Ok, o;) + format
(6)
Where format refers reward score of format, which is designed encourage the model to produce structured outputs, particularly those involving chain-of-thought reasoning.
Question Generator Training We then start to train the question generator Q. In each iteration, we sample training data z E from training dataset D. the sampled i: instruction (I1.... Ix) E I, we will use the question generator Q to generate list of outputs {01, 02. og } which used to replace one instruction Ireplaced E {Ii...., Ja We use the external image edition tool and answer model A to get each augmented data [(2). d1; .n)..... (ga, AG, va)). ]. Due to limited answer model A, we should also maximize the confidence level of the generated answers. The total confidence of the generated sample can be:
k C -= II sim U-11 II To To(ai(f((a:-1)) format
In the meantime, we want maximize the difficulty for answer model A, which minimizes the probability model A can acquire the true answer To(ak (kk vx) relative to original To aa(oo/go, vo). Therefore, the total reward can be defined as:
R I. sim(r;, 11-1) 11 To (o(a:))i((a-1)) format To (a(al19h. vx-) / To (aplgos
(8)
qo. Up)
In order better calculate the Group Relative Advantages [52 and balance the relationship between scores of different magnitudes, we take the logarithm of all the scores and eliminate the common terms in the same group's outputs. To ensure lower bound on performance, we take the maximum of the accuracy and the reciprocal of the number of evaluations, ultimately obtaining:
R'_
log sim (ui, Vi- 1)+
log To
(axx(To(ax)) (T,, Uk), test
(9)
Where test denotes the number of evaluation trials used to assess accuracy, aligning with the number of responses generated in a reinforcement learning batch. The expression using max in the goal ensures at least one correct answer post-question augmentation, preventing variance collapse. The final question generator Q should be able to provide effective image editing without affecting the question, and be able to diversify the connection between the original problem and different instructions regarding the weaknesses of the answer model A.
Mixual Training During training, it is essential to perform quality control on the instructions generated in each round. To ensure the reliability of the editing instructions, we employ filtering strategy that rigorously eliminates instructions that not conform to the required format are prone to introducing hallucinations. In the meantime, to ensure stable training of model A, the confidence C

---

and the accuracy To (ax 9k. Uk) of the generated questions should be maintained within reasonable range. Reference the design of Generative Adversarial Networks [54]. to ensure the answer model A has enough capability, we train multiple steps with the answer model A and one step with the question generator Q. This strategy is analogous the way that training maintains answer model A train using data of appropriate difficulty, the {ri, } for single sample' s outputs (o1.-.., differ therefore model can maintain stable gradient updates during model training.
4.3 Enhanced Evolution with Progressive Training
To further promote the instructional evolution in the training process, it is essential to emphasize progressive training strategies to activate LVLMs' improved reasoning ability. Specifically, we gradually increase the number of sampled instructions k: guiding the model toward longer k+1lstte reasoning chains curriculum learning manner. each training round, the model learns to use k-step multi-modal chain-of- thain-of-thoughts and receives process-level rewards edited multi-modal data requiring larger kt thereby enhancing both the depth and accuracy of its reasoning. This step-wise progression enables the model perform longer reasoning while maintaining correctness at each intermediate step, ultimately allowing it to solve complex multi-modal problems. Algorithm we present training procedure pseudo code.
Throughout training, as the model's 's reasoning capabilities evolve, its confidence executing editing instructions also increases. The framework progressively acquires the ability to support and perform a larger number of instruction edits with greater complexity. This continuous advancement forms a positive feedback loop: improved reasoning capabilities enable the execution of more sophisticated edits, which in turn expose the model to richer and more challenging training signals. Through this iterative refinement, the model gradually approaches its upper bound in reasoning performance. In Appendix D we provide additional theoretical and experimental analyses on the stability and robustness of the evolved instruction generation process.
Algorithm EyolvedGRPO Training Procedure
Require: Initial policy To. training dataset D, outer iterations Round, inner answer model training
steps T, KL regularization weight 3, curriculum step c, sample batch size m, GRPO iteration Round
Ensure: Answer model policy я AA
policy model A (0)
T (0)
ne.
+ ne
2: for iteration t 0, ..... Round - 1 do
3: policy model A (t+1) + A
(0) aQ
(1+1) + =Q
(t+1)
4: editing instruction number - [t/cl
5: for all sample (q, a, E D do
Generate edited sample (gk, Uk ax) through (t) (f) 7: Store (4k, Vk, ax) into new dataset D(t).
6:
n =q
8: end for
9:
for step = ...... do
10:
Sample batch of m samples (s(1) ....., s(..).)))))))) D(t).
11:
Sample G outputs {oj] for each sample 8j and compute Ai for the I-th token.
for GRPO iteration ......A4. : do
13:
Update the policy model A end for end for
(++1) with parameters g(++1) A using the GRPO gradient.
15:
Sample bath of samples (s(1) ..... s (m)) D(t),
17:
Replace by G editing instruction (o; [o for the same instruction editing sample 8j.
18:
Compute A for the 1-th token through group relative advantage estimation.
19: for GRPO iteration = 1,..../4 1 do
(t+1) Update the policy model nq with parameters oft+1)) I the GRPO gradient.
20:
21:
end for
22:
end for

---

EXPERIMENTS
5.1 Experimental Setup
Datasets & Backbones. As ideal data selections should be adaptable to diverse MLLMM instruction datasets, we integrate DataTaailrr with one widely-used datasets to conduct experiments for its effectiveness evaluation: MMK12 [4] dataset, a high-quality and diverse multi-modal mathematical reasoning dataset, being composed of MAVIS [55]. Geo3k [56). RCOT [57]. MultiMath (58) datasets. For the general experimental setup, we adopt Qwen2.5-VL. ----------- as the base model and train it using the GRPO strategy. More details of the experimental setup are shown in Appendix
Benchmarks. We evaluate model performance along two dimensions. First, we evaluate out-of- domain generalization across three visual reasoning benchmarks: Math Verse (59), Vision [60], Math Vista [61] and two textual reasoning benchmarks: GSM8K 622 and MATH500 63] Second, evaluate the performance of EyolvedGRPO across general benchmarks, including MMMU (3).
MMStar [31], and A12D [64]. Additionally, we develop an evaluation suite to consistently assess our trained checkpoints and most open-source RI RI-related checkpoints using vLLM 665 for accelerated inference, while adopting reported results for other models. For parsing generated responses, we employ greedy decoding with GPT-4o I as the evaluation judge. Despite adhering closely the system prompts from the original sources, slight deviations from reported results may persist, which we consider acceptable due to potential variations judge models and evaluation procedures.
Baselines. We use the following baselines: (1) distilling knowledge from stronger external mod- els (e.g., GPT-4o, DeepSeek-R1), Mulberry (66) and Virgo [6 leverage distilled multi-modal chain-of-thought reasoning from more capable teacher models to conduct supervised fine-tuning, thereby enhancing the reasoning capabilities of smaller models; MindGym (10] improves reasoning capabilities by performing curriculum learning based chain-of-thought supervised fine-tuning on set of multi-modal multi-hop questions synthesized by stronger teacher models; (2) collecting
more challenging multi-modal perceptual datasets, VLThikkr [ manually collects a large number of datasets with varying difficulty levels to support stage-wise reinforcement learning, thereby learning guidance; NoisyRollout [14] increase data difficulty by enhancing visual perception: MM- EUMEEARKA 4 facilitates model training by collecting large-scale data and dynamically filtering samples based on their difficulty at each training stage, thereby enabling the model learn from appropriately challenging examples.
enabling curriculum-ste
5.2 Main Results on Multi-modal Reasoning
We report the results of our EvolvedGRPO and other diverse methods shown in Table I Based on the observation of experimental results, we have summarized the following conclusions:
Distillation-based methods exhibit a relatively low upper bound in enhancing reasoning ca- pabilities. We observe that in most benchmarks, Mulberry and Virgo exhibit limited reasoning capabilities by directly fine-tuning on distilled of data. Although MindGYM outper- forms the open-sourced Owen2.5VL-7B-Instrue by employing curriculum learning with synthesized multi-hop questions, it exhibits a limited upper bound of reasoning performance on long-chains benchmarks (e.g., only 68.4 in MATH500) due its reliance on externally generated questions. In contrast, our EvolvedGRPO achieves superior performance in all reasoning benchmarks owing to its progressively improved instructions.
For collection-based approaches, the reasoning performances remain unsatisfactory due to the lack of diverse and complex data. Although these baselines introduce improvements to GRPO-based training. they still underperform EvolvedGRPO in multimodal reasoning benchmarks due to their insufficient data-level enhancements to GRPO. Open VLThinkrr and MM-EUREKA are constrained by the limited availability of noman-annotater data, which prevents continuous data updates during the later stages of training, ultimately resulting in a significant performance gap. NoisyRollout augments the dataset with manually designed visual noise to increase perceptual difficulty, which forces the model to enhance its visual robustness. However, it still exhibits limited improvements in multimodal reasoning.
Our EvolvedGRPO demonstrates strong reasoning capabilities. Furthermore, in Math Vision, the most difficult multimodal reasoning dataset, EvolveGGRPO attains the highest score (30.8), demonstrating its superior capability to handle complex multimodal reasoning tasks. Despite slight
8

---

Table Comparisons with our EvolvedGRPO and other baselines on multi-modal reasoning bench- marks. For fairness, all open-source reasoning frameworks are based on Soven2.5VVL-.. BL--B-Insttree
Vista -mini
MathVision -full
Math Verse -mini
Models
GSM8K
MATH500
Avg.
GPT-4o
63.8
36.8
50.2
94.2
74.6
63.9
Mulberry-77 7B 666
63.1
22.8
39.6
69.1
65.2
52.0
Virgo-7B 36
62.3
24.0
36.7
77.4
71.4
54.4
MindGYM 10
70.3
28.6
48.4
84.1
68.4
60.0
OpenVLThinker-777
70.2
25.3
47.9
78.6
67.4
57.9
NoisyRollout [14
72.9
28.9
52.8
79.0
68.4
60.4
MM-EUREKA
73.0
26.9
50.3
82.5
66.8
59.9
Qwen2.5-VL -nn-ntsuttttt
67.8
24.7
44.5
83.6
67.4
57.6
EvolvedGRPO (Ours)
74.0
30.8
51.8
85.1
73.2
63.0
performance gap with GPT-4o (63.0 vs. 63.9 on average), our method surpasses all other training methods based on Qwen2.5-V -I--ttttttntt exhibiting a distinct advantage. It consistently improves upon the base model across all reasoning benchmarks, highlighting its robust reasoning ability.
5.3 Main Results on Downstream Generalization
Table 2: Main results on general tasks.
Table 2 summarizes the results of our EvolvedGRPO and other baselines on downstream general benchmarks.
Models
MMMU
MMStar
A12D
GPT-4o
Overall, our method generalizes well to unseen down- stream tasks even without task specific training, thanks the improved modii-modal reasoming capabilities brought by our evolved instructions. However, on challeng- ing benchmarks like MMMU, all existing reasoning ap- proaches still fall short compared to GPT-4o, potentially due to the inherent gap between fine- fine-grained perception and deep reasoning. We expect future work to explore richer instruction types that bridge this gap and enable reasoning grounded in fine-grained visual understanding.
Mulberry-7B 66
55.0
61.3
80.1
Virga-7H 36
46.7
MindGYM
64.3
Open hinker
31.9
63.2
82.7
MMMEUREKA 1
52.3
64.1
81.1
50.6
82.6
EvolveGGRPO (Ours)
33.0
62.7
83.3
5.4 In-depth Analysis
Analysis of Model Training Paradigms To investigate our EvolvedGRPO deeply, we study the ablation variants of different factors in Table Specifically, analyze their independence using the following ablation strategy: (1) Base model: we present the performance of the original Qwen2.5- VL-Instruct model baseline for comparison.
(2) SFT we perform Supervised Fine-Tuning on the base model using the K12 dataset for supervised learning.
(3) GRPO: we apply Grouped Relative Policy Optimization the base model using the K12 dataset for reinforcement learning. (4) ProgressiveGRPO: we build upon GRPO based reinforcement learning by introducing curriculum- based dynamic instruction scaling the K12 dataset, which continuously augments the training data with more instructions, without adversarial RL. (5) EvolvedGRPO: the method proposed in this work. It can be observed that: Direct SFT has limited effectiveness in enhancing the model's reasoning capabilities. While direct fine-tuning via SFT improves performance on reasoning benchmarks (+4.9% overall), fails guide the model developing better reasoning strategies. Furtheremore, RL methods also encounter bottlenecks during the scaling process. GRPO demonstrates superior learning performance compared to SFT (+6.6% overall); however, still encounters bottlenecks during the training process. 2) The effectiveness of the GRPO algorithm can be enhanced through multi-stage guided data augmentation. The progressive increase of edited instructions in curriculum learning manner significantly enhances the effectiveness of reinforcement learning training, achieving improvements across all benchmarks (+8.9% overall). 3) The introduction of adversarial reinforcement learning improves both instruction quality and the model's 's reasoning capabilities. This iterative refinement leads to improved training efficiency and progressively stronger reasoning capabilities, ultimately eliciting the model's full reasoning potential( + 12.1% overall).

---

Table Results of ablation study illustrate the effect of individual strategy.
Methods
Vista
MathVision
Math Verse 1
GSM8K
MATH500
Rel. 1
Base Model
67.8
24.7
44,5
83.6
67.4
100.0%
SFT GRPO
69.1
28.4
48.2
84.0
66.7
104.9%
71.9
28.9
49.6
82.9
67.0
106.6%
ProgressiveCRPO
72.8
29.5
51.2
83.7
69.1
108.9%
EvolvedGRPO
30.8
51.8
85.1
112.1%
Scaling Response Length gihh
Scaling Validation Accuracy
540
EuoivedGRPO
teclvatstro
16.1
Responze Lengos
500
800
6:.1
59.7
Velcaaion
59
200
57.6
400
20
Step
Round
(a) Scaling Response Length
(b) Scaling Validation Accuracy
Figure Comparison the Response Length and Validation Accuracy
Scaling Visualization of Progressive Evolved Instructions. In Figure 2 illustrate the changes in reasoning chain length and model performance during training. Our method enables continuous growth in reasoning length. After each round of reinforcement learning, the model progressively adapts to longer reasoning chains, smoothly transitioning from k-step reasoning to k++1-step reason- ing. In contrast, conventional RL-based methods tend to stabilize after the first round of adaptation, with reasoning lengths declining. As the reasoning length increases, our method consistently improves accuracy on the validation dataset. After each training round, the model successfully adapts the current difficulty level while maintaining reasonable accuracy rate. In comparison, the direct GRPO approach shows only marginal improvements beyond the second round, reflecting limited learning progress. We further provide a statistical analysis in Appendix C
6 CONCLUSION
this work, we propose EvolvedGRPO, a progressive instruction evolution framework that constructs increasingly challenging multi-modal reasoning datasets via evolved editing instructions to match the model's growing reasoning capabilities. We conceptualize the evolution multi-move reasoning instructions as multi-step editing across image and text modalities, and enhance it via adversarial rewards that jointly optimize the question generator and the answering model. Extensive experimental results demonstrate that method progressively scales the instruction complexity without external supervision and continuously unlocks LVLMs' reasoning depth and adaptability across various tasks.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (62436007), the Key R&D Projects in Zhejiang Province (No. 2024C01 106, 2025C01030), the NSFC (62272411). and the Zhejiang NSF (LRG25F020001) the Fundamental Research Funds for the Central Universities (226-2025 2025-00117, Ningbo Yongjiang Talent Introduction Programme (2024A-401-G) Zhejiang University Education Foundation Qizhen Scholar Foundation.
10

---

References
[1] OpenAI, A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda,
A. Hayes, A. Radford, A. Madry, A. Baker-Whitcomb, A. Beutel, A. Borzunov, A. Camey, A. Chow,
A. Kirillov, A. Nichol, A. Paino, and ct al., "Gpt-4o system card," arXiv preprint 2111:22410. 2024.
[2] D. Guo, D. Yang, Zhang, J. Song, R. Zhang, R. Xu. Q Zhu, S. Ma, P. Wang, X. Bi, and et al., "Deepseek- rl: Incentivizing reasoning capability in Ilms via reinforcement learning," arXiv preprint arXiv:2501. 2025.
[3] Y. Peng, G. Zhang, M. Zhang, Z. You, J. Liu, Q Zhu, K. Yang, X. Xu, X. Geng, X. Yang, and et al., "Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rulc-based rl," arXiv preprint arXiv:2503.075.. 07536 2025.
[4] F. Meng, L. Du, Z. Liu, Z. Zhou, Q. Lu, D. Fu, T. Han, B. Shi, W. Wang, J. He, K. Zhang, P. Luo, Y. Qiao, Q. Zhang, and W. Shao, "Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforeement learning," arXiv preprint arXiv:2503. 07366, 2025.
[5] W. Huang. B. Jia, Z. Zhai, S. Cao. Z. Ye, F. Zhao, Z. Xu, Y. Hu, S. Lin, and et al.. "Vision-rl: Incentivizing reasoning capability in multimodal large language models," " arXiv preprint arXiv:2503.06749 2025.
[6] Y. Yang. X. He, H. Pan, X. Jiang, Y. Deng. X. Yang. H. Lu, D. Yin, F. Rao, M. Zhu, and et al., "RI- onevision: Advancing generalized multimodal reasoning through cross-modet formalization," arXiv preprint arXiv:2503. 10675, 2025.
[7] H. Shen, Liu, Li, C. Fang, Y. Ma, J. Liao, Q. Shen, Z. Zhang, K. Zhao, Q. Zhang, and et al., "Vim-rl: A stable and generalizable rl-style large lanion-la model," arXiv preprint arXiv:2504.06615 2025.
[8] Y. Deng, H. Bansal, F. Yin, N. Peng. W. Wang, and K.-W Chang, "Openvithinkkrr An early exploration to complex vision- Laiiual reasoning via iterative self improveeeet, arXiv preprint arXiv:2503.13322 2025.
[9] H. Chen, H. Tu, F. Wang, H. Liu, X. Tang, X. Du, Y. Zhou, and C. Xie, "Sft or rI? an early investigation into training rl-like reasoning large vision- vision-language models," arXiv preprint arXiv:2504. 01044//168,, 2025.
[10] Z. Xu. D. Chen, Z. Ling. Y. Li, and Y. Shen, "Mindgym: Enhancing vision-language models via synthetic self-challenging questions," arXiv preprint 2xXx:2553300499, 2025.
[11] J. Zhang. L. Xue, L. Song. J. Wang, W. Huang. M. Shu, A. Yan, Z. Ma, J. C. Niebles, S. Savarese, C. Xiong, Z. Chen, R. Krishna, and R. Xu, "Provision: Programmatically scaling vision-centric instruction data for 2024.
multimodal language models," arXiv preprint arXiv:2412.07000
[12] Y. Liu, Y. Cao. Z. Gao, W. Wang, Z. Chen, W. Wang, H. Tian, L. Lu, X. Zhu, T. Lu, Y. Qiao, and J. Dai, "Mminstruct: high-quality multi-modal instruction tuning dataset with extensive diversity," Science China Information Sciences, vol. 67, no. 12, Dec. 2024. [Online]. Available: http://dx.dii.orr//// 0ttpp/0..000//1112-0224-22
[13] R. Luo, H. Zhang, I. Chen, T.-E. Lin, X. Liu, Y. Wa, M. Yang, M. Wang, P. Zeng, L. Gao, H. T. Shen, Y. Li, X. Xia, F. Huang, J. Song, and Y. Li, "Mmevol: Empowering multimodal large language models with evol-instruct," arXiv preprint ar Xiv:2409. 05840, 2024.
[14] X. Liu, Ni, Z. Wu, C. Du, I.. Dou, H. Wang, T. Pang, and M. Q. Shich, "Noisyrollout Reinforcing visual reasoning with data augmentation," arXiv preprint arXiv:2504. 13055, 2025.
[15] L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang, "Diffusion models: A comprehensive survey of methods and applications," arXiv preprint arXiv:2209.0077 2024.
[16] Y. Zuo, K. Zhang, S. Qu, L. Sheng, X. Zhu, B. Qi, Y. Sun, G. Cui, N. Ding, and B. Zhou, "Tul: Test-time reinforcement lcarning"" arXiv preprint 2504 16084, 2025.
[17] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, "Minigpt-4; Enhancing vision- laiiuul understanding with advanced large language models," arXiv preprint arXiv:2304.1055 2023.
[18] Q. Ye. H. Xu, J. Ye, M. Yan, A. Hu, H. Liu, Q Qian, J. Zhang, and F. Huang, "mplug-owID ow12: Revolution- izing multi-modal large language model with modality collaboration," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 040-133 051.
[19] J. Bai. S. Bai. S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, "Qwen-vl: A frontier large vision- languae model with versatile abilities." arXiv preprint a30iv: 12966, 2023.
11

---

[20] A. Yang. B. Yang, B. Hui. B. Zheng, B. Yu, C. Zhou, Ci, C. Li. D. Liu, F. Huang et al., "Qwen2 technical report," arXiv preprint arXiv:2407. 10607, 2024.
[21] M. He. Y. Liu. B. Wu, J. Yuan, Y. Wang. T. Huang, and B. Zhao, "Efficient multimodal learning from data-centrie perspective." arXiv preprint arXiv:2402.11530 2024.
[22] Q. Yu, J. Li, L. Wei, L. Pang, W. Ye, B. Qin, S. Tang. Q. Tian, and Y. Zhuang, "Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 12944- 12953.
[23] S. Bai, K. Chen, X. Liu, J. Wang. W. Ge, Song. K. Dang, P. Wang. S. Wang. J. Tang, H. Zhong, Y. Zhu, M. Yang. Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Yc, X. Zhang. T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, "Qwen2.5-vl technical report," arXiv preprint arXiv:2502. 13923, 2025.
[24] Q. Yu, Shen, Z. Yue, Y. Wu, W. Zhang. Y. J. Li, Tang, and Y. Zhuang, "Mastering collaborative multi-modal data selection: focus on informativeness, uniqueness, and representativeness; " arXiv preprint arX:::2412.00 2,2.06293, 2024.
[25] Li, S. Tang, L. Zhu, W. Zhang, Y. Yang. T.-S. Chua, F. Wu, and Y. Zhuang. "Variational cross-graph reasoning and adaptive structured semantics learning for compositional temporal grounding." " IEEE Transactions on Puttern Analysis and Machine Intelligence, vol. 45, no. 10, pp. 12 601-12617, 2023.
[26] K. Pan, S. Tang. J. Li. Z. Fan, W. Chow, S. Yan, T.S. Chua, Y. Zhuang, and H. Zhang, "Auto-encoding morph-tokens for multimodal Ilm," arXiv preprint arXiv:2405.01996, 2024.
[27] Q. Y., Li, Y. Wu, S. Tang, W. Ji, and Y. Zhuang, "Visually-prompted language model for fine-grained scene graph generation in an open world." in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2023.
[28] K. Pan, W. Lin, Z. Yue, T. Ao, L. Jia, W. Zhao, J. Li, S. Tang, and H. Zhang, "Generative multimodal pretraining with discrete diffusion timestep tokens," arXiv preprint arXiv:2504. 14666, 2025.
[29] J. Li, K. Pan, Z. Ge, M. Gao. W. Ji, W. Zhang, T.-S. Chua. S. Tang. H. Zhang, and Y. Zhuang, "Fine-tuning multimodal ilms to follow zero-shot demonstrative instructions," " in Proceedingy of the Twelfth International Conference on Learning Representations (ICLR), 2024.
[30] X. Yue, Y. Ni, K. Zhang, T. Zheng. R. Liu, G. Zhang. S. Stevens, D. Jiang. W. Ren. Y. Sun, and et al., "Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi," " in Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[31] L. Chen, J. X. Dong, P. Zhang, Zang, Z. Chen, Duan, J. Wang, Y. Qiao, D. Lin, and et al., "Are we on the right way for evaluating large vision-language models?" arXiv preprint arXiv:2403. 203300, 2024.
[32] Y. Liu, Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Liu, and al., "Mmbench: Is your multi-modal model an all-around player?" arXiv preprint arXiv:2307.06288 2023.
[33] W. Yu, Z. Yang, I. Li, J. Wang, K. Lin, Z. Liu, X. Wang, I... Wang, and et al., "Mm-vet: Evaluating large multimodal models for integrated capahilities," arXiv preprint arXtv:2308.0244: 2024.
[34] L. Ouyang, J. Wu, X. Jiang, Almeida, C. Wainwright, P. Mishkin, C. Zhang. S. Agarwal, K. Slama, A. Ray, and et al., "Training language models to follow instructions with human feedback," Advances in Neural Information Processing Systems, vol. 35, pp. 27 730-27 744, 2022.
[35] H. W. Chung, L. Hou, S. Longpre, B. Zoph. Y. Tay. W. Fedus, E. Li, X. Wang, M. Dehghani. S. Brahma, and et al., "Scaling instruction-fineture language models." arXiv preprini arXiv:2210.11411 2022.
[36] Y. Du, Z. Liu, Y. Li, W. X. Zhao, Y. Huo, B. Wang, W. Chen, Z. Liu, Z. Wang. and J.-R. Wen, "Virgo: A preliminary exploration on reproducing like milm." arXiv preprint (xXXi:2501101904 2025.
[37] R. Zhang, B. Zhang. Y. Li, H. Zhang, Z. Sun, Z. Gan, Y. Yang, R. Pang, and Yang, "Improve vision language model chain-of-thoupe reasoning." 2024.
[38] C. Mitra, B. Huang, T. Darrell, and R. Herzig, "Compositional chain-of-though prompting for large multimodal models." in Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2024, pp. 1440--1443.
[39]] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q V. Le, and D. Zhou, "Chain of thought prompting elicits reasoning in large language models." in Advances in Neural Information Processing Systems (NeurIPS 2022). vol. 35, 2022, pp. 24824-244 8337.4837.
12

---

[40] R. Zhang, B. Zhang. Y. Li, H. Zhang, Z. Sun, Z. Gan, Y. Yang, R. Pang, and Y. Yang, "Improve vision language model chain-of-thought reasoning." arXiv preprint arXiv:2410. 16198, 2024.
[41] B. Luan, H. Feng. H. Chen, Y. Wang, W. Zhou, and H. Li, "Textcot: Zoom in for enhanced multimodal text-rich image understanding." arXiv preprint (rxiv:2404.0979) 2024.
[42] Q Yu, J. Li, W. Ye, S. Tang, and Y. Zhuang, "Interactive data synthesis for systematic vision adaptation via aime-aiges collaboration," arXiv preprint arXiv:2305. 12799, 2023.
[43] K. Pan, Z. Fan, Li, Q. Yu, H. Fei, S. Tang. R. Hong. H. Zhang, and Q. Sun, "Towards unified multimodal editing with enhanced knowledge collaboration," " in Advances in Neural Information Processing Systems (NeurIPS), 2024.
[44] T. Cheng, L. Song, Y. Ge. W. Liu, X. Wang, and Y. Shan, "Yolo-worid: Real-time open-vocabulary object detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 16991-16911.
[45] N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, Ma, H. Khedr, R. Radle, C. Rolland, I.. Gustafson al., "Sam 2: Segment anything in images and videos," arXiv preprint arXiv:2408.00714 2024.
[46] Yu, Z. Wang, Vasudevan, Yeung, M. Seyedhosseini, and Y. Wu, "Coca: Contrastive captioners are image-text foundation models," 2024, arXiv preprint, to be added if available.
[47] J. S. Park, Z. Ma. L. Li, C. Zheng. C.-Y. Hsieh, X. Lu, K. Chandu, N. Kobori, Q. Kong, A. Farhadi, R. Krishna, and Y. Choi, "Robin: Dense scene graph generations at scale with improved visual reasoning." 2024, to appear or preprint, add venue if available.
[48] L. Yang. B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, "Depth anything v2," arXiv preprint Mxiiv:4066099144 2024.
[49] Q. Yu, W. Chow, Z. Yue, K. Pan, Y. Wu, X. Wan, J. Li, S. Tang, H. Zhang, and Y. Zhuang, "Anyedit: Mastering unified high-quality image editing for any idea," arXiv preprint urXiv:2411.15738, 2025.
[50] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," " arXiv preprint arXiv:1707.06344 2017.
[51] Y. Yang, S. Zhang, W. Shao, K. Zhang, Y. Bin, Y. Wang, and P. Luo, "Dynamic multimodal evaluation with flexible complexity by vision-language bootstrapping," arXiv preprint arXiv:2410.08699 2024.
[52] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo, "Deepseekmath: Pushing the limits of mathematical reasoning in open language models," arXiv preprint arXiv:240..03300. 2024.
[53] hiyouga, "Mathruler," https://ghhuu.coom//hgogggagaaau
2025.
[54] I. J. Goodffllow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Farde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial networks," arXiv preprint 1rXiv:1406.2611, 2014.
[55] R. Zhang, X. Wei, D. Jiang, Z. Guo, S. Li, Y. Zhang, C. Tong, Liu, A. Zhou, B. Wei et al., "Mavis: Mathematical visual instruction tuning with an automatic data engine," arXiv preprint (xXiv22407.08733 2024, 2024b.
[56] P. Lu, R. Gong, S. Jiang, L. Qiu, S. Huang, X. Liang, and S.-C. Zhu, "Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning." arXiv preprint arXiv:2105.04103 2021.
[57] L. Deng, Y. Liu, B. Li, D. Luo, L. Wu, C. Zhang, P. Lyu, Z. Zhang, G. Zhang. E. Ding et al., "R-cot: Reverse
problem generation for geometrie reasoning in large multimodal models," arXiv preprint arXiv:2410. 17885, 2024.
[58] S. Peng, D. Fu, I.. Gao, X. Zhong, H. Fu, and Z. Tang, "Multimath: Bridging visual and mathematical reasoning for large language models." " arXiv preprint arXiv:2409.0014 2024.
[59] R. Zhang, D. Jiang, Y. Zhang, H. Lin, Z. Guo, P. Qiu. A. Zhou, P. Lu, K. Chang, P. Gao, and et al.. "Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems?" arXiv preprint arXiv:2403. 14624, 2024.
[60] K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, H. Li, and et al., "Measuring multimodal mathematical reasoning with math vision dataset," in NeurIPS Datasets and Benchmarks Track, 2024.
13

---

[61] P. Lu, H. Bansal. T. Xia, J. Liu. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, J. Gao, and et al.. "Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts," in International Conference on Learning Representations (ICLR), 2024.
[62] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, Hilton, R. Nakano, C. Hesse, and Schulman, "Training verifiers solve math word problems." arXiv preprint arXiv:2110.1166,, 2021.
[63] H. Lightman, V. Kosaraju, Burda, Edwards, B. Baker, T. Lee, Leike, J. Schulman, Sutskever, and K. Cobbe, "Let's verity step by step," arXiv preprint arXiv:2305.20050 2023.
[64] A. Kembhavi, Salvato, Kolve, Seo, H. Hajishirzi, and Farhadi, diagram is worth a dozen images," arXiv preprint urXiv: 160.00960 2016.
[65] Kwon, Li, S. Zhuang. Sheng. L. Zheng, C. Yu, J. E. Gonzalez, H. Zhang, and Stoica, "Efficient memory management for large language model serving with pagedattention," in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[66] H. Yao, Huang. W. Wu, J. Zhang, Y. Wang. S. Liu, Y. Wang, Y. Song. H. Feng, L. Shen, and D. Tao, "Mulberry: Empowering mllm with ol llke reasoning and reflection via collective monte carlo tree search," arXiv preprint arXiv:2412. 18319, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in APPENDIX E.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We give the proof in section 4.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We give experimental setup and implementation details Section 5.1 and Appendix B.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: The codes are provided in supplemental materials. The benchmark data comes from the open-source dataset.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipps.cc/pulli//uuui guwde//ddee/// Cot//ooodeeesiseeeeeeeeee wwwpibmmsssioooooooooooooooi for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we have provided necessary implementation details of our method in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: To ensure a fair comparison with the baseline method, we strictly adhere to the testing settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: we have provided experiments compute resources of in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httph: http://www......aaaa///cc/sscsaa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, follows the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the Broader Impacts in Appendix F.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We include the license of each asset in Appendix B.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

Appendix
Our appendix is organized follows.
More Theoretical Analysis on EVOLVEDGRPO
Section Ap.
Experimental details of our EVOLVEDGRPO (Sccionn B
External Effectiveness Analysis of EVOLVEDGRPO (Section C).
Detailed Evolution Illustration EVOLVEDGRPO Training Pipeline (Section
Qualitative examples of EVOLVEDGRPO and BASE MODEL (Section
Limitations of EVOLVEDGRPO
Section F)
A
More Theoretical Analysis on EvolvedGRPO
A.1 Problem Setup
We provide a theoretical analysis of the convergence behavior of EvolvedGRPO, which alternates training between the question generator policy To and the answer model policy πA under the Grouped Relative Policy Optimization (GRPO) framework.
We next prove that the adversarial training process is convergent for any finite number k instruction editing steps, which allows to generalize the convergence to the entire model training process.
Let TQ be the policy that generates sequence of instructions I (. I&) for given sample = (go. 0o, vo). Let A be the answer model policy generating answer conditioned on multimodal input (g. v). The expected total reward objective is:
I (TA, mq)=
(x. RR' (TTA qk, Uk, ax)]
(10)
where the reward function R' is defined
&
R' format's
log sim c. v2-1) +
Σ
log n ( - log max (TA Uk),
test
(11)
The objective of EvolvedGRPO is to solve the following min-max problem:
min max [[TA, #Q)-
(12)
mg
When optimizing the answer model policy TΛ. the training sample (gk, Uks as) fixed, making most reward components fixed πA. Specifically, the term: Eege vv.))) Σ log sim(us, v-11) Σt log a (a: is decided by old policy Aasi and TQa.s being independent of current policy TA. and can be treated as a constant during training. Therefore, the reward function effectively reduces to maximize Eig .t.)a))) | (g,, vx)]- This simplificatinn allows us to focus the GRPO update on maximizing the likelihood the final answer token under the current dnstruction-condiiin context and baseline for shaping the total reward but do not influence gradients with respect to TA
policy. All preceding terms serve only as
A.2 Proof of Convergence for GRPO
Theorem A.1 (Convergence of GRPO). Let ( GRPO. Then, Then, under the standard assumption that each update approximately maximizes the GRPO objective, the policy sequence ne, converges to stationary policy n².
120 be the sequence of policies updated using
Proof. Let (T(T)) E.- D. [(x y)] denote expected return of policy v and we have:
TORPO (0;0() = Ez- D.yroy (11)) [min (rA(x, y), clip(r, A)A(z, v)) - B ( (13)
where r is importance sampling ratio na, (2.0)
Te(2.9) and A A(x, y) is the normalized groep- advantage.
22

---

Since each update approximately maximizes (f). we have:
JoRpo(O 02) TaRFO 0z).
(14)
Following the standard policy improvement argument, the change in return satisfies:
TT (T(T)))) ENND
E Eynneet (11x)
[I, U) E Bynmo. (11x) [R(x,8)]
(15)
To..: (3/(z)
R(x,, w)
EynD
E By (113)
(1 (R())
(16)
To. (3)(2))
TB+-1 (y)(z)
1) R(x,2)
BrnD, (-12)
(17)
Te, (y/22)
Tects (y/1z)
R(x, v)
2 Bz youter (12)
log
(18)
no. (3)(r)
Since GRPO includes an explicit KL penalty term via clipping, we can apply the Pinsker inequality:
(To, To
|| | ne, ll
(19)
Combine the definition of equation
1
inequality
14 and incquality 19 we can get:
3 To... To, 11.
TT(004))) ) - mT(Fo)) 2 2
(20)
As (TT,))) is upper bounded (since R(x, 4) E [Rmin: Rmaa)) and non-decreasing according to inequality 15, it converges to a limit 17". Thus, we have:
lim To__ - Te. 1l1 O. 1000
(21)
Since the space of probability distributions over discrete outputs is compact under the Li norm, it follows that the sequence To, converges to a stationary policy n².
A.3 Proof of Convergence for EvolvedGRPO
Definition A. (Local Nash Equilibrium). A pair of policies (Ty) T²)) is local Nash equilibrium of J [[Q AA) if there exist neighborhoods NQ and NA such that:
J( TO) S TQ) VIAA E NA. J J( 2 (Q)
(22)
VTQ E NQ..
(23)
Since the policies sequentially optimize their respective objectives, we have:
J( ((+1) (0) - [(T
(1)
(24)
((+1) T _ ((+1)) 2 I (TT (1+1) (1)
(+1))
, ,5
(25)
As we use DKa[Tol|Ta Trer(or,1l 0vct) log Tre (o.
to prevent the model from
To(0,, 14, oic()
T0(0r,119, oz<()
forgetting previously acquired knowledge, while the answer model has the capability to master the reasoing skill in k-step reasoning through GRPO training. We can make the following assumption:
Assumption 1 (Answer Model Consistence). The answer model exhibits monotonic improvement in reasoning performance for all editing instructions generated by the question generator during training. Formally, for any iteration
(++1) [(T .TQ) s J( (T)).
Y TQ:
(26)
23

---

Table Training hyper-parameters of EvolvedGRPO.
Hyper-parameters
Question Generator
Answer Model
LLM Init
1ren2.5-7B-LL-Insteee
Qwen2.5-7B-VL--nnse
KL Penalty
Low Variance KL
Low Variance KL
KL Coefficient
X 10
10-2
Optimizer
AdamW
Adam W
Learning Rate
1 x 10-6
1 10-6
Weight Decay
1 10-2
1 x 10
Numerical Precision
AMP32
AMP32
Gradient Clipping
1.0
1.0
Rollout
5
5
Rollout Temperature
1.0
1.0
Rollout Top-p
0.99
0.99
Rollout Batch Size
512
512
Micro Batch Size for Update
16
16
Micro Batch Size for Experience
64
64
Training Steps
1
30
Resource Usage
4x RTX A6000
4x RTX A6000
Total Epochs
5
Therefore, the accuracy of the answer model is guaranteed to be non-decreasing different instruction editions are applied. Consider the space of k-step instruction editing strategies is finite, while Theorem A. suggested the convergence of the answer model under the condition of finite k-step edition training datasets, the final answer model can also converge. In this situation, the final answer model cannot improve its performance under क by changing its policy π Furtheremore, have:
V VTQ
lim A + (TQ) 1000
(27)
During the evolution of policy TQ and a consistently converges to its optimal mode. Then we only consider the policy ioQ under the final condition TA according to the Theorem A. the TQ also converges to TQ therefore maximize [(T πQ). Finally, (T,, TQ) converge to (T *Q) which is Local Nash Equilibrium. That is, the final equilibrium corresponds is robust, and the questions are maximally adversarial, yet no longer effective in reducing accuracy.
state where the answer model
B Experimental Details
B.1 Detailed Training Hyperparameters
We train our models using GRPO strategy. The both models are initialized with Qwen2.5-VLL-7B. Instruct (23) The detailed hyper-parameterr used during training are summarized in Table 4 In the training process, the details of two different types of instructions are presented in Table
As shown in Table 6 report the time cost of each module in our training pipeline. To reduce the overhead introduced by data augmentation, we adopt the vLLM framework for efficient and paral- lelized inference, which significantly accelerates the instruction editing process. Compared to directly training with the raw model dataset, our method incurs only an additional 10.0% computational cost.
To enhance the model's reasoning capability, we use the template "You FIRST think about the reasoning process as an internal monologue and then provide the final answer. The reasoning process MUST BE enclosed within <think> </think> tags. The final answer MUST BE put in \boxed(]." J." for answer model during both training and evaluation. During reward computation, format reward calculated, and only the content enclosed within "Aboxed(] ]" considered the definitive answer.

---

Table Comparison of representative methods for image and text editing.
Attribute
Image Editing
Text Editing
Input Type
Image + Instruction
Text + Instruction
Output Type
Image
Text
Editing Target
Pixel-level changes
Sentence Addition
Table 6: Resource cost statistics for each stage of the training pipeline using x RTX A6000. Stage 1 involves question generation, Stage trains answer model, and Stage 3 trains question generator.
Training Stage
Training Time (hours)
Stage I:
Instruction Generation
Stage 2: Answer Model Training
60
Stage 3: Question Generator Training
2
B.2 Training Dataset Description
MMK12 encompasses mathematical problems across various knowledge domains, including geome- try, functions, spatial reasoning, and more. Some important categories are introduced as follows:
This task requires models understand function concepts, analyze function graphs and expressions, and apply function properties to solve problems. This type of reasoning develops the model's 's ability to comprehend abstract mathematical concepts, fostering its capability to identify function characteristics, determine critical points, and analyze function behavior.
Function Reasoning:
This task involves applying spatial relationships, geometric theorems, and properties of shapes. Through geometric reasoning training, models enhance their spatial visualization, logical deduction, and formalization abilities for geometry problems, enabling them to solve complex problems both plane and solid geometry.
Geometric Reasoning:
This type of task focuses on understanding flow diagrams and recognizing patterns in visual sequences. Models need to discover pattems, predict rule-bat cule-based changes, understand logical relationships in visual content. This task examines the model's pattern recognition abilities, inductive reasoning skills, and visual logical thinking.
Pattern Reasoning:
Table 7: Comparison of datasets in terms of scope, type, image source, QA source, CoT answer source, and dataset size.
Dataset
Scope
Type
Img. Source
QA Source
CoT Answer Source
Size
MAVIS Gco3k
Geo & Fune
MCQ & FB
Synthetic
Synthetic Engine
GPT-40
20K
Geo
FB
Real World
Real World
None
3K
RCOT
Gen
MCQ
Synthetic
Synthetic Engine
GPT-4o
10K
MultiMath
Diverse
MCQ
Real World
GPT-4o
GPT-4o
6.4K
MMK12 4
Diverse
FB
Real World
Real World
Real World
15.6K
B.3
Evaluation Details
To further assess the reasoning capabilities of the models, we construct a validation set comprising 3,000 instances, sampled from multiple benchmarks with 500 instances drawn from each. Detailed information about the benchmarks is provided in Table The results of the validation experiments reported in Section 5.4 and Appendix
C Effectiveness Analysis of EvolvedGRPO
In this section, we discuss the effectiveness of EvolvedGRPO for improving the reasoning LVL.Ms with low overhead.
25

---

Table Summary of benchmarks for multimodal and mathematical reasoning evaluation.
Benchmark
Task Description
Modality
Size
Source
Math Vista- mini
Visual math reasoning
Image+Ters Text
1.0K
Math Vista
Math Vision-full
Diagram- based math QA
Image+Text Text
3.0K
MATH-Viss Vision
Math Verse-mini
Multiformat math QA
Image+Tere
3.1K
Math Verse
GSM8K
Grade-school math QA
Text
8.5K
OpenAl
MATH500
Competition-level math QA
Text
500
MATH Dataset
Teli
Math QA
Multi-modal
3.0K
Table 9; Effectiveness of each equation.
Methods
MathVision Vista 1
Math
GSMSK
MATH500
Rel.
EvolveGGRPO (Ours)
74.0
30.8
51.8
85.1
73.2
112.1
2 w/o Image Edition
73.4
29.8
51.3
84.5
72.5
110.6%
w/o Edition
72.1
31.8
50.9
81.3
109.7%
4
w/o Image Edition Text Edition
71.9
28.9
49.6
82.9
67.0
106.6%
5 Qwen2 5-VL-7BBIntttruc
71.9
28.9
49.6
82,9
67.0
106.6%
Table 10: Avg performance all domains on EvolvedGRPO disciplinary reasoning.
Models
Physies
Chemistry f
Biology f
Avg.
EvolvedGRPO (Ours)
61.5
65.5
63.9
63.6
OpenneThi
53.8
60.6
65.0
59.8
MM-Eureka
56.2
65.2
65.2
62.2
Owen2.5-VL-7B-Insstuut
45.4
56.4
54.0
51.9
+ GRPO
51.6
57.1
58.6
55.8
+ EvolvedGRPO w/iter
51.4
56.4
57.2
55.0
+ EvolvedGRPO w/iter 2
54.2
60.8
57.6
57.5
C.1 Effectiveness Analysis
To examine the effectiveness each type of instruction, we first remove the image editing instructions and text editing instructions separately, and observe the change in model performance after training. The complete results are presented in Table 9 We can find that both text editing instruction (+4.0%) and image editing instruction (+3.1 %) effective for EvolvedGRPO.
C.2 Experimental Results in a Wider Range of Fields
To demonstrate the framework's ' generality and effectiveness, leverage Owen2.5-VLL-7-- to perform complex editing instructions across various vision-language tasks. EvolvedGRPO continues demonstrate its effectiveness across multiple real-world domains such as physics, chemistry, and biology. The results in Table 10 provide strong evidence that our method effectively decouples visual and textual information diverse areas, leading significant enhancement of the model's comprehensive reasoning abilities.
Furthermore, the gradual improvement in model performance during the first two rounds of training Table [0]also demonstrates the effectiveness of multi round data augmentation training.
D Detailed Evolution Illustration of EvolvedGRPO Training Pipeline
D.1 Theoretical Analysis
For training data that the model has repeatedly encountered, the accuracy can approach 100% over multiple learning iterations for part of the data. In such cases, since each evaluation samples only limited responses and the reward is pariance-normalized. repeated training causes the responses to converge. When all responses are correct, the variance becomes 0 and is thus ignored, lcading to
26

---

Average Reward vs. Training Steps
0.700
GRPO
0.675
EvolvedGRPO
Average Reward
0.650
0.625
0.600
0.575
0.550
0.525
0.500
[10,19]
[20.29]
[30,39]
[40,49]
[50,59]
[60,69] [70,79] [80.89]
Step (training iteration range)
Figure 3: Average reward for the corresponding intervals.
training stagnation. Furthermore, in order to increase accuracy, the model may omit the reasoning process entirely or produce an incorrect reasoning process that nevertheless yields the correct answer, resulting in negative rewards that can cause training collapse.
In contrast, EvolvedGRPO introduce additional informational complexity during multi-round training, thereby increasing the data's 's informational entropy and increasing reasoning complexity. This process reduces the model's 's accuracy, strategically promoting an increase cognitive load that drives the model to engage in more nuanced multi-step inferential reasoning, preventing training stagnation and fostering further reasoning learning. Furthermore, the integration of additional information and escalation of difficulty within the synthetic data forces the model to extend its reasoning processes, impeding direct answer memorization and reliance on spurious memorization of erroncous inferential paths and thereby mitigating the risk of training collapse.
D.2 Experimental Analysis
In this part we calculate the average reward for the corresponding intervals in Figure
We observe that EvolvedGRPO exhibits greater fluctuations during each training round, it can consistently improve after data augmentation, thereby maintaining stable average performance without rapid decline. Specifically, we found that the reward EvolvedGRPO fluctuates continuously, with the average reward sharply decreasing after each data augmentation before gradually increasing. In contrast, the reward growth in GRPO slows down and eventually stagnates.
The analysis confirms that by generating more complex problems, EvolvedGRPO allows for richer variety of high-quality solutions, indicating more diverse and nuanced reward landscape that encourages exploration over exploitation.
D.3 Training Visualization
Since reward signals in multi-modal reasoning are closely tied the depth and structure of the reasoning process, changes in entropy at both the visual and textual semantic levels can effectively modulate the reward landscape. Specifically, altering the number of inference steps peee-ppattcllalyy in extended reasoning chains involving informative reward variations than shallow input perturbations.
4096 tokens-tends tends to induce more substantial and
In this section, we report the detailed evolution process of the first three rounds of EvolvedGRPO in Figure 4 Additionally, the number editing steps is increased once the reward stabilizes and the length of the reasoning chain remains unchanged, allowing for further difficulty scaling.
27

---

40000
(a) Round Neward-Formal
(b) Round Reward-Accuracy
(c) Round KL divergence
(d) Round Reward--Fooeee Formatt
(e) Round II Reward-Accurace
(1) Round II KL divergence
(g) Round III Reward-Formal
(h) Round III Reward-Accuracy
(i) Round III KL Divergence
Figure 4: Detailed Evolution Illustration of Reward-AAcuracy, Reward-Format, KL Divergence for the answer model over Round I, II, and III.
E Qualitative Examples
In Figure 5 demonstrate the advantages of EvolvedGRPO in complex reasoning problems such as geometry and statistics.
Limitations
Due to the limited capabilities of current image editing tools, approach to image manipulation remains relatively simple. In future work, we plan to explore hybrid training of LVLMs with more advanced image editing tools. This integration is expected further enhance the model's reasoning ability and enable more complex and realistic image transformations.
28

---

Question:
Please answer the question and provide the correct option letter, e.g., A, B, C, D, at end.
As shown the figure, P is point outside 30, PA PB intersect c two points C and respectively. It known that central angles of AB and CD are 90.0 and 50.0
respectively, then /P=
(A) 45"
(B) 40"
6
(C) 25"
(D) 20
Answer:
Qwen2.5-VL-7B.
To solve this prablem, we need to use the properties of angles in circle and the relationship between the central angles and the inscribed angies.
The central angle arc W Vroverss((//rowwj/AAe) \\ V 90MMcire W. The central angle of are V (frwwis)(( is V 50M W.
"'Idetify the given information:"
"Use exterior angle theorem:" The exterior angle W Vangle P \y sum the two inscribed
angles Vangle ADB \V and V Vangle WyAn W/n Vangle Vangle Wangle ACD 4crc
+ 25crci 70*Nccrre'n..
"CCorrct approach:"'11
angle V Venngle is difference between the angles formed by the
tangents the chords:'v
\/vi Wangle = 90Meeir 50%/ciic 40*lciir Option: Therefore, the correct
B
EvolvedGRPO:
<thinks
To solve for angle P, we need to use properties the angles circle the relationships
between central angles and the inscribed angles. vnin1. central angle of are 90", the inscribed angle ACB (which is half
central angle) 45" An2. The central angle of
50", inscribed central angle) 25" .n3. Since ACB and CAD are angles same the fact that the external angle
angle CAD (which is half segment, and triangle is equal the of the adj-anta internal angles.
the external angle these two angles, we
Thus, LACB / CAD 45" -55 20"
</think
Option: D
Question: Mary has written all the numbers from to 30 How many times has she written digit 2?
Answer: 13
Qwen2.5-VL-7B:
To determine how many times the digit appears in the numbers from to 30, we need to count occurrences the digit in each place value (units and Venns.. "Count the digit the units place: The numbers that have in the units place are: 12, 22, 32.W In There are such numbers. Vn'n2. "Count the digit in tens place:*
The numbers that have in tens place are: 20, 21, 22, 23, 24, 26, 28, 29.Yn There 10 such numbers. VrinAdding these together, the total number times the digit
appears iscvn\/44 4 10 WirinnSo, Mary written the digit total times.
Option:
EvoivedGRPO:
<thinks
To determine how many times the digit appears in the numbers from 1 to 30, we need to consider the occurrences of the digit 2 in both the tens place and the units place.Inin1. "Units place": The digit 2 will appear in the units place in the numbers 2, 12, 22, So, it appears 3 times.n2 \n2 "Tens place": The digit 2 will appear in the tens place in the numbers 20, 21, 22, 23, 24, 25, 26, 27, 28, 29. So, appears 10 times. Annnnddinnn these together, the digit 2 appears VV3 10 = 13W) times.
</think>
Option:
Figure Example of complex reasoning. Red highlights Qwwen2.5VVL-7I-Innsmm reasoning errors, Green shows ExolvedGRPO-7B: correct answer.
29


Reviews:
[{"review_id": "M2n13E0LGy", "reviewer": "Reviewer_ZaKE", "summary": "The paper introduces EvolvedGRPO, a new framework designed to enhance the reasoning capabilities of VLM. The authors identify a key problem with existing reinforcement learning methods like GRPO: as LVMs improve, the reward scores for different answers to the same multi-modal question tend to converge. This lack of reward variance hinders effective learning. To solve this, EvolvedGRPO progressively generates more complex problems that are matched to the model's current abilities through an adversarial process involving a \"question generation model\" (Q) and an \"answer model\" (A).  The two models are trained adversarially, with Q rewarded for making challenging yet solvable problems and A rewarded for correct answers, while a curriculum learning strategy gradually increases the number of editing instructions to guide the model toward more complex reasoning.", "questions": "See Weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is well-written and easy to follow.\n- The paper identifies and addresses a specific, critical bottleneck in training LVMs.  The proposed EvolvedGRPO framework is an innovative solution that tackles this by progressively increasing the difficulty of training data to match the model's growing capabilities.\n- The authors demonstrate its EvolvedGRPO effectiveness across a wide range of benchmarks.\n\n**Weaknesses**\n- Limited Scope of Demonstrated Instruction Evolution: The text editing instructions are primarily described as performing \"mathematical transformations\" to increase reasoning steps. This approach is well-suited for the mathematical and logical reasoning benchmarks tested, but its applicability to more abstract, qualitative, or common-sense reasoning tasks is less clear. \n- Concern over Quality Control and Cumulative Error: The framework's success is highly dependent on the quality and fidelity of the instructions produced by the question generation model (Q). Because the method increases problem difficulty through multi-step iteration , small errors or logical deviations introduced in early steps can be amplified in subsequent ones, potentially creating samples that are complex in appearance but logically flawed or unfaithful to the original intent. While the paper mentions using a judge model and rule-based checkers for filtering, it does not detail how robust this filtering mechanism is against subtle, accumulating errors that might not violate a simple rule but still corrupt the problem.", "was_revised": true, "final_justification": "The author has addressed my main issues.", "rebuttal": "We sincerely thank you for the valuable comments! We are encouraged to see that our work can enhance subsequent research endeavors. We will address your concerns point by point.\n\n> **Q1**: Limited Scope of Demonstrated Instruction Evolution. \n\nA1: Thank you for raising the important concern regarding the scope of demonstrated instruction evolution in our manuscript. We first leverage Qwen2.5-VL-7B-Instruct to perform more abstract (e.g., conceptual reasoning) and commonsense (e.g., real-world knowledge)-based editing instructions, and then showcase the universal adaptability and scalability of our evolved instruction generation process.\n\n(1.1) Evaluation on More Real-World Vision-Language Tasks Using Qwen2.5-VL\n\nWe conduct instructional evolution on various disciplinary reasoning and general medical QA tasks with Qwen2.5-VL-7B-Instruct.\n\nThe results are presented in Tables 1 and 2. Using Qwen2.5-VL-7B-Instruct as the backbone, EvolvedGRPO continues to demonstrate its effectiveness across multiple real-world domains such as physics, chemistry, and biology, outperforming all baseline methods on both disciplinary reasoning and general medical QA scopes. This confirms that EvolvedGRPO is model-agnostic and effective across various types of real-world vision-language tasks.\n\nTable 1: Avg performance of all domains on EvolvedGRPO disciplinary reasoning\n\n| Models                 | Physics | Chemistry | Biology | Avg. |\n| ---------------------- | ------- | --------- | ------- | ---- |\n| OpenVLThinker| 53.8    | 60.6      | 65.0    | 59.8 |\n| MM-Eureka | 56.2    | 65.2      | 65.2    | 62.2 |\n| Qwen2.5-VL-7B-Instruct | 45.4  | 56.4 | 54.0 | 51.9 |\n| +GRPO    | 51.6    | 57.2 | 58.6| 55.8 |\n| +EvolvedGRPO  | 61.4   | 65.4 | 64.0 | 63.6 |\n| +EvolvedGRPO w/iter 1 | 51.4 | 56.4 | 57.2 | 55.0 |\n| +EvolvedGRPO w/iter 2 | 54.2 | 60.8      | 57.6 | 57.5 |\n\nTable 2: Avg performance of all domains on EvolvedGRPO general medical reasoning\n\n| Models                 | MedXpertQA-MM | OmniMedVQA | Avg. |\n| ---------------------- | ------------- | ---------- | ---- |\n| Qwen2.5-VL-7B-Instruct | 20.3          | 58.4       | 39.4 |\n| +GRPO                  | 23.8          | 61.0       | 42.4 |\n| +EvolvedGRPO           | 24.3          | 62.8       | 43.6 |\n\n(1.2) Analysis on the scalability of EvolvedGRPO\n\nEvolvedGRPO has demonstrated robust performance across multiple domains by employing a single iteration in its operations. As suggested, we have conducted additional experiments with more iterations to examine its scalability. The outcomes showcase the universally adaptable and scalable nature of our evolved instruction generation process, as detailed in the last two rows of Table 1.\n\n\n> **Q2**: Concern over Quality Control and Cumulative Error.\n\nA2: Thank you for your questions. We will address them from three points:\n\n(2.1) In our process reward function, we explicitly penalize large discrepancies between step t and step 0 through model-inferred output discrepancies instead of naive rule-based methods. In this way, we adaptively reduce the cumulative error introduced at each step and mitigate the overall error accumulation in multi-step enhancement, even in more complex physical (+22.5% in Table 1) and medical domains (+10.7% in Table 2).\n\n(2.2) To better illustrate the effectiveness of quality control, we present detailed visualizations of reward dynamics during the multi-round training process in Appendix D. During each round of training, the reward associated with data augmentation initially drops significantly, then gradually increases as training progresses.\n\n(2.3) To better illustrate the reduction of Cumulative Error, we quantitatively analyze the accumulation of errors over time t during the progressive enhancement of batch samples. We measure this by the inverse of the similarity between step t and step 0 using human evaluation. The results are presented in Table 3.\n\nTable 3: Cumulative error during training\n\n| step         | 0    | 30   | 60   | 90   |\n| ------------ | ---- | ---- | ---- | ---- |\n| Accumulation | 1.0  | 1.06 | 1.08 | 1.08 |"}, {"review_id": "zB3KZ7f4aJ", "reviewer": "Reviewer_ZeLT", "summary": "One of the main problems with GRPO is that as a model becomes better, its answers become equally as good/accurate. This paper proposes to resolve the 'reward saturation' problem presented by GRPO by presenting a framework that: \n(a) Generates progressively more difficult multimodal problems by applying image/text edits using a question generator. The question generator is trained adversarially to propose image/text edits that (i) do not modify the original answer and (ii) are hard for an answer model to solve, but are still solvable.\n(b) Utilizes curriculum learning to apply increasing numbers of edits (k) so that the model's reasoning chain lengthens.\n\nThe authors test their method on five multimodal reasoning benchmarks (MathVista-mini, MathVision-full, MathVerse-mini, GSM8K, MATH500) and three broad multimodal benchmarks (MMMU, MMStar, AI2D). Their method, which fine-tunes Qwen2.5-VL-7B-Instruct, shows gains over baselines. In ablations, these gains are mostly attributed to the adversarial curriculum.", "questions": "1. Address the weaknesses.\n2. I really like your method. Have you attempted to use it with real-world vision language tasks?", "limitations": "Sort of. In appendix F, there is a brief mention of using more advanced image editing techniques in the future. However, this may not meet this requirement.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The authors clearly outline the problem that their approach is trying to solve: reward variance collapse and its impact on GRPO training.\n2. The authors introduce a novel approach that integrates adversarial instruction generation and curriculum scaling.\n3. Their method shows improvement across 8 benchmarks.\n4. Ablations that demonstrate complementary effect of both adversarial instruction generation and curriculum scaling.\n\nWeaknesses\n1. The authors do not provide detailed information on their image editing pipeline (aside from stating \"external tools such as Diffusion models\" and similar high level descriptions). It would be advisable to add more details to the appendix.\n2. Results are reported from single runs. No standard deviations or statistical significance tests are provided, and confidence intervals are not present. It would be ideal to report results for >= 3 random seeds with confidence intervals and statistical tests against the strongest baseline.\n3. They use GPT-4o as a judge and source of reward. This can add biases prevalent in GPT-4o, potentially rewarding responses that look most similar to its own, while penalizing those that are different from its own. The authors may wish to perform a robustness study, using a different model (e.g., Claude) or humans as the judge.\n4. Ablations are high-level only. The authors did not isolate lower-level components, such as image/text edits.", "was_revised": true, "final_justification": "The authors addressed all my concerns in their rebuttal and I believe that with the new data/experiments, their paper quality is improved.", "rebuttal": "We sincerely appreciate your constructive and insightful comments! We will explain your concerns point by point.\n\n> **Q1**: More clarification of Image Editing Pipeline.\n\nThank you for the valuable feedback. In Appendix F, we list external image processing tools such as Prisma Art, OpenCV, and Neural Style. It is evident that the current image editing tools lack the capability to modify complex elements, such as auxiliary lines, hence our focus on modifications related to flipping, cropping, and style transfer. Nevertheless, EvolvedGRPO, being model-agnostic and a self-enhancing framework, can integrate more advanced visual editing techniques such as Flux.Kontext, in a resource-efficient manner.\n\nThanks to editing techniques like replacement and rotation from Flux.Kontext, we are able to generate varied physical or medical images, enhancing the multimodal difficulty and thereby assisting EvolvedGRPO in achieving better performance improvements. EvolvedGRPO demonstrates strong performance in both complex physical domains (+22.5% in Table 4) and medical domains (+10.7% in Table 5).\n\n> Q2: Robustness of EvolvedGRPO.\n\nA2: As you suggested, we have reproduced the results in three runs for robustness of our experiments. We evaluate the base model Qwen2.5-VL-7B-Instruct, strongest baseline NoisyRollout, and our method EvolvedGRPO with three different seeds. The results are as follows:\n\nTable 1: Multiple evaluations of NoisyRollout and EvolvedGRPO\n\n| Models       | type                    | MathVista-mini | MathVision-full | MathVerse-mini | GSM8K       | MATH500     |\n| ------------ | ----------------------- | -------------- | --------------- | -------------- | ----------- | ----------- |\n| NoisyRollout | mean                    | 72.6           | 28.1            | 52.8           | 79.0        | 68.4        |\n| NoisyRollout | Standard Deviation      | 0.2            | 0.7             | 0.7            | 0.4         | 0.9         |\n| NoisyRollout | 90% Confidence Interval | [72.1,73.1]    | [27.0,29.3]     | [51.6,54.0]    | [78.3,79.7] | [66.8,70.0] |\n| EvolvedGRPO  | mean                    | 73.7           | 30.4            | 52.2           | 85.0        | 73.3        |\n| EvolvedGRPO  | Standard Deviation      | 0.5            | 0.9             | 0.4            | 0.2         | 0.7         |\n| EvolvedGRPO  | 90% Confidence Interval | [73.1,74.3]    | [29.0, 31.9]    | [51.5,52.9]    | [84.5,85.5] | [72.0,74.5] |\n\n> Q3: Analysis of judgement and reward of EvolvedGRPO.\n\n**A3**: We apologize for any confusion regarding the judgement and reward. We would like to address your concerns from two perspectives.\n\n**(3.1)**: Firstly, we would like to clarify that GPT4o is only used to determine whether the answer result is the correct answer. For evaluation, we extended the results of using human evaluation to perform multiple random seed evaluations and calculate the mean. The full results are shown below:\n\nTable 2: Evaluation of EvolvedGRPO via multiple human annotators\n\n| Models      | type | MathVista-mini | MathVision-full | MathVerse-mini | GSM8K | MATH500 |\n| ----------- | ---- | -------------- | --------------- | -------------- | ----- | ------- |\n| EvolvedGRPO | mean | 74.0           | 30.8            | 51.8           | 85.2  | 73.4    |\n\n**(3.2)**: When using reward for mathematical problems, only functional rule-based mathruler are used to calculate higher required rewards, without using LLM to determine whether the answer is correct. In addition, for more complex tasks, rewards can be computed using the base model, thereby avoiding bias preferences in GPT-4o without relying on external models.\n\n> Q4: Ablations are high-level only.\n\n**A4**: Thank you for the suggestion. We have added ablation studies on the lower-level components to better assess the contribution and importance of each part of the system:\n\nTable 3: Extended Ablation Experiments of EvolvedGRPO\n\n| Models                 | MathVista-mini | MathVision-full | MathVerse-mini | GSM8K | MATH500 | Avg. |\n| ---------------------- | -------------- | --------------- | -------------- | ----- | ------- | ---- |\n| Qwen2.5-VL-7B-Instruct | 67.8           | 24.7            | 44.5           | 83.6  | 67.4    | 57.6 |\n| NoisyRollout           | 72.9           | 28.9            | 52.8           | 79.0  | 68.4    | 60.4 |\n| +GRPO                  | 71.9           | 28.9            | 49.6           | 82.9  | 67.0    | 60.1 |\n| +GRPO & IMAGE EDIT     | 72.1           | 31.8            | 50.9           | 81.3  | 68.8    | 61.0 |\n| +GRPO & TEXT EDIT      | 73.4           | 29.8            | 51.3           | 84.5  | 72.6    | 62.3 |\n| +EvolvedGRPO           | 74.0           | 30.8            | 51.8           | 85.1  | 73.2    | 63.0 |\n\nAs observed, EvolvedGRPO significantly outperforms most ablation settings across all tasks. With the integration of image and text editing, the capabilities of Qwen2.5-VL in logic-based tasks continue to improve, especially after incorporating text editing. This enhancement predominantly stems from the increased complexity of training data due to cascaded text edits, which in turn strengthen the model's reasoning abilities.\n\n> Q5: Have you attempted to use it with real-world vision language tasks?\n\n**A5**: Thank you for your thoughtful suggestions. Thanks to the scalability of EvolvedGRPO, we have successfully applied Qwen2.5-VL-7B-Instruct to real-world vision-language tasks in the next table. In our testing, EvolvedGRPO has excelled in various fields such as physics, chemistry, and biology, consistently outperforming baseline methods in disciplinary reasoning and general medical QA tasks. This confirms the model’s effectiveness and adaptability in real-world applications.\n\nTable 4: Avg performance of all domains on EvolvedGRPO disciplinary reasoning\n\n| Models                 | Physics | Chemistry | Biology | Avg. |\n| ---------------------- | ------- | --------- | ------- | ---- |\n| OpenVLThinker          | 53.8    | 60.6      | 65.0    | 59.8 |\n| MM-Eureka              | 56.2    | 65.2      | 65.2    | 62.2 |\n| Qwen2.5-VL-7B-Instruct | 45.4    | 56.4      | 54.0    | 51.9 |\n| +GRPO                  | 51.6    | 57.1      | 58.6    | 55.8 |\n| +EvolvedGRPO           | 61.5    | 65.5      | 63.9    | 63.6 |\n| +EvolvedGRPO w/iter 1  | 51.4    | 56.4      | 57.2    | 55.0 |\n| +EvolvedGRPO w/iter 2  | 54.2    | 60.8      | 57.6    | 57.5 |\n\nTable 5: Avg performance of all domains on EvolvedGRPO general medical reasoning\n\n| Models                 | MedXpertQA-MM | OmniMedVQA | Avg. |\n| ---------------------- | ------------- | ---------- | ---- |\n| Qwen2.5-VL-7B-Instruct | 20.3          | 58.4       | 39.4 |\n| +GRPO                  | 23.8          | 61.0       | 42.4 |\n| +EvolvedGRPO           | 24.3          | 62.8       | 43.6 |"}, {"review_id": "Ghs8BkVcJF", "reviewer": "Reviewer_nE76", "summary": "This paper introduces EvolvedGRPO, a reinforcement learning framework aimed at enhancing the reasoning capabilities of Large Vision-Language Models (LVLMs). Built upon the Grouped Relative Policy Optimization (GRPO) method, the key contribution of EvolvedGRPO lies in its progressive instruction evolution mechanism that dynamically generates increasingly difficult multi-modal reasoning tasks. It achieves this by iteratively editing both textual and visual instructions in a self-supervised, adversarial way. Two models—a question generator and an answer model—are jointly trained, where the former adversarially challenges the latter with more difficult tasks. Empirical results show that EvolvedGRPO outperforms other open-source baselines and approaches GPT-4o on several reasoning benchmarks.", "questions": "Have you examined whether small perturbations in instructions (e.g., minor edits in visual modality) produce meaningful reward signal changes? Could the reward collapse still persist in late training?\nWhile hallucination reduction is mentioned, how robust is the filtering pipeline?\nCan the framework actively regulate difficulty based on performance plateaus? If so, how is this determined?\nCould you elaborate on the application potential in real-world settings such as visual question answering for medical or industrial imagery?", "limitations": "More detail is needed on how instruction hallucination or adversarial drift is avoided in later stages of evolution.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: Introduces an effective curriculum-style RL method tailored for LVLMs. Combines adversarial data generation with reward shaping to break reward convergence, a common issue in GRPO.\nWeaknesses: The instruction evolution mechanism still relies heavily on implicit assumptions about controllability and reward fidelity. The reward design assumes that small transformations of inputs yield proportionally meaningful reward gradients—this may not always hold in multimodal contexts. The generalization to truly open-ended tasks is unclear.", "was_revised": true, "final_justification": "I have read the author rebuttal and considered all raised points. I will keep the scores.", "rebuttal": "We sincerely appreciate the reviewer's insightful and detailed feedback. We address each of your questions individually below.\n\n> Q1: The instruction evolution mechanism still relies heavily on implicit assumptions about controllability and reward fidelity.\n\nA1: We will focus on analyzing from three aspects: controllability, fidelity, and generalization.\n\n(1.1) Controllability: Our framework enables fine-grained control over reward dynamics by modulating the depth and structure of multimodal reasoning. Increasing the number of inference steps—particularly in long reasoning chains—yields more informative reward variations than shallow perturbations. We further guide the learning process using MLLM-extracted abstract concepts and adversarial editing instructions for text and image editing. The resulting reward trends, which initially decline and then steadily rise across training rounds, reflect the model’s progressive improvement. Appendix D visualizes this evolution, highlighting the controllability of our approach.\n\n(1.2) Fidelity: To ensure data fidelity, we combine a multi-step filtering pipeline with a stepwise Hallucination Estimator to suppress errors and hallucinations during training. In addition, Manual evaluation further confirms the generation of high-quality, semantically valid data throughout the process.\n\n(1.3) Generalization: We conducted additional experiments on several other real-world vision-language tasks, demonstrating that our approach can be effectively adapted to different tasks with minimum modifications.\n\n\n> Q2: Have you examined whether small perturbations in instructions (e.g., minor edits in visual modality) produce meaningful reward signal changes?\n\nA2: We address this concern from two perspectives.\n\n(2.1) Since reward signals in multi-modal reasoning are closely tied to the depth and structure of the reasoning process, changes in entropy at both the visual and textual semantic levels can effectively modulate the reward landscape. Specifically, altering the number of inference steps—particularly in extended reasoning chains involving over 4096 tokens—tends to induce more substantial and informative reward variations than shallow input perturbations. Appendix D visualizes the evolution of reward signals across multiple training rounds, further supporting this observation. For completeness, we plan to integrate these results into the main text in future revisions.\n\n(2.2) We leverage a MLLM to extract abstract concepts and generate adversarial editing instructions, which are then used to guide both text and image editing for multi-modal enhancement. During each round of training, the reward associated with data augmentation initially drops significantly, then gradually increases as training progresses, accompanied by the continuous improvement of the model's capabilities.\n\n> Q3: Could the reward collapse still persist in late training?\n\nA3: Thank you for the concerns about reward collapse. Compared to the original GRPO training, our method not only effectively addresses the issue of reward stagnation caused by repeated training, and also mitigates the reward collapse typically observed after a fixed number of training steps. Specifically, the original GRPO model encounters reward stagnation after approximately 100 steps and collapses entirely after around 600 training steps. In contrast, our EvolvedGRPO framework can maintain stable performance throughout the training process without exhibiting any signs of reward collapse.\n\n> Q4: While hallucination reduction is mentioned, how robust is the filtering pipeline? \n\nA4:  Thank you for the concerns regarding the reduction of hallucinations. \n\n(4.1) Firstly, the filtering pipeline enhances the validation of each step through multi-step disassembly, which can indeed robustly reduce errors. Moreover, the problem generation model incorporates a Hallucination Estimator at each step of reward calculation, ensuring that hallucinations are minimized as much as possible during each training round.\n\n(4.2) To evaluate robustness, we randomly sampled 100 instances and analyzed the distribution of three types of cases across multiple training rounds: (1) valid and effective questions that contribute positively to training; (2) neutral questions that are ineffective but filtered out without impacting training; and (3) erroneous questions that introduce negative effects. The results are as follows:\n\nTable 1: Human evaluation results for hallucination detection\n\n| types  | total | type 1 | type 2 | type 3 |\n| ------ | ----- | ------ | ------ | ------ |\n| number | 100   | 95     | 5      | 0      |\n\n> Q5: Can the framework actively regulate difficulty based on performance plateaus? If so, how is this determined?\n\nA5: The framework can proactively adjust the difficulty. When increasing the difficulty for the question generation model, the reward is computed as the maximum between the actual answer accuracy and `1/test` (where `test` denotes the number of test samples). This guides the difficulty to converge toward `1/test`. As shown in the appendix D, when `1/test = 0.2`, the answer accuracy effectively converges slightly above `0.2`. Additionally, the number of editing steps is increased once the reward stabilizes and the length of the reasoning chain remains unchanged, allowing for further difficulty scaling.\n\n> Q6:  Could you elaborate on the application potential in real-world settings such as visual question answering for medical or industrial imagery?\n\nA6: \n\nWe trained on multiple real-world vision-language tasks by replacing mathematical concepts with their corresponding domain-specific counterparts. With the further development of image editing technology, we embed the Flux.Kontext image editing model into the training framework for image editing, to enable richer and more customized image enhancements. The results are as follows:\n\nTable 2: Avg performance of all domains on EvolvedGRPO disciplinary reasoning\n\n| Models                 | Physics | Chemistry | Biology | Avg. |\n| ---------------------- | ------- | --------- | ------- | ---- |\n| OpenVLThinker          | 53.8    | 60.6      | 65.0    | 59.8 |\n| MM-Eureka              | 56.2    | 65.2      | 65.2    | 62.2 |\n| Qwen2.5-VL-7B-Instruct | 45.4    | 56.4      | 54.0    | 51.9 |\n| +GRPO                  | 51.6    | 57.1      | 58.6    | 55.8 |\n| +EvolvedGRPO           | 61.5    | 65.5      | 63.9    | 63.6 |\n| +EvolvedGRPO w/iter 1  | 51.4    | 56.4      | 57.2    | 55.0 |\n| +EvolvedGRPO w/iter 2  | 54.2    | 60.8      | 57.6    | 57.5 |\n\nTable 3: Avg performance of all domains on EvolvedGRPO general medical reasoning\n\n| Models                 | MedXpertQA-MM | OmniMedVQA | Avg. |\n| ---------------------- | ------------- | ---------- | ---- |\n| Qwen2.5-VL-7B-Instruct | 20.3          | 58.4       | 39.4 |\n| +GRPO                  | 23.8          | 61.0       | 42.4 |\n| +EvolvedGRPO           | 24.3          | 62.8       | 43.6 |"}, {"review_id": "GMnWPEahEP", "reviewer": "Reviewer_moty", "summary": "This paper aims to address the problem of training stagnation caused by reward convergence when fine-tuning LVLMs with RL methods like GRPO. To tackle this challenge, the authors propose EvolvedGRPO, a progressive instruction evolution framework. This framework utilizes an adversarial setup where a question generation model (Q) dynamically creates training samples of increasing difficulty for an answer model (A). Specifically, model Q generates image and text editing instructions to increase the complexity of problems, thereby matching the evolving capabilities of model A. Ultimately, experimental results show that this method achieves state-of-the-art performance on several multi-modal reasoning benchmarks.", "questions": "See Weaknesses", "limitations": "yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Impressive Performance: The paper achieves excellent results on several challenging multi-modal reasoning benchmarks, outperforming other open-source models and even approaching the performance of the closed-source GPT-4O on certain metrics.\n2. Compelling Motivation: The paper's premise is highly valuable. Exploring self-evolving reinforcement learning algorithms to dynamically adjust training data difficulty is an important and promising research direction for pushing model capabilities beyond their current limits and achieving a higher level of intelligence.\n\nWeaknesses:\n\n1. Lack of Clarity and Missing Key Details: For a paper whose core contribution is a new algorithm, the description of the method is surprisingly scattered and vague. It dedicates considerable space to well-known background concepts like SFT and GRPO, which hinders the understanding and reproducibility of its core contribution.\n\n    - In the core reward function, Equation (9), the symbol format' appears without any definition or explanation provided anywhere in the paper. Additionally, the explanation for the test variable is also same problem. \n\n    - The \"Text Editing Instruction\" is a central innovation of the method, yet the main text lacks a detailed description of its specific operations. Readers are forced to infer its meaning from the example in Figure 1, but such a key mechanism should be explicitly and thoroughly described in the text.\n    - The paper's description of the adversarial training process is not systematic. A clear algorithm flowchart or pseudocode block is crucial for understanding the alternating training of the Q-A models and key steps like instruction generation and replacement, but this is absent from the main body of the paper.\n    - The presentation of results in Table 1 could be misleading. Specifically, the 'Avg.' column is calculated and presented even for baseline models where scores for the GSM8K and MATH500 datasets are absent (e.g., OpenVLThinker-7B, NoisyRollout, MM-EUREKA). This creates an inconsistent basis for comparison and could cause readers to misinterpret the overall performance of these baseline models\n\n\n    - Typo: A typo, \"generation generation,\" appears on line 195, which affects the reading experience.\n\n\n2. Mismatch Between Motivation and Method's Effect: The paper's initial motivation is to solve the RL training bottleneck in multi-modal reasoning caused by the entanglement of visual and textual information. However, its core \"text editing\" method—for instance, transforming an answer z into a problem of solving for D = 3*z/4 + 9/4—essentially appends a separate, unimodal mathematical calculation task at the end of the original multi-modal task. This operation does not increase the intrinsic complexity of the visual-textual reasoning itself, but rather appears to train the model's arithmetic capabilities.\n\n3. The paper lacks the necessary analysis to prove that this method actually solves the problem it set out to address: reward convergence and training stagnation. For example, the authors do not analyze whether the reward variance for different responses to the same sample actually increases after using EvolvedGRPO. The logical link between the method's effect and the initial motivation is not established.\nInsufficient Experimental Analysis and Comparison:\n\n4. The paper's image manipulation method (e.g., adding noise or filters) is conceptually very similar to that of NoisyRollout[1]. However, the paper merely cites this work without providing any in-depth comparative analysis or discussing the differences in concept or effectiveness.\nThe paper lacks a dedicated ablation study for the image editing component. From the experiments, we cannot determine the actual performance contribution of adding noise or editing the images. This calls into question the effectiveness of the \"multi-modal\" aspect of the instruction evolution.\n\n[1] NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation", "was_revised": true, "final_justification": "I still think that the paper lacks clarity in its methodological description.\n\nAdditionally, the central claim that the method solves reward convergence and training stagnation remains unsubstantiated. The evidence provided in the rebuttal is unconvincing, and a promise of future analysis is insufficient at this stage.\n\nDue to these fundamental issues, I will maintain my original rating.", "rebuttal": "We sincerely thank the reviewers for their exceptionally thorough and detailed comments!  We will explain your concerns point by point.\n\n> Q1: In the core reward function, Equation (9), the symbol format' appears without any definition or explanation provided anywhere in the paper. Additionally, the explanation for the test variable is also same problem.\n\nA1: Thank you for pointing this out. \n\n`format` refers to a reward score of format, which is designed to encourage the model to produce structured outputs, particularly those involving chain-of-thought reasoning.\n\n`test` denotes the number of evaluation trials used to assess accuracy, aligning with the number of responses generated in a reinforcement learning batch. The expression using `max` in the goal ensures at least one correct answer post-question augmentation, preventing variance collapse.\n\nWe will include a more detailed description in the revised version to improve accessibility and clarity.\n\n> Q2: Clarification of Multi-modal Instruction Editing. \n\nA2: We clarify that the core contribution of EvolvedGRPO lies in its formulation of a model-agnostic, multimodal, self-evolving reinforcement learning framework applicable across diverse domains (e.g., physics, chemistry) using various tools while continuously improving through self-enhancement. Therefore, we emphasize the overall self-evolving pipeline rather than any domain-specific instantiation. For mathematical tasks, EvolvedGRPO adaptively selects appropriate modules for generating text editing instructions and employs external tools (e.g., Prisma Art, OpenCV, and Neural Style) to increase the visual complexity of mathematical images. This part was included in the appendix for brevity; however, we will incorporate a detailed description of the specific tools into the main text in the revised version.\n\n> Q3: A clear algorithmic flowchart or pseudocode is currently missing from the main body of the paper.\n\nA3: The algorithm flowchart is currently included in Appendix A.4: PseudoCode (located in Supplementary Material.zip) due to space constraints. It will be moved to the main body of the paper in the revised version.\n\n> Q4: The presentation of results in Table 1 could be misleading.\n\nA4: In earlier evaluations, we cited results from original papers due to the lack of standardized evaluation protocols. To ensure fairness and provide a comprehensive comparison, we have carefully re-evaluated the results using consistent prompts across all models. The updated results are displayed below:\n\nTable 1: Expanded version of the main table\n\n| Models                 | MathVista-mini | MathVision-full | MathVerse-mini | GSM8K | MATH500 | Avg. |\n| ---------------------- | -------------- | --------------- | -------------- | ----- | ------- | ---- |\n| GPT-4o                 | 63.8           | 36.8            | 50.2           | 94.2  | 74.6    | 63.9 |\n| Mulberry-7B            | 63.1           | 22.8            | 39.6           | 69.1  | 65.2    | 52.0 |\n| Virgo-7B               | 62.3           | 24.0            | 36.7           | 77.4  | 71.4    | 54.4 |\n| MindGYM                | 70.3           | 28.6            | 48.4           | 84.1  | 68.4    | 60.0 |\n| OpenVLThinker-7B       | 70.2           | 25.3            | 47.9           | 78.6  | 67.4    | 57.9 |\n| NoisyRollout           | 72.9           | 28.9            | 52.8           | 79.0  | 68.4    | 60.4 |\n| MM-EUREKA              | 73.0           | 26.9            | 50.3           | 82.5  | 66.8    | 59.9 |\n| Qwen2.5-VL-7B-Instruct | 67.8           | 24.7            | 44.5           | 83.6  | 67.4    | 57.6 |\n| EvolvedGRPO | 74.0           | 30.8            | 51.8           | 85.1  | 73.2    | 63.0 |\n\n> Q5: Mismatch Between Motivation and Method's Effect\n\nA5: Thank you for raising this important question. We will address this from three points:\n\n1. First, we clarify that EvolvedGRPO is designed to autonomously explore and escalate problem complexity while maintaining logical correctness. It uses the visual perception output as an intermediate state to construct longer CoT. The ultimate goal is to effectively disentangle the complex relationship between vision and text, seamlessly integrating perception into subsequent logical steps.\n\n2. At the training level, the subsequent augmentation task (e.g., a calculation) imposes stringent precision requirements on the initial perceptual output (z). Any slight inaccuracy in perception is significantly amplified in the final result (D). By managing this process with a carefully designed reward function and filtering mechanism, we leverage this error amplification effect to generate a stronger, more explicit gradient signal. This, in turn, compels the model to disentangle the visual and textual information with much higher precision.\n\n3. To demonstrate the framework's generality and effectiveness, we leverage Qwen2.5-VL-7B-Instruct to perform complex editing instructions across various real-world vision-language tasks. The results provide strong evidence that our method effectively decouples visual and textual information in diverse scenarios, leading to a significant enhancement of the model's comprehensive reasoning abilities.\n\n   Table 2: Avg performance of all domains on EvolvedGRPO disciplinary reasoning\n\n   | Models                 | Physics | Chemistry | Biology | Avg. |\n   | ---------------------- | ------- | --------- | ------- | ---- |\n   | OpenVLThinker          | 53.8    | 60.6      | 65.0    | 59.8 |\n   | MM-Eureka              | 56.2    | 65.2      | 65.2    | 62.2 |\n   | Qwen2.5-VL-7B-Instruct | 45.4    | 56.4      | 54.0    | 51.9 |\n   | +GRPO                  | 51.6    | 57.1      | 58.6    | 55.8 |\n   | +EvolvedGRPO           | 61.5    | 65.5      | 63.9    | 63.6 |\n   | +EvolvedGRPO w/iter 1  | 51.4    | 56.4      | 57.2    | 55.0 |\n   | +EvolvedGRPO w/iter 2  | 54.2    | 60.8      | 57.6    | 57.5 |\n\n   Table 3: Avg performance of all domains on EvolvedGRPO general medical reasoning\n\n   | Models                 | MedXpertQA-MM | OmniMedVQA | Avg. |\n   | ---------------------- | ------------- | ---------- | ---- |\n   | Qwen2.5-VL-7B-Instruct | 20.3          | 58.4       | 39.4 |\n   | +GRPO                  | 23.8          | 61.0       | 42.4 |\n   | +EvolvedGRPO           | 24.3          | 62.8       | 43.6 |\n\n> Q6: The paper lacks the necessary analysis to prove that this method actually solves the problem it set out to address: reward convergence and training stagnation.\n\nA6: First, we have solved reward convergence by creating a dynamic problem space. As shown in the reward changes provided in the appendix, our adaptive difficulty mechanism ensures that once the model's performance stabilizes on a given problem set, the framework automatically increases the complexity and reasoning chain length. Quantitatively, we further present the gradually changing trends of multiple samples during the enhancement process in the following table:\n\nTable 4: Variables during training\n\n| variables\\step | 0    | 30   | 60   | 90   |\n| -------------- | ---- | ---- | ---- | ---- |\n| variance| 0.17 | 0.14 | 0.17 | 0.16 |\n| accuracy       | 0.22 | 0.21 | 0.23 | 0.24 |\n\nAs shown in the table, EvolvedGRPO prevents the model from converging to a single optimal policy for a static task distribution, forcing it to constantly adapt and learn.\n\nSecond, the practical result of overcoming reward convergence is the alleviation of training stagnation. We provide direct evidence of this by comparing our method against the baseline GRPO framework.\n\n\nThird, as direct evidence against reward convergence, we analyze the reward variance for different valid responses to the same problem. We first randomly sampled a set of N base problems. For each problem, we generated multiple distinct yet valid solutions (e.g., different Chains of Thought) using two methods: (a) for the original problem (baseline), and (b) for the problem after being enhanced by our framework. We then calculated the reward for every solution and computed the variance of these rewards for both sets in the table. We can provide experimental evidence to demonstrate that a complete improvement has been implemented.\n\n> Q7: The paper's image manipulation method (e.g., adding noise or filters) is conceptually very similar to that of NoisyRollout\n\nA7:  Although both our EvolvedGRPO and NoisyRollout edit images, the idea and methodology are different.\n\nIn contrast to NoisyRollout, which injects random noise for specific geometric tasks, our method performs semantically meaningful, complex image edits coupled with multi-round text-image interactions, aiming to enhance model capabilities on general-purpose, real-world datasets. (b) Furthermore, in a direct comparison using only image editing, our method outperforms NoisyRollout by an average of 1.0%, highlighting the benefit of our approach. In addition, we have added comparisons with NoisyRollout and our own ablation studies on the lower-level components to better assess the contribution and importance of each part of the system as follows:\n\nTable 5: Extended ablation experiments of EvolvedGRPO\n\n| Models                 | MathVista-mini | MathVision-full | MathVerse-mini | GSM8K | MATH500 | Avg. |\n| ---------------------- | -------------- | --------------- | -------------- | ----- | ------- | ---- |\n| Qwen2.5-VL-7B-Instruct | 67.8           | 24.7            | 44.5           | 83.6  | 67.4    | 57.6 |\n| NoisyRollout           | 72.9           | 28.9            | 52.8           | 79.0  | 68.4    | 60.4 |\n| +GRPO                  | 71.9           | 28.9            | 49.6           | 82.9  | 67.0    | 60.1 |\n| +GRPO & IMAGE EDIT     | 72.1           | 31.8            | 50.9           | 81.3  | 68.8    | 61.0 |\n| +GRPO & TEXT EDIT      | 73.4           | 29.8            | 51.3           | 84.5  | 72.6    | 62.3 |\n| +EvolvedGRPO           | 74.0           | 30.8            | 51.8           | 85.1  | 73.2    | 63.0 |"}]

Author rebuttal:

Meta-review: Advanced large vision-language models require new efficient  reinforcement learning techniques in particular for multimodal reasoning. Approaches such as  Grouped Relative Policy Optimization (GRPO) assign varying reward scores to different responses from the same prompt to estimate gradients. However, designing such rewards is very challenging in multimodal contexts.  In this paper, the authors introduce  EvolvedGRPO, a framework  that iteratively updates the difficulty of samples progressively using editing with a generation model that  generates multi-modal data editing instructions (texts and images) to augment the dataset.

According to most reviewers comments, the paper is well-written, and addresses a challenging  training issue of LVMs. The proposed approach is also tested using a large variety of experiments. During rebuttal, the authors addressed most of the reviewers concerns on the methodology and implementation details and conducted additional experiments to support the robustness of the approach. They tested the instructional evolution on various disciplinary reasoning and general medical QA tasks with Qwen2.5-VL-7B-Instruct. They also added ablation studies on the lower-level components of the model, the approach consistently outperforming the ablation settings.
Overall, the rebuttal clarified the procedure and the authors added various numerical results which support the performance of the proposed framework in a large variety of settings. A revised version including all comments would be valuable to the ML community, (details on the text and images editing to clarify the algorithm for a larger audience) . The authors are encouraged to clarify the remaining concerns  about reward convergence and training stagnation (at least discuss its long-term viability follwing the discussion with reviewer moty).
Decision: Accept (poster)