DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage
Authors: Haowen Gao
zhenyu zhang
Liang Pang
Fangda Guo
douhongjian
Guannan Lv
ShaoGuo Liu
Tingting Gao
Huawei Shen
Xueqi Cheng
Venue: ICLR 2026 Poster (2026)
Keywords: GRPO
Advantage Vanishing
Reward Sparsity
Multimodal LLM
Difficulty-Adaptive
Primary area: foundation or frontier models, including LLMs
TL;DR: DIVA-GRPO dynamically adjusts problem difficulty and generates tailored variants to stabilize reward signals in GRPO, mitigating reward sparsity and advantage vanishing, improving both training efficiency and reasoning performance in multimodal LLMs.

Abstract: Reinforcement learning (RL) with group relative policy optimization (GRPO) has become a widely adopted approach for enhancing the reasoning capabilities of multimodal large language models (MLLMs). While GRPO enables long-chain reasoning without a traditional critic model, it often suffers from sparse rewards, arising from the scarcity of positive feedback on difficult problems, and from advantage vanishing, which occurs when group-level rewards exhibit high consistency for problems that are too easy or too hard. Existing solutions fall into three categories: sample enhancement and expansion, which may aggravate vanishing advantage due to poor control of difficulty distribution; selective sample utilization, which fails to fully leverage the value of all data; and indirect reward design, which may introduce biased optimization directions due to misalignment between reasoning and the final outcome. However, these approaches overlook a fundamental question: for a given problem, how can we ensure that the within-group reward distribution of responses exhibits enough variance to yield clear optimization signals for each response? To address these issues, we propose DIVA-GRPO, a difficulty-adaptive variant augmentation advantage method that dynamically adjusts the difficulty distribution of variants for each problem from a global perspective. Our method dynamically assesses problem difficulty, samples variants with appropriate difficulty levels, and advantages are computed within both local and global(a problem and its variants) groups using difficulty-weighted and normalized scaling. This design alleviates reward sparsity and advantage vanishing, minimizes data waste, and improves training stability. Extensive experiments on six reasoning benchmarks demonstrate that DIVA-GRPO outperforms existing approaches in both training efficiency and reasoning performance.

Published as a conference paper at ICLR 2026
DIVA-GRPO: ENHANCING MULTIMODAL REASONING THROUGH DIFFICLLTY-AAATIVE VARIANT ADVAN-
TAGE
Haowen Gaol:21, Zhenyu Zhang"i, Liang Pang" Fangda Guo', Hongjian Dou;,
Guannan Lv Shaoguo Liu Tingting Gao Huawei Shen', Xueqi Cheng'
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China
2 University of Chinese Academy of Sciences, Beijing, China
3 Kuaishou Technology, Beijing, China
ga chaowen23s@ www.icttac..... zhargzhhenyod@@kuassshooo pangli angge at.ac.cm
ABSTRACT
Reinforcement learning (RL) with group relative policy optimization (GRPO) has become a widely adopted approach for enhancing the reasoning capabilities of multimodal large language models (MLLMs). While GRPO enables long- chain reasoning without a traditional critic model, it often suffers from sparse rewards, arising from the scarcity of positive feedback on difficult problems, and from advantage vanishing, which occurs when group-level rewards exhibit high consistency for problems that are too easy or too hard. Existing solutions fall into three categories: sample enhancement and expansion, which may aggravate vanishing advantage due to poor control of difficulty distribution; selective sample utilization, which fails to fully leverage the value of all data; and indirect reward design, which may introduce biased optimization directions due to misalignment between reasoning and the final outcome. However, these approaches overlook fundamental question: for a given problem, how can we ensure that the within- group reward distribution of responses exhibits enough variance yield clear optimization signals for each response? To address these issues, we propose DIVA-GRPO, a difficulty-aaaptri
variant augmentation advantage method that dynamically adjusts the difficulty distribution of variants for each problem from a global perspective. Our method dynamically assesses problem difficulty, samples variants with appropriate difficulty levels, and calculates advantages within both local and global (a problem and its variants) groups using difficulty-weighte and normalized scaling. This design alleviates reward sparsity and advantage vanishing, minimizes data waste, and improves training stability. Extensive experiments on six reasoning benchmarks demonstrate that DIVA-GRPO outperforms existing approaches in both training efficiency and reasoning performance. Code is available at htttps://gitthi com/ аааааа wataaaaaa/TAA--a GDTO
INTRODUCTION
Multimodal large language models (MLLMs) (Chen et al. 2024b Hurst et al. 2024 Laurençon et al.
2024
Liu al. 2023 Yin et al. 2024 Wu 2024b Alayrac al. |022 Zhu et al. 2023 Li et al.
2023
have demonstrated remarkable ability to integrate textual and visual information (Pang et al.
2024
for complex reasoning tasks, such as visual question answering (Antol et 2015
Xiao et
Equaa contribution
Corresponding authors

---

Published as a conference paper at ICLR 2026
Previous works
Ours
Expesdill Brodee Space Fres Spece
podete Durety
Problem Variaris
Origin Problem
Origin Probiem Space
Problem
Space
O2
Problem
Origin
Space
Spate
Problem Varlant
Problem Variants
Ereandil hokken
Space
Hard
Easy
Hard
Essy
Hard
Easy
Selective sample
b) Semple enhancement
Train Steps
utilization
and expansion
inffiuutty-dapphee problem expansion
Figure I: (a) Selective sample utilization relies on only a subset of data, leading to underuse. (b) Sample enhancement expands data without difficulty awareness, causing even severe advantage sparsity. (c) Our method adaptively expands the sample space by problem difficulty, ensuring stable difficulty distribution.
2024) and multimodal logical reasoning (Huang & Chang, 2022 Guo et al. 2025). Nevertheless, the
heterogeneous nature of text and visual modalities makes long-chain reasoning challenging. requiring both careful observation and stepwise problem-solving (Xu al. 2024). To address these challenges,
recent studies have explored multimodal chain-of-thougle approaches (Zhou et al. 2024 Chen et al.
2024a; Wang et al. 2025b), including Video-of-Thought (Fei al. Det-CoT al.
Col (Meng et al. 2023), and Grounded-RR RL (Sarch et al.) 2025). which aim to improve reasoning by decomposing problems or leveraging structured strategies. Complementary to these advances, reinforcement learning (RL) has emerged
powerful framework for further enhancing MLLMs: proximal policy optimization (PPO) (Schulman et al. (2017) and direct preference optimization (DPO)
(Rafailov et al.) (202)) are widely used for alignment, while GRPO (Shao al. 2024) advances long-chain reasoning by combining rewards with group-relative advantage estimation, significantly
boosting both alignment and reasoning efficiency.
However, directly applying GRPO to MLLMs faces a critical challenge- vanishing (Huang
et al. 2025| Park et al. [2025] Yao et al. [025] Wang al.. 2025a MMeg al., [2025] [Zhan et al.
2025 Zhang & Zuo 2025). GRPO computes relative advantages by comparing sampled responses for
given problem and then standardizes the advantages across all samples. When problem is too easy (or too difficult) for the current model, obtaining too few positive (or negative) rewards can lead to excessively large advantage values, causing high gradient noise and unstable training. Under extreme conditions, all responses may be entirely correct or incorrect, yielding zero relative ad advantage. This slows learning and weakens the model's reasoning performance across varying difficulties. Existing approaches attempt to address this issue in several ways. One approach is sample enhancement and expansion, which enlarges the problem space by adding prompts to difficult instances or generating diverse text and image variants, though it may exacerbate advantage vanishing due to poor control over the difficulty distribution (Huang et al. 2025 Park et al., 2025 Yao et al. 2025)and introduce hidden biases (Gao et al. 2025). Another approach is selective sample utilization, which prioritizes highly effective samples or disregards those with low learning contribution to improve efficiency; however, this may limit exposure to complex problems or reduce data diversity (Wang et al., 2025a] Meng et al. 2025). Finally, indirect reward design provides finer-grained feedback signals to mitigate reward sparsity, but these rewards may not perfectly align with ultimate task objectives. potentially guiding the model toward suboptimal directions (Zhang et al. 2025) Zhang & Zuo 2025)
Although the above methods can mitigate certain instances of advantage vanishing, they all overlook a critical issue: as shown in Figure (c)), as training progresses, problem difficulty continuously decreases, advantage vanishing intensifies, and model training efficiency steadily declines. To address this, we propose DIVA-GRPO, an adaptive method that dynamically adjusts problem difficulty and variant distribution while preserving semantic consistency, and integrates both local (individual problem) and global (multiple variants derived from the original problem) reward computation. During training, problem difficulty is iteratively assessed based on model responses: simple problems are augmented with complex text and image perturbations, moderately difficult ones with diverse text variants, and hard ones with "think-step" reasoning variants, ensuring effective sample utilization and mitigating advantage vanishing. To balance feedback across different difficulty levels, DIVA- GRPO applies batch z-score normalization and seffiuultyweightte scaling, and further introduces reward-range-bait based rescaling to prevent inflated advantages from minor reward differences. This
2

---

Published as a conference paper at ICLR 2026
enables stable policy optimization, encourages exploration, and enhances long- loni-chain reasoning, while remaining broadly applicable to CeRPO-based methods for improved optimization efficiency.
Extensive experiments validate the effectiveness and efficiency of DIVA- Our model ex- cels across six challenging benchmarks, achieving state-of-the-art performance at the 7B scale, approaching the accuracy of much larger models and proprietary commercial systems. Moreover, DIVA-GRPE GOOO substantially accelerates training. Using the same set of original data, it reaches the optimal performance with over a 2.55x reduction in required steps and delivers more than 1.76x end-to-end speedup in wall-clock time, all while preserving stable and informative advantage signals across difficulty levels. These results highlight the substantial improvements both performance and training efficiency offered by our model.
Overall, DIVA-GRPO overcomes the limitations of conventional GRPO by combining adaptive difficulty assessment, variant generation, and robust advantage estimation. This framework alleviates reward sparsity and advantage vanishing, while enhancing reasoning stability and performance across a wide range of multimodal tasks, from visual question answering to complex logical reasoning. Our core contributions are as follows:
Propose DIVA-GRPO, based on dynamic difficulty assessment, using difficult--aapptive variant generation and advantage sharing alleviate reward sparsity and advantage vanishing.
Introduce joint estimation of local and global advantages, combined with batch z-score normal- scaling, to improve training stability.
ization and tifficulty-weighhe
Present swward-angg-bare based advantage rescaling method, effectively preventing unreasonable advantage inflation and accelerating convergence.
Conduct experiments six mainstream multimodal reasoning benchmarks, demonstrating the effectiveness and superiority of DIVA-GRPO.
2 CHALLENGE: REWARD SPARSITY AND ADVANTAGE VANISHING
We begin by reviewing the key concepts of Group Relative Policy Optimization (GRPO), which form the foundation of our iffficulty-daaptie variant strategy.
Group Relative Policy Optimization (GRPO). Let Q = {q1 ... gN) denote the set of training problems. For a given problem q, the model He generates candidate responses y, = {y1, V2..........3 yk} } through rollouts. Each response yi receives scalar reward r(v:) IR determined by reward function, which can be either a rule-based evaluator (e.g., answer correctness) or a learned reward model. GRPO computes the relative advantage of each response within its group by standardizing its reward:
A(y:) r(y:) 1rr
pb
r(m)), or
Σ j=1
(r(s)) - -
+
where é is a small constant for numerical stability. The policy gradient is then estimated as voL(O) = Egua youna (A(y) Vo log To(y)g)]
Reward Sparsity and Advantage Vanishing. GRPO and its extensions often face reward sparsity: when the base multimodal model has limited capability or the problem is difficult, only few reasoning paths obtain positive rewards, especially early training. They also suffer from advantage vanishing: since relative advantages are computed within response groups, overly hard or easy problems lead to all-correct or all-wrong outputs, yielding zero advantages. As training proceeds, this issue worsens, reducing optimization efficiency and sample utilization.
Motivation. Existing solutions address these problems through (i) sample enhancement and expan- sion, which enlarges the problem space without adjusting intrinsic difficulty; (ii) sample selection and utilization, which improves efficiency but risks discarding valuable hard cases and reducing diversity; and (iii) indirect reward design, which enriches supervision but may bias optimization. While partially effective, these methods cannot guarantee stable reward variance within each group.
This motivates our central question: how can we preserve stable and informative reward variance regardless of problem difficulty? We argue that the key lies in dynamically assessing problem
3

---

Published as a conference paper at ICLR 2026
Dysemiii difficalt odjanmentt
Polacy opriniaatiio
Bun
R
Local
RRange(RR R1..)
A
Qnrry Dipasaia
Q:
Polcy Mode
Global Group Advantage
Local Grorp Advestage
Corn
Af.
Query
Local
Bach Noma
XRange Ri..)
R
Goog
Afia
snginiin Alfendy
Teute Suqgo
Local Advantage
Iitepet Qveryy
Iiflids
Thakk Saps
Had
R1.1
AL
<Iznge
liffually
Aystades Tixt
Viagge
R
Loobal Goog
Beach Norr
A
Rij
Rip)
A
P
A
DiSiuily
Bic X
High Foraloo
Dillicaly-Adsptive Question Expansion
Glubel Advaniage
Figure 2: Overview of the proposed DIVA-GRPO method. For a given question, we dynamically assess its difficulty based on past rollout rewards and adaptively sample variants of different difficulty levels. As shown, when the original question is hard, easier variants are sampled to ensure reward diversity. We then compute local (the question itself) and global (the question with its variants) advantages, and obtain the final advantage through difficulty- aware reweighting and reward-range rescaling to update the policy model.
difficulty and adaptively adjusting the sampling of variants, ensuring every problem produces both positive and negative feedback.
DIFFICULTY-ADAPII VARIANT ADVANTAGE WITH GRPO
To address this challenge, design a dufficulty-adaptiie method, DIVA-GRPO. dynamically assess problem difficulty, adaptively sample semantically consistent variants of different difficulties, and enhance the diversity of rewards within the problem and its variant space. By computing difficulty-weighted local (for the original problem) and global (for the problem and its variants) group advantages, the framework mitigates reward sparsity and vanishing advantage issues. Figure 2 iilussrrtes the overall pipeline.
Its core idea is to
3.1 DIFFICULTY ASSESSMENT OF SAMPLES
prerequisite for implementing adaptive strategies is to assign an appropriate difficulty level each problem. Notably, problem difficulty is neither inherent nor static; rather, it should be dynamically assessed in line with the evolving capability of the model. To this end, DIVA-GRPO introduces a difficulty assessment mechanism based historical rollout outcomes: if most rollouts for problem are correct, it is regarded as relatively easy for the current model; otherwise, it considered relatively difficult. This assessment is recalibrated at every training epoch, ensuring that perceived difficulty adapts as the model improves. The resulting difficulty estimates are then used to guide variant generation and sampling strategies, enabling the model balance exploration and exploitation while maintaining meaningful advantage signals continuously.
We define problem difficulty within the range [Amin Dmax]] where Dmin represents the casiest level and Dmax the most difficult. At the beginning of training, all problems are initialized at the midpoint of this range: Dmid = Pmin and let each variant be rolled out l times. WWe denote the empirical accuracy of these rollouts as & mi Σm Σx ==1 T(ye. is correct), where I[-] is the indicator function. The difficulty then updated according to the following rule:
Let each original problem contain m variants,
2
new clip (Dold 7 17.00.5- (0.5 at), Dmin: D Dmax
where > is learning rate controlling the adjustment magnitude, and clip(-, Dminy D Duax) ensures that the difficulty remains within [Dmiu, ) This update rule guarantees that:
if most rollouts are correct (a + 1). the difficulty decreases;
if most rollouts fail (a 0), the difficulty increases;
if the accuracy is around 50%, the difficulty remains stable.
4

---

Published as a conference paper at ICLR 2026
In this way, difficulty levels evolve smoothly with model perfonmance, allowing more nuanced adjust- ments than the simple "all-correc//lll-wror" wwoogw rule, and ensuring continuous calibration of sampling strategies throughout training. For the hyperparameter 7,, ablation experiments and recommendations for its selection can be found in Appendix D.2
3.2 DIFFCULTY--DAPTIE VARIANT GENERATION
We aim to provide each problem with an appropriate advantage. To this end, we introduce method for generating adaptive variants based on the difficulty assessment of each problem derived above. Let each original problem be denoted as q (g), where Lg is the associated image and T. is the textual description and question. After evaluating the difficulty Dg E [Dmin, Dunx] of each problem,
we generate a set of difficult--peccffic consistent variants V, = {q(2) ,g(2) _(_)....] } for training, ensuring that each variant preserves the original answer Vi while selectively adjusting its difficulty optimize the model's 's learning. By dynamically adjusting the characteristics of each variant according to Dy. we aim ensure that the reward distribution for each problem provides meaningful advantage signals while maintaining a more balanced profile. The sampling method is described in Appendix D.2 and several case studies of the variants are presented Appendix H
Simpler problems (Dg < Dmii): Variants are generated by perturbing both the text and the image: ( = -1,,) .z))). Textual variants T are created by rephrasing, restructuring, or introducing slight linguistic perturbations to the original question, ensuring the answer remains unchanged. These modifications enhance the model's sensitivity to subtle differences, thereby improving robustness. Image variants 7 are generated through perturbation functions such as rotation, Gaussian noise, salt-and -apppper- noise, speckle noise, and blurring. Stronger or multiple perturbations increase difficulty while preserving correctness. Alternatively, the textual information Tg can be embedded within the images in the variant set V, and provided as images to convey the problem information.
Moderate problems (Dy Dmia): Variants are generated by creating semantically equivalent
textual versions: V, {(T,: T((i)) T( is paraphrase of T. These variants maintain the original difficulty while diversifying expression. This allows the model to experience multiple
formulations of the same problem, improving generalization to unseen expressions.
Difficult problems (Dg > Dmia): Variants incorporate partial reasoning guidance: ((()
(Iy: @ ) where R((t) is sequence of intermediate reasoning steps generated and verified by a closed-source model. For more difficult problems, additional reasoning steps are provided hints. This ensures that even challenging problems produce meaningful advantage signals,
mitigating gradient vanishing and promoting gradual mastery of complex reasoning.
3.3 DIMACULTY-EIGHHTED AND NORMALIZED ADVANTAGE BALANCING
Before introducing our balancing strategy, first review the notion of local and global advantages used GRPO GRsed training. While GRPO computes advantages within a single problem, recent Yao et al. (2025) introduces semantically consistent variants to enable broader comparisons. Let an original problem be denoted as = (Tg, Tg), with associated image I, and textual description T, For each set of variants V/ = (1) ..... (N) is constructed, modifying only text or image while preserving the mround-truthe answer. Two types of advantages are defined:
Local advantage: Aucai(y,)) ()), computed for each problem using the standard GRPO formula.
Global advantage: Aglobu ())) computed across all responses in the variant set:
Aglobal (1x ())
rigl"))
tc where Mg and 0g are the mean and standard deviation of rewards
across all responses in Vy,
To ensure that both local and global advantages contribute effectively while accounting for varying problem difficulty, we propose a two-step balancing strategy: (1) normalization to make local and global signals comparable, and (2) difficulty-eiighree scaling to adaptively rescale advantages according to each problem's 's dynamic difficulty coefficient. This design stabilizes optimization by preventing dominance of global advantages and aligning reward signals with problem difficulty.
5

---

Published as a conference paper at ICLR 2026
Formally, after sampling multiple variants for each problem, compute both local and global advantages. However, two issues arise in this process.
(1) Local-Global imbalance. The magnitudes of local and global advantages are unequal: local advantages are computed from l rollouts, whereas global advantages are based on m x k samples. Consequently, global advantages tend to be larger and dominate the optimization, while local signals are underweighted. A detailed analysis of the distributions of the original local and global advantages can be found in Appendix D.6
To address this issue, we apply batch-level z-score normalization
separately:
Aloca (v) Ay) Plocal
Aginhal (v) - Pgglohal
Aglobaly)
Ooocal +
gglobal +
where Plocal, Nooal and Hggloall gloobal are the mean and std of local and global advantages within a batch. This ensures that both signals remain comparable and contribute fully to training.
(2) Difficulty-wiihhted scaling. Existing methods treat easy and difficult problems equally when computing advantages, without accounting for their varying difficulty levels. To encourage the model to tackle harder problems, we introduce sefficulty-weigheee
scaling after normalization.
Let (D 1 denote the difficulty coefficients of the N variants in a problem group Vg. and let Dg M D. be the group-wise mean difficulty. For each response y; associated with variant ( the rescaled advantage is computed as
Aly: (0) = exp l: (D(( - -D)) sgn (g(()) A(ys).
where Ayy: is the normalized advantage, sgn(-) is the sign function, and k > 0 is sensitivity. The reason for using relative difficulty weighting is analyzed and supported by ablation experiments in Appendix D.5 while the analysis of the sensitivity parameter k are presented in Appendix D.I.
Intuitively, when a variant is harder than the group average (D²") > Dy), correct answers (Á > 0) amplified while incorrect ones are softened. Conversely, when variant is easier than average, correct answers are weighweighted and incorrect ones are penalized more heavily.
In this way, the training process achieves iffficulty-ddaptivve optimization, effectively balancing the contributions of both the relative difficulty within a problem group and the magnitudes of advantages. The theoretical validity of this balancing strategy rigorously established in Appendix B where Theorem B. demoonsttasss that reducing gradient variance accelerates convergence, and Corollary B.2 shows that our normalization and weighting strategy significantly decreases variance while preserving unbiased gradient estimates. Moreover, Appendix Dppendix([mmateeeee [mathematicall confirms that the optimization signal is strongest when the ratio of correct to incorrect samples is approximately 1:1, providing a solid theoretical foundation for our dynamic difficulty adjustment mechanism.
3.4 REWARD-RANGE- RANGE- -BASED ADVANTAGE RESCALING
In GRPO-based reinforcement learning, we observed that advantage estimation can become unreliable when the reward range within rollout group is small. If rewards are tightly clustered, standard z-score normalization may exaggerate minor differences, producing misleading optimization signals. As result, the model may over-reward trivial gains while overlooking substantial ones.
Consider a reward scheme where correctly formatted responses receive 0.1 and fully correct re- sponses receive 0.9. Suppose have samples, each rolled out 5 times: Sample A with rewards [0, O, O, O, 0.1] and Sample with rewards [o, 0, 0, 0, 1]. After applying standard 2-score normaliza- tion, both groups assign the same advantage (-0.45) to the zero-reward rollouts and the same high advantage (1.79) to the last rollout. This overestimates the gain from minor formatting correctness in Sample A and underestimates the greater achievement in Sample B.
To address this issue, we propose raward-rnngebbsed advantage rescaling. Let R2 = (ri, .2,....,TT,, denote the rewards of all rollouts for problem q, with the maximum possible reward range Rmaxx We define the reward range and the rescaled advantage as
Arg (max(RRy) min(R.))) R Ramge(ys) Arg A(ys)
where A(y:) is the normalized advantage obtained via standard z-score. Intuitively, this method ensures that the magnitude of the advantage reflects the actual variability in rewards, preventing

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison multimodal mathematical benchmarks. Bold denotes the best perfor- mance among models, underline marks VLMEvalKit (Duan et al. 2024), while results for other models are taken from Mong et al. (2025) and Yao et al. 2025). For cach entry, the score before "/" is our re-evaluation using the officially relcased checkpoints, and the score after "f" reported in the original paper.
overall performance. Evaluation conducted with
Model
Mata
VathVerse
MathVision
OlympiadBench
WeMath
MMK12tesl
Avg.
Sondde..-Sooman
66.8
52.0
41.3
4R.9
72.6
55.3
56.15
GPT-4o
63.8
50.2
30.4
35.0
68.8
68
ol
73.9
57.0
60.3
68.0
98.7
73.9
72. 2.
Gemini2-fash
70.4
59.3
41.3
51.0
71.4
65.2
59.77
In2em &L
64.4
39.5
19.7
12.3
$3.5
45.6
39.17
Qwen-2.5.VT.777
47.9
IntemWVL2.5VLL.. 38-388
71,9
49.4
31.8
32.0
67.5
$8.0
51.77
Qwen-2.5VVT.221
71.7/74.7
49.9
InlemVVL2.55 1 VL-78B
72.3
51.7
32.2
31.1
66.3
61.6
52.53
Qwen-2.5-V1-7211
IntemVL22--8BMMP
68.9
35,5
21.5
7.8
53,5
34,5
36.95
InlemVL22.538B-MMP
73.8
46.5
32.3
25.6
66.2
48.3
48.78
QVQ-728-Presiww
71.4
48.2
35,9
33.2
65.4
61.5
52.60
Adora-7B
73.5
50.1
23.0
20.1
64.2
$8.1
48.17
R1-Onevisioo--B
64.1
47.1
23.5/29.9
17.3
61.8
39.8
42.27
OpenVLThinker-777
70.2
47.9
25.3
20.1
64.3
60.6
48.07
MM-Euccka-7B
71.7/73.0
50.3
26.9
20.1
66.1
64.5
49.93
R1 VL-777
73.5/75.4
52.8
29.5
21.3
67.9
68.8
$2.30
SFT-7B
66.4
49.6
21.3
16.2
57.6
54.3
44.23
DAYA-GRP-7B (Ours)
74.2
57.6
32.1
23.1
69.3
70.2
54.58
6.70
8:70
6.66
0.00
8.5
Siro
6.00
8:55
6:00
00.0
5.58
0.00
8.45
6.60
4.40
6.40
Training Steps
Training
Traliing Steps
(a) RQI: Effectiveness the DIVA-GRPO
(b) RQ3: Effectiveness of RRB General GRPO Methods
(c) RQ4: Impact of DIVA
GRPO on Efficiency and Speed
Figure 3: Effect of Training Steps Model Performance on the Validation Set.
minor differences from being exaggerated and providing a more reliable optimization signal. This rescaling method can be applied independently difficulty-weighter scaling and is compatible with any GRPO-based framework, improving the stability and efficiency of policy optimization.
EXPERIMENTS
In this section, we conduct extensive experiments to evaluate the effectiveness of DIVA-GRPO, complemented by ablation studies assess the contributions of its key components. Our evaluation is driven by the following research questions:
RQ1 (Effectiveness): How does DIVA-GRPO compare recent advanced systems?
RQ2 (Completeness): What impact does removing key components have on model performance?
RQ3 (Generalization): How well does RRB- Res-Reeg generalize to standard GRPO?
RQ4 (Efficiency): What impact does DIVA-GRPO have on training efficiency?
4.1 EXPERIMENTAL SETUP
Benchmarks. Our evaluation covers six diverse benchmarks: Math VistaaL et al., 2023 Math-
Verse(Zhang et al.] 2022)), Math /Wang et al., 2024), OlympiadBenchh(He et al. 2024). We-
Math al. 2024), and MMK12- Men al. 2025).
Baselines. Our comparisons cover three categories of models: (i) closed-source proprietary MLLMs; (ii) open-source base ML,.Ms (ii)) open-source fine-tuned M.L.Ms.
7

---

Published as a conference paper at ICLR 2026
DINS-GRIO
800
5000
6008
300
102
0.06
200
100
GXPO
400
50
100
150
201
100
150
200
100
150
200
Traiing Skeps
Toiting Skeps
Traiiig Stepe
(a) Time per token generation
(b) Average length of response
(c) Time per step
Figure 4: Statistics of time and training speed during training and testing, showing that our method only leads a slight increase in training time per step, primarily due to the increased output length.
(Bai et al. (025) with A learning rate of 10-6 We employ the latest EasyRI framework for training. Specifically, for methods that do not generate variants, we set k = 10, while for methods that do generate variants, we set k 5, using both N and N 3 settings ensure the fairness the experiment. This setup guarantees rigorous and fair comparison across all baselines. Difficulty scores D, are initialized to (Dmin = 1, Paxx = 9) with r = The detailed parameter analysis of can be found in Appendix D Textual reasoning hints are pre-generated offline, while image perturbations are applied online. All textual variants and reasoning sequences are generated offline using GPT-o3.
Implementation. Our method is trained on Swee2.5--L-77B-Imeg
4.2 MAIN RESULTS (RQ1: EFFECTIVENESS)
We report results on 7B-scale models, with the main experiments conducted on the R1-Share dataset. DIVA-GRPO demonstrates superior performance compared to both the base models and other multimodal training approaches. As shown in Table we provide comprehensive comparison against state-of-the-an systems across five widely used and challenging benchmarks.
Our method achieves state-of-the-arr performance across all datasets at the 7B scale, delivering consistently strong results on both Chinese and English benchmarks with an average score of 54.58. Remarkably, on Math Vista, Math Verse, and WeMath, its performance is already on par with the much larger
while substantially outperforming several open-source base models (e.g., Qwen2.5-VLL3225 Intern V.2.5VLL2.--L--781 and proprietary systems (e.g., GPT-4o, Claude 3.7- Sonnet), showcasing notable efficiency and cost-effectiveness. Nevertheless, on more challenging competition-leve mathematics tasks, our method still falls short of ultra-large open-soure, models and cutting-edge edge proprietary systems, likely due to inherent limitations in model capacity and training data coverage. We further explored applying direct SFT to the backbone model using reasoning traces generated by GPT-o3 and found that such direct training only failed to improve performance but even underperformed the base model-further underscoring the effectiveness of our approach. Compared with its backbone model, Qwen2.5-VL-7B. our method consistently improves results across all benchmarks, achieving an average accuracy gain of 8.23 points, which highlights its clear advantages over existing multimodal training frameworks. The discussion of different external models can be found in Appendix D.4
Figure further illustrates the training dynamics: in the early stage, samples have not yet formed clear difficulty distribution, with most concentrated around medium difficulty and exhibiting only small advantages. As training progresses, the distribution gradually expands toward both easier and harder problems, and advantage signals become more distinct across difficulty levels. In later stages, the model maintains informative and balanced advantage signals even on high-difficllyy samples, rather than collapsing into trivial "all-correct" or "all-wrong" states. These dynamics highlight the effectiveness of DIVA-GRPO in sustaining stable and efficient optimization.
4.3 ABLATION STUDY
RQ2 (Completeness): Component Ablation of DIVA-GRPO
We perform ablation studies on the three core components of DIVA-GRPO at the 7B scale using representative benchmarks VMathVista, Math Verse, and MMKI2test) Experiments conducted on 5,000 randomly sampled instances from the MMK12 dataset to ensure timely and fair comparisons. Each 20 steps constitutes epoch, with a total of 10 epochs of training. We evaluate the impact of
8

---

Published as a conference paper at ICLR 2026
Duruity Sampte Divertatton
Denstry Stampte Dreerbuttion
Dunnit Samate
E
biiocolt
bilioms
Adventage
(a) Train stage
Train stage 2
(c) Train stage
Figure 5: 3D kernel density estimation (KDE) surfaces of the joint distribution between problem difficulty and global advantage across different training stages. The surface height reflects the sample density, illustrating how the model's learning dynamics evolve with respect difficulty and advantage.
removing Adaptive Variant Generation (including Global-Local Advantage and Balance), Difficulty Weighting, Reward-RR SawarrdRange-Baae Based Advantage Rescaling (RRB- (RR--Rescaling, and Global-Local Balance (G-L Balance), and compare each variant with the full model.
As reported in Table 2 removing any single component consistently decreases performance, with the full DIVA GRPO model achieving the highest accuracy across all benchmarks. These results indicate that all components contribute complementary gains and none can be omitted without performance degradation, highlighting the necessity of the complete model design.
Furthermore, as shown in 3a we compare DIVA-GRPO with GRPO, GSPO Zheng et al. (222), and DAPO Yu et al. 2025)
Table 2: Ablation study of EIVA-GRPO, showing that each component provides gains and the full model achieves the best performance (accuracy, %).
throughout the entire training process. Ex perimental results demonstrate that, under strict control of time and total response size, our method outperforms GRPO, GSPO, and DAPO terms of both training speed and stability. Additionally, we have evalu- ated the performance our method on Owen3--VL-4B and 8B, with the detailed results presented in Appendix D.3
Method
Vista
Math
MMK12test
Avg.
Generation
70.0
who Diffcuutt-Weighing
69.9
55.7
66.5
61.0
RRB URB-Rescaling
71.5
55.2
54.7
63.8
Balance
70.8
55.4
Full DIVA-GRPO
73.2
56.3
68.8
66.1
RQ3 (Generalization): Generalization of RRB-Reseee BRB--Rescaling
To verify the broader applicability of Advantage Rescal- incorporate it
Table 3: Evaluation of RRB-Rescaling on standard GRPO.
Neward-Range- ing (RRB-Rescaling), into standard GRPO without any other modifications. Table shows that adding RRBB
Method
MathVista
MathVerse
MMK12test
Avg.
GRPO (base)
69.3
$2.8
58.6
60.23
70.0
53.6
62.9
62.17
RRB-Rescaling increases the average accuracy from 60.23 to 62. 17, with improvements observed across all three benchmarks. This is further illustrated in Figure 3b This demonstrates that RRB- Rescaling enhances both stability and performance in general GRPO setting, confirming that this component limited
but can benefit other GRPO-style training frameworks.
RQ4 (Efficiency):
Training Efficiency and Stability
We compare the training performance of DIVA-GRPO and GRPO on Qwen2.5-VL-7B. As shown in Figure 3c| we measure the number of steps required by DIVA-GRPO to reach the test-set optimum within 10 epochs of GRPO training. When GRPO uses n 10 and DIVA-GRPO uses n we ensure constant number of sampled sentences when N 1 and consistent amount of input data when = Figure 4a shows the time consumed per token during generation, Figure 4b) illustrates the average response length, and Figure 4c depicts the time per step. Next, calculate the speedup of DIVA-GRPO in terms of both steps and time.
For N 1, ensure the consistency of the response count, meaning the total number of samples remains equal, while DIVA-GRPO accelerates the steps by a factor of 1.90x. According to the statistics, while the time per token increases by only 13% and the time per step by just 4%, the total time comparison shows that our method consumes an average of 7.98 minutes per step, compared 7.61 minutes per step for the GRPO method. For generating variants and think steps,

---

Published as a conference paper at ICLR 2026
approximately 32 minutes required for every 5000 samples, while the generation of visual variants is almost instantaneous and does not incur additional time costs, as is already included in the training time. Even when factoring in the time for generating variants, our method achieves a 1. 1.76x speedup in terms of time.
For N = 3, we ensure that the data requirements remain unchanged, while DIVA-GRPO acceler- ates the steps by factor of 2.55x. In this case, the average time per step is 12.03 minutes. Even when accounting for the time required to generate variants, our method achieves 1.56x speedup in terms of time. This clearly demonstrates the efficiency of our approach.
RELATED WORK
With the increasing application of GRPO in MLLMs to enhance reasoning, several challenges remain unresolved, particularly sparse rewards and advantage vanishing. Existing solutions can be grouped into three main approaches: sample augmentation and problem-space expansion, sample selection and utilization, and reward design.
Sample augmentation and expansion methods aim to enlarge the problem space using difficulty- aware mechanisms or diverse sample generation to improve exploration and generalization. Noisy Rollout (Liu et al., 2025a) uses image noising to facilitate reinforcement learning, introducing visual diversity to prevent overfitting and improve robustness. Hint-GRPO (Huang al. 2025) adapts hints based on task difficulty, enhancing data efficiency for hard samples. DeepVideo-BB R1 (Park et al. 2025) adjusts difficulty to help variants regain advantage, though it does not reshape single-sample diffcultty distributions, leaving sparse rewards in extreme cases. RI-share VL ( (Yao et al. (2025) expands the space with multiple variants and applies global advantage to shift difficulty, but lacks complete mechanism to mitigate sparsity.
Sample selection and utilization methods focus computation on effective data while ignoring unhelpful samples, improving efficiency but risking premature abandonment of hard cases. VL- Rethinker (Wang al. 2025a) uses selective replay to alleviate reward sparsity and preserves advantage diversity by replaying strong past samples. MM-Eureka (Meng et al., (025) selects prompts with partial correctness, filtering zero-advantage cases stabilize training. While helpful, these methods do not maximize overall sample utilization or provide global optimization view.
Reward design methods aim to provide denser, continuous signals to reduce sparsity and stabilize reasoning. MM-Eureka introduces a hyperparameter À to weight format rewards. R1-VL /Zhngg al. 2025) proposes StepGRPPO, incorporating geasoning-accuraae and step- validity rewards for denser feedback. The introduced indirect rewards may therefore deviate from the primary objective and potentially mislead the model. Deep Video-RI introduces Reg-GRPO, transforming GRPO into regression loss, while GRPO-LEAD (Zhang& & Zuo 2025) weights hard problems in advantage computation via dynamic difficulty awareness. By removing the variance term in advantage estimation, Dr. GRPO (Liu et al.) 2025b) seeks reduce variance and stabilize the learning signal. This shares the motivation with our RRB method, as both aim to provide more stable learning signals. Although these methods mitigate convergence and stability issues, they fail to address the fundamental challenges of sparse rewards and diminishing advantages.
CONCLUSION
In this work, we identified and addressed a fundamental limitation of GRPO-based reinforcement learning for multimodal large language models: reward sparsity and vanishing advantages, which hinder effective long-chain reasoning. To overcome this, we proposed DIVA-GRPO. difficulty- adaptive variant advantage framework that dynamically assesses problem difficulty, generates tailored variants, and computes both local and global advantages with difficulty- -aware normalization and reward-range-based rescaling. This design ensures stable and informative optimization signals across problems of varying difficulty, mitigates reward sparsity, and enhances training stability. Extensive experiments six multimodal reasoning benchmarks demonstrated that DIVA-GRPO consistently outperforms existing CaRPO-based methods in reasoning accuracy, convergence speed, and training efficiency. Overall, our approach provides generalizable framework for improving reinforcement learning in MLLMs, offering a principled way to balance exploration, sample utilization, and advantage estimation for complex multimodal reasoning tasks.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the Strategic Priority Research Program of the CAS under Grants No. 30D06660030, the National Natural Science Foundation of China (NSFC) under Grants No. 62276248, the Key Research and Development Program of Xinjiang Uyghur Autonomous Region Grant No. 2024B03026, the Beijing Nova Program under Grants No. 20250484765, and the Youth Innovation Promotion Association CAS under Grants No. 2023111.
ETHICS STATEMENT
In this study, all data collection and analysis were conducted in compliance with ethical standards, and no human subjects or animal experiments were involved. The research did not use any sensitive personal data privacy information, and all experimental designs and data processing methods adhered to relevant legal and ethical guidelines. Regarding dataset releases, we have strictly followed applicable academic ethics regulations, ensuring that all data and code sharing comply with legal and ethical requirements. There are conflicts of interest in this study, and all funding sources and support have been clearly disclosed. The study ensures transparency, integrity, and fairness, strictly adhering to relevant research ethics and compliance standards.
REPRODUCIBILITY STATEMENT
To ensure full reproducibility of our experiments, we provide all code, configurations, and instructions required to replicate our results in the repository: repository includes seripts for training and evaluation, pre-processing routines, and detailed README documentation describing the dependencies and steps to reproduce the reported results.
I GRPO. The
REFERENCES
lean-Baptinn Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican. Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning, Advances in neural information processing systems, 35:23716-23736 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick. and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425 2233, 2015.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXi::2502.13923 2025.
Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. Mo novel bench- mark for multi-domain multi-step multi-modal chain-of-thouge arXiv preprint arXiv:2405.16473. 2024a.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12)220010, 2024b.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference multimedia, pp. 11198-1 1201, 2024.
Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong- Lee, and Wynne Hsu. Video-of-thought: Step-by-step video reasoning from perception to cognition. arXiv preprint arXiv:2501.032300 2024.
11

---

Published as a conference paper at ICLR 2026
Haowen Gao, Liang Pang, Shicheng Xu, Leigang Qu, Tat-Seng Chua, Huawei Shen, and Xueqi Cheng. Generative ghost: Investigating ranking bias hidden in ai-generated videos. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 6403-6411 Association for Computing Machinery, 2025. doi: 10.1 145/3746602.33754888
Shasha Guo, Liang Pang, Xi Wang. Yanling Wang, Huawei Shen, and Jing Zhang. Geovlmath: Enhancing geometry reasoning in vision-language models via cross-modal reward for auxiliary line creation, arXiv preprint arXiv:2510. 1020, 2025.
Chaoqun He, Renjie Luo. Yuzhuo Bai, Shengding Hu. Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad- bilingual multimodal scientific problems. arXiv preprint arXiv: 14008, 2024.
Jie Huang and Kevin Bhen-Chuan Chang. Towards reasoning in large language models: survey. arXiv preprint arXiv:22 22w. 22.11.11,,,,, 2022.
Qihan Huang, Weilong Dai, Jinlong Liu, Wanggui He, Hao Jiang, Mingli Song, Jingyuan Chen, Chang Yao, and Jie Song. Boosting milm reasoning with text-debiased hint- gint. arXiv preprint arXiv:2503.23905. 2025.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276,, 2024.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better under- standing vision-language models: insights and future directions. arXiv preprint arXiv:2408. 12637. 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19777 19742. PMLR, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 11134499-349111 2023.
Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh. Noisyrollout: Reinforcing visual reasoning with data augmentation. arXiv preprint arXiv:2504. 13055, 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310. w2Xiv22310.02255. 02255, 2023.
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang. Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. CoRR, 2025.
Fanxu Meng, Haotong Yang, Yiding Wang, and Muhan Zhang. Chain of images for intuitively reasoning. arXiv preprint arXiv:2311.09241,, 2023.
Jinhui Pang, Xinyun Yang, Xiaoyao Qiu, Zixuan Wang, and Taisheng Huang. Mmaf: Masked multi-modal attention fusion to reduce bias of visual features for named entity recognition. Data Intelligence, 6(4):1 1 1133, 2024. doi: 1.37747206--7044.122222.0000
Jinyoung Park, Jechye Na, Jinyoung Kim, and Hyunwoo Kim. Decpvideo-rl: Video reinforcement fine-tuning via difficulty-aware regressive grpo. arXiv preprint arXiv:2506.07464, 2025.
12

---

Published as a conference paper at ICLR 2026
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve numan-like mathematical reasoning? arXiv preprint 2rXiv:2407.01284 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning. Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-5374) 2023.
Gabriel Sarch, Snigdha Saha, Naitik Khandelwal, Ayush Jain, Michael Tarr, Aviral Kumar, and Katerina Fragkiadaki. Grounded reinforcement learning for visual reasoning. arXiv preprint MrXiv:2500.23668, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXivv2402.033300 2024.
Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. Vl-rethinker: Incentivizing self-reflection of lanoualanguage models with reinforcement learning. arXiv preprins 08X8: 2025a.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095-95169, 2024.
Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Multimodal chain-of-thought reasoning: A comprehensive survey. arXiv preprint arXiv:2503.12605 2025b.
Yixuan Wu, Yizhou Wang, Shixiang Tang, Wenhao Wu, Tong He, Wanli Ouyang, Philip Torr, and Jian Wu. Dettoolchain: A new prompting paradigm to unleash detection ability of mllm. In European Conference on Computer Vision, pp. 164-182. Springer, 2024a.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma. Chengyue Wu, Bingxuan Wang, al. Deepseek-v12: -fxture-of-eeperts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302. 2024b.
Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal I1m logical reasoning benchmark in visual contexts. arXiv preprint
2024.
Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. setrch-in-thhechaeee Interactively enhancing large language models with search for knowledge-intensive tasks. In Proceedings of the ACM Web Conference 2024, pp. 1362-1373. Association for Computing Machinery, 2024. doi: 10. (0.1144/3589334.66555
Huanjin Yao, Qixiang Yin, Jingyi Zhang, Min Yang, Yibo Wang, Wenhao Wu, Fei Su, Li Shen, Minghui Qiu, Dacheng Tao, al. RI-sharevl: Incentivizing reasoning capability of multimodal large language models via share-grpo. arXiv preprint arXiv:2505.1667. 2025.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11(12)mwwe4033 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source I1m reinforcement learning system at scale. arXiv preprint arXiv:2503. 2025.
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint ar X5v: 20.::2233./37 2025.
Jixiao Zhang and Chunsheng Zuo. Grpo-lead: difficulty reinforcement learning approach for concise mathematical reasoning in language models. arXiv preprint 2009%9250..0966, 2025.
13

---

Published as a conference paper at ICLR 2026
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? In European Conference Computer Vision, pp. 169-186. Springer, 2024.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu. Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.1807,, 2025.
Xiongtao Zhou, Jie He, Lanyu Chen, Jingyu Li, Haojing Chen, Victor Gutiérrez-asullto, Jeff Z Pan, and Hanjie Chen. Miceval: Unveiling multimodal chain of thought's s quality via image description and reasoning steps. arXiv preprint arXiv:2410. 14668, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models, arXiv preprint arXiv:2304. 10592, 2023.
14

---

Published as a conference paper at ICLR 2026
A DETAILED BENCHMARKS AND BASELINES
A. BENCHMARKS
We evaluate six multimodal mathematical reasoning benchmarks:
Math Vista: comprehensive benchmark combining challenges from diverse mathematical and visual tasks. It consists of 6,141 examples, derived from 28 existing multimodal datasets involving mathematics and 3 newly created datasets (i.e., 1QTest, FunctionQA, and PaperQA).
Math Verse: An all-around visual math benchmark designed for an equitable and in-depth evalua- tion of Multi-modal Large Language Models (MLLMs). It includes 2,612 high-qualing multi- subject math problems with diagrams, each transformed into six distinct versions, totaling 15K test samples.
Math Vision: A meticulously curated collection of 3,040 high-quality mathematical problems with visual contexts sourced from real math competitions. Spanning 16 distinct mathematical disciplines and graded across levels of difficulty.
Benh dataset comprising 8,476 math and physies problems sourced from Interna- tional Olympiads, Chinese Olympiads. and the Chinese College Entrance Exam (GaoKao). It features expert-level annotations for step-by-stey reasoning.
WeMath:
Inspired by human- mathematical reasoning. WeMath is the first benchmark specifi- cally designed to explore problem-solving principles beyond end-to-end performance. It meticu- lously collects and categorizes 6.5K visual math problems, spanning 67 hierarchical knowledge concepts and 5 layers of knowledge granularity.
MMK12 test: A dataset focusing on subject-spectes tasks in physics, chemistry, biology, and mathematics. It includes questions with verified answers and is designed to evaluate cobject-speciffe reasoning capabilities.
A.2 BASELINES
Following the taxonomy used in Table I we group baseline systems into three categories:
Closed-source proprictary MLLMs: Claude3.7-Sonnet, GPT-4o, ol, Gemini2-flash.
Open-source base MLLMs: IntemVL2.5- Qwen-2.5- IntemVL2.5-VL-388B IntermW1...5VII-788
Open-source fine-tuned MLLMs: InterNLL2.588-MMPO Intem QVQ-
72B-Previee Adora-7B, R1 R1-Onevision-7B, OpenVL.Thinke:-7B,, MM-Fureka-7BB R1-
VL.-eV
Our proposed system, DIVA-GRPO-7B, instantiated on the same backbone as Qwen2.5-VL-7B for fair comparison.
B THEORETICAL ANALYSIS OF ADVANTAGE NORMALIZATION AND DIFFICULT-WEIGHHTED WEIGHTED BALANCING
Theorem B.1 (Gradient Variance Control). Let g(f) be the stochastic policy gradient estimator
0
A V. Voloogopaals))
where As denotes the advantage function. Suppose E[g(O)] VJ(()) unbiased and Varlg(0)] < co. Then for step size >> O, the expected squared error satisfies
0.+1 - 0 < EE 00 0 1- n + Var(g(O.)).
(I)
This inequality shows that the convergence rate depends critically on the variance of the gradient estimator. Reducing gradient variance improves optimization stability and accelerates convergence.
15

---

Published as a conference paper at ICLR 2026
Corollary B.2 (Advantage Normalization and gradient estimator
Balancing). Consider the adjusted
*
((0) Σ
A(ys) Vo log (V)));
1=1
where the adjusted advantage is defined as
A(y:) exp (K: (k. (D. (D. -DD:-D))gnn(A((u))) A(v:).
Here Aly:) the batch-level normalized advantage (either local or global), D; is the difficulty coefficient of the variant q²),, D the mean difficulty within the group, and k > 0 is sensitivity parameter.
Then:
1. Unbiasedness: Normalization enforces zero-mean scaling, =
2. Variance Reduction: aeffeullty-eighhte scaling adaptively reweights samples. Hard problems amplify correct signals and soften incorrect ones, while easy problems down- weight correct signals and penalize incorrect ones more strongly. This rebalancing prevents domination by outliers, ensuring
Varlğ(0)] I Varíg(0)}.
Substituting into Theorem B.1 yieldds
[110011- 0-1 S EE 0: - (arl
(2)
Hence, the proposed balancing strategy preserves unbiased gradient estimates while reducing update variance, leading to more stable and efficient convergence.
Discussion. Theorem B.1 establishes that variance directly governs the convergence behavior of policy gradient methods. Corollary [B. demonstrates that our sufficulty-weighee and normalized advantage balancing reduces variance while maintaining unbiasedness. Intuitively, this balances con- tributions across problem difficulties and prevents either global or local advantages from dominating optimization, thereby stabilizing training and improving efficiency.
C THEORETICAL PROOF OF OPTIMAL REWARD BALANCE UNDER Z-SCORE NORMALIZATION
To further justify our design choices, we provide a theoretical proof. Our method adaptively assesses the difficulty of each problem and generates variants to keep the overall difficulty moderate for the current model, aiming to maintain an approximately 50/50 ratio of correct and incorrect samples (rather than relying we consider a binary-reward setting, where correct answers receive the maximum reward and incorrect answers receive zero. The proof shows that under this balanced setting, optimization is most efficient: when correct and incorrect samples are equally represented, the estimated gradient aligns most closely with the optimal update direction, leading to more effective model improvement.
single correct incorrect sample produce advantage). For simplicity,
Setup and notation. Consider single batch (or mini-batch) of rollouts in a GRPO-like procedure. Rewards take only two values (0, Rmax). Let be the number rollouts with reward Rmax and set
/ k
[0,
We compute advantages by 7-score normalizing rewards over the batch:
F
A,
or F = pRuaxx
o, VVr(r) Rmux V 1(1 11).
16

---

Published as a conference paper at ICLR 2026
Let 9: V (agss() F denote the per-sample score-gradient vector. We assume the
parameter update direction (up to a positive constant) follows the common form
A
A Di-
T1
Fix reference unit vector v E IRd that represents the desired "optimal" direction. Define
Eig Rmaax
B 0],
the average projection of class gradients onto
Lemma C.1 (advantages for binary rewards). In the above notation the two advantage values are
Riax A+
1-
A
Or
1
In particular, Remax cancels and does not affect the dependence on μ.
Proof. Direct substitution yields
A4 Rinx R.Rmax
1- 1
Ranx v 11(1 11)
Vu(1 - 11)
P
O P.Baxx
1
A_ = Rmax 1n(1 ((1-w)
V - (1)
1 p
Lemma C.2 (batch update projection onto v). Under the class-mean approximation (aggregating samples of each reward class into their class means), the projection of the update onto satisfies (up to positive constant)
v A0 x 5A48+ (1-11)11)) 1)A =V 10] (5) -s
Hence, the absolute projected magnitude is
A & V 11(1 11) 18+ - 8-/.
Proof. Aggregate the sum by reward classes:
A x μA Blg r = Rmax] + (1 1)AA Eig r = 0j.
Projecting onto u and substituting the expressions for A gives
u AA $4 + (1 p)A s_ Vu(1 μ) Vu(1 1) Vu(1 1) (s+ s_).
Taking absolute value yields the stated expression.
Theorem C.3 (optimality of }). For any fixed 84,8_ E R. the absolute projected update magnitude
F(P) == ()(1- -s1
over μ [o, 1] attains its maximum at J. Therefore, under the stated assumptions, batch with half the samples having reward Rmax and half having reward O maximizes the expected update magnitude in direction
Proof. Since |s+ s-1 is independent of μ, it suffices to maximize g(H) = Vu(1 14) on [0, 1]. Note
((4)2 = 111 (1) -1² + μ.
concave quadratic with vertex 3. where it attains the maximum value 1. Hence g(u) is maximized f 1, and is F(n).
17

---

Published as a conference paper at ICLR 2026
Two representative geometric cases. Here discuss two extreme yet representative cases separately.
Corollary C.4 (Case A: opposite-class gradients). Assume that the average class gradients are exactly opposite along the reference direction, i.e.
--=-
Then the projected update satisfies (up to a positive constant)
A0
vu(l p)) (s+ --) 2 μ(1 11) 8+1
and hence the absolute projected magnitude is proportional to
1 A0 x 2|s+ (illllu(1111
Consequently this magnitude maximized
Proof. Substitute s_ = S+ into the general expression T. =0 x ( yields the displayed expression, which is scalar multiple of Vu(1 nJ). The factor Vu(1 14) is maximized μ zo2 hence the result.
() - -s-). This
Corollary C.5 (Case B: orthogonal-class gradients). Suppose the class-mean gradient for the negative class is orthogonal to the reference direction, i.e.
8_ =, 0,
while ++ + O. Then
v AO
v Vu(1 11)
so the absolute projected magnitude is
10 AA
x
|s+|V 1111 -0-1 p),
which is again maximized at ==
Moreover, consider the stronger geometric picture where the mean gradient of the positive class lies along with norm [|g+ 1l = a and the mean gradient of the negative class is orthogonal with norm [Ig-11 3. If the normalized advantage-weighted contributions from the two classes have equal amplitudes (i.e. av μ1) 3 μ)), equivalently is the sum of two orthogonal vectors of equal magnitude, so the angle 6 between the update and satisfies cos à = 1/v2 (independent of μd), while the absolute magnitude of the projection onto v still scales as / - μ4) and is maximized at Y/2
3), then the resulting update vector
Proof. The first statement follows immediately from substituting s 0 into the general projection formula. For the geometric remark, if the two (orthogonal) class-mean vectors have equal weighted amplitudes, their vector sum has length /2 times one amplitude, and the projection onto equals that amplitude; dividing by the total norm gives cos ø = 1/ /2.2. The dependence on fr appears only through the common amplitude factor Vu(1 - 14), which is maximized at μ }
Remarks.
1. The conclusion does not depend on Rmaxi z-score standardization cancels the reward scale.
2. If s_ then v AO > 0 and the update moves toward if s the update moves opposite to v. In either case the absolute magnitude the projection is maximal at μu = }
3. The proof uses simplifying class-mean approximation. In practice each class has internal variance; nonetheless the leading-orde dependence of the expected projection on μ is governed by Vu(1 - μt). Under mild conditions on within-class variance, the qualitative conclusion (maximum at μ = })) remains valid in expectation.
4. For non-binary rewards the algebra changes: one must analyse r=ي for the full reward distribution. The binary case provides clear analytic baseline and shows the key role of z-score normalization in making the effective signal scale proportional to /p(1 - (4).
dr
18

---

Published as a conference paper at ICLR 2026
Variant Description
Gaussian (intensity = 00.45) and Rotate (multiples of + Variant text / Add text to image
Gaussian (intensity 00.45) or Rotate (multiples 30") + Vuriant text Add to image
Gaussian (intensity 0..3 Rotate (multiples of 45") Variant text Add text image
Gaussian (intensity = 0.3) or Rotate (multiples of 90")
Variant text
Original query Variant text
I --hink step
2-think stops
3-think steps
Table C.I: Variants from Hard to Easy.
Table C.2: Performance comparison across different base model.
Model
Mathematies
Physics
Chemistry
Biology
Avg.
Qwen3-VL-od
Qwen3-VL--BB -L-4B-Instrree
65.5
55.5
58.5
56.1
58.9
+GRPO
75.5
68.5
71.0
68.6
70.9
+DIVA-GRPO (Ours)
78.3
70.7
72.7
71.9
73.4
improv.
+2.8
+2.2
+1.7
+3.3
+2.5
Qwen3-VL-SH
Qwen3- Swwn3-VV--B--Ae
VL-8B-Instruct
71.5
64.5
66.5
65.5
67.0
+GRPO
77.5
72.5
74.0
73.2
74.3
+DIVA-GRPO (Ours)
74.3
78.2
78.5
improv.
+4.7
+1.8
+4.2
+4.2
Conclusion. This theoretical analysis rigorously justifies the core motivation of our method. When batch advantages are computed via z-score normalization and rewards are binary {0, Rmax), the expected magnitude of the update projected onto any fixed reference direction v is proportional to / (1(1 - pt), which attains its maximum when μ 1. In other words, a batch balanced with equal numbers of correct and incorrect samples provides the strongest expected directional signal for optimization. This result supports our design choice adaptively generating problem variants to maintain approximately 50/50 ratio of correct and incorrect samples, ensuring that gradient updates most efficiently align with the optimal update direction and thereby maximizing model improvement.
D SUPPLEMENTARY EXPERIMENTS
D. PARAMETER k
In the experimental setup as the ablation study, assess the performance of the proposed method across different values of the parameter k. As shown in Figure D. the model achieves optimal performance when & 0.1. with peak accuracy difference only 0.5%. When difficulty weighting is disabled) or k = 1 (i.e., excessive difficulty weighting). the algorithm exhibits similar performance in the early training stages due the lack of distinction in sample difficulties. However, training progresses, the differences in sample difficulties become more pronounced, and overly extreme weighting results in degraded performance or training instability. When 0.1, the maximum weight applied is roughly 1.5 times the original advantage, while the maximum reduction In(1.5)
(i.e..
about 0.67 times the original value. From the weighting formula, we derive k
max (D -D,)
D.2 PARAMETER AND SAMPLE SAMPLING
In the same experimental setup as the ablation study, we evaluate the performance of the proposed method across different values of the parameter 7) The parameter r) represents the magnitude of change in the difficulty assessment when question is answered correctly or incorrectly. Specifically, when 7 = 2, the sample requires four consecutive correct (or incorrect) answers before difficulty level is updated required for the update, and when = 8, only one correct (or incorrect) answer needed for the update.
the easiest (or hardest) level. When r 4. two correct (or incorrect) answers
19

---

Published as a conference paper at ICLR 2026
AssIano Willive
Traraig Stppe
Traniig Stpe
Tratiing Supe
(a) Under different
(b) Under
(c) Ablation study componens
(d) study compunents
Figure D.I: Effect of different hyperparameters model performance on the validation set.
Table D.3: Performance comparison across different external models on Qwen-2.5-VL. VL-7B.
Model
Mathematics
Physics
Chemistry
Biology
Avg.
Qwen-2.5-VL-7B
Qwen-Plus
78.1
61.5
69.1
70.9
69.8
GPT--3
69.6
70.7
70.2
improv.
+0.2
+0.7
+0.5
-0.2
+0.3
We observe that when the difficulty assessment changes too rapidly, the inherent randomness in the model may result in inaccurate difficulty assessments. Conversely, when the difficulty assessment changes too slowly, the model's performance improvement may be reflected in the difficulty evaluations. As shown in Figure the model achieves the best performance when ry = 4. Based on the difficulty update formula, we suggest that should be set as Dam Dain
Additionally, regarding sample sampling, we configure total of nine difficulty level variants, as detailed in Table(C.1] For sampling, we use a normal distribution to sample from the difficulty level adjacent the target difficulty.
D.3 MODELS OF DIFFERENT SCALES
As shown in Table C.2 we evaluate the performance of GRPO and our method, DIVA-GRPO, on different model sizes and architectures, with the experimental setup consistent with the one described earlier. The experimental results on QQwen3-VL.44-Innsee compared to the Base model, our method improves by 14.5 and 11.5, respectively. Additionally, our method achieves an average improvement of 2.5 and 4.2 over GRPO. These results demonstrate that approach is effective across various model sizes and architectures.
and Qwen3 Vi-e---vt--BB-nntrruc show that,
D.4 GENERATION USING DIFFERENT EXTERNAL MODELS
As shown in Table D.3 generate variants and think steps using Qwen-plus and GPT-o3, and test the performance they achieve under the same training conditions. The results show that the impact of using strong weaker external model the final model training performance is minimal, with GPT-o3 achieving only a 0.3 accuracy improvement over Qwen-plin plus. This can be attributed to our of result supervision, where only samples that can correctly answer the final result selected for training, rather than training on all samples. Additionally, we validate the consistency of the generated question variants with the original questions using Qwen-plus. We sample 1,000 question variants for the experiment and find that 98.8% of the variants are consistent with the original questions. This further demonstrates the reliability of our method. The prompts used in the experiments are shown in Section G
D.5 ABSOLUTE DIFFICULTY WEIGHTING VS. RELATIVE DIFFICULTY WEIGHTING
As described in Section 3.3 our method uses relative difficulty weighting instead of absolute difficulty weighting. This is because the difficulty problem and variants differ across two dimensions. For example, problem with difficulty plus variant with difficulty 8 is not directly comparable to problem with difficulty 9. The variant with increased difficulty simply adds some visual noise, while the problem with inherent difficulty is challenging due to its complex logic. Therefore, applying absolute difficulty weighting intuitively does not make sense. However, for variants of the same original problem, the difficulty ranking is clear: image-noise variant > original problem > variant
20

---

Published as a conference paper at ICLR 2026
Nnwvarro Mirantage Batto
Fand Genie
10
0.8
Avcusay (N))
Density
0.6
16
0.4
COT0
30
Vbroeaa Lovel
2
Training Steps
Advantage Value
(c) The accuracy distribution
(a) Proportion of samples with (b) Origin Global & Local samples with different difficulty non-zero advantages
tages Distribution
levels.
Figure D.2: Analysis of the Effectiveness DIVA-GRPO.
with added think-steps. As result, we adopt relative difficulty weighting, where the difficulty within the same problem can be compared. Additionally, as shown in Figure DDIdd we test both absolute and relative difficulty weighting and find that their impact on the model's final performance is minimal. However, from the perspective of logical consistency, we ultimately choose relative difficulty weighting.
D.6 ANALYSIS OF THE EFFECTIVENESS OF THE METHOD
Proportion of Samples with Non-zero Advantage. Our method dynamically estimates problem difficulty and samples variants of appropriate difficulty, effectively mitigating reward sparsity and improving sample efficiency. As shown in Figure D.2a the proportion of non-zero advantage samples in GRPO and GSPO rises rapidly at the beginning of training, indicating a quick increase in training accuracy. However, as the model becomes stronger, most problems become easy, causing sharp drop in the proportion of non-zero advantage samples and limiting late-stage optimization. In contrast, our method adaptively adjusts problem difficulty, keeping the proportion of non-zero advantage samples stable, ensuring the model continually encounters challenging variants, maintains a balanced reward distribution, and sustains effective training throughout.
Analysis of Origin Local Global Advantages. We first apply z-score normalization within each group for both Local and Global Advantage. Due to the higher sparsity and variance in the Global group compared the Local group, the range of advantages.mmxxxdd)) significantly between groups even after standard normalization, causing the group with the larger range to dominate (see Figure D.2b). To mitigate this effect, we further apply batch normalization separately to the Local and Global advantages, followed by standard normalization within each batch. This additional normalization effectively reduces the discrepancy between the two distributions.
min(adv) can still differ
Analysis of Variant Difficulty. To validate that the difficulty of the variants generated by our method aligns with expectations, we randomly sample 500 examples from the MMK12 test dataset. For each example, we generate one variant at each difficulty level, with the specific variant settings detailed in Table C.I This results in 500 samples per difficulty level. For each sample, we perform 10 rollouts using Qwen-2.5 2.5-VL-7B and compute the final accuracy. The distribution of the final accuracy is shown in Figure D.2c| Although the increase in accuracy is not perfectly linear, the overall trend indicates that our sample variants exhibit the expected pattern: easier variants have higher accuracy, and more difficult variants have lower accuracy. This demonstrates the effectiveness of our method.
21

---

Published as a conference paper at ICLR 2026
Table F.4: Performance comparison across different disciplines in MMK12. Bold denotes the best performance among models, and underline marks the best overall performance.
Model
Mathematics
Physics
Chemistry
Biology
Avg.
Closed-Source Models
Claude3.7-Sonne"
57.4
53.4
55.4
55.0
55.3
GPT-4o
55.8
41.2
47.0
55,4
49.9
ol Gemini2-flash
81.6
68.8
71.4
74.0
73.9
76.8
53.6
64.6
66.0
65.2
Open-Source General Models
Intern V1.2.5-
46.8
35.0
50.0
50.8
45.6
Swen-2.5-VL-72
58.4
45.4
56.4
54.0
53.6
Intern V12.55-V---3
61.6
49.8
60.4
60.0
58.0
Qwen-2.5-VL-32B Intern V1255V12255VL--78 Qwen-2.5-VL--22
71.6
59.4
69.6
66.8
59.8
53.2
68.0
65.2
61.6
75.6
64.8
69.6
72.0
70.5
Open-Source Reasoning Models
26.6
25.0
42.4
44.0
34.5
41.4
42.8
55.8
53.2
48.3
QVQ-72B-Preee Preview
61.4
57.4
62.6
64.4
61.5
Adora
63.6
50.6
59.0
59.0
58.1
R1-Onevision
44.8
33.8
39.8
40.8
39.8
OpenVLThinker
63.0
53.8
60.6
65.0
60.6
MM-Eureka-7B
65.2
64.5
DIVA-GRPO-7B (Ours)
78.3
62.2
69.6
70.7
70.2
E EFFECT OF QUESTION DIFFICULTY ON GRPO OPTIMIZATION
To further validate that providing the model with moder- ately difficult questions is conducive to optimiza- tion via GRPO, we conducted a new experiment. Two sets of 5,000 samples were drawn from the 12K12 training data: the first set consisted of randomly sam- pled instances, while the second set comprised instances of moderate difficulty. Both sets were used to train the model under standard GRPO with rollout parameter of l: = 5. As shown in Figure E training on the mod- erately difficult data enabled faster and more effective model optimization.
Accuracy on Ted
0.45
random sampling
moderatuly diffioult sompling
0.40
20
40
60
80
Training Steps
Figure E.3: Comparison of GRPO optimiza- on randomly sampled versus moderately difficult questions.
CROSS-DISCIPLINARY PERFORMANCE ON MMK
For the MMK12 dataset, we observed an interesting phenomenon: although our model primarily trained on mathematics problems, also demonstrates improved reasoning capabilities in physics, chemistry, and biology. The experimental results are summarized in Table F.4] Our model achieves the best performance among 7B models across all four disciplines and even surpasses many closed- source commercial and open-source general large models. Comparative results for other models can be found in Meng al. (2025).
G PROMPTS USED IN EXPERIMENTS
In this section, we provide the detailed prompts used our pipeline, including variant generation, reasoning, and verification.
22

---

Published as a conference paper at ICLR 2026
G.1 PART I: VARIANT GENERATION
Generate 5 distinct variants of a problem while preserving the exact same correct answer. Variants must include the token <image> and be wrapped in <variant*>, </variant>> tags.
Purpose:
PROMPT VARIANT GENERATION TEMPLATE
Generate 5 distinct variants the following problem that:
1. Preserve the exact same correct
as the original.
2. Use significantly different wording, sentence structure.
3. You can adjjust the sentence length--either making it concise (using streamlined language) or extending it (by explaining the question content in detail or complicating it with advanced language to increase -ifficulty)-butt must ensure correctness.
4. Format of variants as, include <image> in the variants:
[a variant's full toxt]</variant1> <variant2> (Sccond variant's full text]</variant2>
<variant5>>Fifee) variant's full text]]/vaariatt55
Original Problem:
Expected output (example):
<variant1>|... text ... <image> </variant>>
<variant2> [... text <image> </variant2>
<variant 5 [.... text <image> |</variant>>
G.2 PART II: REASONING AND VERIFICATION
PROMPT 2: STEP-BY-STEP REASONING
Generate a stepwise reasoning chain (3-5 steps) with detailed diagram observations, ending with the final answer boxed.
Purpose:
You are mathematician, statistician, and geometer. Below, will present you with math problem along with its accompanying diagram. Please carefully observe the details in the image.
Given the text, images, generate a step-by-step reasoning process that logically leads to the correct result in the \boxed(). Requirements: Flexible step count (3-5 steps) [full problem text and image here]
Expected output (example):
<stepl> observation </stepl>
<step2> inference </step2>
<step3> inference </step3>
<step4> ... conclusion ... </step4>
\boxed [final answer]]
PROMPT 3: ANSWER COMPARISON
Purpose: Compare two numerical answers (answerl, answer2) ignoring formatting. Returns wwweseerrrueeeaaeeeeee equal, else False.
23

---

Published as a conference paper at ICLR 2026
Compare these answers numerically, ignoring any formatting differences:
Answer 1: Jrcaulti'answerl]]]
Answer 2: (result l'anserr2
Extract just the numerical values from each answer and compare them. If the numerical values are the same, return True. Otherwise return False. Reply with <answerptrue</answss or <answer>Falee<</aaae
PROMPT 4: REFLECTION AND REVISION (FROMPT_ SOAK_ARAAAATTTTTISTTTEEETT
Purpose:
Refine reasoning steps after receiving correctness feedback, without directly inserting the correct answer in the steps or reverse-engineering from it.
You are a matemaiician, statistician, and geometer. Please carefully observe the details in the image.
You have already provided the reascning steps and correct above.
Do you think your answer correct? Revise or improve your reasoning steps based on the correct answer. Emphasize!
DO NOT include the correct answer in <step>.
DO NOT reverse-engineer from the answer.
[full problem text and image here]
H CASE STUDY OF TRAIN DATASET
H.1
CASE 1
The figure <image> below shows the graphs of the functions: == = x² 6z + y x² Gr + 8, y =x- 12r 35 in the same Cartesian coordinate system. The most likely graph for y = x² 6x + is
Problem:
1
Figure H.4: Original Problem
DIFFICULT VARIANT
Text and Image Variant: Embed both the original problem description and the question directly into the image, and replace the prompt text with the following fixed instruction:
As shown in the <image>, answer the question according to the figure.
Image Variants:
The image is perturbed with salt noise increase complexity in recognizing the object.
Salt Noise:
Blur:
The image is blurred, making it harder to focus on specific details.
Gauss Noise: The image is perturbed with Gaussian noise.
24

---

Published as a conference paper at ICLR 2026
figure cimage> shows graphs luncions: $5yex122)-15. (2|+5xe88 58yox*[2] Ga+Bss, Styme"/)-12- 2055 the same Cartesian coordinate system. The most iiely graph $8yax122-6x-68
Figure H.5: Domiint Problem
Rotate: The image is rotated, increasing difficulty in spatial reasoning.
(a) Salt Noise
(b) Blur
(c) Gauss Noise
(d) Rotate
Figure H.6: Image Variants: Salt Noise, Blur, Gauss Noise, and Rotate.
Text Variants: Semantically equivalent reformulations of the original problem, preserving both meaning and the correct answer.
Problem Variant 1:
In the Cartesian plane depicted in the figure <image>, four distinct parabolic curves are illustrated, each corresponding to one of the following equations:
x 1, y x² 6z 8, y x² 6r + and x² 12r 35. Your
challenge is to discern and point out the specific curve that is generated by the equation y x² 6r 8.
Problem Variant 2:
25

---

Published as a conference paper at ICLR 2026
Among the provided graphs in the figure <image>, identify which one sponds to the quadratic function = x² 6z + This function, when compared to the others- a unique vertex and orientation. Your task is to select the graph that accurately represents y x²² 6a 8.
x² 1, y x²² + 6x 8, and y x² 12r 35 35-displays
EASIER VARIANT
Think Step:
Step 1:
The image displays four parabolic graphs labeled 1 2 C and ④ Each graph
represents a quadratic function of the form y = ax.² + be c. The general characteristics of these parabolas include their orientation (all open upwards, indicating positive leading coefficient) and their respective vertices.
Step 2: To identify which graph corresponds to the function x²² rewrite this function in vertex form. Completing the square:
6z + 8, we first
x² - 6x + 8 = (x² 6r + 9) - 9 + = (x 3)²
This reveals that the vertex of the parabola is at the point (3, -1).
Step 3: Next, we analyze the positions of the vertices for each graph:
Graph ! has its vertex to the left of the y-axis.
Graph 2 has its vertex the origin very close to it.
Graph has its vertex at (3, -1), as determined from the vertex form.
-
Graph 4 has its vertex the right of the y-axis but not at (3, -1).
Step Thus, the graph with the vertex (3, -1) graph 3 Given the analysis, the function x² - 6z + 8 corresponds to the graph labeled ① is the only graph with the correct vertex coordinates.
H.2
CASE 2
Problem:
<image> Subtract all brown cubes. Subtract all blue cylinders. How many cubes are left?
Figure 11.7: Original Problem
DIFFICULT VARIANT
Text and Image Variant: Embed both the original problem description and the question directly into the image, and replace the prompt text with the following fixed instruction:
As shown in the <image>, answer the question according to the figure.
26

---

Published as a conference paper at ICLR 2026
<image>Once all brown cubes and blue cylinders are taken away. what is the number cubes left?
Figure H.8: Riion-Dominnat Problem
Image Variants:
Salt Noise: The image is perturbed with salt noise to increase complexity in recognizing the object.
Blur:
The image is blurred, making it harder to focus on specific details.
Gauss Noise: The image is perturbed with Gaussian noise.
Rotate: The image is rotated, increasing difficulty in spatial reasoning.
(a) Salt Noise
(b) Blur
(c) Gauss Noise
(d) Rotate
Figure H.9; Image Variants: Salt Noise, Blur, Gauss Noise, Rotate.
Text Variants: Semantically equivalent reformulations of the original problem, preserving both meaning and the correct answer.
27

---

Published as a conference paper at ICLR 2026
Problem Variant 1:
<image> Once all brown cubes and blue cylinders are taken away, what is the number of cubes left?
Problem Variant 2:
<image> Remove all brown cubes and all blue cylinders. How many cubes remain?
EASIER VARIANT
Think Step:
Step 1: Leftmost object: single cube, distinctly green.
Immediately to the right of that cube: large, dark-blue cylinder.
Farther right: two smaller cylinders of a lighter, cyan-blus -luu hue.
Several spheres (purple, gold, yellow) appear, but these are not cubes or cylinders.
No object in the scene exhibits brown color.
Thus, only one cube (green) is present, and there are three blue-family cylinders (one dark, two cyan).
Step 2: Interpret the problem instructions with the observed inventory.
"Subtract all brown cubes." " Because none exist, this operation leaves the cube count unaffected.
"Subtract all blue cylinders."
Step 3:
Although three cylinders satisfy the color condition, removing cylinders changes only the cylinder inventory; cubes remain untouched. Perform the mandated subtractions.
After eliminating zero brown cubes, the cube inventory is the same as noted in Step 1.
Step 4: Removing the three blue cylinders affects only the cylinder category; the cube
category remains exactly before. Summarize the effect of both operations.
None of the specified subtractions alters the original cube count, the final cube tally equals the initial cube tally recorded in Step 1.
I
CASE STUDY OF TEST DATASET
L.1 CASE 1
Problem. shown in the figure, AABC AADE. From the figure we have LB 70%
<C 30", and LDAC 35" Find LEAC.
Choices: A: 40 B: 45
C: 35
D: 25
C
30
B 70
D
E
Figure 1.10: Geometry problem: Math Verse 7.
OUR MODEL. FINAL ANSWER: B: 45
Complete derivation (external, step-hy-step).
28

---

Published as a conference paper at ICLR 2026
1. In AABC, the triangle angle-sum gives LBAC = 180 -B LC 180 70 -30 80%.
By congruence AABC = AADE, corresponding angles are equal: hence LDAE = LBAC 80%.
3. At vertex A, the angle decomposition yields /DAE /DAC ZCAF. Given /DAC = =35" have ACAF 80 35 45
4. Therefore ZEAC = LCAE = 45", corresponding to choice B.
OTHER MODEL (BASELINE) - REPORTED ANSWER: A: 40
Reported derivation (reconstructed from the baseline output).
1. The baseline asserts that by congruence /DAF /BAC (this assertion is geometrically correct in form).
2. The baseline then (incorrectly) reads treats LBAC as 35" and reads LCAD (or LDAC) as 30 (this a misreading of the labels the figure / given data).
3. Using the (incorrect) equality LBAC LCAD + LEAC,, the baseline computes LEAC
35
30 = 5
The baseline then presents an alternative arithmetic attempt: treats some combination as "exterior" or summed angle, computing 35 + +03 = 65%, and then (without a valid geometric justification) performs 65 70% and reports 40 as the final choice.
Explicit error analysis.
Label/misreading error. The baseline's step that sets LBAC 35 is inconsistent with the given values LB = 70 and ZC' = 30%, which imply LBAC = 80". Misidentifying which labeled angle corresponds LBAC the cause.
Invalid arithmetic and unjustified operations. The baseline mixes different angle-sum operations (interior vs. exterior) without geometric justification and carries out arithmetic (e.g.. 65% 70 >
40 in the reported text) that is both algebraically incorrect and lacks geometric meaning in context. Broken logical chain. Because the baseline's premises are incorrect, subsequent deductions (such subtracting summing those incorrect angles) do not follow from valid geometric facts (triangle sugle-sum, congruence correspondence, or correct angle decomposition).
Summary Why Our Model Preferable in This Case.
Accurate visual interpretation: Our model correctly identifies the labels and reads LB, LC, and ZDAC from the figure, avoiding the label-confusion that the baseline exhibits.
Correct and verifiable reasoning: Each step follows basic geometric rule (angle-sum
triangle, congruence corresponding angles equal, and decomposition of an angle vertex), and the arithmetic is straightforward and checkable.
Transparent presentation: The derivation above externalized and suitable for inclusion in an appendix so that reviewers can trace the reasoning and confirm correctness.
1.2 CASE 2
Problem.
Write the set of numbers represented on the number line in interval notation.
Figure 1.11: Interval notation problem: Math Verse 474.
OUR MODEL FINAL ANSWER: (-2, -1]
Complete derivation (external, step-hy-step).
1. Inspect the endpoints shown the number line: one endpoint 2 and the other is at -1.
2. Determine inclusion/exclusion from the marker style:
-2 is marked with open (hollow) circle + -2 not included.
--1 is marked with a solid (filled) circle + -1 is included.
29

---

Published as a conference paper at ICLR 2026
3. Translate inclusion/exclusion to interval notation: Excluding -2 uses a parenthesis "(" at the left.
Including -1 uses bracket 'J' at the right.
Therefore the interval is (-2, -1].
5. In inequality form this corresponds to -2 < <<<s--
OTHER MODEL (BASELINE) REPORTED ANSWER: (-2, 1)
Reported derivation (reconstructed from the baseline output).
1. The baseline identifies the endpoints as 2 and -1.
2. It (incorrectly) interprets both endpoint markers open circles, concluding that neither endpoint included.
3. From that (missinterpretatton it writes the interval using parentheses on both sides and reports -1).
Explicit error analysis.
Visual misinterpretation error. The baseline's 's central mistake is reading the marker at -1 as open when actually solid (closed). This single perceptual/maaping error changes 's' '<' at the right endpoint and thus changes the bracket type.
Notation consequence. Interval notation sensitive to endpoint inclusion: confusing a closed endpoint for open one converts bracket 'I' to parenthesis 'Y' producing different set. Here that error changes the set by including or excluding the point -1
Downstream impact. Because the baseline misreads the marker, any inequality- statement
set-membership statement it produces (e.g.. -2 < x < -1) will be incorrect for points equal to
Summary Why Our Model Preferable in This Case.
Our model correctly maps hollow vs. filled endpoint markers to exclusion vs. inclusion and therefore uses the appropriate parenthesis/bracket combination. Clear, verifiable steps: Each step is an elementary, checkable rule: identify endpoints, read marker style, convert to interval notation, and (optionally) supply the equivalent inequality. This transparency makes verification straightforward for reviewers.
Correct visual-to-symbol mapping:
Precise final expression: The answer (-2, -1] cleanly and unambiguously communicates the set real numbers greater than -2 and less than or equal to 1,, matching the figure where -1 is
solid.
1.3 CASE 3
Problem. Given the figure below, find the slope the line.
Figure 1.12: Line slope problem: Math Verse 591.
30

---

Published as a conference paper at ICLR 2026
OUR MODEL FINAL ANSWER:
Complete derivation (external, step-by-step).
1. The slope of a line passing through two points (I1, V1) and (22,82) is given by:
1/2 21
slope
x2 I1
2. From the graph, identify two points on the line: (- 2) and (1,0). Substitute these points into the slope formula:
slope 0
(-1) = -2
2
Therefore, the slope of the line is
OTHER MODEL (BASELINE) REPORTED ANSWER:
Reported derivation (reconstructed from the baseline output).
1. The baseline identifies two points on the line (1, and (0, (),4).
2. Using the slope formula:
2 2 4 -2 -2
slope
1
The baseline reports the final slope as 22
Explicit error analysis.
The baseline misreads the coordinates of points the line. Using (1,2) and (0, 4) does not correspond to the actual line in the figure. Consequent incorrect slope: Because the points are wrong, the computed slope -2 does not match the true slope of the line, which is -1.
Incorrect point selection:
Logical chain broken: All subsequent reasoning mathematically consistent with the chosen points, but the premise (point selection) is flawed, leading to an incorrect final answer.
Summary Why Our Model Preferable in This Case.
Accurate visual interpretation: Our model correctly identifies the actual points on the line from the figure.
Correct reasoning: Step-by-step application of the slope formula leads directly to the correct answer -1.
Transparent derivation: The reasoning is fully externalized, making easy for reviewers to verify correctness and consistency with the figure.
J CASES THAT FAIL TO GENERATE AN ADVANTAGE
J.1 CASES THAT ARE CONSISTENTLY ANSWERED CORRECTLY
J.1.1 CASE
Problem. As shown in the figure, point P is the intersection the angle bisectors of the interior angle LABC and the exterior angle LACD of AABC. The distance from point P to the line AC cm. What is the distance from point P to the line AB?
Answer: 6
J.1.2 CASE 2
Problem. shown in the figure, in rhombus ABCD, the diagonals AC and BD intersect at point O, and OE || DC intersects BC at point E. If AD = 8 cm, then the length of OE is
cm.
Answer:
31

---

Published as a conference paper at ICLR 2026
A
B
C
Figure J.13: Geometry problem involving angle bisectors.
A
D
B
E
Figure J.14: Geometry problem involving rhombus.
J.1.3 CASE 3
Problem. As shown in the figure, in AABC, points D and E are on AB and AC, respectively. and DE || BC. If AD = 1 and AB = 4, then DE p
BC
A
D
E
B
Figure J.15: Geometry problem involving similar triangles.
1
Answer:
J.1.4
CASE 4
Problem. For any non-zero real numbers a, b, if the operation principle of a @ b is represented by the flowchart shown in the figure, then 3 @ 2 =
Answer: 2
32

---

Published as a conference paper at ICLR 2026
JF
16
Na
-
a
b-1 1
a+1
L11
i
a
Figure J.16: Flowchart operation problem.
J.2 CASES THAT ARE CONSISTENTLY ANSWERED INCORRECTLY
J.2.1
CASE 1
Problem. Figure A is histogram showing the frequency distribution of the monthly income of the staff and faculty of school, where it is known that the frequency of the first group from left right is 80. In the sample, the frequencies of the monthly income (unit: yuan) in the intervals [1000, 1500), [1500, 2000), [2000, 2500), [2500, 3000), [3000, 3500), [3500, 4000) are denoted as A A2...., As respectively. Figure is the flowchart of the algorithm used to estimate the average monthly income of the school's staff and faculty. What is the output of S? (Answer with a number.)
Sudtd xEI 200 (3
000
RB
2
Figure J.17: Complex statistics and flowchart problem.
Answer: 1962.5
J.2.2 CASE 2
Problem.
To process part with the shape shown in the figure. calculate the degree of the inclined angle based on the dimensions indicated (unit: mm). (Use calculator, accurate to 1 second).
E
D
A
140
83
150
124
F
B
Figure J.18: Industrial geometry calculation.
Answer:
22 g' 12"
33

---

Published as a conference paper at ICLR 2026
J.2.3 CASE 3
Problem. Black and white hexagonal floor tiles are arranged according to the pattern shown in the figure; then the number of white floor tiles in the n-th pattern is
M22
$33
Figure J.19: Pattern recognition sequence problem.
Answer:
On
2 1)
4n
J.2.4
CASE 4
As shown in the figure, there is geometric solid composed of a hemisphere and a square pyramid, as shown in the three views. The volume of the solid
Problem.
Figure J.20: Solid geometry from three-view drawing.
V2
Answer:
3
6
J.3
CASE ANALYSIS
Analyzing samples that yield consistently correct or incorrect responses reveals distinct patterns.
Consistently Correct Cases: We observe that our approach does not offer a significant advantage for problems characterized by clear images, simple logic, and well-defined results that are easy to express and verify. This primarily because the visual information is unambiguous, and the solution path is sufficiently straightforward for the base Vision-Languaa Model. Furthermore, the final answers are often simple integers or fractions, which simplifies verification.
Conversely, for problems involving highly complex image content or those where the results are difficult describe verify, achieving correct answer remains chal- lenging even when reasoming steps are provided. This difficulty arises from the inherent complexity of inferring the correct answer through reasoning alone, which is exacerbated by the difficulty in verifying the intermediate and final results, thereby complicating the optimization task.
Consistently Incorrect Cases:
K GENAI USAGE DISCLOSURE
In this work, Large Language Models were utilized to aid refining and polishing the writing. The author(s) affirm that none of the data analysis, methodology development, theoretical contributions in this paper involved content generation or research assistance from the GenAl tool. All aspects of the study, including code and data, were developed independently.
34

---

Published as a conference paper at ICLR 2026
L
LIMITATIONS
Although DIVA-GRPO effectively alleviates reward sparsity and improves performance on multi- modal reasoning tasks, some limitations remain. First, the variant generation process for difficult problems requiring intermediate reasoning steps relies on external models, which may introduce potential biases. Second, sifficulty-eiighee scaling involves hyperparameters that need careful tuning, which reduces out-of-the-box usability. Finally, DIVA-GRIA has been primarily evalu- ated on multimodal reasoning tasks, and generalization to mon-multimodal tasks or tasks with fundamentally different reward structures remains to be explored.
35


Reviews:
[{"review_id": "ODtpwSC9Tz", "reviewer": "Reviewer_ybUJ", "summary": "This paper introduces DIVA-GRPO, a difficulty-adaptive variant of Group Relative Policy Optimization (GRPO), designed to improve multimodal reasoning ability of MLLMs. The key innovation lies in dynamically adjusting the difficulty of training problems and their variants to maintain informative reward signals, thereby mitigating reward sparsity and advantage vanishing in reinforcement learning. Also, DIVA-GRPO computes both local and global advantages using difficulty-aware normalization and reward-range-based rescaling. Extensive experiments demonstrate the effectiveness of the proposed DIVA-GRPO", "questions": "1. How do you ensure that the generated variants (especially for hard problems) are semantically equivalent and answer-preserving? Moreover, how can we verify that the proposed recipe for “easy” and “hard” variants really behaves as intended—i.e., after the perturbations the model’s accuracy on the easy set effectively goes up while that on the hard set goes down?\n2. Since reasoning variants are generated using external models, how do you ensure their quality and consistency? Have you observed any degradation in performance when using different external models or prompts?\n3. Could you provide a sensitivity analysis for the difficulty-weighted scaling parameter k and the difficulty update rate η? How sensitive is the model to these choices, and are there general guidelines for setting them?\n4. What is the computational cost of generating and training with variants compared to standard GRPO? Is the speedup in convergence enough to offset the additional overhead?\n5. Are there specific types of problems or difficulty levels where DIVA-GRPO still struggles? It would be helpful to include examples or failure analysis to understand the method’s limitations.\n6. Do you plan to release the generated variants or the code for variant generation? This would help the community reproduce and build upon your work.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The method is thoroughly developed, with detailed algorithms, theoretical analysis (e.g., variance reduction and convergence), and extensive empirical validation across multiple benchmarks.\n2. The idea of dynamically adjusting problem difficulty and generating semantically consistent variants is well-motivated.", "weaknesses": "1. The method introduces new hyperparameters (e.g., difficulty scaling factor k, learning rate η) that lack robust ablation or automatic tuning. This may hinder out-of-the-box usability and necessitate per-task calibration.\n2. The training flow illustrated in Figure 2 is somewhat confusing. A clearer depiction of how variants are sampled, advantages computed, and the policy updated would improve understanding.\n3. The difficulty of each query is updated after every epoch, but the total number of training epochs is not specified. Clarifying this and analyzing its impact on performance would be helpful.\n4. In Figure 3, training steps range from 0 to 150. It is unclear whether this covers more than one epoch. If not, the difficulty of each query remains near its initial moderate value—raising the question of where the observed gains originate.\n5. Compared to vanilla GRPO, DIVA-GRPO requires additional rollouts on variants. A detailed analysis of the resulting computational cost and trade-offs should be provided.\n6. When the paper reviews the original GRPO, it states that the reward is “rule-based”; however, GRPO itself is not tied to rule-based rewards and can equally be used with a reward model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8wtKAgutdf", "reviewer": "Reviewer_bQCX", "summary": "The paper proposes DIVA-GRPO, a GRPO-based RL framework for MLLMs that (i) dynamically assesses problem difficulty and expands each item into semantically consistent variants with controlled difficulty; (ii) computes local (per-problem) and global (across variants) advantages with batch z-score normalization and difficulty-weighted scaling; and (iii) introduces Reward-Range-Based rescaling (RRB) to avoid exaggerated advantages when reward variance is small.", "questions": "Baselines (GRPO family): Do you have quantitative head-to-head results against other GRPO or GRPO-improvement methods (such as DAPO or other relevant methods referenced in your Related Work), under identical settings (same backbone, datasets, steps)?\n\nScope & transfer: Is this the first time the proposed difficulty-adaptive variant + local/global advantage scheme appears in MLLMs? Could the same recipe be used in text-only GRPO training without images?\n\nFigure details: Why do GRPO curves in Fig. 3(b) vs 3(c) look different?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Method is clearly presented and well-motivated: The paper crisply identifies advantage vanishing and reward sparsity in GRPO, then designs a pipeline that directly targets these failure modes.\n\nThorough experiments and diagnostics: Results span six benchmarks with ablations and a neat speedup study vs GRPO; figures/tables are informative.\n\nGeneralizable component (RRB): The RRB trick improves vanilla GRPO too, suggesting portability beyond this specific framework.", "weaknesses": "Unclear significance under matched baselines: Gains over strong GRPO variants are modest or inconsistent, and baseline setups do not appear strictly aligned.\n\nScope and transfer are not well positioned: It is unclear whether the difficulty-adaptive variant + local/global advantage scheme is truly first for MLLMs, how it relates to similar ideas, and whether the recipe generalizes to text-only GRPO without images.\n\nFigure interpretation ambiguity: The GRPO training curves in Fig. 3(b) vs. Fig. 3(c) differ in shape, but the exact experimental differences are not fully specified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ILmjn1r13m", "reviewer": "Reviewer_aEti", "summary": "The paper proposes a difficulty-adaptive variant advantage framework that (i) dynamically assesses per-problem difficulty from rollout accuracy each epoch; (ii) generates difficulty-matched variants that preserve the gold answer; and (iii) computes both local and global advantages, then stabilizes them via batch z-score normalization, difficulty-weighted scaling, and a Reward-Range-Based (RRB) rescaling that down-weights spurious advantages when reward spread is tiny. Together these steps keep within-group reward variance informative and training signals balanced across difficulties.", "questions": "See the `Weaknesses` part.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is generally well-written and easy to follow, with a clear description of the method.\n2. The paper provides intuitive visual demonstrations to help better understand the paper.", "weaknesses": "1. **Clarify local vs. global advantage magnitudes.** In `Line 263-264`, the paper argues that because global advantages are computed over `m×k` samples, their magnitudes differ from local advantages. However, in `Line 250-252` the global advantage is also normalized, which should mitigate raw magnitude discrepancies. Moreover, the subsequent difficulty-weighted scaling implies the effective magnitude of the global term should also depend on the direction and strength of the difficulty adjustment. Could the authors provide a more detailed and intuitive explanation on this claim and how the proposed pipeline balances the two streams in practice.\n\n2. **Difficulty range and initialization.** The difficulty range is preset to `[1, 9]` with an initial value of `5`. Is there empirical or theoretical justification for this specific choice? Please consider reporting a sensitivity analysis over alternative ranges (e.g., `[0, 1]`, `[0, 5]`, `[1, 7]`) and different initializations to assess stability, convergence rate, and final accuracy. If the choice is largely heuristic, a brief rationale (e.g., numerical stability) would be helpful.\n\n3. **Fairness of efficiency claims.** `Figure 3(c)` reports a `3.17×` faster training speed in terms of *steps*. However, DIVA-GRPO performs **difficulty-adaptive variant generation** at each step, which adds per-step overhead (both compute and memory). Comparing steps alone may therefore be unfair to the GRPO baseline. Please report **wall-clock training time** and **peak memory usage** across runs, and (ideally) throughput metrics (e.g., tokens/sec) to substantiate the claim and isolate where the efficiency gains come from (faster convergence vs. higher per-step cost).\n\n4. **Rollout parity across methods.** In `Line 368-369`, the rollout count is set to `k=5`. If each step also includes `m` difficulty-adaptive variants, then the *effective* number of rollouts per original prompt is larger than `k`. To ensure a fair comparison, could the baseline GRPO be configured with an equivalent total number of rollouts (e.g., set its rollout count to match the sum of local and global samples), or otherwise report an ablation that aligns the **total rollout budget** per step across methods? This will help attribute gains to the algorithm rather than additional sampling.\n\n5. **Relative vs. absolute difficulty weighting.** As noted in `Line 301-302`, the difficulty-weighted scaling uses **relative** difficulty within a group to reweight advantages, whereas many prior works use **absolute** difficulty. It would strengthen the paper to include an ablation comparing relative vs. absolute difficulty weighting (keeping other factors fixed), reporting effects on stability (variance of advantages/gradients), convergence speed, and final performance.\n\n6. **Answer preservation and broader augmentation.** When modifying the text description of the original problem, how do the authors **guarantee** that the final answer remains unchanged? In addition, the current **difficulty-adaptive variant generation** constrains variants to preserve the original answer. Are there plans (or results) for generating **new problems with new answers** under verifiable supervision (e.g., programmatic solvers or formal checkers)? Such a setting could better align with the paper’s stated goal in `Line 232-233` of “improving generalization to unseen expressions”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IGxc8cELt7", "reviewer": "Reviewer_YpGH", "summary": "This paper proposes DIVA-GRPO, a novel reinforcement learning algorithm for multimodal large models designed based on the idea of dynamic difficulty adjustment. DIVA-GRPO dynamically evaluates the difficulty of each problem and generates query variants dynamically according to the difficulty to adjust the optimization signals in the training process. Additionally, this method puts forward Difficulty-weighted scaling and Reward-range-based Advantage Rescaling to optimize the calculation of advantage values during training.", "questions": "In Section 4.3, the w/o Variant Generation setting (Table 2) performs lower than GRPO+Reward-Range (Table 3). Does this imply that the combination of Difficulty-Weighting, RRB, and G-L Balance underperforms using only RRB?\n\nIf the authors’ response sufficiently addresses my concerns, I will consider increasing my score.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Regulating the optimization signals in the training process based on dynamic difficulty adjustment is an intuitive, reasonable, and effective approach.\nDetailed theoretical derivations support the motivation.\nLeading results on multiple benchmarks demonstrate the effectiveness of the algorithm.", "weaknesses": "1. Possible missing related work: There are several studies closely related to the method in this paper and highly relevant. Noisy Rollout uses image noising to facilitate reinforcement learning; Hint GRPO introduces adaptive prompt augmentation; and Dr. GRPO removes the variance term in advantage estimation, which is similar to the RRB in this paper.\n2. The comparison is conducted on only a single base model, Qwen2.5-VL-7B, which greatly limits the applicability of the proposed method. Evaluating across different model sizes and different model families (e.g., LLaVA) would help assess the generality and applicability of the approach.\n3. Potentially misleading “training efficiency and stability” claim. In Fig. 3(c), the efficiency is argued based on the number of steps required to reach a certain performance, which can be misleading. First, DIVA-GRPO requires additional data preparation compared to GRPO. Second, training stability is not observed, and the provided curves suggest that performance gains have not yet converged; could you provide a longer training dynamic until the gains fully converge?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)