Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation
Authors: Yanqi Dai
Yuxiang Ji
Xiao Zhang
Yong Wang
Xiangxiang Chu
Zhiwu Lu
Venue: ICLR 2026 Poster (2026)
Keywords: Mathematical Reasoning
Group Relative Policy Optimization
Question Reformulation
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a MathForge framework to improve mathematical reasoning by targeting harder questions from both algorithmic and data perspectives, including Difficulty-Aware Group Policy Optimization (DGPO) and Multi-Aspect Question Reformulation (MQR).

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) offers a robust mechanism for enhancing mathematical reasoning in large models.
However, we identify a systematic lack of emphasis on more challenging questions in existing methods from both algorithmic and data perspectives, despite their importance for refining underdeveloped capabilities.
Algorithmically, widely used Group Relative Policy Optimization (GRPO) suffers from an implicit imbalance where the magnitude of policy updates is lower for harder questions.
Data-wise, augmentation approaches primarily rephrase questions to enhance diversity without systematically increasing intrinsic difficulty.
To address these issues, we propose a two-dual MathForge framework to improve mathematical reasoning by targeting harder questions from both perspectives, which comprises a Difficulty-Aware Group Policy Optimization (DGPO) algorithm and a Multi-Aspect Question Reformulation (MQR) strategy.
Specifically, DGPO first rectifies the implicit imbalance in GRPO via difficulty-balanced group advantage estimation, and further prioritizes harder questions by difficulty-aware question-level weighting.
Meanwhile, MQR reformulates questions across multiple aspects to increase difficulty while maintaining the original gold answer.
Overall, MathForge forms a synergistic loop: MQR expands the data frontier, and DGPO effectively learns from the augmented data.
Extensive experiments show that MathForge significantly outperforms existing methods on various mathematical reasoning tasks.
The code and augmented data are all available at https://github.com/AMAP-ML/MathForge.

Published as a conference paper at ICLR 2026
HARDER Is BETTER: BOOSTING MATHEMATICAL REASONING VIA DIFFICULTY-AWARE GRPO AND
MULTI-ASPECT QUESTION REFORMULATION
Yanqi 2 Yuxiang Ji³, Xiao Zhang', Yong Wang"! Xiangxiang Chu², Zhiwu Luli
'Golling School of Artificial Intelligence, Renmin University of China
2 AMAA,, Alibaba Group "Xiamen University "Dalian University of Technology
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) offers a robust mecha- nism for enhancing mathematical reasoning in large models, However, we identify a systematic lack of emphasis on more challenging questions in existing methods from both algorithmic and data perspectives, despite their importance for refining underdeveloped capabilities. Algorithmically, widely used Group Relative Policy Optimization (GRPO) suffers from an implicit imbalance where the magnitude of policy updates is lower for harder questions. Data-wise, augmentation approaches primarily rephrase questions to enhance diversity without systematically increas- ing intrinsic difficulty. To address these issues, we propose a two-dual MathForge framework to improve mathematical reasoning by targeting harder questions from both perspectives, which comprises a Difficulty-Awa Group Policy Optimization (DGPO) algorithm and a Multi-Aspect Question Reformulation (MQR) strategy. Specifically, DGPO first rectifies the implicit imbalance in GRPO via difficulty- balanced group advantage estimation, and further prioritizes harder questions by difficulty-aweer aware question-level weighting. Mcanwhile, MQR reformulates ques- tions across multiple aspects to increase difficulty while maintaining the original gold answer. Overall, MathForge forms a synergistic loop: MQR expands the data frontier, and DGPO effectively learns from the augmented data. Extensive experi- ments show that MathForge significantly outperforms existing methods on various mathematical reasoning tasks. The code and augmented data are all available.'
INTRODUCTION
Recently, large language models (LLMs) have demonstrated remarkable reasoning capabilities, fun- damentally altering the landscape of artificial intelligence (Jaech et al., 2024; Comanici et al., 2025; Guo et al., 2025). In this context, reinforcement learning with verifiable rewards (RLVR) has been proven promising training paradigm (Guo et al., 2025; Wen et al., 2025), especially for enhanc- ing mathematical reasoning. It adopts rule-based rewards instead of neural reward models, thereby significantly reducing computational overhead and mitigating the risk of reward hacking.
From an algorithmic perspective, the most representative approach to support RLVR is Group Rela- tive Policy Optimization (GRPO) (Shao al., 2024), which estimates relative advantages of group of responses to the same question. However, we reveal and mathematically prove a critical limita- tion in GRPO and its variants: their advantage estimation function introduces an implicit imbalance where the update magnitudes are suppressed for both easier and harder questions and peak for those of moderate difficulty. The neglect of more challenging yet solvable questions detrimental to RL training. Such questions are ideal training material, as they expose the model's incomplete mastery while also offering at least one correct response for targeted improvement. Therefore, harder ques- tions should be emphasized to focus the model on overcoming its solvable weaknesses, while easier ones necessitate only minimal yet sufficient weighting to prevent forgetting. Zhang & Zuo (2025) also recognize the importance of question difficulty in GRPO, but their method proposes complex difficulty-awar advantage reweighting without rectifying the underlying imbalance.
"Wor done during Yanqi Dai's internship at AMAP, Alibaba Group Project lead: Yong Wang; Corresponding authors: Zhiwu Lu (first) and Yong Wang.
https://gitub..co//AAaaaa /MatFForge

---

Published as a conference paper at ICLR 2026
Meanwhile, from data perspective, traditional augmentation methods for reasoning often generate entirely new question-anan answer pairs (Luo et al., 2023; Li et al., 2023; 2024a), but the quality of the answers is difficult guarantee, especially for competition-level problems. As for those tailored for RLVR, only Liang et al. (2025) explore rephrasing questions while sustaining the original answer to enhance data diversity. However, the question difficulty dimension still lacks attention. Recognizing that solving mathematical reasoning problems requires varying skills, we contend that systematically increasing question difficulty by reformulating them to target and challenge these skills is a crucial approach for pushing the model's performance boundaries.
To address these issues, we introduce a two-dual framework termed MathForge to enhance mathe- matical reasoning by targeting more challenging questions from both algorithmic and data perspec- tives. Specifically, MathForge comprises two key components: a Difficulty-Aaar Group Policy Optimization (DGPO) algorithm and a Multi-Aspect Question Reformulation (MQR) strategy. Al- gorithmically, DGPO first rectifies the implicit imbalance of the update magnitudes GRPO via difficulty-bllanced group advantage estimation, which normalizes group relative advantages by the mean absolute deviation of rewards rather than the standard deviation employed in GRPO. Further- more, DGPO prioritizes harder questions using difficulty- question difficulty is quantified as the negative mean accuracy calculated across all its corresponding responses. Data-wise, MQR reformulates the original questions across multiple aspects to increase their difficulty and diversity, including adding story background, introducing abstract terminology. and nesting sub-problems. A critical constraint is that all reformulations must preserve the original gold answer, so that MQR can maintain the essential mathematical logic of the question and obvi- ate the need for solution regeneration. Overall, our MathForge creates a powerful synergistic loop, where MQR expands the data frontier and DGPO efficiently learns from these augmented data.
question-level weighting, where the
The main contributions of this paper can be summarized as follows:
We introduce a Difficulty-Aaare Group Policy Optimization (DGPO) algorithm, which rectifies the implicit imbalance of GRPO and further upweights more challenging questions.
2. We propose Multi-Aspect Question Reformulation (MQR), a data augmentation strategy tailored for RLVR, which reformulates questions across multiple aspects to increase their difficulty while preserving the original gold answer.
3. Experiments show that our MathForge markedly outperforms existing methods on various models and mathematical reasoning benchmarks, validating its effectiveness and generalizability.
PRELIMINARIES
Notation. In this paper, an autoregressive language model, parameterized by 0, is treated as a policy model, where He and The represent the current and old policies, respectively. For a given query sampled from a question dataset D, multiple responses {o:) are generated using the old policy Te..- A scalar reward rs for each query-response pair (q) is then assigned by rule-based verifier. By default, we only use the accuracy reward, 1 if the response is correct and 0 otherwise. In the context of batch processing, {4.} signifies batch of queries sampled from the question dataset D, and the corresponding responses and rewards are denoted by {oxr} and {r respectively.
Group Relative Policy Optimization (GRPO). GRPO (Shao et al., 2024) is a variant of Proximal Policy Optimization (PPO) (Schulman et al., 2017), which climinates the critic model, and estimates relative advantages of responses within a group of responses the same query. Moreover, Chu al. (2025) and Yu et al. (2025) remove the KL divergence and employ a token-level policy gradient loss to enhance the performance of GRPO. These modifications have been experimentally validated and are more commonly used in practice, becoming the default settings in TRL (von Werra et al., 2020). Specifically,, GRPO optimizes the policy model me by maximizing the following objective:
(f) = [ ~ D, ( ~ Toaa (: (.14)
G lo.l
min [1((0) clip (1), E, E) ACR.]] (1)
G
loil
ACR. T; mean (1ri)e)
na (or, q, (i,
where (0)
TBa (or, q, 01,e))
sid (ri))-)
2

---

Published as a conference paper at ICLR 2026
Here, [i(( denotes the importance sampling ratio of the token Oi.1s and AORS signifies the advan- tage of the response o, obtained by group relative advantage estimation (GRAE). is the number of generated responses to each query (i.e., the group size), and is the clipping range of (0).
3 METHODOLOGY
In this section, we introduce the MathForge framework to enhance mathematical reasoning by con- centrating
challenging questions from both algorithmic and data perspectives. Specifically, consists of two core components: the Difficulty-Award Group Policy Optimization (DGPO) algo- rithm and the Multi-Aspect Question Reformulation (MQR) strategy.
3.1 DIFFICLLTY-AWRRE GROUP POLICY OPTIMIZATION
Although GRPO achieves strong reasoning performance, we mathematically prove that its opti- mization objective is unbalanced with respect to the update magnitudes for questions with varying difficulties, which primarily stems from its group relative advantage estimation (i.e., AGR,i in Equa- tion 2). This imbalance potentially reduces the extent to which the policy updates for more chal- lenging yet solvable questions. However, such questions are ideal training material that expose the model's s incomplete mastery while also offering at least one correct response for targeted improve- ment. Moreover, harder questions may be more complex compositions or reformulations of easier ones, thus mastering harder ones can potentially enhance the model's performance on easier ones.
To resolve this issue, our Difficulty- poses difficulty-bllanced group advantage estimation (DGAE) to normalize the update magnitudes across questions. Secondly, it employs difficulty-aware question-level weighting (DQW) to priori- tize more challenging questions further.
Group Policy Optimization (DGPO) algorithm first pro-
Specifically, the optimization objective of DGPO defined as follows:
JDGPO (0)
E (1)) D, (0si) Towa gs))
B.
load
Apa... clip (0), =,)+ AbG.e]]
min
(3)
E.. E [Osil
==1
1=1
where (0) is the importance sampling ratio of the token osi,s and ÂDG,Es the advantage of the response obtained by DGAE, respectively given by:
Tsi mean ((ra))
(() = Ty (Ozi, Jsr Oct,ee) Thas (Oss,t Je. Osi,e()
ADG
(4)
MAD (railim))
where MAD (11
Σ
1
Fss mean ((ra)i-))
(5)
G
-=1
Here, MAD(-) denotes mean absolute deviation function. Furthermore, X₂ the difficulty-aware weight for the query 4s computed by DQW follows:
exp (D./T)
where D mean ((ra)
A = By BB. exp (D,/T)
(6)
Here, B represents the global batch size, and B, signifies the number of valid queries in the batch. query is considered valid if its G corresponding responses are not uniformly correct or incorrect. Without loss of generality, we assume that the first B, queries in the batch are valid. The token-level average loss is calculated exclusively on valid queries, a procedure we refer to as valid token-level loss averaging. This design is inspired by GPG (Chu et al., 2025) and DAPO (Yu et al., 2025) and is not a key contribution of DGPO. aims to prevent sharp gradient fluctuations caused by inconsistent valid token ratios across batches, thereby ensuring training stability, and also serves the basis for valid query reweighting in the following DQW.
In the following subsections, will describe the two key techniques of DGPO: difficulty-balanne balanced group advantage estimation and difficulty- question-level weighting.
3

---

Published as a conference paper at ICLR 2026
3.1.1 DIFFICULTY- BALANCED GROUP ADVANTAGE ESTIMATION
RIFHICULTY-BLLANCED
Consider single question y and its corresponding responses {o.) the unclipped policy gradient calculated GRPO is as follows:
G
lo:1
AGR, Volir(0)
JGRPO
Σ
04
1=1
t=1
G
lod
(AGR.)
AGR. detach (4u(0)) Ve log (Te (oLe I g o,,)))) (7)
Sgn
t=1
where sgn(.) is the sign function and detach(-) is the stop-gradient operator. The full derivation is provided in Appendix B.1. In this equation, detach ( (0)) and Vo log (To (Oi.e q. (r,<t)) respec- tively represent the importance sampling ratio and likelihood gradient for each token Oi, Crucially, sgn(AGR..) indicates whether the policy тy should be updated to increase or decrease the probability of generating the response while |AGR,il determines the corresponding update magnitude. There- fore, the total update magnitude for single question q can be upper-bounde by the sum of these individual magnitudes across all G responses, i.c., Σ [AGR,al. l. The complete derivation is provided in Appendix B.2. This magnitude has a closed- formm expression, formalized in the following theorem. The complete proof is provided in Appendix B.3.
and -ell-approx
Theorem (Update Magnitude for a Single Question using GRAE). Given a single question q and its corresponding responses { each query-response pair receives binary accuracy reward rs E {0, 1), and p represents the accuracy rate, i.e., the proportion for a reward of Then, the total update magnitude without clipping for the single question when using GRAE satisfies:
G mcan (1ri)s2))
G
AGR.
2GG - p), where p
T4-
std (ri)en))
This total update magnitude varies with respect to the accuracy rate p. reaching its maximum when p = 0.5 and gradually decreasing as p approuches either or
Theorem implies that within a training batch, questions with moderate accuracy rates have a greater influence the policy update, while easier or harder questions have smaller impact. However, we argue that challenging questions, yet have non-zero accuracy rates, warrant greater attention. These questions are ideal for training because they identify areas of the policy model's incomplete mastery while providing at least one correct response for targeted learning. Consequently, to counteract the inherent imbalance of GRAE, develop novel difficulty balanced group advantage estimation (DGAE) strategy. Specifically, the advantage function of DGAE is defined as follows:
mean (ri))en)
G
ADG.:
where MAD (fr)i
Σ
r mean (mi) (9)
MAD (r
Here, the denominator MAD(-) is the mean absolute deviation of rewards across all G responses. This normalization ensures that the total update magnitude for a single question is a constant value, formalized in the following theorem. The complete proof is provided in Appendix B.4.
Theorem 2 (Update Magnitude for Single Question using DGAE), Given single question q and its corresponding responses {or) each query-respons pair receives a reward rú. Then, the total update magnitude without clipping for the single question when using DGAE satisfies:
mean (ri))11)
ADG. =
=
(10)
G mean (ri)(
i=1
Crucially, Theorem 2 removes the binary reward constraint (r: E {0, 1)) in Theorem rendering suitable for a wide array of policy optimization scenarios.
4

---

Published as a conference paper at ICLR 2026
3.1.2 AEPHIULTY-AWARE QUESTION-LEVEL WEIGHTING
Building upon the DGAE strategy, further introduce difficulty aware question-leve weighting (DQW) scheme, which explicitly prioritizes learning from more challenging questions within each training batch. Specifically, DQW assigns a weight A each question ys follows:
exp (D./T)
A B,
B. exp (D./T) where D2 = mean ((ad))))
(11)
Here, D2 is the negative mean reward across all responses of the question 9s. serving as measure of its relative difficulty at the current training stage, and 7 denotes the temperature hyperparameter that controls the distribution sharpness. Compared advantage reweighting of Zhang & Zuo (2025), DQW is simpler and has fewer hyperparameters. Moreover, it is derived based on the analysis of the implicit update magnitude imbalance in GRPO and the balanced advantages of DGAE. This two-step "balance-then-reweighe" procedure offers improved interpretability and controllability.
3.2 MULEI-ASPECT QUESTION REFORMULATION
DGPO enhances mathematical reasoning from an algorithmic perspective by optimizing the learning process on existing data. To complement this, we propose the Multi-Aspec Question Reformulation data-centric solution, which automatically reformulates training questions by a large reasoning model to generate variants that cover more complex and comprehensive aspects. A critical constraint is that all reformulations must preserve the original gold answer. In this manner, MQR can maintain the essential mathematical logic of the question and obviate the need for solution regeneration, thereby placing minimal demands on the reformulator model.
(MQR) approach
Specifically, MQR adds story background, introduces abstract terminology, and nests sub-problems into the original question. The default reformulator model is OpenAI o3, while some smaller open- source models can also competently handle this task. The prompts are provided in Appendix C, and the core instructions for these strategies are as follows:
Core Instructions for Multi-Aspect Question Reformulation
Add a story background that is not related to the core mathematical content of the given question, but seems to be related to the question. If the given question already has such a background, change it to a new, complexer background.
Background:
2. Term:
Invent new, abstract mathematical term to define concept that is central to the given question, and restate the entire question using this term.
3. Sub-Problem: Convert a key numerical condition of the given question which have a definite value into an independent sub-probe uub-problem. The sub-problem may belong to any branch of mathematics (c.g., algebra, geometry, number theory, combinatorics).
The newly generated questions respectively challenge the policy model's ability to: 1) identify criti- cal mathematical information amidst noise; 2) grasp abstract mathematical concepts; and 3) perform reasoning that requires multiple steps and cross-domain knowledge. Successfully solving these more difficult questions provides strong reinforcement signal, compelling the policy model to develop these crucial reasoning skills. Examples of each reformulation aspect are provided in Appendix D.
Overall, the MQR -aggmeeedd data, which combines the original and reformulated questions, serves as ideal training material for DGPO, rendering MathForge synergistic loop where the data extends the model's performance boundaries, and the algorithm efficiently learns from these challenges.
4 EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Models and Datasets. In the main experiments, we train the Qwen2.5-Mathh model (Yang et al., 2024) on the MATH dataset (Hendrycks et al., 2021). To evaluate the model-agnostic effectiveness of MathForge, we conduct experiments on three other models of varying sizes and types: Qwen2.5- Math-1.5B (Yang et al., 2024). Qwen2.5-3B (Team, 2025), and DeepSeek-Mo Mark--Mahh7 (Shao et al., PerpSeck-Math-777 is fine-tuned using 80k data sampled from NuminaMath- CoT (Li et al., 2024c). Furthermore, we apply DGPO in the multimodal domain, training Qwen2.5- VL-3B-Inntruct (Bai et al., 2025) the GEOQA-8k dataset (Chen al., 2025).
2024). For cold start,
5

---

Published as a conference paper at ICLR 2026
Table 1: Comparative results of methods trained on the MATH dataset using Qwen2.5-Math-77
Methods
AIME24
AIME25
AMC23
MATH500
Minerva
Olympiad
Avg./AaRPO
Base Model
12.19
4.79
35.23
48.60
15.07
16.33
22.04
GRPO
20.94
8.44
58.98
72.20
27.76
37.33
37.61
Dr.GRPO
21.04
8.23
58,59
72.05
28.58
35.89
37.40 0.21
GPG
21.98
9.06
59.61
72.05
27.21
37.67
37.93 0.32
DAPO
21.25
8.75
58.20
72.70
29.50
37.22
37.94 +0.33
GSPO
19.38
8.33
60.16
73.00
28.12
37.26
37.7
GRPO-AD
21.56
9.48
59.06
73.25
29.14
37.07
38,26 0..6
DGPO
23.85
10.21
61.02
74.25
31.07
38.33
39.79 22:18
MQR
25.00
11.77
59.38
77.85
31.43
40.81
41.04 +3.43
MathForge 24.58
12.60
59.84
79.95
33.36
42.67
42.17+4.00
Benchmarks. In the text-only experiments, assess models on six commonly used mathematical reasoning benchmarks: AIME24, AIME25, AMC23, MATH500 (Hendrycks et al., 202 I), Minerva (Lewkowycz et al., 2022), and Olympiad (He et al., 2024). To ensure stable results, we perform 32 runs for AIME24, AIME25, and AMC--23, and runs for other benchmarks, reporting the average performance across the respective runs. For the multimodal domain, evaluate on the GeoQA test set (Chen et al., 2021) using greedy decoding. All evaluations are conducted in a zero-shot setting.
Compared Methods. We compare Foth framework against several state-of-the-arr meth- ods: GRPO (Shao et al., 2024), Dr.GRPO (Liu et al., 2025a), GPG (Chu et al., 2025), DAPO (Yu al,, 2025), GSPO (Zheng al., 2025), and GRPO-AD (Zhang Zuo, 2025). For fair algorithm- level comparison, we disable the resampling components in GPG and DAPO, and add the Advantage reweighting for Difficulty (AD) technique of Zhang & Zuo (2025) into the GRPO baseline as GRPO- AD. To isolate the contribution of each component in MathForge, we also evaluate DGPO and MQR separately. The MQR setting refers to training on the MQR-augmented data, including the original and MQR-generate data, using GRPO.
Implementation Details. We used 8 NVIDIA H20 GPUs to conduct all experiments. ensure fair comparison and reproducibility, our implementation is built upon the Open-R1 codebase (Hugging Face, 2025). For the DGPO algorithm, the temperature hyperparameter T' the DQW component is set to 2.0. For the MQR strategy, the data augmentation cost is reported in Appendix E, All other implementation details are provided in Appendix F.
4.2 MAIN RESULTT
Table presents the comparative results of various methods trained on the MATH dataset using the Owen2.5-Math-7. model. In the following, we will analyze the effectiveness of DGPO, MQR, and their combination, MathForge, respectively.
Effectiveness of DGPO. Our DGPO algorithm, when applied alone, elevates the average score 39.79%, substantial gain of 18% over the strong GRPO baseline (37.61%) This result validates our hypothesis that prioritizing more challenging questions through DGAE and DQW effectively enhances the RL training process. By rectifying the update magnitude imbalance of GRPO and explicitly focusing the model on solvable weakness, DGPO fosters a more efficient and targeted optimization. Additionally, DGPO also surpasses other advanced policy optimization techniques, highlighting the superior design and efficacy of our proposed difficulty-wware mechanisms.
Effectiveness of MQR. The use of MQR in training also yields significant improvements, reaching an average score of 41 .04%, which is 3.43% increase over GRPO. This demonstrates the validity of our three question reformulation strategies. By augmenting the training data with questions that introduce narrative noise (Background), abstract concepts (Term), and nested logic Problrro,,m), challenging and diverse learning environment. This substantial performance
MQR creates improvement indicates the effectiveness of compelling the model to develop more robust reasoning skills by tackling these more complex reformulated questions.
Effectiveness of MathF The combination of DGPO and MQR in the full MathForge frame- work achieves the best overall performance, outperforming both individual components and reach-

---

Published as a conference paper at ICLR 2026
Table 2: Comparative results of methods trained on the MATH dataset using varying base models.
Methods
AIME24
AIME25
AMC23
MATH500
Minerva
Olympiad
Avg./GRRP
Qwen2.5-Math-L...
3.65
30.94
34.95
8.55
21.93
17.82
GRPO
11.35
3.96
46.48
64.85
20.13
29.59
29.39
DGPO
11.25
5.73
49.84
65.45
21.14
30.85
30.71+1.32
+ MQR
11.98
5.42
50.08
69.65
23.81
33.67
32.4449..05
MathForge
52.34
70.10
25.74
33.89
33.84 44.45
Qwen2.5-3B
2.81
0.73
22.66
48.65
13.69
19.37
17.99
GRPO
5.31
33.28
63.35
22.89
26.41
25.47
DGPO
6.98
1,56
36.56
65.80
25.28
26.96
27.19 77.19:1.72
MQR
5.10
1.56
39.53
65.20
25.74
29.19
27.72 17.72++2/28 +2/25
MathForge
5.73
1.77
40.70
65.40
28.86
31.59
29.01 +3.34
PrrpSeck-Math-777
0.42
0.10
13.28
31.05
9.56
9.00
10.57
GRPO
0.63
0.10
19.14
41.45
14.71
13.44
14.91
DGPO
1.98
0.42
21.02
41.85
18.93
15.00
16.53
.QRR
1.98
0.83
20.86
44.25
17.00
15.74
16.78 +187
MathForge
3.12
0.73
21.72
43.60
20.68
16.74
17.77 12.80
Table 3: Ablation Results of DGPO trained the MATH dataset using Qwen2.5-Math-77
Methods
AIME24
AIME25
AMC23
MATH500 Minerva Olympiad
Avg./AGRPO
GRPO
20.94
8.44
58.98
72.20
27.76
37.33
37.61
DGPO (w/o DGAE DQW)
20.21
9.06
59.45
72.40
28.58
36.56
37.71 +0.10
DGPO (w/o DQW)
21.77
9.69
60.00
73.45
29.04
37.93
38.65+1.00
DGPO (full)
23.85
10.21
61.02
74.25
31.07
38.33
39.79 +2.18
DGPO (T 1.0)
23.12
9.06
59.45
74.15
30.61
37.78
39.03 11.42
DGPO (T 2.0)
23.85
10,21
61.02
74.25
31.07
38.33
39.79 (2.18
DGPO (T 5.0)
22.81
11.35
60.55
73.80
30.42
38.26
39.53 +1.09
DGPO (T 10.0)
21.35
9.79
62.27
74.55
29.96
37.67
39.27
ing an average of 42.17%. This result highlights powerful synergy between the data-centric and algorithmic components of our framework. MQR provides the ideal training material, diverse and challenging questions that expose the model's limitations, while DGPO capitalizes on this data by ensuring the model focuses its updates mastering these challenges. Additionally, the performance gaps between DGPO and GRPO, as well as between MathForge and MQR, further demonstrate the robustness of DGPO under different query difficulty, MQR makes questions harder.
Monss-Agnos of MathForge. To substantiate the model-agnosticism of MathForge, we fur- ther compare methods across different model sizes and types, as presented in Table 2. MathForge consistently delivers the best performance on all models, and the individual components, DGPO and MQR, also robustly outperform GRPO. This highlights that the principles of MathForge fundamental and not contingent on a specific model, underscoring its broad generalizability.
4.3 ANALYSIS OF DGPO
Ablation Studies. As shown in Table 3, we conduct ablation experiments to isolate the contribution of each component in DGPO. Specifically, the valid token-level loss averaging, DGAE, and DQW components contribute average performance improvements of 10%, 0.94%, and 1.14%, respec- tively. This highlights that DGAE effectively corrects the update magnitude imbalance of GRPO, and DQW provides a significant and complementary benefit by explicitly prioritizing more chal- lenging questions. Additionally, in DQW. The results indicate that T 2.0 yields the best overall performance. A lower temperature may potentially lead to an overly sharp distribution that focuses too narrowly the hardest question batch, while a higher temperature flattens the weighting distribution, diminishing the prioritiza- tion effect of DQW. This confirms that T 2.0 strikes optimal balance, effectively emphasizing
investigate the sensitivity of the temperature hyperparameter T

---

Published as a conference paper at ICLR 2026
Table 4: Synergistic results of DGPO with other policy optimization methods trained on the MATH
dataset using Qwen2.5-Math-75
Methods
AIME24
AIME25
AMC23
MATH500
Minerva
Olympiad
Average
GPG 21.98
9.06
59.61
72.05
27.21
37.67
37.93
DGPO
21.77
10.00
60.00
73.45
30.06
38.26
38.92
DAPO
21.25
8.75
58.20
72.70
29.50
37.22
37.94
DGPO
24.48
9.79
58.75
74.90
31,99
39.56
39,91
GSPO
19.38
8.33
60.16
73.00
28.
37.26
37.71
+DGPO
23.33
10.00
59.14
74.15
30.88
38.41
39.32
Table 5: Comparative results of methods trained the GEOQA-8K A- dataset using Qwen2.5-VL-3B
Instruct in the multimodal domain.
Methods
Base Model
GRPO
DR.G
GPG
DAPO
GSPO
GRPO-AD
DGPO
GeoQA/ActRPO
39.79
57.43
57.96
59.02 +1.99
59.02 +1.09
57.16-or 0.17
58.09 +0.00
59.95 H2.52
difficult questions while maintaining sufficient learning from the entire batch. Because the difficulty score is bounded within (-,,0), setting T = 2.0 ensures that the ratio between the maximum and minimum weights in a batch remains below e/TT / /e- 1/T = 1/2 x 1.65.
Training Dynamics. Figure 1 shows the training dynamics of DGPO ver- GRPO in our main experiments, illustrating the evolution of accuracy rewards and model output lengths on MATH500. As demonstrated in Fig- (a), DGPO consistently outper- forms GRPO after the initial phase, and the performance gap widens as training progresses, underscoring that prioritizing harder questions leads to more substantial and sustained im- provement in accuracy. Mcanwhile, Figure 1(b) indicates that DGPO tends to produce more concise responses, highlighting that DGPO not only improves correctness but also encourages the model to find more efficient and direct reasoning paths, trimming unnecessary verbosity and redundant steps.
0.-
Accuewyy Accurwcy Beward
620
GRPO
0.6
Outtuut
1:00
GEPO
DGPO
19i
Training
(a) Accuracy Reward.
(b) Output Length.
Figure I: Training dynamics of DGPO vs. GRPO evaluated the MATH500 benchmark. Both models are trained on MATH using Qwen2.5-Math-77 -atth-BB.
Compatibility with Other Methods. Our DGPO algorithm primarily introduces improved ad- vantage estimation and an additional question-level weighting scheme, both of which are compatible with most existing policy optimization methods. To demonstrate this, we integrate DGPO with GPG, DAPO, and GSPO, respectively. The combination forms are detailed in Appendix G. As shown in Table this integration yields consistent performance improvements. Particularly, the combination of DAPO with DGPO results in even higher performance than the standalone DGPO (39.91% % vs. 39.79%). This underscores that DGPO addresses a fundamental aspect of the learning process that complements the specific mechanics of other policy optimization methods. In other words, DGPO can function as general enhancement algorithm rather than a monolithic alternative.
Applicability in the Multimodal Domain. To further verify the domain-agnosticisn of DGPO, we apply it to a multimodal mathematical reasoning task. As shown Table 5, DGPO achieves the best performance of 59.95% again, significantly higher than that of GRPO (57.43%). This demonstrates that the core principle our DGPO, prioritizing more challenging questions, is not confined to text- only reasoning. It is a robust and generalizable algorithm for enhancing policy learning wherever quantifiable measure of difficulty (such as accuracy rate) can be established.
4.4 ANALYSIS OF MQR
In this subsection, normalize the total training data volume across all methods for fair compar- ison. Since MQR expands the dataset by factor of four, we achieve this by increasing the training epochs for each method accordingly. As shown in Table 6, we compare the performance of methods
8

---

Published as a conference paper at ICLR 2026
Table 6: Comparative results of methods trained the original data vs. the aug-angmented data using DGPO and varying base models.
Models
Data
AIME24
AIME25
AMC23
MATH500
Minerva
Olympiad
Average
Ori.
26.46
9.17
58.67
74.65
31.62
38.81
39.90
@wen2.5-Math-78
MQR
24.58
12.60
59.84
79.95
33.36
42.67
42.17
Ori.
11.98
5.21
50.62
68.40
24.26
32.59
32.18
Qwen2.5-Math-I.5B
MQR
13.23
7.71
52.34
70.10
25.74
33.89
33.84
Ori.
6.04
1.35
37.66
65.05
25.28
27.93
27.22
Qwen2.5-3B
MQR
5.73
1.77
40.70
65.40
28.86
31.59
29.01
Ori.
2.19
0.21
21.02
43.60
18.29
14.52
16.64
NeepSeck-Math--.
MQR
3.12
0.73
21.72
43.60
20.68
16.74
17.77
Table 7: Ablation Results of MQR on the MATH dataset using Qwen2.5-Math-75
Data
AIME24
AIME25
AMC23
MATH500
Minerva
Olympiad
Avg./Don.
Original
26.46
9.17
58.67
74.65
31.62
38.81
39.90
MetaMath- Rephrasin
25.21
11.35
59.45
76.70
31.71
39.93
40.73
Original + Background
25.52
10.73
58.59
77.50
32.90
40.48
40.95 1.05
Original + Term
25.52
11.15
58.98
77.75
33.09
40.93
41.24 1.84
Original + Sub-Problem
26.67
10.94
58.75
77.05
34.38
41.36
41.53
MQR
24.58
12.60
59.84
79.95
33.36
42.67
42.17 +2.27
trained on the original data versus the -QR-auumeetted data using DGPO and varying base models. MQR consistently yields superior results than the original data across all models, confirming that its effectiveness stems from the qualitative enhancement of the data, not merely an increase in volume. Additionally, we assess the quality of the generated data in Appendix H.
Difficulty Assessment. We first conduct a direct comparison of question difficulty by evaluating the accuracy of -rren..5-Math-7B-mmme
the subsets of MQR-augmented data. The accuracy rates are 79.77% on Original, 77.31 % on Background, 76.87% on Term, and 72.04% on Sub-Problem, confirming the increased difficulty of reformulated questions and the effectiveness MQR.
Ablation Studies. To assess the individual contributions of our three reformulation strategies, we conduct ablation studies where each strategy utilized separately. MetaMath MetaMath-Rephrrainn (Yu al., 2024) is also included sample 22.5k data from its total 50k rephrased questions, combined with the original data for train- ing. The results, as presented in Table 7, all trained using DGPO. Each strategy independently improves performance over both the Original and the MQR approach, which combines all three strategies, achieves the highest average score of 42.17%. This underscores clear synergy, where these diverse strategies produce a more substantial improvement than any individual component in mathematical reasoning.
baseline, which uses GPT-3.5- Turbb to simply rephrase questions. We
SeaMMatt-Rephrasing baselines. Crucially,
Training Dynamics. Figure 2 illus- trates the training dynamics of DGPO when trained on the original MATH
Accumecy Benwurd
0.75
0.7
0.70
dataset versus the MQR-augmented dataset. As presented in Figure 2(a), the consistently lower training accu- racy on the -QQR-augeented data ex- hibits that the reformulated questions are substantially more challenging. Despite this increased difficulty, the model trained with MQR ultimately achieves superior accuracy on the un- seen MATH500 benchmark, as depicted in Figure 2(b). This "train harder, test better" phenomenon suggests that the more challenging questions of MQR result in robust, generalizable reasoning ca- pabilities, enhancing performance while preventing overfitting.
Angumeg
0.48
0.8
10
40
196 140 Training Step
160
180 350
10
40
70
160 180 350 Training Step.
100
130
(a) Training.
(b) Evaluation.
Figure 2: Training dynamics of Original MQR train- ing and evaluation data. Both models are trained on MATH and evaluated MATH500 using Qwen2.5-Math-77

---

Published as a conference paper at ICLR 2026
Table 8: Comparative results of MQR using varying reformulator models the MATH dataset.
Reformulators
AIME24
AIME25
AMC23
MATH500
Minerva
Olympiad
Avg./Aon
Original
26.46
9.17
58.67
74.65
31.62
38.81
39.90
Qwen2.5-776 --n--tuuu-ttttt
25.10
11.98
58.67
76.85
33.00
40.96
41.0% +1.09
Qwen3-30B-Ae 1amn3-303--33B-Thnnnnngg
25.73
12.29
59.84
78.85
33.18
41.22
41.85+1.95
OpenAl
24.58
12.60
59.84
79.95
33.36
42.67
42.17
Generality to Less Capable Reformulators. The reformulator model is only required to reformu- late questions rather than solve them, thereby imposing lower demands on its reasoning capabilities. To assess the generality of MQR to reformulator models with less capabiliit, we utilize two smaller (Team, 2025) and -oeen--30B-A3B-Thinne
and open-source models: Powen..5-BB-Insse et al., 2025a). As shown in Table 8, while the most capable OpenAl o3 reformulator achieves the best results, the other two models also deliver substantial gains over the original data. This indicates that even moderately capable models can effectively generate challenging question reformulations that enhance mathematical reasoning within the MQR strategy.
(Yang
RELATED WORK
Reinforcement Learning. Policy optimization has become standard for post-training large lan- guage models to enhance their reasoning capabilities (Jaech et al., 2024; Guo et al., 2025; Team et al., 2025). Building upon Proximal Policy Optimization (PPO) (Schulman et al., 2017), Group Relative Policy Optimization (GRPO) (Shao et al., 2024) proposes highly efficient critic-less paradigm us- ing group relative advantage estimation. This spurred line of research focused on refining GRPO's stability and performance. For example, Dr.GRPO (Liu et al., 2025a) removes the length bias and PPO-objective bias in GRPO's advantage estimation. GPG (Chu et al., 2025), DAPO (Yu et al., 2025), and GRPO-LEAD (Zhang & Zuo, 2025) address issues in reward design, advantage estima- tion, and oversampling, while GSPO (Zheng et al., 2025) and GMPO (Zhao et al., 2025) introduce alternative optimization objectives. Besides, another line of work (Dai al., 2025; Yue al., 2025; Liu et al., 2025b) proposes more complex pipelines, such as value models or prompt refinement, RI. for reasoning has been widely explored and adopted across diverse applications (Sun al., 2025; Ji et al., 2025; 2026; Yang et al., 2025b;e; Li et al., 2025; Xiong et al., 2025). Additionally, other work also improves reasoning for large models from different angles (Wang et al., 2026a;b).
Data Augmentation. A parallel line of work improves mathematical reasoning from a data-centric perspective. One strategy involves generating entirely new, high-quality problem-solution pairs us- ing powerful teacher models, showing that synthetic data can rival human-curated datasets (Luo al., 2023; Li al., 2024b;a). Another strategy, more aligned with our work, focuses on reformu- lating existing questions while preserving the original answer. Approaches like MetaMath (Yu et al., 2024) and PersonaMath (Luo et al., 2024) achieve this by rephrasing problems or adopting specific personas. Moreover, advanced approach employs self-play, where the model generates its own challenging questions from solutions, fostering continuous self-improvement (Liang et al., 2025).
CONCLUSION
In this paper, we propose MathForge, comprehensive framework designed enhance mathemati- cal reasoning by targeting harder problems from both algorithmic and data perspectives. MathForge two-fold: the Difficulty-wware Group Policy Optimization (DGPO) algorithm rectifies the up- date magnitude imbalance and prioritizes challenging questions, while the Multi-Aspect Question Reformulation (MQR) strategy augments training data with more difficult, yet answer- mnswerrpreserving. question variants from multiple aspects. Extensive experiments demonstrate that this synergistic combination significantly outperforms existing methods across various models and benchmarks, un- derscoring our core principle that "harder is better" in mathematical reasoning.
ACKNOWLEDGMENTS
This work was supported in part by National Natural Science Foundation of China (62376274, 62437002).
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics, ensuring ethical compliance throughout all stages of the research. The MlQR-augmenter data was constructed by reformulating problems from the public MATH dataset. This process and the source data do not involve any personally identifiable informa- tion sensitive content, thereby mitigating privacy concerns. The primary goal of research is to enhance the mathematical reasoning capabilities of Al models, pursuit with significant potential benefits for scientific research, engineering, and education.
REPRODUCIBILITY STATEMENT
To ensure the full reproducibility of our research, we will make our code and the MQR-augmenter dataset publicly available. Our implementation built upon the Open-R1 codebase (Hugging Face, 2025). Comprehensive details regarding the experimental setup, including model configurations and all hyperparameters, are described in Section 4.1 and further elaborated in Appendix For the MQR strategy, the exact prompts used for generating the augmented data provided in Appendix C, and illustrative examples of the reformulated questions are presented Appendix D.
REFERENCES
Shuai Bai, Keqin Chen. Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang. Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint urXiv:2105.1 12221/05..4411 /45/7, 2021.
Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than $3, httph:////// http::::/gttttbtooee cow.b//.pommDeeeeeeeeeeo Agent/ R1-V, 2025. Accessed: 2025-02-02.
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint 2.Xxi::504402546. 2025.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Oni Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Muzhi Dai, Chenxu Yang, and Qingyi Si. S-grpo: Early exit via reinforcement learning in reasoning models. arXiv preprint arXiv:2505.0766 076666 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Pciyi Wang, Xiao Bi, al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Chaoqun He. Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. /4008, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0383 2021.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: //github. com hugg ing face open-rl. r1.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16721 2024.
11

---

Published as a conference paper at ICLR 2026
Yuxiang Ji, Ziyu Ma. Yong Wang, Guanhua Chen, Xiangxiang Chu, and Liaoni Wu. Tree search for Ilm agent reinforcement learning. ar preprint arXiv:2509.21240, 2025.
Yuxiang Ji, Yong Wang, Ziyu Ma, Yiming Hu, Hailang Huang. Xuecai Hu, Guanhua Chen, Liaoni Wu, and Xiangxiang Chu. Thinking with map: Reinforced parallel map- sappauumented agent for geolocalization. arXiv preprint vrXiv22601.05433 2026.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Chen Li, Weiqi Wang, Jingeheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706, 2024a.
Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. Mugglemath: Assessing the impact of query and response aug- mentation on math reasoning. arXiv preprint arXiv:2310.05506, 2023.
Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. Mugglemath: Assessing the impact of query and response aug- mentation on math reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10230-10258, 2024b.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9).9, 2024c.
Renda Li, Hailang Huang, Fei Wei, Feng Xiong, Yong Wang, and Xiangxiang Chu. Adacurl: Adap- tive curriculum reinforcement learning with invalid sample mitigation and historical revisiting. arXiv preprint 25..:::221000477.. 2025.
Xiao Liang, Zhongzhi Li, Yeyun Gong, Yelong Shen, Ying Nian Wu, Zhijiang Guo, and Weizhu Chen. Beyond pass @ 1: Self-play with variational problem synthesis sustains rlvr. arXiv preprint arXiv:2508. 14029, 2025.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint 25XIv:2503.20783, 2025a.
Ziru Liu, Cheng Gong, Xinyu Fu, Yaofang Liu, Ran Chen, Shoubo Hu, Suiyun Zhang, Rui Liu, Qingfu Zhang, and Dandan Tu. Ghpo: Adaptive guidance for stable and efficient Ilm reinforce- ment learning. arXiv preprint arXiv:2507.1063 106288 2025b.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qing- wei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583 2023.
Jing Luo, Longze Chen, Run Luo, Liang Zhu, Chang Ao, Jiaming Li, Yukun Chen, Xin Cheng, Wen Yang, Jiayuan Su, et al. Personamath: Boosting mathematical reasoning via persona-driven data augmentation. arXiv preprint arXiv:2410.01504 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (oo:::707700440 2017.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Rumxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepscekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint wxXiv:2402.03300 2024.
Zhongxiang Sun, Qipeng Wang, Haoyu Wang. Xiao Zhang, and Jun Xu. Detection and miti- gation of hallucination in large reasoning models: A mechanistic perspective. arXiv preprint arXiv:2505. / /2886, 2025.
12

---

Published as a conference paper at ICLR 2026
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025.
Qwen Team. Qwen2.5 technical report, 2025.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. hwwps:ttp//.mmm //g/ttttttttt github..hb...o www.coo/ooooooooo. http://hgggggacce///o// /trr 2020.
Yifei Wang, Yueqi Wang, Zhenrui Yue, Huimin Zeng, Yong Wang, Ismini Lourentzou, Zhengzhong Tu, Xiangxiang Chu, and Julian McAuley. Fasa: Frequency-awary sparse attention. arXiv preprint arXiv:2602.03152. 2026a.
Zengbin Wang, Xuecai Hu, Yong Wang, Feng Xiong, Man Zhang, and Xiangxiang Chu. Every- thing in its place: Benchmarking spatial intelligence of text-to-image models. arXiv preprint arXiv:2601.20354 2026b.
Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint 2rXiv:2506.1422 2025.
Feng Xiong, Hongling Xu, Yifei Wang, Runxi Cheng, Yong Wang, and Xiangxiang Chu. Hs-star: Hierarchical sampling for self-taught reasoners via difficulty estimation and budget reallocation. arXiv preprint arXiv:2505.1 19866, 2025.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5 math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. /2 2024.
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
Wenkai Yang, Jingwen Chen, Yankai Lin, and Ji-Rong Wen. Deeperitic: Deliberate critique with large language models. arXiv preprint (xXiv25505.00662 2025b.
Wenkai Yang, Weijie Liu, Ruobing Xie, Yiju Guo, Lulu Wu, Saiyong Yang, and Yankai Lin. Laser: Reinforcement learning with last-token relf-rewarding. arXiv preprint arXiv:2510.14943 2025c.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In International Conference on Learning Representations, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv: 14476, 2025.
Yu Yue, Yufeng Yuan, Qiying Yu. Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05//88 2025.
Jixiao Zhang and Chunsheng Zuo. Grpo-lead: A difficulty-awarr reinforcement learning approach for concise mathematical reasoning in language models. arXiv preprint arXiv:2504.096% 2025.
Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shao- han Huang, Lei Cui, Qixiang Ye, al. Geometric-mean policy optimization. arXiv preprint arXiv:2507.20673, 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao. Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507. 18071, 2025.
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used LLMs assist in polishing the writing of this paper. Its use limited improving grammar, clarity, and style. All core intellectual contributions, including the proposed methods, experimental design, and analysis, were conceived and executed by the human authors.
B PROOFS
B. FULL DERIVATION FOR GRADIENT OF GRPO
Consider a single question q and its corresponding responses {o:] the unclipped policy gradient calculated in GRPO is follows:
G
lo.1
AOR, (o)
9GRPO
Ei
los|
1
G
1o.1
To (or, q, 0z,{(()
AGR V
Li=1
|o:|
Tew (0t, 9, 0z,(()
1o:1
1
AOR.
detach (To (0r, q, 01,<1))
V detach (ne (ar q, o1,))))
To (os, q, 0i,<e)
Li=1
10:1
Teau (0i, q, 01,ct)
l0:1
1
AGR, detach (Z)) Vo log (na (oi. q. 0i,ct))
Σ
l0s1
i=1
1=1
las1
(AGR.))
|ARRS
detach (Ix(()) Vo log (mo (oa, 4 oi,))))
8gn
Σ
(12)
where sgn(-) is the sign function and detach(-) is the stop-gradient operator.
B.2 FULL DERIVATION FOR THE TOTAL UPDATE MAGNITUDE OF GRPO
The PPO/GRPO-styve gradient for a fixed question q can be written (ignoring token length differ- ence, clipping, and importance sampling terms) as:
G Σ
G
((9)
A log log (0:
Σ
Ågg.
(13)
G
G
1=1
By the triangle inequality, the gradient norm satisfies:
G Agg
C
[( = T
S
A.llgi
(14)
Since all gradients Ågg are generated from the same question and tend together improve the policy on that specific query, their directions are positively correlated. Such directional alignment implies limited mutual cancellation, causing the triangle inequality be nearly tight.
Moreover, as all responses in batch are sampled from the same policy with the same temperature and the same or similar math prompt, the variation in ||gal] typically much smaller than the varia- tion [ Under this mild assumption, E: AA,| serves as a tight upper bound and a faithful proxy for the question-level update strength, but is not exact equality.
14

---

Published as a conference paper at ICLR 2026
B.3 PROOF OF THEOREM 1
We provide a proof of Theorem (Update Magnitude for Single Question using GRAB) below.
Proof. By definition, the total update magnitude is the sum of the absolute values of the advantages:
Σ ri mean (tri),e))
Σ ri mean (tr:),=))
Σ AGR,i
(15)
std ((ri)
std (fri)
For binary rewards ri E {0,1), the mean value is the accuracy rate = 6 Ei Ti and the standard deviation is vp(1 pp). Substituting these gives:
Σ |ra pl Vp(1- pp
Σ ACR.
(16)
The numerator can be decomposed based on the reward values. There are Gp terms where and G(1 - p) terms where Ti Therefore:
Gpl1 pl G(1-p)|0- pl
AGR.
Vp(1- -p(1-p)
1=1
Gp(1 p) + G(1-P))
(since p E (0, 1))
Vp(1
2Gp(1- -p)
Vp(1 p)
2G 2GVp(l p).
(17)
B.4 PROOF OF THEOREM 2
We provide proof of Theorem 2 (Update Magnitude for Single Question using DGAE) below.
Proof. By definition, the total update magnitude is the sum of the absolute values of the advantages:
r mean ((r))
AbaS
Σ
(18)
r; mean ((r))
Since the denominator, j==1 [r mean (fr;) constant with respect to the summation index i and non-negative, we can move outside the outer summation:
Σ r mean( mean((r:)) (r:))e))
ApaI
- r3 mean (ean((::) G.
(19)
C PROMPTS FOR MQR
We provide the detailed prompts for MQR below.
General Prompt for Question Reformulation
want you to act as an expert Math Question Rephraser.
Your goal is to rephrase a given math question so it becomes more challenging for large AI models while remaining logically sound and fully comprehensible humans. The
15

---

Published as a conference paper at ICLR 2026
rephrased question MUST yield exactly the same final answer the original.
You should complicate the given question using the following method: (instruction)
You must strictly adhere to the following constraints:
The final answer MUST remain unchanged.
The rephrased question should be no more than 100 words longer than the given question. Preserve the original interrogative verb (e.g., "find", "determine", "compute... " "evalu- ate").
Use LaTeX for all mathematical expressions.
Output only the rephrased question (no hints, solutions, explanation, commentary).
#Given Question Start#
{question]
#Given Question End#
Specific Instruction for Background Question
Add story background that is not related to the core mathematical content of the given question, but seems to be related to the question.
If the given question already has such background, change it new, complexer back- ground.
Possible background themes include, but limited to, the following: history, culture, geography, nature, occupation, daily life, sports, art, science fiction, and adventure. Astron- omy is explicitly excluded.
The background should be presented as natural parts of the question statement, ensuring the rephrased question is coherent and self-contained.
Specific Instruction for Term Question
Invent new, abstract mathematical term to define a concept that is central to the given question, and restate the entire question using this term.
'The term should be presented as natural parts the question statement, ensuring the rephrased question is coherent and self-contained.
Specific Instruction for Sub-Problem Question
Convert key numerical condition of the given question which have definite value into an independent sub-problem.
-The sub-problem may belong to any branch of mathematics (e.g., algebra, geometry, num- ber theory, combinatorics).
The sub-problem must be self-contained, have unique solution, and its solution must yield exactly the value required for the original question.
The sub- sub-problem should be presented natural parts of the question statement, ensuring the rephrased question is coherent and self-contained.
D AUGMENTED DATA OF MQR
We provide examples of questions generated by MQR below, with the highlighted parts representing the main modifications made according to the reformulation strategies.
Original Question
Berengere and her American accepts both euros and American dollars. They want to buy cake, but neither of them has
student Emily are at a bakery in Paris that
16

---

Published as a conference paper at ICLR 2026
enough money. If the cake costs 6 euros and Emily has American five-dollar bill, how many euros does Berengere need to contribute to the cost of the cake if euro = 1.25 USD?
Question using Background Reformulation
In the bustling Montmartre district of Paris, Berengere-a culinary historian compiling notes on classic French desserts-and her visiting American friend Emily, an anthropology stu- dent documenting European food customs, wander into the venerable pâtisserie "Le Temps Sucré." " They decide to purchase famed gâteau Saint-Honors that the proprietor has priced 6 euros. Emily searches her travel wallet and discovers only single crisp five-dollar bill, while Berengere carries euros exclusively. A sign by the register lists the day's exchange rate 1 euro 1.25 USD. To complete the purchase, how many euros must Berengere contribute?
Question using Term Reformulation
Define the "euro-gap" of a prospective purchase as the non-negative difference, measured in euros, between an item's 's listed euro price and the euro-denominated value of the funds already on hand to pay for it. Berengere and her American forcign- exchange student Emily visit Parisian bakery. The cake they wish buy is priced at 6 curos. Emily only an American five-dollar bill, and the prevailing conversion rate is euro = 1.25 USD. Determine, in euros, the euro-gap gap that Berengere must cover complete the purchase.
contribute
Question using Sub-Problem Reformulation
Berengere and her American foreign-exchange student Emily are at a Paris bakery that ac- cepts both euros and U.S. dollars, but neither of them alone pay for the desired cake. Before the exchange rate is revealed, solve this independent task: Find positive integers æ and y that satisfy
x x+ = and
41.
Let r be the ratio of the larger of x and y to the smaller. The cashier states that El is worth exactly U.S. dollars. The cake costs E6, and Emily offers single $5 bill. Using the exchange rate r defined above, how many euros must Berengere contribute so that together they can pay for the cake?
E DATA AUGMENTATION COST OF MQR
The average token usage per question 255.05 input tokens, 820.27 output reasoning tokens, and 138.33 output reformulated question tokens. Therefore. the total cost for generating 22.5k reformu- lated questions of the MATH dataset is approximately $184.
F IMPLEMENTATION DETAILS
This section provides detailed information on the training and evaluation configurations used in our experiments.
For all reinforcement learning experiments, responses were generated with a temperature of 1.0 and maximum completion length of 1024 tokens. During evaluation, used generation temperature of 0.6, a top-p value 0.95, and set the maximum new tokens to 4096.
F.1 MATH
For experiments trained on the MATH dataset, used the following system prompt to guide the model's 's reasoning process: "Please reason step by step, and put your final answer within (boxed()." The maximum prompt length was to 512 tokens. For each prompt, generated 8 responses and
17

---

Published as a conference paper at ICLR 2026
used a training batch size of 32. The reward was based binary accuracy, where correct final answer yielded reward of and an incorrect one yielded O.
Model-specific hyperparameters, including learning rate, number of epochs, gradient accumulation steps, and total training steps, are detailed in Table The table specifies configurations for training on both the original 7.5k MATH dataset and the 30k -QR-augmented dataset.
Table 9: Hyperparameter settings trained on the MATH dataset using varying base models.
Learning
Gradient
Training
Models
Epochs
Rate
Accumulation
Steps
Qwen2.5-Math-7
+MQR
le-6
230 4
Owen2.5-Math-L.58
5e-7
1
230
+MQR
le-6
4
230
Qwen2.5-3B
Se-7
1
230
+MQR
le-6
4
230
DeepSeek-Math-
le-6
2
1
468
+MQR
le-6
1
937
For the cold start of DeepSeek-Math- with a learning rate of 2e-6, a batch size of 32, and gradient accumulation steps 8, resulting in total of 40 training steps.
sampled 80k data from NuminaMath-Con to fine-tune
F.2 GEOQA-8K
For the multimodal experiments on the GEOQA-8K dataset using Qwen2.5-VI. I-ntructtt we per- formed preprocessing step remove non-standard units from the gold answers to facilitate con- sistent reward calculation. Consequently, the system prompt was adjusted to: "Please reason step by step, and put your final answer without units in (boxed()."
The training was configured with a maximum prompt length of 2048 tokens and 8 generated re- sponses per question. The model was trained for 2 epochs using a learning rate of 1e-6 and a batch size of 32. We set gradient accumulation steps to 1, resulting in a total of 480 training steps. The reward mechanism was the same binary accuracy metric used in the text-only experiments.
G COMBINATION FORMS OF DGPO AND OTHER METHODS
This section details how DGPO is integrated with other policy optimization methods.
G.1 GPG
The integration with GPG involves replacing its original advantage formulation with our DGAE and incorporating the DQW scheme. Specifically, the policy gradient objective of GPG is retained, but the update for each token is now scaled by the difficult-bbllnneed advantage Apa.st Further- more, the loss contribution of each question modulated by the difficulty- weight A,- The normalization is also adjusted to average over valid tokens. The optimization objective is as follows:
JaPG-DaPO
{g.)
D, (oss)e
TT(- 9x)
lowd
|
log (or, 1, q, ADG,
(20)
==
s=1
where ÂDG, the advantage of the response o; obtained by DGAE given by:
Tsi mean (rar)
Apg
(21)
MAD (r
18

---

Published as a conference paper at ICLR 2026
and X, is the difficulty aware weight for the query 9, computed by DQW as follows:
exp (D.T))
where D2 mcan (frllia)
A, B,
(22)
Ee exp (D,
G.2 DAPO
For DAPO, the combination preserves its core PPO-style clipped objective and its use of a composite reward signal (aceuracy plus length penalty, i.e., Tsi = Tace,si + Mength,ss). We replace DAPO's original advantage estimation with our DGAE (i which is calculated using this composite reward. Crucially, the difficulty score D2 for our DQW scheme is computed only using the accuracy component of the reward (s). This design choice ensures that the question weighting focuses purely on the logical difficulty of the question, rather than being conflated with the verbosity of the responses. The optimization objective is as follows:
(DAPO-DGPO(O) = (g=)
D. fooilim TT(- q.)
B,
losl
min (O) clip (Is))0), 1 Elows 1 + thigh) ApG,St }
Σ
Ei
(23)
where (0r is the importance sampling ratio the token Ozi, and ÂpG the advantage of the response o; obtained by DGAE, respectively given by:
Tsi mean (rac)smn)
To (Osi, As Ost,ct)
Teae
ApO.et
(24)
Thaa (Oos,e 9:. Oss,
MAD
(rat
and X2 is the difficulty-aware weight for the query ys computed by DQW as follows:
exp (D,/T)
A = By
EB exp (D./T)*
mean (fracc if mean
(F i=]
where D₃
(25)
if mcan (Face,st {im)
=0
Here, B, signifies the number of valid queries in the batch. A query is considered valid if its rewards for G corresponding responses
completely equal. For questions where all corresponding responses are incorrect (i.e., accuracy reward is 0), no positive learning signal is available in the current question. Consequently. we deliberately set corresponding difficulty score. D.. to its floor value of -1. This prevents the model from dedicating excessive attention to instances that offer no constructive gradient for policy improvement.
G.3 GSPO
The integration with GSPO is performed the sequence level, aligning with GSPO's fundamental design. GSPO's sequence-level importance sampling ratio (S.;) is preserved. The update for each sequence is then driven by our DGAE, ÂDG,si The question-level weighting X, is also applied to modulate the influence of each question on the total loss. The loss is averaged over the number of valid questions. which aligns with the sequence-level nature of both GSPO and our DGPO. The optimization objective is as follows:
JaSPO-DCPO = {g, ) ~ D, ( ~ no( [.)]
min (0) clip (S.. (0), 1+ AbG.o0 (26)
By ==1
i=1
19

---

Published as a conference paper at ICLR 2026
where (0) is the sequence-level importance sampling ratio of the response Osts and ADo is the advantage the response o, obtained by DGAE, respectively given by:
Tai mean (Tanlimn)
To (Osi,t Us Ost,ce)
5,,(0) =
ADG.st
(27)
Te (Osi,t
Go Oss,et)
MAD (Fx))
and X, is the difficulty- aware weight for the query 9, computed by DQW as follows:
exp (Xp(T./T)
A = B, E exp (D./T)"
where D, = mean (ra))ia)
(28)
H QUALITY ASSESSMENT OF MQR
We utilized the OpenAl o3 model to determine whether reformulated question is mathematically equivalent to the original question. this context, mathematical equivalence defined as the ca- pacity to yield the same final answer. The specific prompt used for this evaluation is as follows:
Prompt for Quality Assessment of MQR
You expert in mathematics and logic.
Your task is to meticulously analyze and compare versions of mathematical problem: an "Original Question" and a "Rewritten Question". Your primary objective is to determine if these two questions mathematically equivalent. For the purpose of this task, "mathematically equivalent" means that both questions, when solved correctly, will yield the identical final numerical answer symbolic solution.
Equivalence Verdict: Start with clear and unambiguous "Yes" or "No". 2. Detailed Justification: If they are equivalent, explain why the changes in wording, structure, or given information do not alter the underlying mathematical operations the final result. If they not equivalent, pinpoint the specific change in the rewritten question that alters the problem's mathematical core. Explain how this change leads different solution answer.
Please structure your response as follows:
#Original Question Start# {question]
#Original Question End#
#Rewritten Question Start#
(rewritten question} #Rewritten Question End#f
We randomly sampled 100 questions from each of the three categories reformulated questions, which yielded equivalence rates of 99% for Background, 97% for Term, and 97% for Sub-Problem, respectively.
In MQR, failed reformulation means that the resulting question becomes unsolvable or has new answer different from the original answer. In math reasoning RLVR, the answer space is open-ended, extremely large, and requires exact canonical matching (e.g., exact integers, simplified fractions, or normalized symbolic expressions). Therefore, it is highly improbable that the policy model would reason incorrectly and happen provide the same answer that of the original question. Therefore, the multiple responses to the corrupted question would be uniformly incorrect (i.e., all rewards = 0). Under GRPO and its variants (including our DGPO), such questions invalid queries yielding no update gradients, thereby providing no harmful training signals.
20


Reviews:
[{"review_id": "cU3LmrrF0s", "reviewer": "Reviewer_Ft8M", "summary": "This paper proposes MathForge, a framework that enhances mathematical reasoning in large language models by focusing on harder questions from both algorithmic and data perspectives. The authors identify a critical limitation in Group Relative Policy Optimization (GRPO): its advantage estimation introduces an implicit bias where policy updates are suppressed for both easier and harder questions, peaking only at moderate difficulty. To address this, they introduce Difficulty-Aware Group Policy Optimization (DGPO), which employs difficulty-balanced group advantage estimation (DGAE) using L1 normalization and difficulty-aware question-level weighting (DQW) to prioritize challenging questions. Complementing this algorithmic improvement, they propose Multi-Aspect Question Reformulation (MQR), which generates more difficult question variants by adding story backgrounds, introducing abstract terminology, and nesting sub-problems—all while preserving the original answer. Experiments across multiple models (1.5B-7B parameters) and six mathematical reasoning benchmarks demonstrate consistent improvements, with the full MathForge framework achieving a 4.56% average improvement over the strong GRPO baseline on Qwen2.5-Math-7B.", "questions": "Difficulty distribution sensitivity: How does DGPO's effectiveness depend on the difficulty distribution of the training data? If the dataset already skews heavily toward hard or easy questions, would DGPO still provide similar benefits, or is there an optimal difficulty distribution where this method works best? Have you analyzed the interaction between dataset composition and DGPO's prioritization mechanism?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper makes several notable contributions. First, it provides rigorous mathematical analysis with formal proofs (Theorems 1 and 2) demonstrating that GRPO's advantage estimation creates an inherent bias favoring moderate-difficulty questions over harder ones, which is a valuable theoretical insight into a widely-used algorithm. Second, the proposed solutions are well-designed and complementary: DGPO corrects the algorithmic bias while MQR enriches training data with harder questions, creating a synergistic framework. Third, the experimental validation is comprehensive, spanning multiple model sizes, architectures, both text-only and multimodal domains, and six benchmarks, with consistent improvements demonstrating strong generalizability. Fourth, the MQR strategy cleverly preserves answer equivalence (avoiding costly solution regeneration) while targeting distinct reasoning skills through three reformulation aspects. Finally, the demonstration that DGPO is compatible with other policy optimization methods (GPG, DAPO, GSPO) and improves them further suggests it addresses fundamental aspects of the learning process rather than being narrowly applicable.", "weaknesses": "Despite its strengths, the paper has several limitations that warrant consideration. The central premise that prioritizing harder questions is universally beneficial lacks deeper justification—while GRPO under-weights them, the paper doesn't definitively establish when or why this is detrimental, or whether all types of difficulty contribute equally to learning. The MQR approach's reliance on OpenAI o3, an exceptionally powerful and potentially expensive model, raises concerns about reproducibility, scalability, and accessibility for the broader research community. Although MQR achieves 97-99% equivalence rates, the 1-3% of potentially incorrect reformulations could introduce noise, yet the impact of this corruption on training is not analyzed. Some improvements, particularly DGPO alone (+2.18%), are modest relative to the added computational complexity and implementation overhead. The temperature hyperparameter T=2.0 for DQW appears somewhat arbitrary despite ablations, and guidance on setting this across different scenarios would strengthen practical applicability. Finally, the paper doesn't discuss potential failure modes or scenarios where prioritizing harder questions might harm performance, such as when models still struggle with fundamental concepts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sWqCUHIFqZ", "reviewer": "Reviewer_PMhx", "summary": "This paper mainly introduces 3 techniques on top of GRPO:\n1. Difficulty-Balanced Group Advantage Estimation (DGAE), which guarantees the sums of the estimated advantage magnitude across different groups are the same.\n2. Difficulty-aware Question-level weighting (DQW) to encourage training on hard questions, which is similar to the AD technique(Advantage-reweighting for Difficulty) in arxiv:2504.09696.\n3. Multi-Aspect Question Reformulation (MQR) strategy to use a prompt asking OpenAI O3 to rewrite the prompt problem in three different ways that maintain the groundtruth answer.\n\nThe authors do experiments and ablation studies to show that the models trained with the three techniques on the MATH dataset are better in performance on the benchmarks. However, from my point of view, the training and evaluation setups are questionable, and do not match the mainstream practice. As a result, the conclusions drawn by this paper are not completely reliable.", "questions": "My questions have been listed in the weaknesses. I appreciate if the authors can clarify the confusions and provide more experimental details mentioned in weakness 4.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The idea of DQW is intuitive and reasonable. Compared to the advantage-reweighting for difficulty technique used in arxiv:2504.09696, DQW is simpler and has fewer hyperparameters for tuning. \n\n2. The evaluation is comprehensive in the aspect that it is done on various benchmarks, including AMC, AIME, MATH500 and the Olympiad.", "weaknesses": "1. The three techniques introduced in the paper are all not completely novel. The biased estimation issue is well-known in the community and has been handled in the Dr.GRPO work. The difficulty-aware policy optimization is also a topic already touched in arxiv:2504.09696. The Multi-Aspect Question Reformulation (MQR) strategy belongs to question augmentation methods via LLMs that have been investigated in depth since arxiv:2210.11610. \n\n2. The theoretical analysis of DGAE is not convincing to me. In subsection 3.1.1, the authors say \"... $|\\hat{A}_{GR,i}|$ determines the corresponding update magnitude. Therefore, the total update magnitude for a single question q can be defined as the sum of these individual magnitudes across all G responses\". However, it is obvious that the total magnitude is not equal to the sum of the individual magnitudes. As a result, using the sum of the advantage estimation magnitude as the measure for the estimation bias is not convincing. In contrast, Dr.GRPO solves the biased estimation problem by simply removing the standard variation normalization.\n\n3. The presentation is not clear enough. For example, in the introduction part, it is said that \"MathForge comprises two key components: a Difficulty-Aware Group Policy Optimization (DGPO) algorithm and a Multi-Aspect Question Reformulation (MQR) strategy\". However, in Table 2, it is indicated that Mathforge is a framework more than DGPO+MQR, and section 3 provides no further details on what Mathforge is.  Also, DGPO differs from GRPO not only in DGAE and DQW, but also in integrating part of the techniques of DAPO, which confuses the readers about the contributions of DGPO. \n\n4. The experimental setup is not appropriate, which leads to some conclusions contradictory to previous work, such as the performance degradation of Dr.GRPO and GSPO. The base model series, Qwen2.5-Math models, are known to be continual-pretrained on Qwen Math Corpus v1&v2 with a context length of 4096. However, this paper performs training and evaluation with a 1024 maximum completion length, which (1) makes the evaluation performance significantly lower than that reported in the Qwen2.5-Math technical report (e.g. 35.0% on MATH500 in this paper vs. 49.8% on MATH in Qwen's report for Qwen2.5-Math-1.5B) (2) forces the model to reason with fewer tokens as shown in Figure 1(b), which is contrary to the common sense that longer thinking length indicates better reasoning ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M9rUJuxpVf", "reviewer": "Reviewer_jCMd", "summary": "This paper argues that Group Relative Policy Optimization (GRPO) and its variants exhibit a critical limitation: their advantage estimation introduces an implicit bias where the magnitude of policy updates is lower for harder questions. And the authors propose a Difficulty-Aware Group Policy Optimization (DGPO) algorithm to debias the magnitude of policy updates. For data-centric viewpoint, this paper points out that existing augmentation approaches primarily rephrase questions to enhance diversity, without systematically increasing their intrinsic difficulty and propose a Multi-Aspect Question Reformulation (MQR) strategy. Extensive experiments demonstrate the method outperforms existing methods on various mathematical reasoning tasks.", "questions": "1. The definition of DGAE is similar to GRPO but replace second central moment to first moment of denominator and total magnitude is 1 proved by B.3, what is the meaning of multiplying G.\n2. could GRPO has better performance then DGPO if given queries near 50% accuracy since it has the largest advantage magnitute.\n3. Notice this paper use a strong model gpt-o3 to augment dataset in MQR, what is the success rate and cost to generate new queries and is it capable to employ on less capable models. A comparison with baseline rephrasing method is preferred. \n4. what is the potential explaination in Fig. 1(b) the output length is reduced significantly than GRPO.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proves that traditional GRPO has an implicit bias and could be resolve with DGPO in small modification. \n2. This paper propose a data synthesis method MRQ which shows performance improvement.", "weaknesses": "Lack of depth analysis between GRPO and DGPO under different settings, e.g. queries in different difficulties, different types of datasets. More is discussed on questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xzm44spQ5a", "reviewer": "Reviewer_HcCo", "summary": "The paper targets mathematical reasoning by emphasizing harder problems from both the optimization and data sides within a unified framework, MathForge. MathForge addresses this with a new training algorithm and a targeted augmentation pipeline designed to push models toward robust, generalizable reasoning.\n\nAlgorithmically, DGPO (Difficulty-Aware Group Policy Optimization) introduces (i) difficulty-balanced group advantage estimation (DGAE) and (ii) difficulty-aware question-level weighting (DQW), which upweights questions with lower mean accuracy within a batch.\n\nOn the data side, MQR (Multi-Aspect Question Reformulation) augments each problem—while preserving the gold answer—across three axes: adding narrative background (noise tolerance), introducing abstract terminology (conceptual grasp), and nesting sub-problems (multi-step, cross-domain reasoning).", "questions": "See `Weaknesses` part.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is generally well-written and easy to follow, with a clear description of the method.\n2. The paper provides intuitive visual demonstrations to help better understand the paper.", "weaknesses": "1. **Limited novelty.** Several design choices appear incremental relative to prior work. For example, restricting the token-level loss averaging to only the number of valid queries in the batch (`Equation (3)`) is conceptually very close to DAPO’s online filtering, which also discards uninformative samples to stabilize updates. Likewise, the proposed Difficulty-Aware Question-Level Weighting overlaps with established difficulty-prior methods that upweight harder items [1]. \n\n2. **Claims that lack sufficient evidence or run counter to intuition.** The discussion around “sharp gradient fluctuations” on `Line 144–145` seems at odds with the usual motivation for valid-query filtering (faster training and fewer zero-gradient issues). It is also unclear why all-correct/all-incorrect groups (zero-gradient) would *induce* fluctuations rather than increase variance. Please clarify this with quantitative diagnostics: gradient-norm/variance traces across steps, per-batch valid-token ratios, and an ablation toggling the valid-query criterion to show its true effect on stability. In addition, the argument on `Line 178–181` that samples with “at least one correct response” are especially valuable may conflate genuine learnability with chance hits (guessing). A more reliable analysis could incorporate token-level entropy, answer-consistency across rollouts, and response-pattern statistics (e.g., self-consistency or calibration curves) to disentangle “solvable but hard” from “lucky” cases.\n\n3. **Training cost and efficiency.** From the ablations in `Tab.1`, MQR appears to drive a substantial share of the gains, yet the efficiency implications are unclear. Please report a training cost–performance gain analysis across multiple augmentation data scales, including training time, needed memory. It would also be helpful to control for total training data size (or time) when comparing proposed method with others.\n\n4. **Fairness and completeness of comparisons.** In this paper, comparisons are only provided on GEOQA-8k, which raises two concerns: (i) the works for other algorithms may not conduct experiments on this dataset, and (ii) there is no control for equal data volume or sampling, which can confound conclusions. Please add broader and fairer comparisons: match total training data sizes across methods, use more base training dataset (e.g., beyond GEOQA-8k) to demonstrate robustness. \n\n5. **Title–claim alignment (“HARDER IS BETTER”)**. While the paper introduces Multi-Aspect Question Reformulation, it does not provide solid evidence that the augmented items are *actually harder*. Beyond the qualitative examples and the accuracy gains, please include direct hardness validation to substantiate this claim. \n\nReferences:\n\n[1] GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models. https://arxiv.org/abs/2504.09696", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)