GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning
Authors: Xiangxiang Chu
Hailang Huang
Xiao Zhang
Fei Wei
Yong Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Model Reasoning
Multimodal Model
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement Learning (RL) can directly enhance the reasoning capabilities of large language models without extensive reliance on Supervised Fine-Tuning (SFT). In this work, we revisit the traditional Policy Gradient (PG) mechanism and propose a minimalist RL approach termed Group Policy Gradient (GPG). Unlike conventional methods, GPG directly optimizes the original RL objective, thus obviating the need for surrogate loss functions. By eliminating the critic and reference models, avoiding KL divergence constraints, and addressing the advantage and gradient estimation bias, our approach significantly simplifies the training process compared to Group Relative Policy Optimization (GRPO). Our approach achieves superior performance without relying on auxiliary techniques or adjustments. As illustrated in Figure 1, extensive experiments demonstrate that our method not only reduces computational costs but also consistently outperforms GRPO across various unimodal and multimodal tasks.

Published as a conference paper at ICLR 2026
GPG: A SIMPLE AND STRONG REINFORCEMENT LEARNING BASELINE FOR MODEL REASONING
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, Yong Wang
AMAP, Alibaba Group
Project Page: https: //g / thub ctm www..//////////PPA..
ABSTRACT
Reinforcement Learning (RL) directly enhance the reasoning capabilities large language models without extensive reliance on Supervised Fine- Tininn (SFT). In this work, revisit the traditional Policy Gradient (PG) mechanism and propose a minimalist RI approach termed Group Policy Gradient (GPG). Unlike conventional methods, GPG directly optimizes the original RL objective, thus obviating the need for surrogate loss functions. By eliminating the critic and reference models, avoiding KI. divergence constraints, and addressing the advantage and gradient estimation bias, our approach significantly simplifies the training process compared to Group Relative Policy Optimization (GRPO). Our approach achieves superior performance without relying on auxiliary techniques or adjustments. illustrated in Figure 1, extensive experiments demonstrate that method not only reduces computational costs but also consistently outperforms GRPO across various unimodal and multimodal tasks.
INTRODUCTION
Owen 5-Math-TT
Owen Muth- intruct
Openilaasheen Zers------ 7B
Simpleéi
70
60.2
15.0
20.0
46.7
47
53.5
30
0.1
23.7
20
23
3.0
ANEE4
OlympiadBench
AMC223
76.2
Performe S8
70
CGNG
M.0
$1.8
21.6
11.1
Visual Reasoning
Geometry Reasoning
Ressoning Grounding
Classfication
Figure 1: Performance comparison on unimodal reasoning tasks, with extended validation on multimodal reasoning. (Top) GPG achieves substantial performance gains over state-of-the-art (SOTA) baselines across diverse mathematical benchmarks, demonstrating its core effectiveness for linguistic reasoning. (Bottom) The method also generalizes robustly multi-mode settings, outperforming other RL methods and further validating its broad applicability.
Large Language Models (LLMs) have achieved substantial advancements, progressively narrowing the gap towards achieving Artificial General Intelligence (AGI) (OpenAI, 2024; Guo al., 2025; Bai al., 2025; Wu al., 2024; Yao al., 2024). Recently, LLMs, exemplified by OpenAl ol (OpenAI,

---

Published as a conference paper at ICLR 2026
2024) and DeepSeek RI (Guo et al., 2025), adopt strategy of generating intermediate reasoning steps before producing final answers. This approach markedly improves their efficacy in domain- speciffc tasks, such mathematical reasoning (Jia et al., 2024; Gao et al., 2024; Lai et al., 2024; Lightman et al., 2023; Huang et al., 2024). The remarkable success of this technology is mainly attributed to the Reinforcement Fine-Tunning (RFT) method (Schulman et al., 2017; Shao et al., 2024; Yu et al., 2025; Li et al., 2024; Hu, 2025; Dai et al., 2026; Ji et al., 2026a; Li et al., 2026; Xiong et al., 2025). Through the application of RFT. the models allocate additional time to "deliberate" prior to generating answers, thereby constructing intricate reasoning chains and subsequently enhancing overall model performance.
In contrast to Supervised Fine-Tuning (SFT), which involves training models on fixed onput-output pairs to mimic correct responses, RFT introduces an iterative process that incentivizes models to generate coherent and logically structured reasoning paths. RFT leverages RI. techniques, such as Proximal Policy Optimization (PPO) (Schulman et al., 2017) and GRPO (Shao et al., 2024) to optimize decision-making during the generation of intermediate steps. Specifically, PPO ensures stability by constraining policy updates, preventing new strategies that deviate significantly from established behaviours. In contrast, GRPO enhances this process by evaluating performance across groups of actions, encouraging consistent improvements in reasoning quality. This dynamic and feedback-driven approach enables models think more deeply, resulting in nuanced answers that better handle complex reasoning tasks compared to the more rigid and lahel-dependent training of SFT.
Despite the significant success of PPO in enhancing reasoning quality, it still suffers severely from the enormous resource consumption required during training. PPO necessitates the development and integration of both a critic model and a reference model, which not only complicates the training process but also substantially increases computational demands. Consequently, there is a growing trend toward simplifying the PPO method. For instance, ReMax (Li et al., 2024) removes the critic model by introducing a baseline value, which reduces the training GPU memory usage and accelerates the training process. Besides, GRPO eliminates the need for a critic model and utilizes normalized rewards within a sample group.
In addition to these methods to improve efficiency and stability, a very recent and concurrent work Dr. GRPO (Liu et al., 2025a) studies the details of reward and loss normalization and states GRPO tends to generate more tokens. However, although it reveals the reward bias in the advantage function, observe that its performance did not significantly outperform GRPO.
Let's review the birth of PPO. PPO was proposed primary evaluation benchmarks, where the policy network typically learns both visual representations and the control policy. In the LLM era, however, the policy is an LLM that already possesses strong representations from pretraining and SFT. Removing unnecessary components is important for scalability, which motivates rethinking simplified RL methods. Notably, PPO itself is simplification of TRPO (Schulman et al., 2015), which in turn builds on the policy-gradient algorithm. A major weakness of policy gradients is high variance, which can be mitigated by (i) using a value-function baseline in advantage estimation and (ii) sampling more trajectories- both common practices in the weweeeraaeeerrrr post-training training for L.LMs. Thus, it natural build a streamlined method for reasoning.
general RL algorithm, with Atari games as
In summary, our key contributions are as follows:
We revisit the design of policy gradient algorithms (Sutton et al., 1998) and propose simple RL method that retains minimal RL components. Unlike conventional approaches, our method directly optimizes the objective function rather than relying on surrogate loss.
Our approach eschews the necessity for both a critic model and a reference model. Moreover, it imposes distributional constraints. These characteristies confer substantial advantages for potential scalability.
We analyze and demonstrate the reward bias inherent in existing advantage functions and reveal the limitations of simplistic debiasing methods. Our exploration of the gradient estimate bias phenomenon has led us to propose a simple yet accurate gradient estimation (AGE) technique. To mitigate the potential issue of large variance in gradient estimation when the proportion of valid samples is excessively small, we introduce simple thresholding mechanism to ensure minimal partition of valid samples is maintained, followed by resampling.
2

---

Published as a conference paper at ICLR 2026
Extensive experiments demonstrate that GPG achieves SOTA results across various unimodal and multimodal visual tasks.
2 METHOD
2.1
PRELIMINARY AND TASK FORMULATION
RL a computational approach to learning through interaction, where an agent seeks to maximize cumulative rewards by selecting optimal actions within environment. The RL problem is typically defined by policy Te, which maps states to actions, and aims to optimize the expected return. The core idea behind policy gradient methods is use gradient ascent iteratively adjust the policy parameters. The learning objective is maximizing the return J (0),
[(0) max E
Σ
re
The policy gradient theorem (Sutton et al., 1998) proves that the above problem be converted into estimating the gradient,
VOJ Eg VV log no (a, s²)QTM (s;, a;)l,
where QT (st, 0z) is the action-value function, representing the expected return when taking action a, in state 8, and following policy vo thereafter.
To reduce the variance, the advantage function A (st, az) is often used, leading to the policy gradient update rule:
VeJ EE Ve log (a 8c) "e (x),
One-step advantage estimation can be mathematically formulated as (Sutton et al., 1998):
A (8e, ur) Q (8t, V (s:),
(4)
where ST (sz) is function of st. In principle. v* (s:) can take any functional form. One commonly employed function is the value function, which represents the expected return when starting from state 8, and following policy wa. While GAE (Schulman et al., 2018) offers more sophisticated approach to balance bias and variance advantage estimation, we find that in the context of model reasoning, one-step estimation is sufficiently effective for achieving good performance. This simplicity is particularly advantageous scenarios where computational efficiency is paramount.
Given sequence of questions and instructions, the model is tasked with generating corresponding answers. Subsequently, rewards are returned based on predefined reward models or hand-crafted rules. Our objective leverage these reward signals to optimize our policy, thereby enhancing the model's ability to generate accurate and contextually appropriate responses.
However, designing or obtaining accurate rewards for intermediate steps is nontrivial (Guo al., 2025). To address this challenge, we simplify our problem as follows. Given question and prompt we sample an action a from policy we and obtain a final reward signal r. Note that the policy distribution Ty is modeled in an autorcgressive manner. In this setting, we can leverage policy gradient methods to optimize the policy.
2.2 GROUP POLICY GRADIENT
Our proposed method, Group Policy Gradient (GPG), designed to address the issue of high variance in policy gradient estimation in the absence of value model. By leveraging group-level rewards, GPG stabilizes learning and enhances the robustness Specifically, GPG utilizes the mean reward within each group to normalize the rewards, thereby effectively reducing variance. This approach eliminates the need for a traditional value model, thereby simplifying the training process and enhancing computational efficiency. The name "Group Policy Gradient" reflects our method's core mechanism of utilizing group-level mean rewards to stabilize and optimize learning.
reinforcement learning training.
3

---

Published as a conference paper at ICLR 2026
The core objective of GPG is defined as:
G
loc
JoPG (0) = E ((..)-DDoo.....
log To 1,,0, < <
(5)
i=1
where o, represents the individual responses in the group G, and the advantage of the i-th response is calculated by normalizing the group-level rewards {
A mean( (R.)
(6)
Frorm
Foor is an optional normalization technique, which commonly applied in conjunction with reward clipping to mitigate the impact of unexpected outlier values. One widely adopted practice to employ standard variance normalization within a training batch (Mnih et al., 2016; Schulman al., 2017). This approach helps stabilize the training process by reducing the variance of the reward signal, which is particularly important when dealing with environments where the magnitude of rewards vary significantly, such in different Atari games. By normalizing the reward signal, the model becomes less sensitive to extreme values, thereby improving the robustness and convergence of the training algorithm. However, in the reasoning tasks involving large models, the reward is typically well-defined and does not suffer from the same variance issues observed in other environments. As for the Math reasoning problem, it is common practice award the right answer with 1.0 and the wrong answer with 0.0.
We utilize basic Math Reasoning setting of SimpleRL from open-rl (Face, 2025), using only the MATH-lightews dataset to facilitate rapid experimental validation. Specifically, we remove the format reward and only enable the accuracy reward for simplicity.
Models
Average AIME24 MATH-500 AMC23 Minerva OlympiadBench
Qwen2.5-Man Mathh-B
30.9
13.3
57.6
45.0
14.7
23.7
GPRO
43.7
16.7
73.4
62.5
30.2
35.7
GPG horm=l,0=
43.9
23.3
76.3
52.5
30.1
37.4
GPG Porm ( R(o)),a=1) B
45.3
23.3
73.6
60.0
30.5
39.3
GPG (norm std ((() B-al
44.1
23.3
74.2
52.5
30.9
39.7
GPG Poorm
47.8
30.0
75.0
62.5
33.1
38.2
GPG =1,
B-M 3eh 0.6)
48.3
30.0
76.2
62.5
34.2
39.0
Dr. GRPO
43.7
26.7
74.6
50.0
30.1
37.3
Table I: Math reasoning results on Qwen2.5-Math-75 model. f: reproduction use the released code.
The critical component: A has been underexplored in prior research in reasoning. This gap in the literature highlights the need for further investigation of the role and impact of Åç,, within reasoning tasks. There are two unresolved problems.
The Å. should not introduce reward bias. Otherwise, bias deviates from the original problem formulation. GRPO (Shao et al., 2024) formulates Frorm std R(o)), which is essentially a function of s, in Equation 2 and explicitly introduces the reward bias. Since we aim to solve the original problem, we don't want apply surrogate or bias. However, As shown in Table 1, if remove this bias item, i.e. Fnorm = 1, it (43.9%) cannol clearly outperform GRPO (43.7%), which opposite the observation of concurrent work Dr. GRPO (Liu al., 2025a).
Examples of all right or wrong responses within a group introduce bias for the estimation of the gradient. Given a training batch of batch size B3 let the gradient of the i-th sample be denoted as gi- Without loss of generality, assume that the first M examples within the batch are all right or wrong responses within a group. The standard backpropagation (BP) algorithm estimates the gradient as:
ΣB g 21
Σ Σ g However, the first M examples are not valid for gradient estimation and contribute zero gradient. Therefore, the more accurate gradient estimation (AGE) can be written as:
B
1
M+
gi
B
B
g B
ag, a B - M
B-M
M
Mwat.///aaa////eooooooooossiinaa rlamml
4

---

Published as a conference paper at ICLR 2026
It should be noted that the value a is not constant and it varies across different sample batches. We also illustrate a with different steps in Figure 2, which indicates the necessity of gradient correction. As for multi-GPU training, to achieve more accurate gradient calculations, it is advisable to gather all non-zero gradient samples across all GPUs and compute the average gradient uniformly. This approach can be implemented through custom gradient aggregation function, which leads to increased communication overhead. Instead, we derive another equivalent format, which doesn't t require extra cost, and we provide the proof in Section A. Therefore, given a batch sample, the objective can be written as
(opa(() = (f)
(8)
As shown in Table 1, our method achieves an average score of 47.8%, being equipped with AGE.
10C
Easy probrems
0.35
famward 513
5.30
02-
Value
0.25
Sample
0.26
Policy Iterattion
Policy iteration step
Figure 2: (Left) The proportion of easy problems with all rewards are O, hard problems with all rewards are I within rollout group. (Right) The standard deviation of reward across steps.
In a scenario where we reject the M examples and resample responses in a manner similar to the approach presented in recent work (Yu et al., 2025) until M equals 0, is set to However, this particular setting is not training-efficient The reason is that the training time is constrained by the worker that takes the longest collect the desired examples. In contrast, our proposed method demonstrates superior efficiency. Moreover, it can automatically adjust the loss based on the performance of the sample batch.
We also evaluate a setting of reward normalization of GRPO, where Fnor==std| ((o)) and show the result in Table to dive into the source of the improvement. We plot the std of the reward in Figure 2. Note that the std is calculated by averaging the std of each group, whose value ranges from 0.10 0.35. And varies from 1.5 to 4.0. The reward normalization of GRPO provides such diving std (within group) mechanism, which has some gradient correction effect.
outperforms Fnorm==la==] by 1.4% average score. This motivates us
Components
Value Models
Reference Models
Surrogate Loss
Policy Constraint
PPO
GRPO
X
TRPO
V
X
GPG
X
X
X
X
Table Comparison of reinforcement learning algorithms (in reasoning) with various components.
Thresholding minimal partition of valid samples and resampling to reduce variance. While our approach provides an unbiased estimation of the gradient, it may encounter issues with high variance when the proportion of valid samples excessively low. To mitigate this, we introduce a threshold 1 for the proportion of valid samples. When this proportion falls below the given value, 3c. a...
accumulate the valid samples into the resampled subsequent batch until the proportion exceeds the threshold. This strategy effectively reduces the variance of the gradient estimation, thereby enhancing the stability and convergence rate of the model training process. It is worth noting that this strategy further improves the performance, as demonstrated in Table
RL algorithms vary significantly in their approaches tackling variance and optimizing policies. Two key components in many RL algorithms are surrogate loss and policy constraints. We summarize the main comparisons among various frameworks in Table 2. Our method stands out by preserving
5

---

Published as a conference paper at ICLR 2026
the simplest form, which not only ensures of implementation but also maintains high efficiency and effectiveness.
3 EXPERIMENTS
All experimental settings are meticulously controlled to ensure fair comparisons. We closely adhere to the hyperparameters employed by GRPO, despite their suboptimality for our approach. Notably, our method consistently outperforms GRPO across all tasks, achieving superior performance with clear margins. These results underscore the robustness and efficacy of our proposed method.
3.1 EXPERIMENTAL SETUP
Dataset and Benchmarks. In the unimodal scenario, we utilize datasets from multiple sources such as open-sl, open-rs (Dang & Ngo, 2025), and MATH-lighteval (Hendrycks et al., 2021) for D-bieek-RR--sssii----eeee
training. Specifically, we train the
base model with the open-sl dataset, resulting in the GPG-RS1 model. Similarly, training with the open-rs dataset produces the GPG-RS3 model. Furthermore, we perform ablation studies using the MATH--ighteval dataset on the Owen2.5-Mathe base model. To compare the overall performance on the 7B model, we utilize the dataset from (Yu et al., 2025), and the detailed setting is shown in Section B.1.
These datasets encompass wide range of problem types and difficulty levels. To assess the reasoning capabilities of the models, we employ five distinct mathematics-focuse benchmark datasets: AIME24, MATH-500 (Lightman et al., 2023; Hendrycks et al., 2021), AMC23, Minerva (Lewkowycz et al., 2022), and OlympiadBench (Huang et al., 2024).
In the multimodal case, handle variety of tasks. Specifically, for the visual reasoning task, utilize approximately 12, 000 samples from the SAT dataset (Ray al., 2024) for training and perform evaluations on the CV-Bench dataset Tong et al. (2024). In addressing the geometry reasoning task, by following R1-V (Chen et al., 2025), we train on around 8, 000 samples from the GEOQA training set (Chen al., 2025) and subsequently evaluating performance on the GEOQA test set (Chen et al., 2022). For both the classification and reasoning grounding tasks, we follow Visual-RFT to conduct few-shot classification training on Flower102 (Nilsback & Zisserman, 2008), Pets37 (Parkhi et al., 2012), FGVCAircraft (Maji et al., 2013), Car196 (Krause et al., 2013), respectively. Additionally, training is conducted on 239 samples from the LISA training set (Lai et al., 2024). All evaluations carried out using the corresponding test sets associated with these training sets.
Implementation Details. Our approach is broadly applicable across a wide range of reinforcement learning tasks. To demonstrate its versatility and efficacy, we conduct experiments encompassing both unimodal and multimodal scenarios. These experiments are performed on NVIDIA H20 GPUs and NPUs from China. For each experiment, code base, ensuring consistent training and evaluation procedures. The implemented GPG method refer to Algorithm and more detailed settings can refer to Appendix B.
adhere strictly to the implementation of original
Algorithm 1 Group Policy Gradient (GPG)
Input: [shape: (B, G, C, dim)] Model Output, Reward, 3.h
Collecting samples and calculate Å and a based on Equation 6 and Equation 7 until < Pex Calculate log no (o) pher__oken_logps based on and model m
2:
3: loss < <- log (o) Á a
Output: loss
3.2 UNIMODAL TASK EVALUATION
To evaluate method, select two models: strong 1.5B distilled SFT model (DeepSeek-R) -1- Distill-Qwen-1.5B) and 7B base model.
Mathematical Reasoning using 1.5B model (A strong SFT model). Compared with other 1.5B distilled models, our models exhibit superior performance with average accuracy 55.7% of GPG-RSI, illustrated in Table 3. Additionally, GPG-RS1 and GPG-RS3 shows strong results in AMC23 with

---

Published as a conference paper at ICLR 2026
Distilled 1.5B Models
Average AIME24 MATH-500 AMC23 Minerva OlympiadBench
48.9
28.8
82.8
62.9
26.5
43.3
Sill-3-1.5B-Previee
51.6
32.5
84.4
66.7
29.0
45,4
Open-RS1
53.1
33.3
83.8
67.5
29.8
50,9
Open-RS3f
52.0
26.7
85.4
70.0
27.9
50.2
GPG-RS1
55.7
33.3
87.6
77.5
29.4
50.5
GPG-RS3
55.5
33.3
85.0
80.0
26.8
52.4
Table 3: The zero-shot pass performance of the 1.5B models distilled by DeepSeek-R across five mathematical reasoning benchmarks. f: reproduced results using released codes. t: results from (Dang & Ngo, 2025).
7B Models
Average AIME24 MATH 500
AMC23 Minerva OlympiadBench
43.8
13.3
79.8
50.6
34.6
40.7
Dwen2.5-Muth-TB
30.9
13.3
57.6
45.0
14.7
23.7
Qwen2.5-Muth-7B (no lemplate)
38.2
0.2
69.0
45,8
21.3
34.7
rStar-Math-7B (Guan al., 2025)
26.7
78.4
47.5
47.1
Eurus-2-7B-PRIME (Cui et al., 2025)
26.7
79.2
57.8
38,6
42.1
Oat Dat-Zero-7B (Liu al., 2025a)
51.4
80.0
62.7
30.1
41.0
Oat Zero-BB (Liu et al., 2025a)f
47.8
30.0
80.6
55.4
29.0
44.0
OpenReasoner- Deeeassonn---ee--7 Zero-7B 8k (Hu et al., 2025)
45.9
13.3
82.4
54.2
31.6
47.9
SimpleRI-7Zer-711 (Zeng et al., 2025)*
46.6
26.7
78.2
60.2
27.6
40.3
GPG-Zero-7B
57.7
36.7
84.6
82.5
39.0
45.8
Table 4: The zero-shot pass @ performance of the 7B models across five mathematical reasoning benchmarks. t: reproduced results using the released code. $: results from (Dang & Ngo, 2025), * results from (Liu et al., 2025a).
a score of 77.5% and 80.0%, obviously surpassing Open-RS 67.5% and 70.0%. Both GPG-RS1 and GPG-RS3 demonstrate competitive performance across various benchmarks, particularly excelling in MATH-500 with scores of 87.6% and 85.0%, and OlympiadBench with scores of 50.5 and 52.4%.
Mathematical Reasoning using 7B model. As illustrated in Table 4, GPG-7B achieves average score of 57.7% and outperforms other baselines with clear margins. This exceptional performance is further highlighted in the AMC23 and Minerva, where GPG-7B attained leading score of 82.5% and 39.0%, exceeding SimpleRL-Zero-7B by impressive margins of 22.3% and 11.4%, respectively. Moreover, GPG-7B consistently exhibits superiority across most benchmarks, outperforming the recent state-of-the the-arte art method, Oat-Zero-7B. by an average of 6.3%.
Models
GEOQA
Qwen225-VV-3B-Imsttuuu
35.41
GRPO
47.48
114
725
GPG
$1.33
Table 6: 4-shot Results on Four Fine-grained Classification Datasets. GPG shows consistently better results than GRPO on 4 classification datasets.
Table 5: Geometry reasoning re- sults on GEOQA. GPG is better than GRPO.
Models
Total
Count
Relation
Depth
Distance
Models
mloUjes
mloUva
gloU
Qwen2-VL-2B VL-2B
31.38
54.69
22.46
0.16
31.66
Qwen2-VL.25
26.9
30.1
25.3
+ SFT
$7.84
60.02
68.92
$5.00
45.83
SFT
28.3
29.7
25.3
+ GRPO
59.47
59.64
66.76
54.16
56.66
+ GRPO
37.6
34.4
34.4
+ GPG
76.15
66.62
83,23
81.66
75.50
GPG
51.8
51.3
50.4
Table 7: Visual reasoning results on CV-Bench (Tong et al., 2024), which shows GPG training on base model has overall better performance over GRPO and the base model.
Table 8: Reasoning grounding results on
LISA (Lai et al., 2024). GPG surpasses GRPO in reasoning grounding.

---

Published as a conference paper at ICLR 2026
3.3 MULTIMODAL TASK EVALUTAION
We further evaluate our method on several very recent multimodal benchmarks, most of which report results based on GRPO.
Geometry Reasoning. In addition to visual reasoning, MLLMs exhibit notable proficiency in geometry reasoning. To evaluate the efficacy of the GPG method in this domain, we employ an experimental setup similar to that used in RI-V (Chen et al., 2025) using the GEOQA (Chen et al., 2022) dataset. The results, presented in Table 5, indicate that the GPG method achieved score of 51.33% surpassing the GRPO's score of 47, 47.48 by 3.85% points. This demonstrates the superior performance of the GPG method in addressing complex geometric reasoning tasks.
Classification. Beyond the evaluation of reasoning tasks, we also assess the enhancement of the GPG method over GRPO in image perception tasks. As shown in Table 6, the GPG method achieves an average score of 89.0% across four classification datasets, surpassing GRPO by 7.1% points. Additionally, our method consistently produces improvements across all four classification datasets, underscoring its superiority in image perception tasks.
Visual Reasoning. We initially evaluate the GPG method using the CV-Bench (Tong et al., 2024) visual reasoning dataset, strictly adhering to the parameter settings of VisualThinker-R R1-nker--R-Zroo.... As illustrated in Table 7, the GPG method demonstrates a significant improvement in performance. Specifically, it attains a score of 76. 15% on CV-Bench, representing an increase of 16.68% points compared to the 59.47% score achieved by GRPO.
Reasoning Grounding. The final critical aspect of evaluating MLLMs involves precisely identifying objects according to user requirements. To this end, we employ the Qwen2-VL-25 model for grounding tasks using the LISA dataset (Lai et al., 2024), with the results presented in Table 8. In comparison to the GRPO method, the GPG approach demonstrates a substantial enhancement, improving all metrics by over 14.0% points. This significant improvement underscores the superiority of the GPG method in object localization, leading to considerable advancements in reasoning and perception capabilities.
3.4 ABLATION STUDY AND DISCUSSION
Case Study and Training Analysis. We present the reasoning processes of GPG and GRPO, as illustrated in Figure 4 (supplementary). Compared to GRPO, the GPG approach demonstrates a more comprehensive and accurate reasoning capability, whereas GRPO exhibits errors in formula analysis. Consequently, GPG arrives at the correct solution, while GRPO produces an incorrect result. In Figure 3, we present range of real-time training metrics to illustrate the effectiveness of GPG straightforward yet strong RL algorithm.
Sensitivity on Group Size. We study the effect of the number of generations within group. As shown in Table 11, increasing the group size from 2 to 16 leads to progressive improvements across most metrics. Specifically, the Average performance improves steadily with larger group sizes. We choose 8 to achieve a good tradeoff between training cost and performance.
Comparison with Various RL Methods. We attempt to explain the differences between GPG and other RL methods in the simplest way. As shown in Table 14, it can be seen that the loss of GPG does not include the "CLIP term" and the "KL divergence". Its form and calculation are the simplest, and as discussed in Section 3.2, its performance is better than other methods.
Comparison with DAPO (Yu et al., 2025). We meticulously control the experimental settings and rigorously reported the results in Table 9. All models are trained on the same dataset and for the same number of steps (1 (1100). In contrast to DAPO (Yu et al., 2025), which incorporates proposed components, method focuses exclusively achieves superior performance with reduced training and data costs. DAPO, which constructs fully valid batches through dynamic sampling, often requires more batches and may waste valid samples in the final batch. In contrast, our method avoids these inefficiencies, ensuring optimal resource utilization and enhanced performance.
the accuracy reward. Despite this, approach
KL constraint, In principle, our method is designed to optimize the original reinforcement learning (RL) problem directly. And it's bit strange without imposing any distribution constraints. Despite
8

---

Published as a conference paper at ICLR 2026
Method
Average AIME24 MATH-500 AMC23 Minerva OlympiadBench Training Cost Data Cost Memory
DAPO-7B
56.0
30.0
84.6
82.5
34.9
47.8
1x
1x
28G
GPG-Zero-7B
57.7
36.7
84.6
82.5
39.0
45.8
0.45x
x.39
24G
Table 9: Comparison with DAPO (Qwen-7B Math). Ours is simpler, stronger and resource efficient.
this, we conduct an ablation study to evaluate the impact of adding distribution constraint. The results are presented in Table 13. Our findings indicate that incorporating such a constraint negatively impacts performance.
Limited by space, we provide more ablation studies in Section B.2.
3.5 IMPACT AND LIMITATION DISCUSSION
Achieving advanced general intelligence critically depends on augmenting the reasoning capabilities of models, with efficient and scalable reinforcement learning methods serving a cornerstone. Our proposed approach investigates a minimalist strategy that aims to enhance reasoning capacity through simplicity and efficiency, thereby potentially facilitating the development of scalable systems. However, given the constraints of our computational budget, we do not evaluate our method on extremely large models.
RELATED WORK
Large Model Reasoning. Recent advancements in both LLM and Multimodal Large Language Model (MLLM) increasingly focus enabling models to simulate human-like, stepwise reasoning processes. In the field of LLMs, researchers have pioneered methods such as Chain-of-Thoughee (CoT) prompting (OpenAI, 2024; Wei et al., 2022; Kojima al., 2022; Ye al., 2025), Tree-of- Thought (Yao et al., 2023), Monte Carlo Tree Search (Feng et al., 2024; Xin et al., 2024; Trinh al., 2024), and the construction of complex SFT datasets (Muennighoff al., 2025), enhance performance in reasoning tasks. Notably, approaches such as DeepSeek-R1 (Guo et al., 2025) employ large-scale RI. with format-specific and result-oriented reward functions, guiding L.LMS toward self-emerging, human-like, complex CoT reasoning with significant performance improvements in challenging reasoning tasks. Meanwhile, MLLMS convert inputs from various modalities into unified LLM vocabulary representation space for processing and exhibit superior performance in vision understanding tasks (Wu et al., 2024; Liu et al., 2023; Chen et al., 2024; Google, 2023). Building advancements in LLM reasoning, the research community collectively applies the DeepSeek-R1 I methodology to MLLMS to enhance their visual reasoning capabilities, yielding remarkable progress (Zhang et al., 2025; Liu et al., 2025b; Chen et al., 2025; Zhou et al., 2025; Yuan et al., 2026b; Ji et al., 2026b; Wang et al., 2026; Yuan et al., 2026a).
Reinforcement Learning. RL has driven significant progress in sequential decision-making, with policy gradient methods being fundamental to optimizing stochastic policies. The REINFORCE algorithm (Williams, 1992) establishes early principles for gradient-based policy updates in trajectory- driven tasks. However, its high variance poses challenges for scalability. To address this, subsequent research focus on stabilizing policy optimization processes. Trust Region Policy Optimization (TRPO) (Schulman et al., 2015) introduces constrained updates via quadratic approximations to ensure monotonic improvement. This approach further refined by PPO (Schulman al., 2017), which employed clipped objective functions simplify the optimization process. Subsequent studies seek to enhance the PPO algorithm (Zheng et al., 2023) or elaborate on its implementation (Engstrom al., 2019). PPO achieves widespread use in language model alignment and robotic control. However, the algorithm's 's dependence on conservative policy updates or heuristic clipping thresholds can undermine its exploration potential in favour of stability. which poses significant challenge in complex domains requiring dynamic strategy adaptation.
I Limited by space, more related work is discussed in Section C.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
In this paper, we introduce GPG, which effectively addresses the critical challenges inherent in reinforcement fine-tuning approaches such as PPO and GRPO. By directly incorporating group-based decision dynamics into the standard PG method, GPG simplifies the training process and significantly reduces computational overhead without sacrificing reasoning quality. This breakthrough provides more efficient framework for training advanced LLMs capable of complex reasoning, thereby contributing to more resource-effective and scalable artificial intelligence systems.
REPRODUCIBILITY STATEMENT
We have taken the following steps ensure the reproducibility of our empirical results: (1) We provide comprehensive description of all experimental setups, including the datasets used and their corresponding benchmarks, in Section 3.1. (2) Our implementation builds upon several publicly available code repositories. For unimodal tasks, we utilize the VERL. framework (Sheng et al., 2024), Open-rl (Face, 2025), and Open-rs (Dang Ngo, 2025). For multimodal tasks, we adopt SuualThinker-RI-ZOn (Zhou et al., 2025), RI-V (Chen et al.. 2025), and Visual-RFT (Liu et al., 2025b). These repositories have been adapted to suit our purposes and to facilitate replication by the research community. (3) Detailed training configurations including hyperparameters, evaluation protocols, and specific adaptations applied to each base framework are thoroughly documented in Appendices B.I. (4) In compliance with the double-blind review policy, we have made our full implementation, along with training and evaluation scripts, publicly accessible through an anonymous repository. This ensures that all reported results be reproduced without revealing the authors' identities.
REFERENCES
Marcin Andrychowicz, Anton Raichuk, Piotr Starczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters for on-policy deep actor-critic methods? large-scale study. In International conference on learning representations, 2021. 16
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingho Liu, Eric P. Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning, 2022. URL https: arxiv. org abs 11115.114511 6, 8, 16
Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than $3, https: https://githubs com/ -eee//Deeeeggg/////eee R1-V, 2025. Accessed: 2025-02-02. 6, 8, 9, 10, 16
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024. 9
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.
Yanqi Dai, Yuxiang Ji, Xiao Zhang, Yong Wang, Xiangxiang Chu, and Zhiwu Lu. Harder is better: Boosting mathematical reasoning via difficulty-wware grpo and multi-aspect question reformulation. ICLR, 2026.
Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small 1lms: What works and what doesn't, t, 2025. URL pettts://arree wogg /250/ 621999 9. 6, 7, 10, 16
10

---

Published as a conference paper at ICLR 2026
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International conference on learning representations, 2019.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl. https://github... huggingface/open- r1, 2025, Accessed: January 2025. 4, 10,
Xidong Feng. Ziyu Wan, Muning Wen, Stephen Mareus McAleer, Ying Wen, Weinan Zhang, and Jun
Wang. Alphazero-like tree-search can guide large language model decoding and training, 2024.
URL wwt.t:tpp////. /wwxivvvvvvv.////va//eee/e/aaevv org/ abs 2309 17179.
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07983 2024. 2
Google. Gemini: A family of highly capable multimodal models, 2023. 9
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small lims can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.045 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501.12944 2025. 3,9
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03884 2021. 6, 17
Chloe Ching-Yo Yun Hsu, Celestine Mendler- Dünner, and Moritz Hardt. Revisiting design choices in proximal policy optimization. arXiv preprint wrXiv:2009.10897 2020. 22
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models, 2025.
URL https:://aaaate.e
org/ abs 0aos/2500 2500. 032 62.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint ar arXiv2200.222000
2025. 7
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36: 62991-63010, 2023. 17
Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyuman- shan Ye, Ethan Chern, Yixin Ye, et al. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai. Advances in Neural Information Processing Systems, 37:19209-
19253, 2024. 2, 6
Yuxiang Ji, Ziyu Ma, Yong Wang, Guanhua Chen, Xiangxiang Chu, and Liaoni Wu. Tree search for Ilm agent reinforcement learning. ICLR, 2026a. 2
Yuxiang Ji, Yong Wang, Ziyu Ma. Yiming Hu, Hailang Huang, Xuecai Hu, Guanhua Chen, Liaoni Wu, and Xiangxiang Chu. Thinking with map: Reinforced parallel map-augmented agent for geolocalization. arXiv preprint 2tt::2600..55333 2026b.
LI Jia, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi. Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, et al. Numinamath, 2024.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. Large language models are zero-shot reasoners. In S. Koyejo. S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.). Advances in Neural In- formation Processing Systems, volume 35, pp- 22199-22211 Curran Associates, Inc., 2022. URL ht https: htttps://proeeddings neurins. cttppppsccpppper__eeeee http://ooeeeeiiee.uu...o...aaaaa file/ 2 Pare Conf--eeeee-- .df..

---

Published as a conference paper at ICLR 2026
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine tie-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013. 6,
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579-9589 9589, 2024. 2, 6, 7, 8, 16
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer. Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 45:3843-38577 2022. 6
Renda Li, Hailang Huang, Fei Wei, Feng Xiong, Yong Wang, and Xiangxiang Chu. Adacurl: Adaptive curriculum reinforcement learning with invalid sample mitigation and historical revisiting. AAAI, 2026. 2
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quar Qu--nQann Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models, 2024. URL hwtptttp///// /aaxiviv org/ abss/ / 2231 0 0505. 2
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023. 2, 6
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, (1:33892-349115 2023.
Zichen Liu, Changyu Chen, Wenjun Li. Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783,
2025a. 2,4,7, 7, 17
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025b. 10, 16, 18, 19
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1rXiv:1306.51577 2013. 6, 7, 16
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928-19999999 1998-197..0 PmLR, 2016. 4
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang. Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint 2rXiv:2501.1933 2025. 9
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE, 2008 6, 7, 16
OpenAL Learning to reason with llms, 2024. learning-tor to--tto-oie coooreaoo--- with- l1ms /. 1,9
URL https::/ index/
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3508 IEEE, 2012. 6, 7, 16
Arijit Ray, Jiafei Duan, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A. Plummer, Ranjay Krishna, Hao Zeng, and Kate Saenko. Sat Spatial aptitude training for multimodal language models, 2024. URL https: /ttp//xxvvvo///// org/ 2412. 2112.077555 6
12

---

Published as a conference paper at ICLR 2026
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1897. PMLR, 2015. 2, 21
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 2, 4, 9, 17,
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018. URL https: http///axxvvvoooo abs/1506. 02 82. 3, 17
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepscekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024. 2, 4, 17,22
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhí framework. arXiv preprint arXiv: 2409. / 19256, 2024. 10, 16
Richard S Sutton. Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. 2,
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang. Shusheng Yang, Adithya Iyer, Xichen Pan, Ziteng Wang, Rob Fergus, Yann LeCun, and Saining Xic. Cambrian-l: 1: A fully open, vision-centric exploration of multimodal Ilms, 2024.
URL httpt:t//////. http///axvvvv.. org/ abs/ / 2406 6860. 6, 7, 8,8, 16
Tricu Trinh, Yuhuai Wu, Quoe Le. He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 2024. doi: 1.11038//4158--23--6666
Yu Wang, Yi Wang, Rui Dai, Yujic Wang, Kaikui Liu, Xiangxiang Chu, and Yansheng Li. Urban seminttc segmentation with vision-language reasoning. arXiv preprint arXiv:2601.10477, 2026. 9
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 25:24824-24837, 2022.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-v12: Mfxxure-of-experts -siounlanguage models for advanced multimodal understanding. arXiv preprint arXiv:2412. 2024. 1,
Huajian Xin, Z. Z. Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du, Wenjun Gao, Qihao Zhu, Dejian Yang, Zhibin Gou, F. Wu, Fuli Luo, and Chong Ruan. frepseek-prover-ee learning and monte-carlo tree search, 2024. URL https:/
Harnessing proof assistant feedback for reinforcement www.aaivvvvv.......a.aa...a wwtp:ii.rr//b///44400 ab00 08152.
Feng Xiong, Hongling Xu, Yifei Wang, Runxi Cheng, Yong Wang. and Xiangxiang Chu. Hs-star: Hierarchical sampling for self-taught reasoners via difficulty estimation and budget reallocation. EMNLP, 2025." 2
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:1 1809-11822, 2023.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianci Cai, Haoyu Li. Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint a40i 8000 2024.
13

---

Published as a conference paper at ICLR 2026
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for 03387. 9
reasoning, 2025. URL https:://axxii www.wawwww.....aaoo.ooooooooooot
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai. Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source IIm reinforcement learning system at scale, 2025. URL htwps: /ttes///rxxiii http::/s:///axiii /ttttp:///rrai..ooggaa///////// 14476. 2, 5, 6, 8, 16, 17
Zhenlong Yuan, Chengxuan Qian, Jing Tang, Rui Chen, Zijian Song, Lei Sun, Xiangxiang Chu, Yujun Cai, Dapeng Zhang, and Shuo Li. Autodrive-12: Incentivizing reasoning and self-reflection capacity for vla model in autonomous driving. ICI., 2026a.
Zhenlong Yuan, Xiangyan Qu, Chengxuan Qian, Rui Chen, Jing Tang, Lei Sun, Xiangxiang Chu, Dapeng Zhang, Yiwei Wang, Yujun Cai, et al. Video-star: Reinforcing open-vocabulary action recogmition with tools. ICLR, 2026b.
Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. httpp::/bkustt-aat
not site sestte/simlerrrrreeee
2025. Notion Blog.
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. RI-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503. 21..5503..2223. 12937, 2025. 9
Rui Zheng, Shihan Dou, Songyang Gao, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Limao Xiong, Lu Chen, Zhiheng Xi, Yuhao Zhou, Nuo Xu, Wenbin Lai, Minghao Zhu, Rongxiang Weng, Wensen Cheng, Cheng Chang, Zhangyue Yin, Yuan Hua, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. Secrets of rlhf in large language models part i: Ppo, 2023. 9
Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1- zero's "aha moment" in visual reasoning on a 2b non-sft model, 2025. URL https: //arxiv.. org/ brg/abs/ 220./55333.0 05132. 10, 16
14

---

Published as a conference paper at ICLR 2026
A ANALYSIS OF DISTRIBUTED GRADIENT AVERAGING WITH INVALID SAMPLES
A.1 PROBLEM FORMULATION
Consider distributed training setup where:
A batch of B samples is evenly distributed across N GPUs, with each GPU processing = B/N samples.
For the i-th GPU, the first M, samples produce zero gradients (invalid samples), while the remaining (K M,) samples generate valid gradients.
Let Moral Σ i= Mj denote the total invalid samples, and S B MMosa the effective valid samples.
Let Ii, represent the gradient of the j-th valid sample on the i-th GPU. We define the valid gradient sum for GPU i as:
G
(9)
M
The conventional distributed averaging approach in PyTorch computes gradient estimate:
1
GryTorch
G
(10)
B
whereas the theoretically correct gradient should be:
N
1
Guue
G₁
(11)
S
2m1
Proof. Step 1: Conventional Approach Derivation
Each GPU calculates its local mean using the assigned sample count K (not valid samples):
G3
Groccll
(12)
K
Global averaging then gives:
G
N
CiyyTorhh
G
docal
G
(since N K = B)
(13)
K
Step 2: True Gradient Computation
The correct gradient averages over only valid samples:
Gime
G
(S= B
Mota)
(14)
5
Observe the proportional relationship:
1 N
GryTorch
G;
(15)
B
im
s
Σ
G
(16)
3
S
Gime
(17)
B
n
15

---

Published as a conference paper at ICLR 2026
B MORE EXPERIMENT DETAILS
B.1 EXPERIMENT SETTINGS
Training setting 7B based on dataset from (Yu et al., 2025). We employ the VERL framework (Sheng et al., 2024) with a global batch size of 144 prompts. For each prompt, we generate responses and use only accuracy-
rewards. Our implementation strietly follows Algorithm 1. We optimize the network using the AdamW W optimizer with a constant learning rate of 1x 10 and weight decay of 0.1. The threshold value Beh. is set to 0.6. We trained the model for 1100 steps utilizing 48 NPUs sourced from China.
To evaluate the unimodal reasoning capabilities of our proposed method, we utilize publicly available code repositories: Open-rl (Face, 2025) and Open-rs (Dang & Ngo, 2025). These reposito- ries are selected due to their extensive coverage of various reasoning scenarios and their ability to present substantial challenges that effectively assess the reasoning capabilities of advanced models. The DeepSeek-Re R1-Distilll-Wue and open-rs datasets, as reported in the repository (Dang & Ngo, 2025), resulting in the GPG-RS1 and GPG-RS3 models, respectively.
1.5B model is trained for 100 and 50 global steps using the open-sl
For multimodal tasks, we have selected three renowned frameworks our code base: VisualThinker- RI-Zero (Zhou et al., 2025), RI-V (Chen et al., 2025), and Visual-RFT (Liu et al., 2025b). These frameworks cover variety of tasks, including visual reasoning, geometric reasoning, and image perception. The use of distinct code bases enables a comprehensive assessment of the performance enhancements achieved by method across different tasks. Specifically, for the VisualThinker-Rl Zero framework, we evaluate the results of the GPG approach on the CV-Bench (Tong et al., 2024).
Additionally, evaluate our method on the GEOQA dataset (Chen et al., 2022) based RI-V. Finally, for tasks related to image perception, such as classification (Nilsback & Zisserman, 2008; Parkhi al., 2012; Maji al., 2013; Krause et al., 2013) and reasoning grounding (Lai et al., 2024). we examine the performance of GPG using the Visual-RFT framework.
B.2 MORE ABLATION EXPERIMENT RESULTS
Bth
Average
AIME24
MATH-500
AMC23
Minerva
OlympiadBench
0.6
48.3
30.0
76.2
62.5
34.2
39.0
0.8
48.6
33.3
73.6
67.5
29.4
39.3
Table 10: Ablation on different Bt, using Qwen2.5 Math 7B.
Group Number
Average
AIME24
MATH-500
AMC23
Minerva
OlympiadBench
2
41.9
16.7
71.6
60.0
25.0
36.0
4
43.3
20.0
73.2
55.0
29.8
38.5
8
45.3
23.3
73.6
60.0
30.5
39,3
16
47.3
26.7
74.6
65.0
32.4
37.8
Table 11: Ablation on different group size (wo AGE) using Qwen2.5 Math 7B.
Model
MMLU
C-Eval
38.31
32.91
+ GPG
38.53 (+0.22)
33.29 (+0.38)
Table 12: Evaluation of GPG on MMLU and C-Eval.
Reward Normalization. We study the role of reward normalization and show the result in Table 13. Normalization within batch is common practice in the RL training process (Andrychowicz et al., 2021). The results of the experiment show that reward normalization within a group is better than the batch.
Comparision various RL methods. We compare the main component of various RL methods in Table 14 and illustrate the evolution from GPRO to GPG in Table 15.
16

---

Published as a conference paper at ICLR 2026
Foorm Average
AIME24
MATH-500
AMC23
Minerva OlympiadBench
Group Batch
45.3
23.3
73.6
60.0
30.5
39.3
44.9
23.3
72.2
55.0
35.3
38.5
43,9
23.3
76.3
52.5
30.1
37.4
Table 13: Ablation on reward normalization using Qwen2.5 Math 7B.
RL Method
Loss Function
Advantage Function
(Sclalan et 2017)
Zwo min
ma(o) FoLse (o)
Al
me(o) Tewn
applying GAE (Schuinin et al., 2015) based on rowand and the crilic model.
where A computed by
GRPO (Shio et al, 2024)
Casro
min
Noto)
A.CLIP A
A anc Riot
meaan (R(o))
Thaa
(Liu di,, 20256)
Coccaro
Cora
R(o) moant R(o))
DAPO (tu 2025)
(o(o) Rauce (0)
d.
Touia A
Ty(o)
Rto) muuun R(a))
(Bd()))
GPG
log Tylo)
mo (R(o) mean (Rao((((o)))
Table 14: Comparison of various RL methods, explain in the simplest form.
Compared with the GRPO baseline, Group A replaces the loss with a policy-gradien (PG) loss and removes the KL divergence term. It thus applies a policy-gradient algorithm with group rewards, as in GRPO. Group B corrects an error in reward normalization and uses the proper formula; however, its performance degrades. We attribute this degradation to gradient bias, which Group mitigates via a-scaling, yielding improved performance. Group D further improves performance by imposing minimum valid-sample proportion threshold, which serves as a reruutio-reductionn mechanism.
Models
Javeraze Value Model::Reefeeence Models|Suroguee Loss Policy Coustruint Debiased Gradient Variance Reduction
Muth 7B
30.9
GPRO
GPGUE.... will [to)), PPGrGoupp
GPGI
43.9
GrGif
D. GPG(E 1.u Au 0.6)
48.3
X
8
Table 15: Math reasoning results on Qwen2.5-Math-7B model.
Robust evaluation across multiple runs. To more robustly evaluate our method, we additionally report the mean and standard deviation in Table 16. Our method consistently shows clear advantages over other baselines, consistent with the trends in Table 4.
B.3 EVALUATION ON GENERAL BENCHMARKS
One potential concern for GPG is that the performance gains specialized reasoning benchmarks might come at the cost of degrading the model's 's general capabilities. To investigate this, we conduct an additional evaluation on two widely used general benchmarks that are unrelated to the reasoning datasets used in training: MMLU (Hendrycks et al., 2021), which covers 57 subjects spanning STEM, humanities, social sciences, and other fields, and C-Eval (Huang et al., 2023), a comprehensive Chinese evaluation suite consisting of 52 diverse disciplines.
We evaluate the DeepSeek RI Ri-distill- Sweilll-ween11... 1.5B and the same model after being trained by GPG. The evaluation is performed using the OPENCOMPASS framework, ensuring identical settings for fair comparison. As shown in Table and 18, GPG achieves consistent improvements on both MMLU (+0.22) and C-Eval (+0.38), indicating that it not only boosts also enhances performance on general-purpos evaluations. Detailed results for each sub-domain are provided in Table 17 and Table 18. These findings confirm that GPG's improvements reasoning tasks do not compromise the model's general capabilities, and in some cases even slightly enhance them. Therefore, GPG can be regarded as safe and broadly applicable method.
benchmarks but
specialized
We also report zero-shot evaluation results on code generation and general QA tasks in Table 19, where our method still outperforms GRPO.
17

---

Published as a conference paper at ICLR 2026
Model
Avg AIME24 AMC23 MATH 500 MINERVA
pass / (Acc/Sid)
Oat-Zero
52.0 31.6/8.8 66.5/7.6
79.5/1.8
30.5/2.8
Marus-2-7BPRIINA
48.9 16.7/6.3 62.5/7.5
79.6/1.8
37.1/2.9
Saan-Reasnner-Zero-ee
46.3 15.8/6.3 55.0/7.9
82.2/1.7
32.2/2.8
Owen-2.5-Matte 1B-SimleRL--Zer
49.4
29.2/8.5 60.6/7.8
76.6/1.8
31.3/2.9
GPG-Zero-7B
58.7
31.7/8.8 80.6/6.1
85.3/1.6
37.4/3.0
pass 3 (Acc/Stcl
Oat-Zero
59.4 40.0/8.8 74.4/6.7
85.6/1.6
37,4/2.9
Eurus-2-25 2-iis-2---PPRIN--
58.7 27.5/7.4 76.3/6.9
86.9/1.5
44.3/3.0
Open-Reasoner-Zeree
54.8 20.8/6.9 68.8/7.8
88.1/1.5
41.5/3.0
1ren-225-Math-7--Simpeett
58.9 36.8/9.1 70.0/6.7
86.0/1.3
42.8/3.0
GPG-Zero-7B
64.7 41.5/9.2 85.0/5.7
89.0/1.4
43.4/3.0
pass @3 (Acc/Std)
Oat-Zero
62.2 42.5/8.9 78.8/6.7
87.1/1.5
40.5/3.0
Murrus2-7BB-PRINAT
62.1 30.9/8.2 80.6/6.7
89.3/1.3
47.6/3.0
Open-Reasoner-Zeero-11
60.4 27.5/7.4 78.1/6.9
90.5/1.3
45.3/3.0
Qwen-2.5-Math- 7--i-n----L---------
62.7 41.6/9.1 74.4/6.7
88.8/1.3
45.9/3.0
GPG-Zero-7B
66.2 41.7/9.2 86.3/5.7
90.9/1.3
46.0/3.0
Table 16: Pass@3 and @ results- as mean + standard deviation computed over four random seeds for the Qwen2.5-7B base model. GPG consistently shows clear advantages over other baselines, consistent with the trends in Table 4.
B.4 PROMPT AND REWARD FUNCTION
Prompt for Reasoning. In the process of reinforcement fine-tuning, specific instructions are incorporated into the system prompt. These instructions encourage the model to generate intermediate reasoning steps, thereby facilitating the reasoning capabilities of the model. An example of this approach is provided below (Liu et al., 2025b):
System Prompt for Reasoning for 1.5B Model
A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.c., <think reasoning process here </thnk><aaswwe>> answer here </answer>
System Prompt for Qwen 7B Reasoning
<lim_ moin_sstrttssstemmmnntt You are a helpful assistant.<Cin <eime-lliiiiiiie wwwpeeeenciinsstrrtaeeaaeoae
can read five pages of her fiction book or two pages of her history textbook seven min- utes. If Kelly wants to read thirty pages of each book, for how many minutes in to- tal must Kelly read?InPlease reason step by step, and put your final answer within startbaassittannnn" hatiinnmnmuuuuiiiistaaaaiisannnn
hoxed.<lim, <lii__endddnnliin wwweeenndddnnneeeeeeeee
Reward Function. For most tasks, we use the accuracy and formatting reward functions. For the grounding task, the Intersection over Union (loU) reward function is utilized. For the Qwen 7B setting, we only use the accuracy reward.
Accuracy: the model's output is consistent with the ground truth, a reward of 1.0 is awarded.
Formatting: the format the model output is "<think></think> <answer></answer>" a reward of 1.0 is granted.
18

---

Published as a conference paper at ICLR 2026
MMLU Datasets
Deepseek-RI-Disee -i--iisill---------555
+GPG
Accuracy Gain
college biology
24.31
24.31
college chemistry
32.00
32.00
0.00
college computer science
21.00
21.00
0.00
college mathematics
32.00
32.00
0.00
college physics
28.43
30.39
+1.96
electrical engineering
50.34
50.34
0.00
astronomy
36.84
38.16
+1.32
anatomy
35,56
35.56
ahstract algebra
27.00
27.00
machine learning
37.50
37.50
0.00
clinical knowledge
41.8
42.64
+0.75
global facts
31.00
30.00
-1.00
management
43.69
43.69
0.00
nutrition
38.89
38.89
0.00
marketing
58.55
58.55
professional accounting
29.08
28.72
-0.36
high school geography
44.95
45.45
+0.50
international law
45.45
47.11
+1.66
moral scenarios
24.13
24.36
+0.23
computer security
39.00
39.00
0.00
high school microeconomics
44.96
46.22
+1.26
professional law medical genetics
27.71
28.16
+0.45
46.00
46.00
professional psychology
33.5
33.66
+0.16
jurisprudence
39.81
39.81
0.00
world religions
33,92
33.33
-0,59
philosophy
41.48
42.12
+0.64
virology
40.96
40.96
0.00
high school chemistry
38.42
39.41
+0.99
public relations
42.73
42.73
high school macroeconomics
42.82
43.08
+0,26
human sexuality
48.85
48.85
0.00
elementary mathematics
37.57
38,10
+0.53
high school physics
24.50
23.84
-0.66
high school computer science
42.00
42.00
0.00
high school european history
40.00
40.61
+0.61
business ethics
43.00
43.00
moral disputes
37.57
37.28
-0.29
high school statistics
50.00
50.46
+0.46
miscellaneous
44.32
44.57
+0,25
formal logic
29.37
29.37
0.00
high school government and politics
38.34
38.86
+0.52
prehistory
32.72
33.33
+0.61
security studies
43.67
43.67
0.00
high school biology
44.52
44.19
-0.33
logical fallacies
38.04
38.04
0.00
high school world history
42.62
43.04
+0.42
professional medicine
38.60
38.97
+0.37
high school mathematics
30.00
30.74
+0.74
college medicine high school history
32.37
32.95
+0.58
35.78
35.78
0.00
sociology
47.76
48.26
econometrics
32,46
32.46
0.00
high school psychology
42.94
42.75
-0.19
human aging
36.32
36.32
0.00
foreign policy
56.00
57.00
+1.00
conceptual physics
40.43
39.57
-0.86
AVERAGE
38.31
38.53
+0.22
Table 17: Comparison of performance metrics across general MMLU datasets.
IoU: Consistent with Visual-RFT (Liu et al., 2025b), the reward value is derived from the calculated scores bounding boxes generated by the model.
19

---

Published as a conference paper at ICLR 2026
C-Eval Datasets
Deepseek-RI-Distill--Neee
GPG
Accuracy Gain
computer network
9.09
9.09
0.00
operating system
42.86
43.36
+0.50
computer architecture
28.57
27.93
-0.64
college programming
50.00
50.00
0.00
college physics
16.67
17.49
+0.82
college chemistry
42.86
42.86
0.00
advanced mathematics
42.11
41.83
-0.28
probability and statistics
38.89
39.30
+0.41
discrete mathematics
20.57
21.05
+0.48
electrical engineer metrology engineer
38,89
38,89
0.00
12.00
13.40
+1.40
high school mathematics
20.78
21.67
+0.89
high school physics
47.18
50.67
+3.49
high school chemistry
49.00
50.25
+1.25
high school biology
22.22
22.54
+0.32
middle school mathematics
26.67
26.34
-0.33
middle school biology
39.00
38.37
-0.63
middle school physics
42.86
45.93
+3.07
middle school chemistry
33.33
32.47
-0.66
veterinary medicine
46.15
47.13
+0.98
college economics
57.89
57.89
0.00
business administration
15.38
19.88
+4.50
marxism
30.82
30.82
+0.00
mao zedong thought
16.67
17.06
+0.39
education science
45.45
45.16
-0.29
teacher qualification
55.56
57.68
+2.12
high school politics
21,43
29.24
+7.81
high school geography
27.27
28. 11
+0,84
middle school politics
29.56
29.56
0.00
middle school geography
4.73
4.73
0.00
modern chinese history
25.00
25.00
0.00
ideological and moral cultivation
39.98
39.98
0.00
logic law
46.67
46.02
-0.65
26.67
20.33
-6.34
chinese language and literature art studies professional tour guide
15,38
14.79
-0.59
35.71
35.71
0.00
20.00
20.00
0.00
legal professional
7.14
6.66
-0.48
high school chinese
41.67
41.67
0.00
high school history
41.67
42.21
+0.54
middle school history
9.09
9.09
0.00
civil servant
47.37
47.84
+0.47
sports science
62.48
62.48
0.00
plant protection
33.33
34.33
+1.00
basic medicine
44.44
43.74
-0.70
clinical medicine urban and rural planner
42.86
42.16
-0.70
57.14
61.44
+4.30
accountant
23.53
22.77
-0.76
fire engineer
41.67
41.67
0.00
environmental impact engineer tax accountant
19.05
18.34
-0.71
30.75
29.22
-1.53
physician
25.00
24.79
-0.21
AVERAGE
32.91
33.29
+0.38
Table 18: Comparison of performance metrics across general C-Eval datasets.
20

---

Published as a conference paper at ICLR 2026
Code
General QA
Method
MBPP
MBPP+
HellaSwag (atCn))m/stderr)
TruthfulQA (mc2/stderr)
GRPO
24.60%
21.96%
41.914/0.492
47.307/1.516 50. 50.146/1.533
GPG
26.19%
23.81% %
42.551/0.493
Table 19: Zero-shot results on code generation and general QA tasks using Qwen-1.5B.
B.5
ZERO-SHOT EVALUATION ON GENERAL QA AND CODING TASK
Loss
Reward
Format Reward
Completion
0.10
20.0
0.75
2000
244
1100
0.30
-02
22
-0.3%
25
40
50
03
220 140
123
100
28
40
66
50
300
LSE
349
165
40
Policy iteration
60
06
200
120
240
100
23
40.
60
05
163
220 343 300
Figure 3: Comparison of GPG(blue curves) and GRPO(gray curves) in terms of training loss, rewards and completion length. Experiments are based on DeepSeek-RI RI-Distill-Qwen same as Table 3.
B.6 CASE EXAMPLE
We show case study in Figure 4.
Solve follow ing muth problem efficiently and elearly. The line of your response should be the following format: "Therefone, the final answer I hope correct" (without quotes) where [ANSWER is the final number expression that solves the problem. step by step
User:
points the hyperbula J7 y" the greatest real number less RD" for all such rhombi. Assistant: Let's solve this step by step.
answering.
diagonals
intersect the
Ground Truth: 480
GPG:
GRPO:
<thinks> ABCD rhombus with diagonals intersecting the origin, we know that AB CD2 = AD + BC and the diagonals right angles.
points A. R.
on the hyperbola,
gives the equations: 2f Assume that (avv0,.bv2)) G (-ay20, - bv24), due to the symmetry of the rhombus.
bisect
equation the hyperbola
24
To find spocific courdimates (x. of points A, B, 30 hyperhola, we solve the hyperhola equation these points satisfynng the thombus propesties. To find BD', we takse (x) and (2s.)t) on hyperbola such that the diagonals bisecl each other and cross at the origin. Utilize the symmetry that rhombus provides, where diagonals the rhombus are equal.
given as
transversely symmetric origin. From
hyperbola properties: (e<ta)" (N/28) (b-28) (a22)) Thus solving both yield simpler veriex compoment: Expression: 2v20 24 2444. Bounding details, max schievable setting confirms: Hence, BD' = 420 + 24) gives: Calculated
completo max achiovable filting cross: 176, <think>
We derive and solve expressions for the distances between these points (ix., BD, which encompasees symmetric properties):
<answer> Therefore, the final answer is: correct. </inwwer>
176 hope that
-Caleulate the squared distnnes of each diagonal intersecrion.
Consider maximum length fits within geometric
properties of the thombus:
Henos, BD vxi yf x5 y3
After considering all possible values within mage dictated the hyperbolic equation and rhombus properties, we find the greatest
number. <think>
<answer> Therefore, the final snswer [400] hope correct.
<lanswrr
Figure 4: Comparison of GPG and GRPO in mathematical reasoning task based on DeepSeek-R1- Distill-Qwen-1.5BB model trained Open-rs dataset: test case from AIME24 dataset.
C MORE RELATED WORK
Proximal Policy Optimization. PPO (Schulman et al., 2017) addresses the inherent optimization instability of Trust Region Policy Optimization (TRPO) (Schulman al., 2015) through clipped
21

---

Published as a conference paper at ICLR 2026
surrogate objective. Formally, let the probability ratio between the updated policy my and the previous policy Tes be defined as
To (a)())))))
(0)
(18)
uu (at st)
where de and denote the action and state at timestep t, respectively. While TRPO maximizes the surrogate objective
JTRPO (0) E. [r)(
(19)
under a Kullback-Leible (KL) divergence constraint, PPO reformulates this via clipped mechanism. Here, Ag represents the estimated advantage function quantifying the relative value of action a in state st. The PPO objective is defined as:
J (0) = Ex min (r),))A,,
- -(1())
(20)
where the clip operator restricts (0) to the interval [] e, controlling the policy update magnitude. This constraint prevents excessive policy deviations that could degrade performance.
, 1 + e]. with being a hyperparameter
To further stabilize training and promote exploration, the composite objective incorporates three components: 1) Clipped policy gradient term (CL)P(O)) Value function loss:
VF E, (Vo(ss) - Vunger(st)) (st))²
(21)
where (s,ss is the vtate-value function estimator and (ssge($:) denotes the target value computed via temporal-difference methods, 3) Entropy regularization:
H(s,, mo)
Σ acA
To (e(a log (alss))
(22)
with A being the action space, which prevents premature policy convergence by encouraging stochas- ticity.
The complete objective integrates these terms
JPPO (0) R. (0) + C2H,(s (0))
(23)
where > 0 and > 0 are coefficients balancing policy optimization, value estimation accuracy, and exploration. Crucially, PPO replaces TRPO's computationally intensive second-order KL constraints with first-order gradient clipping, enabling efficient large-scale implementations while preserving monotonic policy improvement guarantees, as rigorously established through surrogate objective monotonicity analysis (Hsu et al., 2020).
Group Relative Policy Optimization. GRPO (Shao et al., 2024) establishes policy gradient framework that eliminates dependency explicit value function approximation through comparative advantage estimation within response groups. The method operates by sampling multiple candidate outputs for each input question and constructing advantage signals based relative rewards within these groups. For given question P(Q), the algorithm generates G responses {oi,..., oG from the current policy Tew then computes token-level advantages using intra-group reward comparisons.
The advantage term År, for the t-th token in the i-th response defined as the deviation from the group average reward:
Å R((o) - T R(o,),
(24)
where R(-) denotes the reward model's evaluation. This design inherently aligns with the comparative training paradigm of reward models, which typically learn from pairwise response rankings.
The optimization objective integrates clipped probability ratios with explicit KL regularization. Defining the token-level probability ratio as:
(opoo.l (,, 0c,ct)
(0))
(25)
To. (0i, 44 og,er)
22

---

Published as a conference paper at ICLR 2026
the clipped surrogate objective constrains policy updates through:
Tz.t (0) min (rr,,())A.,,, clip(rr..(0), 1 e, +.1+c,1+... e)A4.). -))1+)
(26)
Diverging from PPO's implicit KL control via reward shaping, GRPO directly regularizes policy divergence using an unbiased KL estimator:
Ter(oo 0i.tt) log (e, 1,
DRL. [mo] [er]
(27)
To(os, 41 01,ct) To(0r,e 01,ct)
The complete objective combines these components with regularization coefficient B::
(O) Bgg(or)
Glod Σ
(T (0) (0) -I1DxI. [ |ieet)
(28)
23


Reviews:
[{"review_id": "rNY8XmdbZW", "reviewer": "Reviewer_u5um", "summary": "The paper introduces Group Policy Gradient (GPG), a simplified reinforcement learning (RL) algorithm designed to improve reasoning in large language models (LLMs) and multimodal models. Building on policy gradient methods, GPG eliminates the need for both a critic model and reference model, while directly optimizing the original RL objective rather than a surrogate loss (see Section 2.2). The method normalizes rewards at the group level and incorporates an Accurate Gradient Estimation (AGE) correction (Equation 7) to handle bias introduced by invalid samples. Extensive experiments demonstrate that GPG achieves superior results over GRPO and PPO across multiple unimodal reasoning benchmarks (e.g., AIME24, AMC23, MATH-500) and multimodal reasoning tasks (e.g., visual reasoning, grounding, classification) — see Tables 4–8. Overall, GPG is presented as a minimalist yet strong baseline for RL-based reasoning, offering better performance with reduced computational cost (see Table 9).", "questions": "Normalization Effect: In Table 1, normalization with Fnorm = std{R(o)} improves performance. Could the authors clarify whether this acts primarily as a variance reduction mechanism or indirectly biases the gradient toward higher-reward samples?\n\nScalability: How does GPG perform with larger-scale LLMs (e.g., >70B) where group sampling is more diverse? Are communication overheads from multi-GPU gradient aggregation (see Section A.1) significant in such settings?\n\nReward Generality: Would GPG remain stable with dense or continuous rewards (e.g., reward models for factuality or helpfulness) instead of discrete accuracy-based rewards?\n\nRelation to PPO Variants: How does GPG compare to ReMax and Reinforce++ (Hu, 2025) in terms of both variance and convergence rate? Including such baselines would clarify its contribution to the broader RL-for-LLMs landscape.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality: The paper’s originality lies in revisiting classical policy gradient formulations (Section 2.1–2.2) to develop a direct optimization approach that avoids surrogate losses. Unlike PPO and GRPO, GPG fully removes the need for a critic and reference model (Table 2), and introduces a theoretically motivated correction term (AGE, Equation 7) to address gradient estimation bias. This balance between conceptual simplicity and empirical strength is a novel contribution to RL for reasoning.\n\nQuality: The experimental design is comprehensive and sound. GPG is evaluated across diverse tasks and scales, from 1.5B to 7B LLMs (Tables 3–4) and from unimodal to multimodal settings (Tables 5–8). The paper controls for fairness by using the same hyperparameters as GRPO (Section 3.1), demonstrating clear performance gains. Additionally, ablations on normalization (Fnorm) and resampling thresholds (β_th) provide insight into the method’s behavior (Section 3.4, Table 11).\n\nClarity: The paper is clearly structured, with intuitive explanations of the GPG mechanism and its differences from existing methods (Table 2). Equations (5)–(8) and Figures 2–3 effectively illustrate the derivation and behavior of the algorithm. The visualizations on page 5 show variance reduction and adaptive correction, supporting the theoretical claims. However, the write-up, particularly the introduction, could benefit from additional polishing. Additionally, the layout of the submission should be improved (e.g., on first page there is no space after the table caption).\n\nSignificance: The approach’s simplicity makes it reproducible and scalable. By removing KL constraints and critic networks, GPG significantly lowers the barrier for applying RL to reasoning tasks. The empirical results — particularly the improvements on reasoning-heavy datasets (AMC23: +22.3% over SimpleRL, Table 4) — suggest the method could serve as a new baseline for RL-based fine-tuning in both language and vision-language models.", "weaknesses": "Limited Theoretical Depth: While the paper provides a compelling intuitive justification for GPG, the theoretical analysis is shallow. The AGE correction is empirically validated but not formally proven to ensure unbiasedness under stochastic sampling.\n\nAblation Scope: The ablation studies mainly focus on hyperparameters within GPG (e.g., group size, normalization), but lack comparisons to alternative simplifications such as ReMax or DAPO variants without dynamic batching. This makes it difficult to isolate whether improvements come primarily from AGE, normalization, or the removal of surrogate losses.\n\nEvaluation Scale: The experiments are limited to ≤7B parameter models due to compute constraints (Section 3.5). The authors acknowledge this, but it raises questions about the method’s scalability to larger models (e.g., 70B or multi-billion-parameter MLLMs).\n\nReward Design Simplification: The authors employ binary accuracy rewards (1.0/0.0) for math reasoning (Section 2.2), which may not generalize to tasks with continuous or ambiguous reward signals. The paper could benefit from demonstrating robustness under more complex reward formulations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ptVhrup78h", "reviewer": "Reviewer_gf7C", "summary": "This manuscript proposes GPG, a minimal post-training algorithm for LLM reasoning. The authors argue that a minimal REINFORCE objective with group-wise mean reward as baseline can outperform other algorithms with PPO/GRPO-style surrogate objective, reference models, KL divergence constraints, or critic. They also propose a simple technique, AGE, to remove those examples with no contribution to the gradient and reduce the bias of gradient estimation.", "questions": "Please refer to the weaknesses. If the authors can address them properly, the reviewer may consider raising the score accordingly.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The method is simple, but the experimental results are surprisingly good. \n* The authors conducted broad experiments across different task domains.", "weaknesses": "**Major:** \n* There is no theoretical grounding for stability without the surrogate objectives, constraints, and critic. After reading this paper, the reviewer still cannot understand why this simple algorithm would work. The reviewer is not saying that the authors should provide some theorems and prove them, but I think it is important to provide some understanding about:\n    * What problems and challenges the previously proposed techniques are trying to deal with, and why they are needed in previous practices.\n    * Under what circumstances the simple algorithm would work, and what the failure cases are. The reviewer is not doubting the experimental results provided by the authors, but supposes that this simple algorithm will encounter failures such as divergence and mode collapse. To convince people and make a large impact on the community, the authors should be very specific about the scope of this simple algorithm.\n\n* The authors provided comparisons of RL algorithms in Table 2 and 14 to better understand their differences (which is nice), but it is not reflected in the experiments. To truly understand the discrepancies and the contributions, the authors should:\n    * Provide evaluation results of each of the algorithms mentioned in Table 2. Maybe part of them are already presented in Table 3 or 4, but the reviewer finds it hard to get a clear understanding because the evaluated algorithms are not explained. \n    * Equally importantly, it would be better if the authors could provide ablation studies on each of the four factors mentioned in Table 2, i.e., degrading gradually from PPO to GPG. The reviewer believes this will help the readers better understand the mechanism of each factor and why this simple algorithm works.\n\n**Minor:**\n* The main experiments are provided in pass@1. The reviewer is interested in pass@3, 5, etc.\n* DAPO has already proposed to remove the KL constraints and introduced dynamic sampling, where the latter is very similar to the AGE technique proposed in this paper. The authors should either be clear that these are not novel contributions of this paper, or provide an intuitive analysis of why the removal of KL constraints and the introduction of dynamic sampling have different motivations than the previous work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y0g14mJVAr", "reviewer": "Reviewer_S5T4", "summary": "This submission proposes GPG, a minimalist RLHF/RFT approach that directly optimizes the original policy-gradient objective without relying on surrogate losses, critic/reference models, or KL constraints. To address gradient dilution when groups are entirely correct or entirely incorrect, the paper introduces three key components: a group-mean baseline, an accurate gradient estimation (AGE) factor $\\alpha=\\frac{B}{B-M}$ that rescales gradients when the effective batch shrinks, and a minimum valid-ratio threshold with accumulation/resampling. Experiments cover mathematical reasoning and several multimodal tasks, with ablations on $\\alpha$, thresholding, and group size. Overall, the method is simple and practical, and it improves upon GRPO-like baselines in the reported settings.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work gives a clear and simple alternative to GRPO-style training. It treats the \"all-correct/all-wrong\" group issue as a key problem and fixes it with light tools (group mean, AGE, a small threshold/accumulation rule). These parts keep the method close to plain PG while cutting engineering overhead from reference/critic/KL. \n\n2. The paper explains why using data-dependent normalization can add reward bias and why removing it helps. It shows an unbiased estimator and an equivalent multi-device implementation via loss scaling. The AGE factor is tied to the share of valid samples; the appendix explains the distributed averaging equivalence in a direct way.\n\n3. The authors test both unimodal math tasks and several multimodal tasks (visual reasoning, geometry QA, classification, grounding), and shows consistent gains over GRPO. The ablations over $\\alpha, \\beta_{th}$, and group size are helpful to read. A brief comparison against adding a KL constraint is also reported.", "weaknesses": "- The design largely follows established ideas in variance reduction, importance weighting, and baseline correction. While the engineering is simple and effective, the overall contribution reads more like a practical consolidation and optimization of an existing paradigm rather than a breakthrough in theoretical framework.\n\n- The paper proves unbiasedness and shows distributed equivalence, but it does not give variance bounds, stability, or convergence guarantees. The settings of $\\alpha$, $\\beta_{\\text{th}}$, $G$ are chosen by ablations. Adding even light bounds or citing close results and mapping assumptions would strengthen the theory.\n\n- The current evaluation focuses on math reasoning and a few multimodal tasks. Adding code-generation benchmarks (e.g., HumanEval, MBPP, APPS) and general QA benchmarks (e.g., ARC, HellaSwag, TruthfulQA) would better show generality and the case for “replacing GRPO.” Results are mostly pass@1/accuracy; please also report pass@k and multi-seed mean +(-) std to reflect stability. The simplicity advantage can be further strengthen if the authors can include computational results like runtime and VRAM usage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pSr6jV2A7h", "reviewer": "Reviewer_t9S6", "summary": "This paper proposes a minimalist reinforcement learning framework, Group Policy Gradient (GPG), which directly optimizes the original policy-gradient objective without relying on surrogate losses, reference models, or KL constraints. The method mitigates the gradient-dilution issue caused by all-correct or all-wrong samples within a group through a group-mean baseline, an unbiased scaling factor,  and a valid-sample ratio threshold. This design simplifies the RLHF training pipeline while maintaining effectiveness.", "questions": "1. Could you provide additional theoretical analysis or intuitive explanation of why the proposed objective (direct policy gradient with group normalization) may accelerate convergence compared to surrogate-based methods? For instance, does removing the KL term implicitly allow for larger policy updates or better credit assignment?\n\n2. It remains unclear whether GPG maintains stability in generating longer responses (e.g., multi-step or chain-of-thought reasoning). Could you analyze performance stratified by response length or reasoning complexity to clarify whether the method remains robust compared with GRPO/DAPO in more difficult reasoning tasks?\n\n3. Is there any principled guidance on how to choose the hyperparameter?  For example, $\\beta_{th}$. How sensitive is the training process or convergence speed to its value?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a minimalist reinforcement learning framework (GPG) that directly optimizes the original policy-gradient objective, removing the need for surrogate losses, reference models, and KL constraints. This design substantially simplifies the reinforcement fine-tuning process.\n2. The paper clearly derives GPG from the original policy gradient theorem. The analysis of the “reward bias” problem is thorough, and the introduction of the AGE and thresholding mechanisms provides an unbiased—but variance-controlled—implementation under the absence of surrogate losses and KL constraints.\n3. The experiments are extensive, covering both mathematical reasoning and multimodal benchmarks, demonstrating the method’s generality. GPG outperforms GRPO and other baselines on multiple datasets, and its comparison with DAPO highlights its data efficiency.", "weaknesses": "1. While the paper highlights that GPG avoids surrogate losses and KL constraints, it provides limited theoretical comparison with existing methods regarding the objective formulation, convergence properties, or stability guarantees. The current justification primarily relies on empirical evidence (e.g., the claim that “adding KL degrades performance”) without analytical support.\n\n2. The experiments are mainly conducted on mathematical and visual reasoning datasets. It would be beneficial to include broader tasks, such as general QA and code-generation datasets to demonstrate generality.\n\n3. No code or implementation link is currently provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)