Geometric-Mean Policy Optimization
Authors: Yuzhong Zhao
Yue Liu
Junpeng Liu
Jingye Chen
Xun Wu
Yaru Hao
Tengchao Lv
Shaohan Huang
Lei Cui
Qixiang Ye
Fang Wan
Furu Wei
Venue: ICLR 2026 Poster (2026)
Keywords: large language model
reinforcement learning
stability
Primary area: reinforcement learning
TL;DR: a stabilized variant of GRPO

Abstract: Group Relative Policy Optimization (GRPO) has significantly enhanced the reasoning capability of large language models by optimizing the arithmetic mean of token-level rewards. Unfortunately, GRPO is observed to suffer from unstable policy updates when facing tokens with outlier importance-weighted rewards, which manifest as extreme importance sampling ratios during training. In this study, we propose Geometric-Mean Policy Optimization (GMPO), with the aim to improve the stability of GRPO through suppressing token reward outliers. GMPO is plug-and-play—simply replacing GRPO's arithmetic mean with the geometric mean of token-level rewards, as the latter is inherently less sensitive to outliers. GMPO is theoretically plausible—analysis reveals that both GMPO and GRPO are weighted forms of the policy gradient while the former enjoys more stable weights, which consequently benefits policy optimization and performance. Experiments on multiple mathematical reasoning benchmarks show that \Ours-7B improves the average Pass@1 of GRPO by up to 4.1%, outperforming many state-of-the-art approaches. Code is available at https://github.com/callsys/GMPO and verl.

Published as a conference paper at ICLR 2026
GEOMETRIC-MEAN POLICY OPTIMIZATION
Yuzhong Zhao
Yue Liu
Junpeng Liu?
Jingye Chen Qixiang Ye!
Xun Wu
Yaru Hao
Tengchao Lv
Shaohan Huang'
Lei Cui*
Fang Wan
Furu Weift
!UCAS CCUKK HKKUST
*Microsoft Research
htps://aka.ms/GeneralAA
ABSTRACT
Group Relative Policy Optimization (GRPO) has significantly enhanced the rea- soning capability of large language models by optimizing the arithmetic mean of token-level rewards. Unfortunately. GRPO is observed to suffer from unsta- ble policy updates when facing tokens with outlier importance- weighted rewards, which manifest as extreme importance sampling ratios during training. study, we propose Geometric-Mean Policy Optimization (GMPO), with the aim to improve the stability of GRPO through suppressing token reward outliers. GMPO is plug-and-play sand-play-implll replacing GRPO's arithmetic mean with the geometric mean token-level rewards, as the latter is inherently less sensitive to outliers. GMPO is theoretically plausible-analysis reveals that both GMPO and GRPO are weighted forms of the policy gradient while the former enjoys more stable weights, which consequently benefits policy optimization and performance. Experiments on multiple mathematical reasoning benchmarks show that GMPO-7B improves the average Pass @ of GRPO by up 4.1%, %, outperforming many state-of-the-art and verl.
this
approaches. Code is available at http:://gthuu..oo//allls////////
Range importance sampling ratio: log(o.
GRPO
min cl)p((((()) Flow, thigh) ) )A]}
GMPO (Ours)
|o]
min [px(0)A, clip(((()) <low, Chigh } 1M
GRPO
Unstable policy updates!
sgn(A)
GMPO (Ours)
-15
Training steps
Figure Comparison between GRPO and our GMPO. GRPO optimizes the arithmetic mean of leken-level rewards while GMPO optimizes the geometric mean (left). When training with GRPO, the importance sampling ratio (p)(8) to unstable policy updates. In contrast, GMPO enjoys more stable importance sampling ratio and fewer outliers (right).
seaia (o. ) frequently reaches extreme values, leading
INTRODUCTION
As test-time scaling becomes key research focus the large language model community, recent post- training methods have increasingly sought to extend chain-of-thought (CoT) generation to enhance reasoning capabilities. Recent advances, such as Group Relative Policy Optimization (GRPO) (Shao et al., 2024), leverage multiple sampled responses per input prompt to compute relative rewards and advantages (Á in Figure 1, left), leading to notable improvements in reasoning performance. By maximizing the arithmetic mean of teven-level rewards, GRPO has achieved strong results on complex tasks such mathematics, code generation, and multimodal reasoning.
Equal contribution. Corresponding authors. Work done during internship Microsoft Research.

---

Published as a conference paper at ICLR 2026
Token-level rewards
log(pr(0))
Arithmetic-Ment (Sensitive outliers)
High varioace
Outher Importance weighted rewards
Geometric-Mean
variance
(Robust outllers)
Figure 2: Compared to the arithmetic mean, the geometric mean is more robust to outliers and yields importance sampling ratio distributions with lower variance.
During GRPO training. the importance-weigee reward for each token given by (e( where v,(o)]0,0e0) This ratio plays key role the importance sampling ratio Ps (0) is defined as Pe(0) Tr ("
in PPO (Schulman al., 2017) and GRPO, ensuring that policy updates are grounded in data from the current policy Tg. Large deviations of (+() from indicate excessive policy shifts, leading to overly aggressive updates and instability. Constraining the ratio within a reasonable range is therefore critical for stable and reliable training.
As shown in Figure (top left), the objective of GRPO involves the arithmetic mean of token-level rewards, which is sensitive to outliers (Figure 2). As training progresses (Figure 1, right), the range of p:(0) under GRPO expands, leading to unstable policy updates and degraded model performance. To mitigate this, GRPO applies clipping range (Elow, éhigh) ) to restrict large deviations of pc(O). However, this constraint leads to limited exploration and an early deterministic policy, which can hinder the scaling process (Yu et al., 2025).
To alleviate the instability while enhancing GRPO's exploration capability, we propose Geometric- Mean Policy Optimization (GMPO), shown in Figure 1 (bottom left). GMPO takes full advantage of the geometric mean, which inherently less sensitive to outliers and yields importance sampling ratio distributions with lower variance (Figure 2). During training (Figure 1, right), the range of GMPO 's (e( remains stable, exhibiting fewer extreme values than GRPO. With GMPO, we can maintain stable policy optimization while allowing larger clipping range promote greater exploration.
To further emphasize the advantages of GMPO, we provide detailed theoretical and experimental analyses to justify its training objective. First, we show that GMPO 's objective produces narrower value range than GRPO's, indicating reduced training variance and more stable policy updates. Second, from a gradient perspective, GMPO provides more balanced update signal and exhibits greater robustness to outlier values of the importance sampling ratio (#). Third, as training progresses, GMPO maintains smaller KL divergence from the pre-trained model and higher token entropy than GRPO, indicating enhanced stability (via smaller KL) and greater policy exploration (via higher entropy).
Extensive experiments on language, multimodal, and agentic reasoning tasks demonstrate the advan- tages of GMPO over GRPO. Specifically, on five mathematical reasoning benchmarks of varying difficulty (AIME24 (Li et al., 2024), AMC (Li et al., 2024), MATH500 (Hendrycks et al., 2021), Minerva (Lewkowyez al., 2022), and OlympiadBench (He al., 2024)), GMPO improves the average @ass@1 I accuracy by 4.1% 11 (63.4% vs. 59.3%) using the Qwen2.5-7B model, compared to GRPO. In addition, GMPO improves the Pass@1 accuracy by 2.1 1 (96.7% vs. 94.6%) MATH500 using the Qwen-32B (Yang er al., 2025) Mixture-of-Expert model. On the Geometry3K multimodal reasoning benchmark (Lu et al., 2021). GMPO increases the average Pass@ I accuracy by 1.4% (54.7% vs. 53.3%) with the Qwen2.5-VL-78 model. On the ALFWorld agentic reasoning benchmark (Shridhar al., 2020), GMPO increases the overall accuracy by 13. 1% (85.9% vs. 72.8% with the Qwen2.5-1.5B model.
The contributions of this study are summarized as follows:
We propose Geometric-Mean Policy Optimization (GMPO), which stabilizes the GRPO algorithm by maximizing the geometric mean of token-level rewards.
We conduct thorough theoretical and empirical analyses, showing that GMPO improves stability while enhancing exploration relative to GRPO.
2

---

Published as a conference paper at ICLR 2026
GMPO-7B consistently outperforms GRPO-7B across diverse reasoning scenarios, de- livering notable improvements in accuracy: 4.1% higher on five mathematical reasoning benchmarks, 1.4% higher the Geometry3K multimodal reasoning benchmark, and remarkable 13. % improvement on the ALFWorld agentic reasoning benchmark.
BACKGROUND
2.1 RELATED WORKS
Reinforcement learning (RL) has become key approach for post-training large language models (LLMs), with verifiable rewards enabling significant reasoning improvements, as demonstrated by DeepSeck-RI (Guo al., 2025a). Building on Proximal Policy Optimization (PPO) (Schulman et al., 2017), numerous variants have been developed to enhance efficiency and performance.
GRPO (Shao et al., 2024; Guo et al., 2025a) eliminates the need for computationally expensive value models while maintaining strong results across mathematics, coding, and QA benchmarks. GPG (Chu et al., 2025) further simplifies optimization by eliminating surrogate losses, critics, and KL constraints. Several extensions address rollout selection or bias correction: SRPO (Zhang et al., 2025c) uses history resampling, DAPO (Yu et al., 2025) employs dynamic sampling, GRPR (Liu et al., 2025) mitigates length bias, and OPO (Hao et al., 2025) provides an optimal baseline to reduce gradient variance. Reward shaping and advantage estimation are also actively explored. EMPO (Zhang al., 2025b) incorporates semantic entropy, AAPO (Xiong et al., 2025a) introduces advantage momentum, and BNPO (Xiao et al., 2025) adaptively normalizes rewards via a beta distribution. Seed-GRPO (Chen et al., 2025) scales policy updates by question uncertainty, while GRPO-lead (Zhang & Zuo, 2025) addresses reward sparsity through length-dependent accuracy, explicit penalties, and difficulty-awarr reweighting. Efficiency-driven methods include CPPO (Lin et al., 2025) (pruning low-advantage completions), S-GRPO (Dai et al., 2025b) (early exit to cut redundancy), Ada-GRPO (Wu et al., 2025) (adaptive reasoning formats), and GVPO (Zhang et al., 2025a) (analytical KL-constrained weighting). GRPO-À (Dai et al., 2025a) dynamically switches between
and length- agnostic rewards to avoid collapse. Further methods improve rollout usage. PODS (Xu et al., 2025) trains only on informative subsets of parallel rollouts, while RePO (Li et al., 2025) retrieves diverse off-policy samples via replay. RAFT (Xiong al., 2025b) trains solely on positive samples yet rivals GRPO. INTUITOR (Zhao et al., 2025) eliminates external rewards by using model self-certainty, and PRIME (Cui et al., 2025a) provides a scalable RL framework for reasoning. Exploration- focused techniques include the 80/20 rule (Wang et al., 2025), which emphasizes high-entropy minority tokens, and entropy-based advantage augmentation (Cheng al., 2025). Complementary to algorithmic advances, data-centrie approaches have also proven crucial. Fzen-Reasoner--Zero (Hu et al., 2025) curates 129k diverse, high-quality samples with curriculum learning. Eurus (Yuan al., 2024) contributes large-scale alignment dataset and novel reward modeling.
Despite rapid progress. the stability of RL for LLMs remains rarely explored, even though it is essential for developing reliable and scalable post-training systems. While several GRPO variants enhance stability through better baseline estimation (OPO (Hao et al., 2025)), reward shaping (GRPO- lead (Zhang & Zuo, 2025)) or reward normalization (BNPO (Xiao et al., 2025)), the underlying stability of the RL process remains persistent challenge. Our work offers a complementary perspective these methods by introducing robust aggregation operator for token-level rewards, providing an orthogonal approach to achieving more reliable and scalable post-training systems.
2.2 PRELIMINARY
The Group Relative Policy Optimization algorithm was initially proposed in DeepSeek-math (Shao et al., 2024). The core idea is to estimate the baseline using relative rewards within group of rollouts, which reduces the computational cost of the critic model and improves training stability. Specifically, for each question q from the training set Q. GRPO samples a group of rollouts {or 02, . from the old policy mesa and calculates the corresponding rewards (ri, "*, )ra). Then the policy
3

---

Published as a conference paper at ICLR 2026
model Na is optimized by maximizing the following objective:
(To) = (0:)), (-14)
G
lod
min (l.:(0),,, Clow; Chigh) A.1] (9DIL meat) (1)
0
where (i,)(0) Aeaia (0..ee)))..))) and A; >--mean (aa (0,) represents the importance std( (2:1))
sampling ratio of the t-th token in the i-th rollout based on the current policy and old policy meaa Å; is the advantage of the i-th rollout and is calculated by normalizing the rewards within the same group according to GRPO. (Clow, Chigh) ) denote the clipping thresholds and DKL (Tg |l Tyet) is the KL regularization term. Following Dr. GRPO (Liu et al., 2025), ignore DKL (Ie Tret) for simplicity and memory saving. The objective of GRPO equivalent to the arithmetic mean token-level rewards (we ignore the clipping range term for clarity), which can be formatted as:
(11))
G
G
O
(0)
(2)
In practice, the rollouts are sampled from the old policy Thou To approximate policy updates as if they were based on rollouts sampled from the current policy no, the normalized advantage Å, of each rollout is weighted by the importance sampling ratio (o))
GEOMETRIC-MEAN POLICY OPTIMIZATION
As shown in Figure (rright), we observe tokens with extreme importance sampling ratios during GRPO training, indicating unreliable policy updates. This instability arises because GRPO's 's objective sensitive outlier values of importance-weigee and further amplify the variance of importance sampling ratios.
rewards, which drive aggressive policy updates
To solve that, we propose coometric-Mean Policy Optimization (GMPO), stabilized variant of GRPO. Instead of optimizing the arithmetic mean of token-level rewards as shown in Equation 2, GMPO maximizes their geometric mean:
Tompo(Fo)
Mo (1l)
G Σ II II...()))A)) क..
(ggn(A))
(3)
where sgn(A,) ensures the correct optimization direction, returning when A, 0 and otherwise. TeMPo (TO) has a narrower value range than JGRPO(TO) (To), which can be derived as follows:
lod
[TmMPO(Ta)l (000 (:k)
1
Σ
(o.( 1221
T
1=1
lod
< E ( (.10)
Σ
Pi.e (0)
= (JārPo(Te)
(4)
G
10
This narrower range suggests that the training process of GMPO experiences lower variance in the optimization objective, which can be viewed as evidence of more stable policy updates. Compared to JCRPo (ma), JGMPO (TTA) is less sensitive to outliers because the geometric mean is inherently more robust to outliers than the arithmetic mean. As result, JGMPO(RA) (Te) provides more reliable policy updates and maintains a more stable range of importance sampling ratios as shown in Figure (rright). By expanding Equation and incorporating the clipping range term from PPO (Schulman et al., 2017) at the token-level, we can derive the complete objective function of GMPO as follows:
JAMPO(TT) = E E. mo (-19)
1
min (0) ( flow, high Ad]
sgg(( (A))-
(5)
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 GMPO Training objective
gmpa_loss pneb_p probs, mask, advantage, epsilonn0.4):
new_probe [I, 11: Token probabiiities
sodel
old_probs [I, 11: Token probabilities from the old model
mask (I, Indicates valid padded) cokens
advantage [1]: Advancage or normalized reward for the sequence
epsilon [1jz Controis the ciipping range
Clipping token-level Clipping wider
nev_l og_proba, pld_log_probs = torch. (new_poro)))) torch. torch.logtoles logtold agn 1.0 if advantage Migned_logrrate = agn [new_ see__log_probs old_ og_probs)
else -1.0
clipped Lorch. tta.posssgeeee__oooo_ooeo. min_ min_log_rati og_ratio
-epsilof, epeilon)
el ppeed
min_los_ratio syn nin_iog__attoo t reometrie-Mean Policy optinization
importance_samplingg.e. sampling_ratio Lozch. exp log___no_______iiao ratio [maak] (amkk)ssumdd /maak. loas -advantage importance_aane aaaeee_ammmmm______ee_e ig__rati
return loss
GMPO is straightforward to implement, and its pseudo-code is given in Algorithm 1. For numerical stability, both the product and clipping operations in Equation are carried out in log space.
To better understand why GMPO is more stable than GRPO, we show that GMPO is more robust to tokens with extreme importance sampling ratios from a gradient perspective. Specifically, given question q and rollout og, the gradients of JCRPo (Te) (Equation 2) and JéMPo (Te) (Equation 3)
with respect to the model parameter 0 are as follows!:
(t)
A (log(T)(oL.144...k)))
(6)
Gol
G-
9.0r
lagl
V VoJomPo (TO)
(O. [oा
AA Valog(To(oxalgeoccc)))
(7)
9.0
G
0:
The term A, V Velog(Te(OLLlg) or,<1)) quantifies the influence of the generated token oi, the parameters 0, which corresponds to the standard policy gradient (Sutton et al., 1999). The gradients of both objectives weights. For TePPo (Te), the weight of the token o₁, includes its individual importance sampling ratio (f). An extreme (f,) will cause the token gradient to be too large small, resulting in aggressive policy updates. For TGMPO (TT), the weight of the token or, includes the geometric mean
weighted sums of the policy gradients of the generated tokens, but with different
of all the ratios (II1--) (0) | in the same sequence, which provides a more balanced update signal and more robust to outlier values.
Beyond the proposed training objective, we demonstrate the effectiveness of the following key designs in GMPO:
(i) Clipping at token-level. Unlike vanilla GRPO in DeepSeek- math (Shao et al., 2024), DeepSeek- [o.|
[==1 Pi, (0)) and clips outliers at the sequence-level, i.e., clip [| (0), flow, Enigh). . The term | (f) also appears in the objective of GMPO (Equation and 5). However, instead of applying clipping at the sequence-level as in DeepSeek-RR -11, we find it more effective to perform clipping at the teken-level, as shown in Equation The rationale as follows: (1) Clipping the token-level is more stable than at the sequence-lever level. As shown in Figure 3, the sequence-level clip -aMpO-se--clip-ee-00 e.o4)) has larger importance sampling range than the level clip (GMPO (e -0. cd.4)), which makes more prone to creating extreme gradients during optimization. (2) Sequence-level clipping is too aggressive compared to loken-level clipping. Once triggered, it sets the gradients of all tokens in the sequence to zero, potentially discarding valuable update signals from informative parts of rollouts.
R1 (Guo et al., 2025a) maximizes the sequence-level reward (II
'The clipping range term is omitted for clarity. Detailed derivations are provided in Appendix A.
5

---

Published as a conference paper at ICLR 2026
The maximum log(p((0))
The minimum (0)(
GMPOle eet)
Unstable
Stable
GMP00le g*t)
L4-
GMPO)- +*)
-33
GRYGIO.8. 1.23
-22
12
-25
GMPOke ei))
GMPOle e4,
GMPO
GMPOC km]
GMPD cip-te-"n. GA
Unstable
Stable
251
26
24
Training steps
Training steps
Figure 3: The range of importance sampling ratio (+( with respect to different clipping ranges and training steps. A wider range indicates less stable policy updates. Compared GRPO with clipping range of (0.8, 1.2), GMPO demonstrates greater stability, even with a larger clipping range of (E 0.4 e..d). All curves are smoothed for clarity.
(ii) Clipping wider. As illustrated DAPO (Yu et al., 2025). the clipping operation can limit exploration and cause early deterministic policy, which hinders the scaling process. To encourage exploration without sacrificing stability, DAPO uses a clip-higher strategy. which slightly expands the clipping range (Elow, Ehigh) from (0.8, 1.2) (0.8, 1.28). As shown in Figure 1, we visualize the maximum and minimum importance sampling ratios at each training step for both GRPO and GMPO. The key observations are: (1) As training proceeds, the importance sampling ratios span wider range, indicating more aggressive policy updates and increased instability. (2) Compared to GRPO, GMPO preserves a narrower range of importance sampling ratios, suggesting more stable updates. (3) For GMPO, expanding the clipping range from (e-0.2 e0.2) to (-o0, +o0) increases instability policy updates. Based on these findings, we balance training stability with exploration by setting clipping thresholds (Elow, Chigh) in Equation 5 to (e 0.4 larger than both GRPO and DAPO, encouraging greater exploration and improving performance.
(0-0.4 eo.4). This range is significantly
EXPERIMENT
4.1 IMPLEMENTATION DETAIL
Model. We evaluate the algorithm's 's performance both language-only, agentic, and multimodal reasoning tasks. For language-only tasks, following Dr.GRPO (Liu er al., 2025), we use Qwen2.5- Math-1.5B (Yang et al., 2024), Swen2.5-Mathh-
DeepSeek-RI R1 --istill-Qwenn 7B (Guo et al., 2025b) and Qwen3-32B (Yang et al., 2025) as our base models to assess performance on mathematical tasks. For agentic tasks, GiGPO (Feng et al., 2025). For multimodal tasks, we use Qwwe2.55-VL-Insruucct as the base model to train GRPO and GMPO, and evaluate their performance on geometry reasoning tasks.
Qwen2.5 1.ve225-Instrruct11555 (Yang et al., 2024) the base model following
(Bai et al., 2025)
Training. For language-only tasks, following the setup of Dr.GRPO (Liu al., 2025), we use MATH (Hendrycks et al., 2021) Levels 3-5 as the training dataset for models under 7B, which contains 8,523 mathematical problems. For each question, we generate rollouts and cap the model's maximum response length at 3,000 tokens. During each RL training round, the old policy Taai produces 1,024 rollouts, and the current policy Ty is updated 8 times with batch size of 128. For Mfxture-of-FFane -o-eets models, we use DeepScaleR (Luo et al., 2025) and CountDown (Pan, 2024) as the training dataset, with further details provided in Appendix E. For multimodal tasks, we follow the setup of EasyR (Zheng et al., 2025) and use Geometry3K 3K (Lu et al., 2021) as the training dataset. For agentic tasks, we follow the setup of GiGPO (Feng et al., 2025) for training and inference, with further details provided in Appendix D. For mathematical problems, rewards are verifiable: "1" for correct responses and "0" for incorrect ones. Our method is mainly compared with Dr.RPO and GRPO, under the same experimental setup in Tables 1, and 2.

---

Published as a conference paper at ICLR 2026
Table Comparison of GRPO and GMPO for language models (a), multimodal models (b), Mixture- of of-Experts models (c), and agentic models (d).
Language Model
AIME24
AMC
MATH500
Minerva
Oly.
Avg.
GRPO-1.58 Shao et al. (2024)
23.3
49.4
75.2
25.7
39.0
42.5
GMPO-1.5B (Ours)
20.0
53.0
77.6
30.1
38.7
43.9
GRPO-7B Shao et al. (2024)
40.0
59.0
83.4
32.4
41.3
51.2
GMPO-7B (Ours)
43.3
61.4
82.0
33.5
43.6
52.7
GRPO-7B Shao et al. (2024) (R1-Distill)
43.3
67.5
89.0
39.7
56.7
59.3
GMPO-7B (RI-Distill, Ours)
46.6
78.3
91.4
37.9
62.5
63.4
(a) Five mathematical reasoning benchmark.
Multimodal Model
Geometry3K
MoE Model
MATH500
GRPO-7B (Shao al., 2024)
53.3
GRPO-32B (Shao et al., 2024)
94.6
GMPO-7B (Ours) (b) Geometry3K benchmark.
54.7
GMPO-32B (Ours)
96.7
(c) MATH500 benchmark.
Agentic Model
Pick
Look
Clean
Heat
Cool
Pick2
ALL
GRPO-1.5B Shao al. (2024)
85.3
53.7
84.5
78.2
59.7
53.5
72.8
GMPO-1.5B (Ours)
93.1
78.6
81.0
88.2
82.1
89.5
85.9
(d) ALFWorld benchmark.
Evaluation. We evaluate our method on five mathematical reasoning benchmarks of varying difficulty following Dr.GRPO (Liu et al., 2025), one multimodal reasoning benchmark following EasyR1 (Zheng et al., 2025), and one agentic reasoning benchmark. The mathematical benchmarks include: AIME24, which consists of 30 schh-school-level olympiad problems from the American Invitational Mathematics Examination 2024; AMC, which contains 83 intermediate-difficul multiple-choice problems; MATH500, subset of 500 problems from the original MATH dataset covering algebra, geometry, and number theory; Minerva (Lewkowyez et al., 2022), which features 272 graduate-level problems requiring multi-step reasoning; and OlympiadBench (Oly.) (He al., 2024), a collection of 675 high-difficulty olympiad problems. These benchmarks collectively cover broad spectrum of problem types and difficulty levels. For multimodal reasoning, we use Geometry3K (Lu al., 2021), visual question answering dataset consisting of 601 geometry-focuset problem-solving questions. For agentic reasoning, we evaluate on ALFWorld (Shridhar et al., 2020), an embodied environment designed to assess the ability of LLM agents to perform multi-step decision-makinn We primarily use the Pass metric for comparative analysis, which evaluates whether single generated response to problem meets the required criteria. For language tasks, we set the temperature to 0.0 and generate one answer per question, following Dr.G (Liu et al., 2025). For the multimodal task, set the temperature 0.5 and generate 16 answers per question.
4.2 PERFORMANCE
Table 2 present comprehensive evaluation of our GMPO approach against established reason- ing methods across multiple benchmarks. Our method demonstrates consistent and substantial improvements over strong baseline systems.
Language-only tasks. GMPO demonstrates consistent improvements across different base models. With Qwen2.5-Math-1 1.5B, it achieves 43.9% average performance, outperforming GRPO by 1.4% and Dr.GRPO by 1.8%. Similar gains observed with Qwen2.5-Mathe +1.3% vs. Dr.GRPO) and DeepScck-RI- ---SScki----istllll- Qwen-BB (+4.1% vs. GRPO, +1,9% vs. DrGRPO). In the stability-sensitivv Mixture-of- Ex-eertss (MoE) setting with Qwen3-3288 GMPO achieves 96.7% accuracy on MATH500, outperforming GRPO by 2.1% Additional results for MoE models are provided in Appendix E.
(+1.5% GRPO,
Multimodal tasks. Using Qwen2.5 Drven25-VV--Instrre -L-Instruct- 7B as the base model, GMPO surpasses GRPO by 1.4% Geometry3K, highlighting its potential for broader application in multimodal tasks.
7

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of GMPO and state-of-the-arr methods on mathematical reasoning benchmarks.
Model
AIME24
AMC
MATH500
Minerva
Oly.
Avg.
Qwen2.5-Math-1 1.58 (Qwen et al., 2025)
16.7
43.4
61.8
15.1
28.4
33.1
Owen2.5-Math-I. 1attt-1-.S--Instrrre (Qwen al., 2025)
10.0
48.2
74.2
26.5
40.2
39.8
Oat-Zero-1.5B (Liu et al., 2025)
53.0
74.2
25.7
37.6
42.1
GMPO-1.5B (Ours)
20.0
53.0
77.6
30.1
38.7
43.9
Qwen2.5-Math- (Qwen et al., 2025)
16.7
38.6
50.6
9.9
16.6
26.5
SimploRL-Zeer-7BB (Zeng et al., 2025)
26.7
60.2
78.2
27.6
40.3
46,6
PRIME-ZZen-7BB (Cui al., 2025a)
16.7
62.7
83.8
36,0
40.9
48.0
Openneeasonerr-eeno-71 @ 3k (Hu et al., 2025)
13.3
47.0
79.2
31.6
44.0
43.0
OpenReeasoner-Zeno-T1 @ (Hu et al., 2025)
13.3
54.2
82.4
31.6
47.9
45.9
Eurus- 7B (Yuan et al., 2024)
16.7
62.7
83.8
36.0
40.9
48,0
GPG-7B (Chu et al., 2025)
33.3
65.0
80.0
34,2
42.4
51.0
Oat-Zero-7B (Liu al., 2025)
43.3
62.7
80,0
30.1
41.0
51.4
GMPO-7B (Ours)
43.3
61.4
82.0
33.5
43.6
52.7
Oat-Zcro-7B (Liu al., 2025) (R1-Distill)
50.0
74.7
89.6
37.5
55.7
61.5
GMPO-7B (RI-Distill, Ours)
78.3
91.4
37.9
62.5
63.4
Table 3: Comparison of objectives and their performance under same training settings.
1: 1
E min (O) clip((((()) flow (high))A)
2: | (0)A
[o|
sgn(A)
3:
min
[[]
[ (e)) clip (e()), clow, thigh) { sAgn(A)
[o]
4:
[
min
clip(p: Clow, hhggh )
sgn
(A)
5:
II
|o| [=
min (0), cpip(p, (0), Flow, Ehigh ) ssg((A)
Training objectives
AIME24
AMC
MATH500
Minerva
Oly.
Avg.
(Pre-RL model)
16.7
38.6
50.6
9.9
16.6
26.5
I (GRPO)
40.0
59.0
83.4
32.4
41.3
51.2
(without clip)
40.0
63.9
80.6
33.5
43.7
52.3
3 (with seqpc
46.6
57.8
80.2
34.2
44.3
52.6
4 (without norm)
36.6
67.4
82.0
29.8
44.1
52.0
5 (GMPO)
43.3
61.4
82.0
33.5
43.6
52.7
Agentic tasks. Using Qwen2.5- -nnnnn.tttstttut--... 1.5B as the base model, GMPO achieves a 13.1% perfor- mance gain over GRPO on ALFWorld, demonstrating its potential in open-world agentic tasks.
4.3 ABLATION STUDIES
Table 3 presents an ablation study the key modifications in GMPO relative to GRPO. The effect of the clipping thresholds is presented in Table 4, and training statistics are shown in Figure 4.
Geometric mean vs. Arithmetic mean. The performance of GRPO and GMPO is reported in lines 1 and 5 of Table 3, respectively. GRPO achieves an average performance of 51.2% by optimizing the arithmetic mean token-level rewards. In contrast, GMPO improves this 52.7%. outperforming GRPO by 1.5%, by optimizing the geometric mean instead. In row 4 of Table 3, we test removing the normalization term 1 from the training objective, similar Dr. GRPO (Liu et al., 2025). This Tol
results in 0.7% drop in average performance (52.0% vs. 52.7%), suggesting that the normalization term is crucial for maintaining optimal performance.
Clipping strategy. The performance of GMPO without clipping, with token-level clipping, and with sequence-level clipping is shown in lines 2, 3, and 5, respectively. The corresponding ranges of
8

---

Published as a conference paper at ICLR 2026
Table Influence of the clipping thresholds on model performance.
Clipping thresholds (clow Chigh)
AIME24
AMC
MATH500
Minerva
Oly.
Avg.
1
(e 2.22)
36.6
60.2
84.2
35.7
45.0
52.4
2
(c 0.4 c00a)
43.3
61.4
82.0
33.5
43.6
52.7
3
(c 0,88 c...)
40.0
60.2
82.2
33.5
44.7
52.1
4
(-00, +00)
40.0
63.9
80.6
33.5
43.7
52.3
(8) MATH Level 3-Level
(6) Deep
GRPO -04 eb-1)
Aw
2.21
Entropy
GMPO
GRPO
Entropy
1:0
GMPO
Entropy collapse due to aggressive policy updates
8.5
Training steps
Training steps
(c) MATH Level 3-Level
(d) SeepSealeR-Mot
KL divergence
GRPO
Grad norm
GRPO GMPO
Mymma Whlh
GMPO
Unstable pollcy updates
Training
Training steps
MATH Level 5-Level
DeepScaleR-MoE
1.K
Val score
Val scote
GRPO GMPO
Entropy collapse causes performance plateau
GRPO
GMPO
Triining steps
Training steps
M
(h) CountDwnn MoE
630
GRPO
GMPO
Val score
05
Unstable updates performance collapse
01
GRPO
GMPO
L08
20
300
430
500
Training steps
Training steps
Figure 4: Analysis of entropy, KL divergence, gradient norm, validation score over training steps. (a-b) GMPO maintains higher entropy than GRPO, whether trained on MATH Level 3-Level 5 or DeepScaleR dataset. (c-d) GMPO maintains more stable gradient and a smaller KL divergence from the pre-RL model than GRPO. (c- h) GMPO outperforms GRPO in validation scores across language-only and multimodal tasks, for both dense and Mixture-of -f-eres models.
importance sampling ratios are shown in Figure 3, labeled as GMPO (e-0.4, e. GMPO-seq-clip- (e (..4.4, ,,044, and GRPO(0.8. 1.2). Clipping the sequence-level achieves similar performance to token-level clipping but results in larger range of importance sampling ratios. Therefore, we use token-leve clipping strategy. Removing the clipping range term (GMPO (-oc, +oo)) leads to excessive fluctuations in the importance sampling ratio during training, which affects stability and results in a 0.4% decrease in average performance (52.3% vs. 52.7%).
Influence of the clipping thresholds. To find the optimal clipping thresholds for GMPO, train the model with different clipping thresholds, as shown in Table and Figure A larger clipping range encourages exploration but introduces instability optimization, which ultimately affects performance. To balance stability and performance, we set (Gow: Chigh) in Equation 5 to (e 00. c,4), which has stable range of importance sampling ratios and achieves the best performance.
Exploration capability. As noted in (Cui et al., 2025b), language models in reinforcement learning often trade off entropy for short-term performance. Premature entropy collapse can cause performance

---

Published as a conference paper at ICLR 2026
to plateau. As shown in Figure 4 (a-b), visualize the mean token entropy of GMPO and GRPO when training the policy model at MATH Level 3-Level 5 and the more challenging mathematical dataset DeepScaleR. GRPO's s entropy drops rapidly during training, limiting exploration and causing performance to plateau (Figure 4 (e-g)). As shown in Figure (a), applying a wider elipping range for GRPO temporarily encourages exploration, but the entropy still declines quickly over time. This behavior arises because GRPO optimizes the arithmetic mean of token-level rewards, which is sensitive to outliers. Consequently, it can generate aggressive updates that sharply reduce entropy while offering only marginal performance gains, hindering both exploration and scalability.
In contrast, GMPO employs the geometric mean, which is robust outliers. This allows maintain stable, moderate entropy, enabling consistent exploration throughout training and resulting in higher rewards and better overall performance than GRPO, as shown in Figure 4 (e-g).
Training stability. As shown in Figure 4 (c-d), we visualize the gradient norm during training and the KL divergence between the current model my and the reference model Tref- Tref is initialized as the base model before RL training. As training progresses, GMPO maintains stable gradient and low KL divergence from reference model, indicating greater training stability and a lower risk of overfitting. In contrast, GRPO exhibits unstable gradients and large KL divergence, suggesting unstable learning and greater tendency to drift away from the reference model.
Validation scores. Figure 4 (e-h) shows validation scores under different training settings. Figures (c) and (f. g) correspond to Tables I, while results on CountDown are detailed in Appendix E. GMPO consistently outperforms GRPO in validation scores across language-only (e, h) and multimodal (g) tasks, for both dense (e, g) and Mixture-of- of-Experts models (f, h).
5 CONCLUSION
We propose GMPO, a stabilized variant of GRPO. By optimizing the geometric mean of token-level rewards and enlarging the clipping range of the importance sampling ratio, GMPO only alleviates instability in policy updates but also enhances exploration capabilities, as evidenced by a narrower objective value range, more stable gradients, and consistently lower KL divergence and higher token entropy throughout training. Extensive experiments on language-only and multimodal reasoning benchmarks demonstrate that GMPO outperforms GRPO in terms of both stability and reasoning capacity. This work sets the stage for future research on developing more reliable and scalable reinforcement learning systems.
ACKNOWLEDGMENT
This work was supported by the National Natural Science Foundation of China (NSFC) under Grant 62472402, 62225208 and 62176260, CAS Project for Young Scientists in Basic Research under Grant No. YSBR-117.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace. Julia Kreutzer, Olivier Pietquin, Ahmet Ustün, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. In ACL. pp. 12248- 12267, 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 13325 2025.
Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3):
1997.
Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization. arXiv preprint arXiv:2505112346. 2025.
Daixuan Cheng. Shaohan Huang. Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. /4758, 2025.
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546, 2025.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456. 2025a.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. 2025b.
Muzhi Dai, Shixuan Liu, and Qingyi Si. Stable reinforcement learning for efficient reasoning. arXiv preprint arXiv:2505. 18086, 2025a.
Muzhi Dai, Chenxu Yang, and Qingyi Si. S-grpo: Early exit via reinforcement learning in reasoning models. arXiv preprint 2XXx:25055006866 2025b.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-grout policy optimization for Ilm agent training. arXiv preprini 2rXiv:2505510978 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025a.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability 1lms via reinforcement learning. arXiv preprint arXiv:2501. 2025b.
Yaru Hao, Li Dong, Xun Wu, Shaohan Huang. Zewen Chi, and Furu Wei. On-policy with optimal reward baseline. arXiv preprint arXiv:2505.253: 233588 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlynpiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Lun-Wer Wei Ku, Andre Martins, and Vivek Srikumar (eds.), ACL, pp. 1828-38500 August 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Jingcheng Hu, Yinmin Zhang, Qi Han. Daxin Jiang, Xiangyu Zhang, and Heung. Yeung Shum. Open-reasonne-zzert An open source approach to scaling up reinforcement learning on the base model, arXiv preprint 20Xi2 2025.

---

Published as a conference paper at ICLR 2026
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer. Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. NeurIPS, 35:3843-38883 3857, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 2024.
Siheng Li, Zhanhui Zhou, Wai Lam, Chao Yang, and Chaochao Lu. Repo: Replay-enhanced policy optimization. XrXiv preprint arXiv:2506.09) 09340, 2025.
Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group
relative policy optimization-based reasoning models. arXiv preprint arXiv:2503.22332, 2025.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprini urXiv:2503.20783, 2025.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2005.04165, 2021.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing ol -preview with a 1.5 b model by scaling rl. Notion Blog, 2025.
Jiayi Pan. Countdown-task--3ttd dataset, 2024. Accessed: 2025-02-21.
Qwen, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https:/ /aa/iviiivvv.//.. org/ alss 11412.15111
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: N.KY:v/707.06444 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Mohit Shridhar, Xingdi Yuan, Mlee-Alex Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, arXiv preprint arXiv:2010.03766 2020.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NeurIPS, volume 12, 1999.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang. Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.0193) 2025.
Siye Wu, Jian Xie, Yikai Zhang, Aili Chen, Kai Zhang, Yu Su, and Yanghua Xiao. Arm: Adaptive reasoning model. arXiv preprint wwXiv:2505.20255 2025.
Changyi Xiao, Mengdi Zhang, and Yixin Cao. Bnpo: Beta normalization policy optimization. arXiv preprint arXiv:2506. N.x:22506.02864. 2025.
12

---

Published as a conference paper at ICLR 2026
Jian Xiong, Jingbo Zhou, Jingyong Ye, and Dejing Dou. Aapo: Enhance the reasoning capabilities of Iims with advantage momentum. arXiv preprint arXiv:2505. 1426, 2025a.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to IIm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504.1 1343, 2025b.
Yixuan Even Xu, Yash Savani, Fei Fang, and Zico Kolter. Not all rollouts are useful: Down-sampling rollouts IIm reinforcement learning. arXiv preprint arXiv:2504. 138/8, 2025.
An Yang, Beichen Zhang, Binyuan Hui, Bofci Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvemment arXiv preprint arXiv:2409. 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.08 000,00988,,, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 2025.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing Ilm reasoning generalists with preference trees. arXiv preprint arXiv:2404.00 02078, 2024.
Weihao Zeng, Yuzhen Huang. Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
Jixiao Zhang and Chunsheng Zuo. Grpo-lead: difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models. arXiv preprint arXiv:2504.0969, 2025.
Kaichen Zhang, Yuzhong Hong, Junwei Bao, Hongfei Jiang, Yang Song, Dingqian Hong, and Hui Xiong. Gvpo: Group variance policy optimization for large language model post-training. arXiv preprint arXiv:2504. 0ete:22044.99999 19599, 2025a.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised Ilm reasoning incentivization. arXiv preprint arXiv:2504.05882, 2025b.
Xiaojiang Zhang, Jinghui Wang. Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojic Wang, Yinghan Cui, Chao Wang, Junyi Peng, al. Srpo: A cross-domain implementation of large-scale reinforcement learning on 1im. arXiv preprint arXiv:2504. /4286, 2025c.
Xuandong Zhao, Zhewei Kang, Aosong Feng. Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505. 2025.
Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng. Dongdong Kuang, and Yuwen Xiong. Easyrl: An efficient, scalable, multi-modality rl training framework. /tgp::////thub/./////// hiyouga/ (Eaaaa//aaaaaaaa 2025.
13

---

Published as a conference paper at ICLR 2026
Appendices
A GRADIENT DERIVATION
To better understand why GMPO more stable than GRPO, we analyze its robustness to tokens with extreme importance sampling ratios from gradient perspective. Specifically, we first derive the gradient of the importance sampling ratio Pa. with respect to the model parameter in Lemma 1. Building on this result, then derive the gradients of GRPO and GMPO with respect to 0 in Lemmas 2 and 3. For clarity, the clipping range term is omitted in the gradient derivation.
Lemma: 1. The derivative of the importance sampling ratio is:
V (0) = plo(o))olog(mooonn o.,())
(8)
Proof.
V (0p),(() V To(0i, 01,
Toau (0i,t 08,c1)
VoTo(Os,1l;,, 01,e))
Tana (0s,elg. 01,(1)
T0(01,14 01,ct)
V 01,0))
Teni (0s,el. 0s,(1)
To 10(05,19.00,00)
(t,( V (olog(To or,er))
Lemma: 2. The derivative of the GRPO objective is:
V VoJGRPO(TA)
(0) A (4)
(9)
G- los
Proof.
lo.|
V VoJorro
V
Pr. (9)
9.0,
G
iml
0
1=1
lo.1
- (P)
G. lo,1
1=1
(0) A. Volog(mo(os.ll0...c 0i,ee)
G.
Lemma: 3. The derivative of the GMPO objective is:
[o:
lod
Σ
(L)(()
A (00( (10)
G.
9.0;
t=1
14

---

Published as a conference paper at ICLR 2026
Proof.
G
lod|
V VoJGMPO(TH))
V
Pt.x (0),(0)A.))
(A.)
9.09
1=1
Joal
pc.(())
A
10:
1
Pi,z
(0)
.A. V
())
G lo:l
Jo,1
los1
[o.1
1
Pe.x(0)
104
A
ps.((0) V Vep.x(()
G los
1o.1
Jos1
Jo.1
1
(II
(o))
1n4
A
PL. V Velog(m((ornlg.o...))
G lo:1
1=1
0vl
10:1
(r,)
A.
Volog(To(ousl))04.c)))))
G
10:
k=1
B THEORETICAL CONNECTIONS WITH GRPO
As the foundation of PPO (Schulman et al., 2017) and GRPO Shao et al. (2024), TRPO (Schulman et al.. 2015) establishes a monotonic improvement guarantee for general stochastic policies with trust region constraint. Consequently, under mild assumptions, GRPO inherits these desirable properties from TRPO.
In this section, we show that, within trust region, GMPO is O(8²) Lipschitz-stable perturbation of GRPO, where denotes the maximum token-level ratio deviation across the sampled trajectories. As result, GMPO preserves GRPO's monotonic-improvemeee and convergence guarantees up to (65)) error (Bertsekas, 1997), making it principled optimization objective.
Our argument proceeds in two steps. First, Lemma 4 quantifies the difference between GMPO and GRPO when the policy update remains in a small trust region. Then, Lemma 5 shows that this discrepancy is O68²). establishing that GMPO is Lipschitz-staiee perturbation of GRPO and therefore inherits its local theoretical guarantees.
To (os, clq.0,.s) Lemma: 4. Let (0) Re (2.0,.)) be the importance ratio and define the token-level
deviation (0) (0) 1. The trajectory-level mean deviation is (,( = |o.1 Eial (c+( and define the population variance Var;(f) |o:| Σlal (8),(()) 8, (0)) Assume that the update lies in a trust region where (0), < 8 for all i, I and some
Then, as + O, we have
G
A
(TMPO(TT) = TēRPO(TO) - E
Var:(0)
+ 0(09).
(11)
2
1=1
Proof. The training objectives of GRPO (JeRPo (To)) and GMPO (TeMro(To) are defined as (clipping range term and KL regularization term are omitted for clarity):
TēRPo(F0) = B Me (e (1))
(0)A:
-
ToMPo(FO) Mea
Pi. (0)
sggn(Ag)
~e (+1g)
G
15

---

Published as a conference paper at ICLR 2026
Then deduce the difference between the two objectives as follows:
10:1
A:l) मेा sgn(A.))
Jomro (To)
E
(0))
l0,1
1
log(1 + 61.(811)
E
A exp
loil
locl
E
A
exp
|ool
1
1o (0) - 5m
+0, +
Toylor expaision log(1 +.))
G
lod
E
A exp (d.(0)
1
(0) + +000))
G
2 2
0:
l0
E
A (1+
1
6 (0) + (o:(0)
(0))()))) +
2
0s
Taylor expansion cocp(x))
Var:(0) - 1 5.((0)² + -0(00)++0)00)))))))))))))))))))
E
A (1+)
E
(1+ (10)
(a): + 066%))
G
10.1
C
A
E
Pu.: (O)
E
Var Var,()) (0) +0
loul
2
Im1
1-1
C Var,(0)
(To)
+ +
which proves the stated relation in Equation 11.
Lemma: 5. Assume that: (1) the update lies in a trust region where (0) & 8 for all i, t and some 8 > O. (2) the gradient of importance sampling ratio is bounded || Vop,, (0)l S K8, (3) the
advantage A; is bounded [A,] S. Amax- Then GMPO is an ((82) Sosschizz-bunnder perturbation of GRPO, which means that there exist constants C1, C2 > 0 such that for all 0 in the trust region
JāmPo (0) (G)P s 52
(12)
V (0) VoTERPO S O
(13)
Proof. We start from the expansion in Lemma 4:
A
(T TGRPO(TO) E
Vor
+ 0
G
2
where
1 |eu| 5,()))
|o.|
1
Var;(0) jo,1
(0)
8,(0)
8 (
(1,(()) (r) 1.
|op
t=1
16

---

Published as a conference paper at ICLR 2026
Table 5: Comparison of reward aggregators and their performance under same training settings.
Reward Aggregator
AIME24
AMC
MATH500
Minerva
Oly.
Avg.
Interquartile Mean
36.7
60.2
79.6
29.0
43.4
49.8
Arithmetic Mean
40.0
59.0
83.4
32.4
41.3
51.2
Harmonic Mean
36.7
56.6
83.4
36.0
45.9
51.7
Geometric Mean
43.3
61.4
82.0
33.5
43.6
52.7
Since the update in trust region with (6,,(()) S
we can bound the trajectory-level deviation:
l0:1
1
[8,(0)]
(0)
(0),
8,
0i
0i
t=1
[o:1 (0) (())))))))))
lo:
0 S
Var,(0)
(0((0)))))))))))))))00))0)) <
Σ
(25)2 482
0a
los 1=1
Hence, the difference in objectives satisfies
1 A (o) 0(69) 22
(TmPo(TT) JGRPO (mo)
E
G
1=1
which proves Equation 12. To prove Equation 13, we differentiate the expansion:
G A
VoJeMrO(Eo) = VoJGRrO(TO)
Vo Var,
+ 018%)
G
2
i=1
Now,
1 |od]
(ar V
(0), 5.(0))²
[0:
|os|
2
(0) 6,(0)) (Vods,((0) vo)(())).
Since (6.,(0) -0(()),0)) s 26 and (V)), = V oo... & Kō, we can bound
0.
2
Vor,
28 2K8 8K8.
0
Therefore,
V VoJeMrO(TO) VoJeroo (TT) 4K A 0(82).
(14)
which proves Equation 13.
C ADDITIONAL ABLATION STUDIES
As shown in Table 5, compare the performance and importance sampling ratio ranges of several reward aggregators, including a classic subset of power means (c.g., arithmetic, geometric, and harmonic means) well as the interquartile mean, formally defined in Definitions 1 and 2.
Among these choices, the geometric mean achieves the strongest overall performance. The arithmetic mean used in GRPO is overly sensitive to large outliers, resulting in unstable importance sampling
17

---

Published as a conference paper at ICLR 2026
The maximum log(p:(0))
The minimum log(p:
Geometncc Mean
Unstable
Stable
Arithmetric Mean
-10
Interquarti
-15
+25
62-
Geometric Mean
Arthmmetiic Nean
Harmonic
Unstable
Stable
Interquentle
130
200
251
300
200
250
309
350
Training steps
Training steps
Figure 5: The range of importance sampling ratio (0) with respect different reward aggregators and training steps. All curves are smoothed for clarity.
ratios and degraded optimization behavior. The harmonic mean, while yielding the most stable ratios, performs worse than the geometric mean used GMPO. Finally, the interquartile mean stabilizes the arithmetic mean by filtering outliers. However, its performance falls short of the arithmetic mean, indicating that extreme values might earry meaningful learning signals; overly aggressive trimming removes useful information and, as result, degrades model performance.
Definition 1 (Power Mean). Given non-negative samples {a, k and an exponent p E RR the power mean defined as
1/p
a
(p(
x
P p 0,
71
and in the limit case
1
Mu( exp
log(x:)
71
Special cases include:
M1 Arithmetic Mean, Mo Geometric Mean,
M-1 Harmonic Mean.
Definition 2 (Interquartile Mean). Given sample {r.) with order statistics T(1) S... S (n)) the interquartile mean (IQM) is the trimmed mean that removes the lowest 25% and highest 25% of the sample:
[0.75m|
1QM
I(1)
[0.75m]
[0.25n]
+
[0.2mm]
D PERFORMANCE ON AGENTIC REINFORCEMENT LEARNING TASKS
To better demonstrate the stability advantages of GMPO over GRPO, we conduct post-training experiments on agentic reinforcement learning (RL) tasks. Following the experimental settings of GiGPO (Feng et al., 2025) and utilizing 1...2.55-..B-Istttrre (Qwen al., 2025) as the base model, we train and evaluate LIM agents on ALF (Shridhar et al., 2020). ALFForld is an embodied environment designed assess multi-step decision-making, featuring 3,827 task instances across six common houschold activity categories: Pick & Place (Pick), Examine in Light (Look), Clean & Place (Clean), Heat & Place (Heat), Cool & Place (Cool), and Pick Two Place (Pick2). All RL training methods, including sourced from the verl-agents repo (Feng et al., 2025).
method and baselines,
identical hyperparameter configurations
As shown in Table 6, GMPO achieves significant 13.1% performance gain over GRPO ALFWorld. Furthermore, GMPO demonstrates comparable performance even when compared to GiGPO, a method specifically designed for agentic RI. tasks.
18

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of GMPO and state-of-the-ar methods on mathematical reasoning benchmarks.
Agentic Model
Pick
Look
Clean
Heat
Cool
Pick2
ALL
Closed-Source Model
GPT-4o Gemini-2.5-Pro
75.3
60.8
31.2
56.7
21.6
49.8
48.0
92.8
63.3
62.1
69.0
26.6
58.7
60.3
Open-Source Model
Qwen2.5-1 1..0255-1..BB-B 1.5B-Instruct Qwen al. (2025)
5.9
5.5
3.3
9.7
4.2
0.0
4.1
PPO-1,5B Schulman et al. (2017)
64.8
40.5
57.1
60.6
46.4
47.4
54.4
RLOO-1155B Ahmadian al. (2024)
88.3
52.8
71.0
62.8
66.4
56.9
69.7
GRPO-1.515 Shao et al. (2024)
85.3
53.7
84.5
78.2
59.7
53.5
72.8
GiGPO-1.5B Feng et al. (2025)
94.4
67.5
94.8
94.4
79.8
76.4
86.7
GMPO-1.5B (Ours)
93.1
78.6
81.0
88.2
82.1
89.5
85.9
Table 7: Training settings for GMPO and GRPO on Mixture-of-Expere models. Qwen2.5-200M is small-scale language model adapted from the Qwen2.5 series (Qwen al., 2025). "Bs / Mini Bs" denote the batch size and mini-batch size used during training, respectively. "E/AAct. E." indicate the total number of experts in the model and the number of experts activated per token, respectively.
Training dataset
Eval dataset
Base model
Bs./Mini Bs.
E./Act. E.
DeepScaleR
MATH500
Qwen3-32B Yang al. (2025)
128/64 256/128
128/8
CountDown
CountDown(Vall Qwen2.5-200M Bai (2025)
E PERFORMANCE ON MIXTURE-OF-EXPRRE MODELS
To better demonstrate the stability advantage of GMPO over GRPO, we conduct post-training experiments on Nixtrreoff-xxpee (MoE) models, where stability particularly critical. The experiments are performed on the DeepScaleR (Luo et al., 2025) and CountDown (Pan, 2024) datasets, with detailed training settings provided in Table 7. Specifically, DeepScaleR consists of approximately 40,000 unique mathematics problem-answer pairs compiled from AIME (Li et al., 2024), AMC (Li et al., 2024), Omni-MATH dataset, and Still dataset. CountDown consists of arithmetic puzzles where models combine given numbers using basic operations to reach a target, commonly used to test algorithmic reasoning and step-by-step problem solving. We reserve a subset of the CountDown dataset for model evaluation.
CountDown. As shown in Figure 6(a)(c), we visualize the KI. divergence and gradient norm during GMPO and GRPO training. GMPO consistently maintains lower KL divergence from the reference model and a steadier gradient norm than GRPO. Consequently, GMPO achieves stable validation scores, whereas GRPO collapses after about 250 steps, as shown in Figure 6 (e).
DeepScaleR. As shown in Figure 6 (b),d), GMPO achieves higher entropy while steadier gradient norm than GRPO. Consequently, GMPO achieves higher validation scores as shown in Figure 6 (f).
F ANALYSIS OF THE NORMALIZATION FACTOR IN THE GEOMETRIC-MEAN
Unlike vanilla GRPO in DeepSeek-math (Shao et al., 2024), DeepSeek-RI (Guo et al., 2025a)
maximizes the sequence-level reward [o: Pt. (,)) The term [|!2 Pi. (0) also appears in the objective of GMPO (Equation 5). Unlike DeepSeck-RA. R1, GMPO introduces an additional power-based normalization term: o, which we find is critical for GMPO objective.
shown in Figure we visualize the range of sequence-leve importance sampling ratios from trajectories that yield positive rewards during GRPO training. Without the normalization term, these sequence-level importance sampling ratios can become very large, especially as the response length increases. This phenomenon ultimately leads unstable policy optimization, which in turn degrades the model's final performance.
19

---

Published as a conference paper at ICLR 2026
(a) CountDown-MoE
(b) DeepScaleR MoE.
KL. divergence
01300
GMPO
/yy Aw
0:300
GRPO
Entropy
GMPO
0.00
0230
200
20
300
500
206
200
259
Training steps
Training stops
CountDown-MoE
DeepScaleR MoE
Grad norm
GMPO
Grad norm
GMPO GRPO
Training steps
Training sleps
(c) CounDown-MooE
(f) DeepSealeR-MM
Val score
Unstable updates cause performance collapse
GMPO
Val
GMPO
GRPO
GRPO
Truining steps
Training sleps
Figure 6: Analysis of entropy, KL divergence, gradient norm, and validation score over training steps Mixture-of- -xperts models. (a) GMPO maintains smaller KL divergence than GRPO. (b) GMPO maintains higher entropy than GRPO. (c-d) GMPO maintains more stable gradient norm than GRPO, suggesting more stable policy optimization. (e-f) GMPO achieves higher validation score than GRPO.
|o] II (e)) t=1
log
Large ratios caused by long responses
importance sampling ratio
Sequence-leva. level
20
500
1000
1500
2000
Response length |o|
Figure 7: Sequence-level importance sampling ratios from trajectories that yield positive rewards during GRPO training. Without normalization, these ratios can become highly unstable, especially as the response length increases.
20

---

Published as a conference paper at ICLR 2026
G USE OF LARGE LANGUAGE MODELS
During the writing process, we consulted large language models (LLMs) for word choice suggestions enhance readability. The final manuscript was carefully reviewed by humans prevent any potential inaccuracies or misleading information generated by the LLMs.
21


Reviews:
[{"review_id": "tTh5sdc6lj", "reviewer": "Reviewer_f72r", "summary": "This paper addresses the critical challenge of training instability in GRPO. The authors identify the root cause of this instability as the GRPO objective's reliance on the arithmetic mean of token-level rewards, which is highly sensitive to outlier importance sampling ratios. To remedy this, the paper proposes Geometric-Mean Policy Optimization (GMPO), which replaces the arithmetic mean with the geometric mean. The central hypothesis is that the geometric mean, being inherently more robust to extreme values, will effectively dampen the influence of outlier tokens.", "questions": "- Why does the geometric mean objective enable a wider clipping range to be effective without causing instability? Is there a principled way to determine the optimal clipping range in conjunction with GMPO, or is it simply another sensitive hyperparameter that requires careful tuning? \n\n- Have the authors analyzed the computational and memory overhead of this operation compared to the simple summation in GRPO, particularly for very long reasoning chains (e.g., >2000 tokens)? Is there a point at which this overhead becomes a significant practical concern? A complete analysis would include profiling the computational time and memory usage of the GMPO loss function versus the GRPO loss function for varying sequence lengths to quantify any potential overhead.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The problem is clearly defined and motivated. The authors narrow their focus to a specific, plausible mechanism: the sensitivity of the arithmetic mean aggregator in the GRPO objective to these outlier IS ratios. \n\n- The method is practical and easy to implement. The loss is a small rewrite of GRPO. The authors enhance this practicality by providing clear pseudo-code in Algorithm 1, which details the implementation in log-space for numerical stability.1 This transparency is crucial for reproducibility and allows the community to build upon the work easily.\n\n- The paper substantiates its claims with extensive empirical evidence across a variety of benchmarks and model scales.", "weaknesses": "- The core novelty of the paper lies in the application of the geometric mean to the GRPO objective. While this application is new in this specific context, the underlying idea of using a robust statistical estimator to handle outliers is a foundational concept in statistics and data analysis. This weakness is compounded by a failure to justify the choice of the geometric mean over other standard robust estimators that designed to be less sensitive to outliers and could plausibly offer similar or even superior stabilization benefits.\n\n- The theoretical analysis presented in Section 3 and Appendix A does not provide formal convergence guarantees, nor does it connect the method to more rigorous optimization frameworks that are commonly used to analyze the stability of policy gradient methods.\n\n- A key design choice highlighted in the paper is the use of clipping range for the importance sampling ratio, which is substantially larger than the typical range used in GRPO. The authors justify this choice empirically, showing in their ablation study (Table 5) that this specific range yields the best performance among the options tested. However, the paper fails to provide a deeper explanation for why this is the case. \n\n- The evaluation can be further enhanced. There are several alternatives to improve GRPO stability, such as GSPO, DAPO, or Dr.GRPO. Given how close GMPO is to sequence-ratio weighting and clip-widening, this is critical. Also, the evaluation is mostly math. It will be more convincing if the authors can demonstrate code tasks, tool-use, or larger open-ended reasoning. The improvement seems marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ua1XmvaEjA", "reviewer": "Reviewer_vzMt", "summary": "The paper proposed GMPO, which replaces the arithmetic mean in GRPO with the geometric mean to achieve better  stability of training.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method is easy to follow, and the paper writing is clear.", "weaknesses": "* Judging from Table 4, it seems that the proposed method's gain is quite marginal. The token-wise clip also seems not so effective.\n* The major technical change from GRPO seems to be token-level clipping + geometric mean vs group arithmetic mean. The technical novelty is low.\n* Although discussion is present, comparisons is missing against the method that controls training stability by carefully setting clipping ratio (e.g. DAPO)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OdZIIy2bpX", "reviewer": "Reviewer_VUr8", "summary": "The paper proposes Geometric-Mean Policy Optimization (GMPO), a modification of Group Relative Policy Optimization (GRPO) designed to improve training stability for large language models. The core issue identified with GRPO is its use of the arithmetic mean of token-level rewards, which makes it sensitive to outliers in importance sampling (IS) ratios and leads to unstable policy updates. GMPO addresses this by replacing the arithmetic mean with the geometric mean, which is inherently more robust to outliers. The authors provide theoretical analysis showing that the GMPO objective has a narrower value range and results in more balanced gradient updates. Empirically, through experiments on several mathematical and multimodal reasoning benchmarks, the paper demonstrates that GMPO consistently outperforms GRPO, achieving significant performance gains (e.g., an average Pass@1 improvement of up to 4.1%). The paper includes extensive ablation studies that validate the key design choices of GMPO, such as token-level clipping and the normalization factor in the geometric mean.", "questions": "1.  In line 266, it says \"clipping at the token-level, as shown in Eq 3\". I didn't see the clipping operation in formula 3; should this refer to Eq 4?\n2. What is the intended value of $sgn(\\hat A_i)$ when $\\hat A_i=0$? The current code treats it as negative; the math does not specify it. This edge case can occur because the group based advantage in GRPO is standardized. Please clarify. \n3. Can you report GRPO with the exact same per token clipping in log space and the same wide range $(e^{-0.4},e^{0.4})$ so that the only difference is arithmetic vs geometric aggregation? That would isolate the effect of the geometric mean from the effect of wider exploration.\n4. In Table 3 you compare against several external methods such as PRIME Zero and OpenReasoner Zero. Were those numbers reproduced under your pipeline or copied from their papers which might have used different data or evaluation protocols?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The core idea of replacing the arithmetic mean with the geometric mean is simple, elegant, and well-motivated. It directly targets a plausible source of instability in GRPO (sensitivity to outlier rewards) with a classic statistical tool known for its robustness. The \"plug-and-play\" nature of the modification makes it highly practical.\n2.  The empirical evaluation is comprehensive and convincing. The authors demonstrate consistent improvements over GRPO across multiple model sizes (1.5B, 7B, 32B MoE), different model architectures (dense and MoE), and different domains (language-only and multimodal reasoning). The performance gains are substantial and statistically significant.\n3.  The ablation studies presented in Section 4.3 are excellent. They systematically dissect the contributions of each component of the proposed method: the geometric mean itself, the token-level clipping strategy, the clipping range, and the normalization term. This analysis provides strong evidence for the authors' design choices and adds depth to the paper's claims.\n4.  The paper is generally well-written and clearly structured. The motivation for the work is established effectively in the introduction, and the figures, particularly Figure 1 and Figure 4, provide strong visual support for the claims regarding improved stability (tighter IS ratio range, higher entropy, lower KL divergence) and its connection to final performance.", "weaknesses": "1.  The manuscript has numerous small but distracting typographical and formatting errors.\n    *   The text is missing a character at the beginning of lines 190 and 202.\n    *   The formulas in the derivation on lines 182-189 lack eq number.\n    *   Several plots in Figure 4 are missing x-axis labels (e.g., plots c, e, g), which should be labeled \"Training steps\" for clarity.\n2.  The related work section, while comprehensive, reads like a long list of recent methods without sufficient structure or synthesis. It does not adequately position GMPO with respect to other methods that also aim to improve stability, such as those using adaptive reward normalization (BNPO) or variance reduction techniques (OPO). The paper would be stronger if it discussed the conceptual differences and potential synergies with these alternative approaches, rather than solely focusing on the direct comparison with GRPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DDzaufiZcZ", "reviewer": "Reviewer_WSoW", "summary": "This paper proposes Geometric-Mean Policy Optimization (GMPO), a simple and effective variant of GRPO for stabilizing RL-based fine-tuning of LLMs. While GRPO maximizes the arithmetic mean of token-level importance-weighted rewards, GMPO instead maximizes its geometric mean, which is theoretically less sensitive to outliers.\n\nThe paper provides:\n\n- A clear analysis showing that GMPO has a narrower objective range and smaller variance in importance sampling ratios.\n- A gradient-based justification showing geometric averaging provides smoother updates.\n- Extensive experiments on reasoning benchmarks (AIME24, AMC, MATH500, Minerva, OlympiadBench, Geometry3K, and MoE settings).\nGMPO improves average Pass@1 by 4.1% on GRPO-7B and shows improved entropy stability and lower KL divergence during training.\n\nThe paper is clearly written and supported by sufficient experiments. However, I have several reservations about its theoretical depth and novelty (see the weaknesses section). In particular, the connection between the proposed theory and the empirical behavior of the model remains somewhat unclear. Additionally, the idea is closely related to **Group Sequence Policy Optimization (GSPO)**, which seems to share a similar motivation and formulation. Depending on how ICLR treats concurrent submissions and the arXiv paper, this overlap may further reduce the perceived novelty. Given these concerns, I lean toward a negative recommendation at this stage.", "questions": "1. Importance sampling, not important sampling.\n\n2. This is a major concern. In Section 3 (line 262), the authors state that GMPO performs clipping at the token level, which is different from GRPO. However, when comparing Equations (4) and (1), the clipping operations appear to be applied inside the summation (or product) over the token index t. Furthermore, by examining the gradients in Equations (5) and (6), we can see that GRPO’s rho depends explicitly on t, whereas GMPO’s weight is normalized across all t.\n    \n    In other words, GRPO effectively operates in a token-level manner, while GMPO behaves more like a sequence-level method, which contradicts the authors’ claim that GMPO’s token-level behavior.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Simple but intuitive idea.**\n    \n    Replacing arithmetic with geometric mean provides a clear and theoretically motivated way to suppress reward outliers, which is an elegant modification of GRPO with minimal implementation overhead.\n    \n2. **Extensive experiments.**\n    \n    Results across multiple reasoning and multimodal benchmarks (including MoE settings) are consistent and demonstrate improvement without introducing extra hyperparameters or architectural changes\n    \n3. **Comprehensive ablation.**\n    \n    The paper carefully studies token-level vs. sequence-level clipping, clipping range, normalization, and entropy behavior, which strengthens empirical credibility.\n    \n4. **Readable and well-structured.**\n    \n    The presentation is clear, the pseudo-code and figures are well-designed, and the comparisons with Dr.GRPO, DAPO, and GRPO variants are complete.", "weaknesses": "1. **Limited conceptual novelty.**\n    \n    While the geometric mean idea is intuitive and effective, it is arguably a *minor variation* of GRPO rather than a fundamentally new algorithmic contribution. The theoretical part mainly formalizes this intuition but lacks deeper insight into the connection to existing stability mechanisms (e.g., adaptive learning rate, variance regularization). The existence of GSPO further weakens the novelty of this paper.\n    \n2. **The analysis is not quite clear to some extent.** The main results of analyzing the gradient space of GMPO and GRPO lie in equations 5 and 6, where we see that the main difference is the reweighting coefficient of the gradient to log-pi. Then, how do the results relate to the model’s improved behaviors in:\n\n    - The more stabilized dynamics are shown in Figure 3. One potential explanation is the insensitivity to the outliers of the importance sampling ratio; then, would vanilla GRPO+filter on this ratio solve the problem? If not, where are the benefits of applying the geometric mean here?\n\n    - Figure 4 provides many good signs that GMPO indeed stabilizes the model’s training. Could the authors provide more analysis on why GMPO can make them happen, e.g., why the entropy is not decaying, and why the gradient norm is more stable? I believe that only claiming to the insensitive to the outlier is not enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)