Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning
Authors: Vaishnavi Shrivastava
Ahmed Hassan Awadallah
Vidhisha Balachandran
Shivam Garg
Harkirat Behl
Dimitris Papailiopoulos
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning for LLMs
LLM Reasoning
Efficient Reasoning
Policy Optimization
Primary area: reinforcement learning
TL;DR: GFPO: sample more outputs, filter by length/efficiency, and optimize only on the survivors—curbing chain-of-thought length inflation while matching GRPO-level accuracy.

Abstract: Large language models trained with reinforcement learning on verifiable rewards often inflate response length—trading brevity for accuracy. While longer reasoning can help on hard problems, many extra tokens are filler: verbose text making little progress. We introduce GFPO (Group Filtered Policy Optimization), which curbs this length explosion by sampling larger groups per problem and only training on responses filtered by (1) length and (2) token efficiency (reward per token). By sampling more during training time, GFPO teaches models to think less at inference time. On Phi-4-reasoning, GFPO cuts GRPO’s length inflation by up to 85\% across STEM and coding benchmarks (AIME 24/25, GPQA, Omni-MATH, LiveCodeBench) while preserving accuracy. We find that GFPO also outperforms Dr. GRPO in both accuracy and length reduction and generalizes across model sizes and families. We further propose Adaptive Difficulty GFPO, which allocates more training exploration to harder problems, yielding better efficiency-accuracy trade-offs on challenging questions. With only a 7\% increase in training time, GFPO reduces end-to-end latency by $\sim$30\%, cutting response time on hard queries by 90 seconds. GFPO trades modest training-time increases for lasting gains in inference—an effective recipe for efficient reasoning.

Published as a conference paper at ICLR 2026
SAMPLE MORE TO THINK LESS: GROUP FILTERED POLICY OPTIMIZATION FOR CONCISE REASONING
Vaishnavi Shrivastava Ahmed Awadallah Vidhisha Balachandran"
Shivam Garg" Harkirat Behl"
Dimitris Papailiopoulos
Microsoff Research University of Wisconsin-Madison
ABSTRACT
Large language models trained with reinforcement learning on verifiable rewards often inflate response angth--trading brevity for accuracy. While longer reasoning can help on hard problems, many extra tokens are filler: verbose text making little progress. We introduce GFPO (Group Filtered Policy Optimization), which curbs this length explosion by sampling larger groups per problem and only training responses filtered by (1) length and (2) token efficiency (reward per token). By sampling more during training time, GFPO teaches models to think less at inference time. On Phi-4-reasoning. GFPO cuts GRPO's length inflation by up 85% across STEM and coding benchmarks (AIME 24/25, GPQA, Omni-MATH, LiveCodeBench) while preserving accuracy. We find that GFPO also outperforms Dr. GRPO in both accuracy and length reduction and generalizes across model sizes and families. We further propose Adaptive Difficulty GFPO, which allocates more training exploration to harder problems, yielding better efficiency- accurrayy trade-offs on challenging questions. With only 7% increase in training time, GFPO reduces end-to-end latency by ~30% cutting response time on hard queries by 90 seconds. GFPO trades modest training-timg time increases for lasting gains inference effective recipe for efficient reasoning.
GRPO
GRPO
14000
Token Efficiency GPPO
Ai
R R(g,0;) mean ((.,,)))..RR996)))))
Adaptive Dufficuity GFPO
Average Response Length
std sta(R(gge)) R(q,6))
12000
xp(oui g.0ca)) Taulau 1.9000] Taa g.m))))
zafow 9.02a))
CaRoo
clipp
e, 1, AM
10000
mn
GFPO S REJECTIONSAMPLECA.. metric), m: (iES)
8000
6000
A,(()) (m) R(g,o;) mean R(q.0,, R(g.0)()) mi
std R(9.00 R(9,0)))
4000
2 (m)) clipp
a, 1,900
men (culsace))
20
40
60
80
100
Training Step
Figure 1: Left: GFPO introduces simple yet powerful modifications to GRPO: sample more responses during training (t G), rank them by a target attribute (e.g.. length, token efficiency), and learn only from the top-inseeting the advantages of the rest to zero. This selective learning functions implicit reward shaping, steering the policy toward desired behaviors. Right: When optimizing for length or token efficiency, GFPO curbs GRPO's length inflation- letting the model think less at inference-time by sampling more at training- vaining-timee- time time-thilee maintaining its core reasoning capabilities.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Reinforcement learning from verifier rewards (RLVR) methods such as GRPO (Shao et al. 2024
and PPO (Schulman et al. 2017) have been pivotal for test-time scaling-enabling models like O3 (OpenAl 2025) and DeepSeek-RI (Guo et al., (025) to "think longer" and achieve state-of-the-ar results on challenging reasoning tasks such as AIME and IMO. Yet longer chains are not always better: prior work shows that long responses don't It correlate with correct seswers-shorter responses can even be more accurate. For instance, [Blachandran al. (2025) report that DeepSeek-R produces responses nearly 5x longer than Claude 3.7 Sonnet on AIME 25 with no accuracy gain, while Hassid al. (2023) find that QwQ-32B's shortest responses outperform random ones by 2% while using 31% fewer tokens.
One may suspect that longer responses simply reflect harder problems. However, by comparing correct and incorrect responses to the same AIME 25 questions with Phi-4-reasoning- pooponnnnnggussaasoe (Abdin (et al., 2025), we find the opposite: in 72% of cases, the longer responses are more likely to be wrong. This suggests that verbosity is not just a byproduct of difficulty but a distinct failure mode.
Works such as Dr. GRPO (Liu al. 2025) and DAPO (Yu et al., 2025) apply token-level nor- malization address this failure mode. However, even with these methods, response length for mhi-4-reasoning-plae balloons from 4k to 14k tokens within 100 GRPO steps. We hypothesize that while loven-level normalization penalizes long incorrect outputs, it also amplifies rewards for long correct ones reie-oreeifonnggggooo verbosity in models already SFTed for hep-by-step reasoning (Abdin et al.
2025
Guo et al. 2025).
Motivated by these observations, our goal is to train efficient reasoning models: ones that preserve GRPO's accuracy while producing far shorter reasoning chains. Our contributions are as follows:
GFPO (Group Filtered Policy Optimization): A variant of GRPO that samples larger groups of candidate chains increase exposure to desirable outputs, filters them based target metric, and learns only from the filtered subset. GFPO optimized for response length reduces GRPO's length inflation by 46-71% across AIME 24/25, GPQA, Omni-MATH, and LiveCodeBench, with no loss in accuracy ($4.1)
Token Efficiency ($4.2); Defined as the ratio of reward to response length allows longer chains when justified by higher rewards. GFPO with this metric cuts length inflation by 71-85%.
Adaptive Difficulty GFPO 4.3 A dynamic variant of GFPO that allocates more exploration to hard problems using unsupervised difficulty estimates, striking better balance between efficiency and accuracy.
Out-of-Distributic Generalization ($4.4) We demonstrate that GFPO preserves accuracy while curbing response length even for out-of-distribution tasks.
Our experiments further show that GFPO improves upon Dr. GRPO, delivering both higher ac- curacy and shorter reasoning chains ($4.6). GFPO also generalizes across model families and sizes- inssssdiccnnin Phi-4- and the DeepSeek-RL RDDisilll Qwen and Llama models at 7B, 8B, and 14B scales-cutting response lengths while consistently preserving GRPO-level accuracy ($4.7).
Our best GFPO variant increases training time by only 7%, yet reduces inference latency by nearly 30% for hard problems compared to GRPO. This translates to long responses arriving ~00 seconds faster for users-an an immediate and substantial benefit ($5.3). GFPO provides favorable train-test trade-off, delivering materially lower inference latency with only marginal additional training cost.
2 GROUP FILTERED POLICY OPTIMIZATION
Group Relative Policy Optimization (GRPO; Shao et al. (0244) simplifies Proximal Policy Opti- mization (PPO; Schulman et al. (2017) by removing the value model and instead using the average reward of sampled responses as a baseline, while retaining PPO's s clipped surrogate objective.
We propose Group Filtered Policy Optimization (GFPO), simple yet effective method for targeted policy optimization of desirable response properties. GFPO samples a larger group of candidate responses per question, broadening the response pool to include more candidates with desirable traits,
2

---

Published as a conference paper at ICLR 2026
and then explicitly filters for these traits when computing the policy gradient. While it may seem natural to directly encode desirable attributes such as brevity or informativeness into the scalar reward, doing so for multiple traits can be challenging, especially when conrectness must already be captured.
Data filtration instead serves as an implicit, ficxible form of reward shaping- aakin to iterative self- improvement methods that use selective sampling to amplify specific model behaviors (Zelikman et al. 2022). After this explicit filtering step isolates the preferred responses, standard rewards are then used solely to compute relative advantages within the selected group. Thus, GFPO optimizes for multiple desirable properties (e.g., length and accuracy) simultaneously, without requiring complex reward engineering. Since our goal is reduce the response length inflation in RL, we focus on using GFPO to optimize for shorter responses while matching GRPO's 's accuracy.
Given a question g, we sample large set of responses g {oi..... OG from the current policy, Rather than training equally on all responses, GFPO applies selection step based on user-specified metric to filter a subset of size & of the most desirable responses train on. We compute metric score for each response and sort accordingly, selecting the top-k responses to form the retained subset C G. We define a binary mask m E {0, 1]G, where m, = indicates a selected response and my 0 indicates a rejected response.
Formally, we define the GFPO objective as:
G loi
(0) = E P(Q)), foth No (0)))
min
(m)) ri A,
cllip(r.,, - 1-
+ (m))
EC l0:1
-BDKL(TT (Bmia) + Entropy(Te)
where
m
REIECTIONSAMPLEGG k metric, order), my [1ES)
(m))
R(g. - 1 -
R(),
To(oi, q, 01,(1)
m4,
Fú, To (0r, q, 01,ct)
and B DDL (To ( denotes the KL penalty.
We normalize the advantages for responses in the selected subset S using the mean and standard deviation of the response-level rewards in S. This enables meaningful comparisons among responses already exhibiting the desired property. ensuring GFPO prioritizes the highest-reward responses within the filtered subset. Responses not in receive zero advantage, effectively excluding them from influencing policy updates. Thus, GFPO's primary intervention is at the level of advantage estimation, making compatible with any GRPO variant such as DAPO (Yu et al. 2025), Dr. GRPO (Liu et al. 2025). or GRPO with the Dual-Clip PPO loss (Ye al. 2020).
While GFPO is general- purpose and can accommodate various scoring metrics, our experiments specifically leverage metrics aimed at reducing response length inflation:
Response Length: Training on short responses directly encourages brevity.
Token Efficiency (reward (reward/length): Training on highly token-efficient responses encourages succinetness, but still allows longer responses if sufficiently "justified" by proportionately higher rewards.
Other such as factuality, diversity, or external quality scores- could also be integrated into GFPO to optimize different attributes of interest.
2.1 Adaptive Difficulty GFPO. We introduce Adaptive Difficulty GFPO which allocates more training signal harder questions. At each step, we estimate difficulty from the average reward of sampled responses lower averages indicate higher difficulty.
To scale the number of retained responses k, we maintain streaming summary of prompt difficulties using a lightweight t-digest, which approximates quartiles over past rewards. New questions are
Note we use the DAPO token-level loss aggregation for both GFPO and GRPO which is the default choice in verl. We employ slightly modified version of the clipped surrogate policy gradient loss introduced in prior work [[ al. 2025), which reduces training instahilities caused by negative advantages and large policy ratios.
3

---

Published as a conference paper at ICLR 2026
bucketed into difficulty levels, and assigned lo = 4 (easy), k = 6 (medium), or k (hard/very hard) of 16 samples. During a short warmup period, all questions use & = 8 to avoid unstable estimates. The number of buckets and k per bucket are hyperparameters. See Appendix for more details.
Our method sharpens filtering on easy prompts while encouraging exploration on harder ones. To our knowledge, this is the first RLVR method that adapts group size based on question difficulty.
SETUP
Model. We build on Phi-4-reasoning (Abdin et al. 2025), 14B-parameter Phi-4 model (Abdin et al. 2024) extensively SFTed on synthetic 03-mini reasoning traces in STEM, but never trained with RL. We refer to Phi-4-reasoning as the SFT baseline. To assess GFPO's generalization across model families and scales, also evaluate GFPO Distill-Llama-8B, and
DeepSeek-R1-
We compare our GFPO trained models with mhii--reaaoning-phee (Abdin al. 2025). which is trained with GRPO and DAPO's token-level loss aggregation on top of the Phi-4-reasoning model. We refer this the GRPO baseline. We match the training setup of Phi-4-reasoning, and
Baseline.
use slightly modified clipped surrogate objective for training stability ($2). For the DeepSeek-RI- -e-Seek---- Distill models, we likewise train GRPO baselines as described below. We also compare GFPO with Dr. GRPO (iiu et al. 2025). a prominent alternative for token- leken-efficient reasoning.
Dataset. RL training uses 72k math problems from the same corpus as Abdin al. (025). With 100 training steps and batch size 64, models see only 6.4k problems identical to the GRPO baseline.
Reward Function. We adopt the GRPO baseline reward: weighted sum (i) a length- -ware accuracy reward Race and (ii) 5-gram repetition penalty Reep
aware binary
R Wace LENGTIISCALE (Ras)) + Wrep Weep Heeps
R [-
(2)
Accuracy is 0/1 based on extracted final answers, with GPT-4o fallback if regex extraction fails. Formatting violations receive the minimum reward. Long correct responses penalized by cosine reward scaling. However, this length penalty is insufficient to prevent GRPO's length inflation.
Training Configuration. For the 14B models, we train with verl (Sheng et al.
2024) 32 H100s,
with global batch size of 64, for 100 steps. We use Adam with learning rate x 10 cosine warmup (10 steps), temperature T 1.0, KL regularization (3 0.001), and entropy coefficient (7 0.001). Models are trained with a 32k context, reserving 1k tokens for the prompt. GRPO uses group size G GFPO increases G {8, 16, 24) to expose the model to more candidates, but retains only k < 8 responses for policy gradients, ensuring fair comparison. For the 7B and 8B models, we train with 16 H100s, with a learning rate of 1 10 and 16k context. All other hyperparameters match the 14B training configuration. See Appendix A.7 for details on training dynamics.
(32 samples), GPQA RRein et al. 2024) (5 samples), Omni-MATH (Gao et al. 2025) (1 sample), and LiveCodeBench (8/24- 1/23) (Jain et al. (024) (3 samples). Responses are sampled at T 0.8, top-p=0.95 with 32k max length, with a maximum of 1k prompt tokens. For the -i-nSeee--R-Dssillll -ewen models, we evaluate at T = 0.6 with 16k max length for the 7B model. Final answers are extracted via regex, with GPT-4o fallback. LiveCodeBench tests OOD generalization to code, which is unseen during RL training.
Evaluation. We evaluate on: AIME 25/24 (AMEE 2025 2024
We report pass@1 accuracy, raw response length L, and excess length reduction (ELR):
LGRPO LGFPO
ELR
T.RPPO LSFT
Statistical significance is tested using the Wilcoxon signed-rank test (Wilcoxon) 1992).
4
RESULTS
4.1 GFPO Reduces Length by Sampling More and Retaining Less. An initial question is whether rejection sampling alone, without increasing sampled responses, suffices to shorten reasoning chains. To examine this, evaluate Shortest 6/8 GFPO, which retains the six shortest responses from
4

---

Published as a conference paper at ICLR 2026
Omni- MATH
LiveCode Bench
AIME 25
AIME 24
GPQA
Average
Len
Len Inf ())
% Len
% Len Inf (1)
% Len
Acc
Len
Inf (J)
Inf (1)
Inf (L)
Inf (4)
SFT
N/A
N/A
N/A
N/A
N/A
69.2
9.5k
N/A
GRPO
0.0
0.0
0.0
0.0
0.0
72.1
13k
0.0
Dr. GRPO
43.6
48.5
65.1
71.6
7.2
70.1
11.5k
47.2
6 of
1.8
9.5
11.5
-5,5
7.0
72.7
12.9k
4.8
8 of 16
23.8
33.0
23.7
31.5
36.5
73.4
12k
29.7
of 16
25.6
35.6
38.8
43.7
37.2
72.3
11.8k
36.2
16
38.0
46.8
47.3
43.2
72.0
11.5k
44.2
54.4
52.7
$2.2
51.9
59.4
71.7
11.1k
54.1
6 of 24
41.0
44.9
48.6
58.2
42.7
72.2
11.4k
47.1
of 24
46.1
59.8
57.3
71.0
57.0
72.3
11k
58.2
Token Eff.
70.9
84.6
79.7
82.6
79.7
71.7
10.2k
79.5
Adaptive Diff.
50.8
52.9
41.7
35.1
49.4
72.9
11.4k
46.0
Table 1: Pass @1 Accuracy, Response Lengths, and Length Inflation Reduction. Across all bench- marks, GFPO cuts length inflation while matching GRPO accuracy (no significant difference under Wilcoxon signed-rank test). Sampling more responses is key and lowering k/G effectively controls length. Token Efficiency delivers the largest reduction in length inflation (79.5%) at GRPO-level accuracy, and Adaptive Difficulty outperforms shortest k/G at equal compute. On LiveCodeBench (OOD coding), GRPO lengthens chains without accuracy gains, whereas GFPO shortens them and sometimes improves accuracy (e.g., 8/16, 4/24). GFPO also outperforms Dr. GRPO, with higher accuracy and larger excess-length reductions. Pass@1 accuracy uses 32 (AIME-25/24), 5 (GPQA), 1 (Omni-MATH), and 3 (LCB) samples. See Table for per-dataset response lengths and pass@1
group of eight. While accuracy remains comparable GRPO on AIME 25/24, GPQA, and Omni- MATH, length reductions are minimal (1.8-11.5%) and even negative on Omni-MATH (+5.5%). This indicates that subsampling within small response groups offers little efficiency benefit.
Substantial improvements emerge once the group size is increased. With Shortest 8/16 GFPO, which filters the shortest half of candidates, excess length is reduced by 24-37% across benchmarks without statistically significant accuracy loss. Further decreasing the number of retained responses strengthens this effect: Shortest 6/16 and 4/16 achieve an additional 2-22% reduction relative to 8/16. Scaling the group size amplifies these gains-for example, increasing from 8/16 to 8/24 yields 20-30% additional reduction, while 4/24 results in an additional 4% reduction over 8/24 (Table
Taken together, these results indicate that the decisive factor is the retention fraction k/G (Figure 6). Decreasing this Praction-eithe by reducing k or increasing G-coonistentlly shortens reasoning chains. 4/16 and 6/24 both retain 25% of responses, and their length reductions are nearly iden- cical-confirming that k/G is the key factor. Sampling from a larger group offers only slight additional benefit, as seen with 6/24. Beyond a point, however, returns diminish: moving from 8/24 to 4/24 yields only marginal additional gains. The strongest reductions are observed at retention fractions of about 25-33 23%3... We offer additional guidance for practitioners on tuning and scaling and G in Appendix AA.5
4.2 Reinforcing Token Efficiency. Reducing the k/G ratio eventually stalls learning- faaniingfiiling to deliver meaningfully shorter chains beyond certain group size. To break this ceiling, we introduce Token Efficiency GFPO, which ranks responses by reward-per-token chains only when their rewards justify the added cost. Token Efficiency GFPO filters for high reward-per-token responses- ttpically short correct chains, plus some long correct and long incorrect ones. Within this short correct chains receive the strongest positive gradients, long correct ones are modestly penalized, and long incorrect ones are sharply cut back, providing more direct length control than shortest-k, which relies the KL penalty to implicitly suppress late-token probabilities.
(t//oo.)-aaorring longer
With & 8, 16, this method yields the largest length reductions across tasks 70.9% (AIME 25), 84.6% (AIME 24), 79.7% (GPQA), 82.6% (Omni-MATH), and 79.7% (Live-
5

---

Published as a conference paper at ICLR 2026
AIME
72
70
Accuracy (%g
Accuracy (Ng
Accuracy (%)
70
6
08
88
09
64
11000
12000
13000
14000
10000
11000
12000
13000
7000
8000
5000
10000
Response length
Response length
Response length
MATH
LiveCadeBench
Average
09
Accuracy (%6)
Accuracy (9)
Accuracy (9%)
88
86
37
10000
11000
12000
11000
12000
13000
14000
10000
11000
12000
13000
Response longth
Response longth
Respance longth
SFT
GRPO
Dr. GRPO
Shortest 8/16
Shortest 4/24
Taken Efficiency
Adaptive Difficulty
Figure 2: Pareto Trfde-off Between Accuracy and Response Length. For all benchmarks except AIME 25, at least one GFPO variant strictly dominates GRPO achieving both higher accuracy and shorter responses (green region above and the left of GRPO). For AIME 25, GRPO attains the highest accuracy, but several GFPO variants, while taking non-significant accuracy dips, remain Pareto-optimal because their responses are shorter, and no other method is simultaneously more accurate and more concise. On average, Shortest 4/24, Adaptive Difficulty, and Shortest 8/16 are strictly Pareto- superio to GRPO with Token Efficiency close behind. Dr. GRPO generally falls outside the Pareto frontier-yielding lower accuracy and longer responses than GFPO.
CodeBenchh--outperrrrnng outperforming shortest-k at similar or smaller G (Table These gains come with small, non-significant drops in accuracy. Still, Token Efficiency GFPO consistently delivers the sharpest token savings without compromising accuracy-showee reward-per-token to be powerful proxy for concise reasoning.
4.3 Adaptive Difficulty GFPO. Beyond improved rejection metrics, we introduce Adaptive Diffi- culty GFPO, which varies the retained group size by question difficulty, allocating more training to harder problems. We estimate difficulty via per-question average reward, compute quartiles each step, and assign questions four buckets: very hard (0-25%) hard (25-50%). medium (50-75%) and easy (75-100%). From G=16 samples, retain 8, 8, 6, and 4 shortest responses in these buckets, respectively, giving an average 6=6.5 and making Shortest 6/16 GFPO natural baseline.
Adaptive Difficulty GFPO achieves stronger excess length reductions than Shortest 6/16 AIME 25 (51%% % vs. 26%), AIME 24 (53% vs. 36%), GPQA (42% vs. 39%), and LiveCodeBench (46% vs 36%) though Shortest 6/16 is more effective Omni-MAn
(44% 35%). Even against the more aggressive Shortest 4/16, it performs better on AIME 25 (51% % vs. 38%), AIME 24 (53% vs. 47%), and LiveCodeBench (49% 43%) (Table D). It also delivers the highest accuracy GPQA (71%) and on the hardest AIME 25 quartile (27%) compared to GRPO and other GFPO variants (Figure 3b).
4.4 Distri-Disttibbttion Effects of GFPO. Our RL training recipe is geared towards enhaneing mathematical reasoning performance. To investigate potential adverse effects of GFPO's s bias toward shorter responses, assess out-of-distribution generalization on the LiveCodeBench coding benchmark. Note that coding is not part of our RL training set. GRPO inflates response length even out-of tittof--dstributtion cotttiionnouuuuts grow from 10.3k tokens (SFT) 13.9k, while accuracy stagnates (57% vs. 58%) (Table 4.. This verbosity is undesirable, especially without accuracy gains. GFPO counters this: Token Efficiency reduces excess length by 80%, and Shortest 8/24 trims 57% while modestly improving accuracy to 59% (vs. 58% SFT, 57% GRPO). GFPO not only reins in unnecessary length but can also enhance out-of-distribution generalization.
4.5 Accuracy-Length Pareto Comparison. Figure 2 shows the accuracy-length frontier. On four five benchmarks, at least one GFPO variant is strictly Pareto-superion to GRPO (green region), demonstrating that GFPO can yield both shorter and more accurate answers. Even on AIME 25, where GRPO is slightly more accurate, GFPO variants remain on the Pareto front by offering meaningful length reductions without significant accuracy loss. Aggregated results (bottom-right)

---

Published as a conference paper at ICLR 2026
AIME
AIME
GPQA
% Len
% Len
Len
Acc
Len
Acc
Len
Acc
Len
Inf (L)
Inf (L)
Inf (1)
Deepseek-RI
SFT
37.0
11.4k
N/A
49.5
10.7k
N/A
47.4
7.5k
N/A
Distill
GRPO
40.2
13.3k
0
51.5
11.9k
0
48.4
8.3k
Qwen
of 16
39.1
11.9k
63
52.4
11k
39.6
7.9k
48.9
7B
GFPO
DeepSeck-RR R1
SFT
29.8
12.7k
N/A
44.3
12.1k
N/A
45.7
7.5k
N/A
Distill
GRPO
33.0
10.4k
0
47.9
10.3k
0
45.5
7.6k
0
Ilama
of 16
33,3
10.4k
Undef
48.5
10.0k
Undef
45.7
7.3k
319
8B
GFPO
Deepseek-RI Distill
SFT
48.2
12.3k
N/A
64
10.8k
N/A
56.4
N/A
GRPO
51.9
16.1k
0
69.3
13.9k
0
56.8
8.8k
Qwen
8 of 16
52.0
13.7k
61.9
68.5
12.5k
44.9
57.4
8.4k
22.8
14B
GFPO
Table 2: GFPO on DeepSeek-R1 Distill Models. GFPO generalizes to DeepSeck-RI-De RI-Distil models of the Qwen and Llama families at the 7B, 8B, and 14B scales, consistently reducing excess response length while preserving GRPO-level accuracy. Notably for the R1-Distill Llama model, both GRPO and GFPO reduce length below the SFT model while improving accuracy for AIME. GFPO also shortens length for GPQA.
highlight Shortest 4/24, Adaptive Difficulty, and Shortest 8/16 as the most consistently concise and accurate, with Token Efficiency trailing in accuracy by narrow margin.
4.6 GFPO vs Dr. GRPO. We compare GFPO with Dr. GRPO (Liu et al., 2025) which removes GRPO's per-response length normalization. Across all tasks, Dr. GRPO delivers lower accuracy and longer Desponses-GFPE improves accuracy by 1-3% on AIME 24, AIME 25, GPQA, and Omni-Math with 10-70 more excess length reduction than Dr. GRPO. (Table I Table 4.
This gap follows directly from Dr. GRPO's design: without length normalization, gradients scale with token count, so long trajectorics dominate updates. Since our models are already SFTed on chain-of-though and biased toward long traces, Dr. GRPO mainly learns select among long trajectories rather than shift probability mass toward shorter ones. While this may help in cold-start RL, GFPO's explicit preference for concise, high-quality samples is better suited reasoning models with SFT priors. We also observe less stable training under Dr. GRPO, with spikes in KL divergence, gradient norm, and entropy loss. Meanwhile, GFPO matches GRPO's stability (Figures 9).
4.7 Generalization Across Model Families and Sizes. To evaluate cross-model generalization, we apply GFPO to the DeepSeek-RL -I--issill Qwen and Llama families 7B, 8B, and 14B scales GFPO cuts length inflation by 63%, 39.6%, and 48.9% on AIME 25, AIME 24, and GPQA, respectively, while closely tracking GRPO's s accuracy. The 14B model shows the same trend: GFPO reduces inflation by 61 61.9%, 44.9%, and 22.8% on the three benchmarks without compromising accuracy. For DeepSeek- Bireekk-R--iiiii---iie RR--istill-Llam--B surprisingly both GFPO and GRPO shorten responses below the SFT baseline on AIME 24 and 25 while still improving accuracy- GFPO providing equal (AIME 25) or greater (AIME 24) length reduction. On GPQA, GRPO slightly lengthens responses, whereas GFPO shortens them. GFPO generalizes across model families and scales, suppressing length inflation while maintaining GRPO-level accuracy.
(Table 2 On DeepSeck-RI
5 ANALYSIS
We analyze performance of GFPO models on AIME 2025 by measuring question difficulty as 1 1- SFT accuracy, which captures how hard each problem is for the SFT model prior to RL. Problems are partitioned into quartiles (easy-very hard) to study how GFPO affects length and accuracy across difficulty and accuracy of long responses fixed difficulty. We also examine which parts of responses GFPO trims, with qualitative comparisons to GRPO for AIME 25 and GPQA in Appendix AA22
5.1 Length and Accuracy Across Problem Difficulty. On AIME 2025, response lengths grow steeply with problem difficulty- from roughly 4k tokens on easy questions to over 20k on very hard ones (Figure 7) GFPO consistently reduces this verbosity across all quartiles (Figure 3a). Token
7

---

Published as a conference paper at ICLR 2026
100
Shortest B/16 G-PO
Excess Lengin Reduction [%6
100
laken Erficiency GHPO
GFPO
30
Adaptive Difficulty GRRO
Accuracy (%)
80
60
60
40
40
20-
Easy
Modium Difficulty Quartile
Hars
Very Hard
Easy
Nedium
Hard
Very Hard
SFT
Shottest 8/16 GFPO
Tuken Efficiency GFPO Adeptiiv Difficulty
Difficulty Quartile
GHPO
Shortest 8/24 GP
(a) Excess Length Reduction Across Problem
(h) Accuracy Across Problem Difficulties.
Difficulties.
Figure 3: Excess Length Reductions and Accuracy Across AIME 25 Problem Difficulties. (a) GFPO reduces excess length across all difficulties. Token efficiency has the strongest overall reductions-winn outputs more brief than the SFT model easy questions. Shortest 8/24 has the best reductions on very hard questions. (b) Adaptive Difficulty and Shortest 8/24 have the best accuracies.
Efficiency GFPO delivers the largest overall reductions, exceeding 1 120% excess length reduction on easy problems-produeingg shorter outputs than the SFT baseline, while maintaining accuracy. Its impact diminishes on harder problems (56-79% reduction) because the token efficiency criterion permits longer responses when justified by higher rewards. Adaptive Difficulty GFPO follows the opposite trend: modest gains easy questions (38%) but substantially stronger reductions very hard ones (60%), effectively suppressing the "long tail" of overly verbose outputs. Shortest 8/24 also consistently surpasses Shortest 8/16, achieving the strongest reductions very hard problems.
Accuracy patterns mirror these differences (Figure 3b). All methods perform near-perfectly on easy problems, while GRPO and GFPO both improve Efficiency's reductions come with small, statistically insignificant accuracy dips. Adaptive Difficulty, by contrast, matches or exceeds GRPO accuracy across easy, medium, and very hard questions (e.g.. 90% vs. 88% on medium; 27% vs. 27% on very hard) while simultaneously reducing length by up to 60%. Its only shortcoming appears on "hard" questions, where filtering occasionally removes useful longer responses. This can be mitigated by increasing the group size: for example, Shortest 8/24 fully recovers GRPO's 73% accuracy on hard questions while producing substantially shorter outputs.
the SFT baseline harder ones. Token
5.2 Accuracy of Long Responses under GFPO. Reasoning models often produce less accurate answers response length grows, but this effect is entangled with problem difficulty-harder questions naturally elicit longer chains. To isolate verbosity, we fix difficulty and examine accuracy by response length on AIME 2025. Using SFT per-question accuracy as a difficulty proxy, we partition responses to hard and very hard problems into length quartiles and plot accuracy (Figure 4b).
harder
Accuracy falls steadily with length even under fixed difficulty. On hard problems, most models peak in the mid-length range (12k-16k tokens, Table suggesting sweet spot: long enough for reasoning but short enough avoid over-thinking, Beyond this, accuracy drops consistently. GFPO variants outperform GRPO in the longest bin (67% vs 52% on Hard; 20% vs 17% on Very Hard, Table as their longest responses are both shorter (20k vs 24k on Hard; 27k 28k on Very Hard) and more accurate. On very hard problems, degradation is sharper. Adaptive Difficulty and Token Efficiency briefly improve from short mid-low bins, but all methods decline at longer lengths. Token Efficiency and Shortest 8/24 show the steepest drops, likely from reduced exposure to long chains. Adaptive Difficulty the most robust, maintaining stable accuracy across bins. By contrast, SFT degrades little with length but rarely solves hard problems, producing flat yet low curve.
We complement this with an absolute-length analysis across models (Figure 4a GFPO shifts substantial mass away from the long tail (220k tokens), cutting it from 32% under GRPO 22-23%, and increasing the share of 15k responses. These shorter chains often solve harder problems: the <5k bin, GFPO's prompt difficulty is ~9x higher than GRPO's (16-18% vs 2%) with only minor accuracy loss (100% + 97%). Lower accuracy in GFPO's 's longest bins reflects that most solvable prompts are handled in shorter bins; the remaining long chains correspond to the hardest cases.
8

---

Published as a conference paper at ICLR 2026
0.34
0.22
0.15
0.11
0.17
SFT
84
75
48
19
0.24
0.17
0.12
0.15
0.32
GRPO
100
93
87
78
32
2%
33%
42%
51%
52%
60
Shortest 8/24
98
0.211
80
0.17
73
66
29
Accuracy
Very Hard
12%
35%
46%
52%
53%
40
Acounsy (S1
Adaptive Difficulty
97
61
2B
18%
1.1
40%
49%
52%
20
0.30
0.21
0.17
0.10
0.23
Efficiency
16%
35%
49%
49%
50%
langth (withir cifficuity)
20k-ak
SFT
Shortest 5/15 arFO
Token Efficiency GFPO
Absolute Longth Bin
GR
Shortest 8/24 3FPO
Adaptive Difficulty GFPO
(a) Accuracy, Response Share, and Prompt Diffi- culty by Response Length. Each cell shows accuracy (center), response share (top left), and prompt difficulty (bottom right; avg difficulty (1 S FFac))) of prompts corresponding responses in cell, for fixed response length range.
(b) Accuracy Relative Length Hard and Very Hard Problems. very hard problems, Adaptive Difficulty is most robust. Token efficiency and Shortest 8/24 drop in the longer bins, likely due to aggressive filtering.
Figure 4: Accuracy Across Response Lengths for AIME 25. (a) GFPO cuts long-tail verbosity (32% to 22% outputs 2 20k tokens) and solves hard problems with shorter responses (~9x harder prompts solved with S 5k tokens). (b) Accuracy declines with increasing response length even at fixed difficulty. On hard problems, most models peak at 12k-166 tokens, while GFPO variants outperform GRPO in the longest bin by producing shorter, more accurate long responses.
% Latency Overhead
Method
Step Time (m)
% 1 Step Time
Latency (s)
% J Latency
SFT GRPO
196.8
28.5
0.0%
315.1
0.0%
8 of 16
35.8
25.7%
272.9
13.4%
35.7%
Token Eff.
30.4
6.8%
225.0
28.6 %
76.2%
Adaptive Diff.
36.8
29.5%
255.7
18.9%
50.2%
Table 3: Train-Test Trade-off. Training step time vs. end-to-end latency for GRPO and GFPO variants. Token Efficiency GFPO reduces latency by ~29% with only 7% increase training time, eliminating three-quarters of the latency overhead introduced by GRPO over SFT.
Together, the relative and absolute-length analyses show verbosity-not difficulty-is the main driver of GRPO's long-chain emoos. GFPO mitigates this by solving harder problems more succinetly while maintaining or improving accuracy. Among variants, Shortest 8/24 and Adaptive Difficulty achieve the best Sullance-suustantially shortening responses while preserving performance, Further gains may be possible by tuning the G ratio for Token Efficiency and Adaptive Difficulty.
5.3 Train Test-time Trade-off.
We compare GRPO and several GFPO variants on training cost and inference latency (Table see Appendix AA.4 details on calculations). Latencies are averaged over AIME 24, AIME 25, and GPQA, capturing the long tail of hard problems. GRPO slows inference dramatically, raising latency from 196.8 seconds for SFT to 315.1 seconds. Although GFPO samples the responses GRPO, we find this cost is almost completely offset as the learned policy produces shorter responses than GRPO at most time steps. Token Efficiency GFPO delivers nearly identical training cost (+6.8%) step time, ~3.2 extra hours) while cutting latency by 28.6% (315.1 225.0 s), eliminating over three-quarters of GRPO's ' overhead relative to SFT. In practical terms, users wait about 90 seconds less per hard query- substantial improvement. Other GFPO variants also reduce latency but at 29-66% higher training cost. Token Efficiency GFPO provides the clearest Pareto improvement: much faster responses for nearly the same training time.

---

Published as a conference paper at ICLR 2026
5,4 What is GFPO trimming? To analyze where GFPO's length savings arise, annotate AIME 25 traces from five models- SFT, GRPO, Shortest 8/24 GFPO, Token Efficiency GFPO, and Adaptive Difficulty setup), Solution (developing candidate solutions), Verification (checking intermediate results), and Final (answer statements). GRPO inflates mid-trace reasoning compared SFT on AIME 25 the Solution segment expands from 6.5k to 8.3k tokens, and Verification from 1.9k to 3.1k (Figure (5) GFPO reverses this trend: Shortest 8/24 GFPO shrinks the Solution phase from 8.3k 6.6k tokens (1 94.4% of excess length), trimming many digressive solution attempts. It also reduces Verification from 3. to 2.3k tokens (1 66. 7% of excess length), cutting the redundant, circular checks in GRPO.
GPT-4o. Each trace is segmented into Problem (problem
RELATED WORK
GRPO Loss Modifications. Several works refine GRPO's s loss normalization to better handle token efficiency and stability. Dr. GRPO (Liu al. 2025) normalizes by the longest chain in the batch, and DAPO YYu et al. 2025) by the total token count-both amplifying penalties on long incorrect outputs. GFPO adopts DAPO's normalization (as in verl (Sheng al. 2024). TRL (von Werra jet al. 2020)) but shows that normalization alone cannot prevent verbosity: it penalizes long failures yet also rewards long successes. GFPO instead modifies the advantage function-ffitering which to normalization and compatible with variants like Dr. GRPO.
chains count for
Length-Aware Penalties. Another line of work directly penalizes verbosity, e.g., capping rewards beyond a token limit (Hou et al. 2025), applying adaptive or solve-rate scaled penalties (Su & Cardie 2025; Xiang al., 2025). or optimizing toward target lengths (Aggarwal Welleck 2025).
Such reward engineering can reduce length but often harms accuracy or requires careful tuning.
Avg Token Counts by Section for AIME25
GFPO sidesteps explicit penalties: its rejection
3333
pato Stunso ape ca9o
step implicitly shapes which outputs drive learn-
Average Token Count
-320
Toenn Eridanoy
ing.
Inference- Time Interventions. Reasoning length can also be managed without retraining. Prior work includes voting over the shortest m of l responses (Hassid et al. 2025), using "bud- get forcing" phrases to stop generation (Muen- nighoff al. 2025). or halting once answers stabilize (Liu & Wang 2025 Yang et al. (2025).
0303
4303
3303
Probees
bototion
Leritcation
Piool
These inference-time approaches are comple-
Section
mentary to GFPO.
Figure 5: Average Token Counts by Reasoning Segment. GRPO inflates the Solution and Ver- ification phases relative SFT. GFPO variants markedly reduce this excess. On AIME 25, Short- est 8/24 cuts Solution length inflation by 94.4% and Verification by 66.7%.
Rejection Sampling Methods. Rejection sam- pling has been applied post-training for length control, e.g., fine-tuning or DPO on shortest cor- rect outputs (Kim et al., 2024), or stabilizing RL updates via prompt filtering Yu et al.) 20255: Xiong al. 2025). GFPO integrates rejection
within RL itself: sampling larger groups and updating only on top-ranked chains by length or reward efficiency. Related methods use rejection for contrastive pair selection (Khaki al. 20244,, con- strained decoding (Lipkin et al. 2025), or efficient Best-of-NV search (Sun et al. (024). GFPO differs in baking this principle directly into the policy update, avoiding distillation explicit penalties.
7 CONCLUSION
We present Group-Filtered Policy Optimization (GFPO), simple extension of GRPO that samples more responses and trains only on those aligned with target attributes. This filtering serves as implicit reward shaping, curbing length inflation without complex reward design. Across multiple models and challenging benchmarks, GFPO preserves GRPO-level accuracy while improving efficiency: with only 7% increase in training time, it lowers end-to-end inference latency by nearly 30%, yielding 90-second faster response times on hard STEM queries. GEPO shifts modest additional compute to training deliver decisively faster test-time reasoning without sacrificing response accuracy.
10

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY STATEMENT
We detail our training and evaluation setup in Section including our model, baselines, training configurations and infrastructure, evaluation metrics and datasets. We provide further details on how we compute our training time and inference latency estimates in Appendix A.4
ACKNOWLEDGMENTS
We would like to thank Guoqing Zheng. Suriya Gunasekar, Piero Kauffmann, Mojan Javaheripi, Safoora Yousefi, and the AI Frontiers team at Microsoft Research for valuable discussions and feedback throughout this project.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li. Weishung Liu, Caio T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024. URL httpt:
org/ abs 2212 08905
Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, Piero Kauffmann, Yash Lara, Caio César Teodoro Mendes, Arindam Mitra, Besmira Nushi, Dimitris Papailiopoulos, Olli Saarikivi, Shital Shah, Vaishnavi Shrivastava, Vibhav Vineet, Yue Wu, Safoora Yousefi, and Guoqing Zheng. Phi-4-reasoning technical report, 2025. URL https: pectps://arxiv...o /bbs/ 1504.21318
Pranjal Aggarwal and Scan Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning, 2025. URL http::/arxix orglabs//503.04697, 2025.
AIME. Aime 83-24.
[88__2024] 2024.
Accessed: 2025-03--7
AIME. Aime 2025. Accessed: 2025-03-17.
htt.:///hhgggoooe....//aatttta/a 2025.
Vidhisha Balachandran, Jingya Chen, Lingjiao Chen, Shivam Garg, Neel Joshi, Yash Lara, John Langford, Besmira Nushi, Vibhav Vineet, Yue Wu, et al. Inference-time scaling for complex tasks: Where we stand and what lies ahead. arXiv preprint arXiv:2504.00294 2025.
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma. Liang Chen, Runxin Xu, et al. Omni-math A universal olympiad level mathematic benchmark for large language models. ICLR. 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Michael Hassid, Gabriel Synnaeve, Yossi Adi, and Roy Schwartz. Don't overthink it. preferring shorter thinking chains for improved Ilm reasoning. arXiv preprint arXiv:2505.17813 2025.
Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of-thought of Ilms via reinforcement learning. arXiv preprini 011::22004.01266. 2025.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint 2001v:2403.07974, 2024.
11

---

Published as a conference paper at ICLR 2026
Saeed Khaki, JinJin Li, Lan Ma, Liu Yang. and Prathap Ramachandra. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. arXiv preprint 4rXhv:2402.10038, 2024.
Sunnie SY Kim, Q Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, and Jennifer Wortman Vaughan. ii'm m not sure, but...": ": Examining the impact of large language models' uncertainty expression on user reliance and trust. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 822 222-835,22 2024.
Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2001.088 10050000833,,.. 13, 2025.
Benjamin Lipkin, Benjamin LeBrun, Jacob Hoover Vigly, João Loula, David R MacIver, Li Du, Jason Eisner, Ryan Cotterell, Vikash Mansinghka, Timothy O' O'Donell,, et al. Fast controlled generation from language models with adaptive weighted rejection sampling. arXiv preprint arXiv:2504.05410, 2025.
Xin Liu and Lu Wang. Answer convergence signal for early stopping in reasoning. arXiv preprint arXiv:2506.02536. 2025.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Perey Liang. Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, arXiv preprint 1rXiv:2501.19993 2025.
OpenAL. Introducing openai o3 and 04-mini. Latroodciin--33-add-------
https://openat..cmm/m
April 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference Language Modeling, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. In Proceedings of the 5th International Conference on Learning Repre- sentations (ICLR), 2017. CRL/https:/ tttttp::/oppereeiieee hwtp:/wwwwwee..foueeeee http:/oorrmmmdddleeeeellllla
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang. Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. /ttptttt////axi.orr///// ors/
03300
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. arXiv preprint arXiv: 2409. 19256, 2024.
Jinyan Su and Claire Cardie. Thinking fast and right: Balancing accuracy and reasoning length with adaptive rewards. arXiv preprint arXiv:2505. 18298, 2025.
Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, and Andrea Zanette. Fast best-of-n decoding via speculative rejection. Advances in Neural Information Processing Systems, 37:32630-32653 2024.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. pettps://githae http::/ttth.bo.ooo/gggogggggee/
2020.
Frank Wilcoxon. Individual comparisons by ranking methods. Breakthroughs in statistics: Methodology and distribution, pp. 196-202. Springer, 1992.
12

---

Published as a conference paper at ICLR 2026
Violet Xiang, Chase Blagden, Rafael Rafailov, Nathan Lile, Sang Truong, Chelsea Finn, and Nick Haber. Just enough thinking: Efficient reasoning with adaptive length penalties reinforcement learning. arXiv preprint arXi:22506.022555 2025.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504. 11343, 2025.
Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Qiaowei Li, Zheng Lin, Li Cao, and Weiping Wang. Dynamic early exit in reasoning models. arXiv preprint rrXiv:250411885 2025.
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, al. Mastering complex control in moba games with deep reinforce- ment learning. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 6672-6679, 2020.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint Sttt::2500.1447777 2025.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-11544 15488, 2022.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1
LLM USAGE.
We use LLMs for some editing and rephrasing paper text.
A.2
QUALITATIVE EXAMPLES
AIME 25 Il Problem 1: GRPO v/s Token Efficiency GFPO
User:
Six points A, B, D, and F lie straight line in that order. Suppose that G is point not the line and that AC 26, BD 22, CE DF 33, AF 73, CG 40, and DG 30. Find the area of AGGE.
GRPO
<think>... So triangle BGE: B (18,0) G (58, 24), E (57, 0). Its base BE distance from 18 to
then what is the height? The triangle's base is along x-axis and G has y coordinate 24, but note that not vertically above the segment? Actually, base BE: endpoints (18, 0) and (57, 0). The distance from C to line BE simply the absolute coordinate if G's coordinate lies between 18 and 57? But here
58, G is the right of E. But the formula for using base and height with base along line is the distance from G the line containing BE. But the line containing BE is horizontal (y 0). So the distance [24] 24, regardless of the x x-coordinate, So area = base height
39 24 468.
Wait, 39 24 936, half is 468. But wait, is that the answer? Let's check: base BE length 57
39, height 24. area
468.
But the problem's 's answer is expected to be an integer? Possibly yes. let's let's check step 1: "Six points
A, C2 C. D. E, F lie in straight line in that order." Our computed positions are consistent: A = 0,
B 18, C
D = 40, E 57, F = 73. Check distances: AC 26, BD 40 18 22.
CE 57 26
DF
33, AF 73, correct. Now then triangles: CG
distance from C = (26, 0) to G (58, 24) equals (32)2 24 /1024 + 576
1600 40,
and DG / /58 40)2 24 v18² + 576 = /324 + 576 /900- = 30, so correct. Now
triangle BGE: vertices B (18, 0). G (58, 24), E (57, 0). The base BE has length
39, and the height the vertical distance from the x-axis 24, so answer 468.
468.
However, double-check: It might be that G could have negative y. But then area is same magnitude, absolute value. Therefore answer: 468.
But wait, check problem statement: "Find the area triangle BGE." Not necessarily with base BE horizontal? But we computed that. I want to check there's any trick: Possibly the triangle is not oriented with base along the line containing B and F because then the altitude would he the distance from G that line. And indeed the line BF horizontal, the altitude is the absolute value of G's g-coordinate. computed G's y-coordinate as 24. So final answer:
Wait, but what about possibility = 58? Let's double-check the r-coordinate from difference of circle equations: We had circles: (x 26) + 1²² 40 1600, (a 40) + y² 30 900.
Subtracting: (z 26)² (x 40)² 700. Expand: (x² 52r + 676) (x² 80z 1600)
-52x + 676 + 80r 1600 28x - 924 = 700, so 28x = 1624, x = 58. So that's consistent. So
yes. Thus final answer: 468. </think>
Token Efficiency GFPO
<think>... Now we want the area of triangle BGE. The vertices are B = (18, 0), G (58, 24)
(58, 24), and E (57, 0). The area is given by & absolute value of cross product of BC and BE etc. Let's compute vectors: BO
(58 18, 0) (40, 24). BE
(57 18, 0) (39, 0). Area = 1/ det (deL(BG, BE [40 24 39) 1 936 468.
Thus answer: 468. So final answer: 468. </think>
14

---

8r. Published as a conference paper at ICLR 2026
Comparison:
The GRPO response recomputes the area of ABGE six times, while Token Efficiency GFPO computes it only once.
AIME 25 Problem 8: GRPO v/s Token Efficiency GFPO
User:
Let & be real numbers such that the system [25 20i 5 and 1z = - k| has exactly one complex solution 2. The sum possible values can be written m where m and relatively prime positive integers. Find m Here V-I.
GRPO
<think>... Equate squares: (2- (x-
(x-k)* k)² (y- 3) Expand: (a (4+ 1))²
(x - 4)² = (xr 15--] k)² 2. 1. (x k) + 16, wait check: actually, let's s expand properly. Let
A = 1 k, B = k. then equation becomes (N A)2 + y² (I B)² + (y 3) Expand
LHS: (z -A²² ====²
2Ar A and then plus y² LHS x² - A² v². RHS: 2.4
(G-B))+(y-33" ( (y
x² =
B
Cancel x² and
yields +2A --22++1² -6y+9. Rearranging: -2Ar 4
or 2 - AAx 6y (A*² -B= -9) 0. But note that A² -²² BA+ B) and
A+ = = A) 6y (16 8k 9) 0.c.c 2 A) 6y 8k)
and A-B (4+ - s, A² B2 4k4+2k) 16
Then our equation 2
but note that A k (4+ -4. Then 2(B - A)æ = 2( 4)xx Sx, so the equation
becomes - 8r + 6y + =k or 8z 6y 7+ 8k.
Alternatively, can get the perpendicular bisector of the segment connecting (4 and (k, 3). Let's 's check the midpoint: M = ( 0-3)
(_+2* 2 = (2 + k, 3/2). The slope of the line joining these two points from (4 + 0) to (k, equals x-(411)
- So perpendicular bisector slope is the positive reciprocal: 1. and it passes through M. Its equation is y - 3/2 } (x (2 + k:)). Multiply both sides by 3: 3y 9/2 4(x
k). Multiply both
sides by 2: 6y 9 = 8(x 2 k) = Sx 16 8k. Then rearranging: 6y = 8cr 16 8k, so
8x 6y 16 + 8k + 9 25 + 8k:
Let's check.....
equation
(4 14+1)) k)) + (y- 0)2 (x k)) + (v 3)2 Expand
k 4)2 y (x" 2x(4- k} (4+ k) v² The RHS is (xr - k)2 + (y 3) =x² 2kr k2 + y² 6y + Cancel and y²² from both sides to get -2(1 -2(1+ +
LHS: (x 4 k)² +y²
2kr + 9 -22kr + k² 6y + 9. Bring terms one side: -2(1 kj. + (4 k)
Combine the
2(4 k)rr 2hr -or 2kr + 2hr
Then the constant terms: (4 + k))² k2 expanding
k)² k² 8k + 16. subtracting k2 gives Sk + 16, and subtracting 9 gives 8k + 7. So the equation becomes -8r + 6y + (8k + 7)
equivalently 8x 6y 8k +
</think>
Token Efficiency GFPO
<think>... Squaring both sides yields: (a (4 + k))² - 1² (a k)² + (y - 3) Expand:
(u² - 2(4 k)w + (4 k)² + y²) (2² 2k + h2 + y² Gy + 9). Cancel x² and y leaving:
-2(4 + k)rr
k)²
2k k2 6y 9. Simplify: Bring terms: -2(4 k)r
k)²² 2kr 1²2 6y
Combine terms: 2 k)r + 2hr -2(4 + k) -2 4x Sr. So equation becomes: - Sz + (4 k)2
Simplify constant: 2 -1: (16+8k+ - = 16+ 8k. So the equation is: - -xx+ (16+8k)+++) -9---9990 - 0, or
+ Gy -
Sx + Gy + (8k + = 0, but check: 16 + 8k 9 = 7+ 8k. Multiply by - 8r 6y (8k + 0.
in standard form: 8x 6y 8k 7. ..... </think>
Comparison:
GRPO response recomputes and checks the perpendicular bisector equation times (only instances shown), while Token Efficiency GFPO computes it only once.
15

---

Published as a conference paper at ICLR 2026
AIME 25 II Problem 5: GRPO v/s Shortest 8/24
User:
Suppose A ACC has angles LBAC 84", LABC 60', and ZACB 36". Let D, E, and be the midpoints of sides BC, AC, and AB, respectively. The circumcirele of DDEF intersects BD, AE, and AF at points G, H, and J, respectively. The points G. D. E. H, J. and F divide the circumeircle of ADEF into six minor measured degrees.
shown. Find DE 2 HJ + + FG, where the ares
F
D
GRPO
<think> We need compute tices triangle DEF in order: says circle triangle DEF, and then the points G, E, H, J, are arranged in order along circle. So the vertices of triangle DEF are D, E, F. But note the circle has six points in order, and the six ares are (are D E),, then are H Wait, but check the given: They mention "the points G, D, E, H, J. and F divide the circumcirele of triangle DEF into six minor arcs, shown." That means the circle order is: G. then D, then E, then HI then J. then F, and then back to But we need to careful: The circumcirele triangle DEP
the circumeircle of triangle DEF. Let's denote the
vertices D, E, F as well. So the ares are between adjacent points in the circle order. The six ares are: are GD, are E, are EHH, are HJ, are JF are FG. But the problem statement says "the points G, D. E, H. J, and F divide the circumcircle of triangle DEF into six minor
shown." But then they ask "find DE + 2 are HJ 3 are FG.." But wait, which are are FG? The ares. need check which ares are which from the order: They are given in the diagram: The points in order are: G, then D, then E, then H, then J, then F, then back to G. So the ares are: are GD, are DE, are EH, are H., are J., are FG. But the problem asks for are DE, then 2 (are HJ), then (are FG). But note that are FC is the from F G. but in ordering, the from careful: the order the sequence. So we need to compute these are measures, but they are not trivial. We need to find angles related to triangle ABC and midpoints D, E, F. Let coordinates: A, B, C. But maybe we properties: The circle through midpoints of sides of triangle is the medial circle, but it's necessarily the circumcircle of the medial triangle. but that's exactly what it is: The circumeirele triangle D.F,, where D, E, F are midpoints of BC, AC, AB respectively, is the nine-point circle triangle ABC. </think>
G the are FG. but the are from to G. which indeed
D-E----J-F- so are
last are
Shortest 8/24 GFPO
<think> We need to compute ares on the circumeircle of triangle DEF. Let's denote the ver- tices of triangle DEF in order: It says circle of triangle DEF, and then the points G, D, E, H, J, arranged in order along circle. So the vertices of triangle DEF are D, E, F. But has intersections with lines BD, AE, AF Let's re-read the problem statement. Given triangle ABC sides BC, AC, AB respectively.
the circle
with angles A = 84, B = 60, = 36. D, E, are midpoints The circumcircle of triangle c D F intersects BD, AE, E, AF at points G, H, and J respectively. The points C. D. E, H, J. divide the circumcircle of triangle DEF into six minor ares, shown. I want sides BC, AC, AB
label positions: We have triangle ABC. but then midpoints respectively. Now, note that triangle DEF the medial triangle of ABC. And the circumcirele the medial triangle is known to the nine nine-point circle of triangle ABC. </think>
E.
16

---

Published as a conference paper at ICLR 2026
Comparison:
The Shortest 8/24 GFPO response quickly recognizes that ADFF is the medial triangle and its circumeircle the nine-point circle, while GRPO response gets bogged down are bookkeeping before arriving at the same conclusion.
AIME 25 Problem 9: GRPO v/s Adaptive Difficulty GFPO
User:
The parabola with equation y x²² 4 is rotated 60 degrees counterclockwise around the origin. The unique point the fourth quadrant where original parabola and its image intersect y-coordinate s-vE
Find c
GRPO
<think>
But before that, we can attempt to factor the quartic if possible. It is quartic in a. But perhaps substitution might simplify it? Alternatively, note that might be "nice" value? Alternatively, try find the intersection by considering symmetry. Altematively, consider the fact that the intersection point is unique one the fourth quadrant. Maybe we can try solve directly by eliminating, u. Alternativel,, consider using rotation in reverse: For an intersection point (z, y). we have r - and also (y + 4)??? Not sure. </think>
Adaptive Difficulty GFPO
<think>
I think need solve for æ explicitly from the quartic. Alterattvely, (x, y) be the intersection point. Then (æ, y) satisfies: y x² and also (a, y) satisfies equation of the rotated parabola, iee. if we rotate (I, y) by -6 degrees, we get a point on the original parabola: So let (u, v) ( + Zy. - - bu). Then y u 4. </think>
the approach:
Comparison:
The GRPO response jumps between several different allernatives-ffattriimn the quartic, trying substitutions, symmetry arguments, vague rotation ideas without committing to plan doesn't reduce uncertainty, whereas the Adaptive Difficulty GFPO defines single clear approach to take.
17

---

Published as a conference paper at ICLR 2026
GPQA: GRPO v/s Token Efficiency GFPO
User:
Consider this annihilation process p - 2 2A- The antiproton is slowly moving and mac" = 300MeV.
What is the velocity of particle A?
GRPO
<think>... Actually, maybe the process is in the lab frame where the antiproton is slow relative to the proton. But wait, which is slowly moving: "The antiproton slowly moving" means that maybe the proton moving? Wait check: + p one them slow, but which onc? "The antiproton slowly moving" means that the proton is moving fast? But then the question states: "Consider this annihilation process: + f + 2 2A"". The wording "The antiproton is slowly moving" implies that the initial momentum is coming from the proton, which is fast. But then the question asking: "What is the velocity of particle A?" Possibly meaning the outgoing particles have velocity determined by energy-momentum conservation in the center-of-mass frame.
Alternatively, the problem might be set in the center-of- mass frame. Let me think: "Annihilation process" usually is considered in the center-of-man frame. But then "The antiproton is slowly moving" might he hint that the proton is rest? Because usually one would take p is rest, and the anti-proton slow relative to p? Actually, "slowly moving" means small velocity. antiproton is slow, then maybe the proton has kinetic energy? But then the process might be not exactly symmetrie. Alternatively, maybe the assumption is that the antiproton is slow relative to the lab frame. That means the proton is rest? Because then the antiproton slow? That is contradictory: proton at rest, then the anti-proton must be moving we are the frame?
Maybe the intended scenario is that the annihilation takes place in the lab frame where the antiproton almost rest, and the proton moving with velocity such that the center-of-man energy just above threshold. But then threshold for production of 1A Am 1200 MeV.
That doesn't work. Mayhe the intended scenario is that both the proton and antiproton are nearly at rest in the center-of-mass frame...
</think>
Token Efficiency GFPO
<think>... Possibly the antiproton is slowly moving, so the reaction is in the lab frame, and partiele is rest, but then the total kinetic energy available is difference the rest masses? But careful: the antiproton "slowly moving" that means is non-relativistic? But then the proton might be at rest or antiproton at rest? Let me re-read: "The antiproton is slowly moving." So perhaps rest? Actually, which is rest? The phrase "slowly moving" might indicate that the antiproton is nearly at rest relative the proton? But then the total energy available is the mass difference, but then momentum conservation forces the final products to share momentum such that they have cqual velocity magnitude. But which velocity? Let me analyze the reaction in the center-of-mass frame </think>
the proton
Comparison:
The GRPO response repeatedly rehashes whether the reaction is in the lab frame center-of mass frame, while Token Efficiency GFPO response correctly makes assumption of center-of mass frame once.
18

---

Published as a conference paper at ICLR 2026
GPQA: GRPO v/s Adaptive Diffccuity GFPO
User:
A quantum mechanical particle of mass moves in two dimensions in the following potential, as function of (r, V(r,)) krr + kr cos" (0). Find the energy spectrum.
GRPO
<think>... Let x = r cos 0, y = r r 0. Then y² = x² + s², cos" 8 =² if + O. So
potential becomes: (( v) = [/²(=² v²) Ik(x² + 1²) ( 1k(p² + + + Aaze
kkx² + "jky² + krr (2)kx² Let's check: jkko² + kar 2kx², plus k remains. So potential becomes: ((2, y) 2kr² kky. That is two-dinensional harmonic oscillator with different frequencies along a and y. But wait, does that match? Let's recalculate: Given
(rr, 0)
//2 krr² - 1 krr cos" = fk(a² + +²) coo A(x" + )²) But cos 8 x/r so
cos" 0 = =² =²+y2 So then V = 1/((²* + +²) "k(2" + =²) s²+y² = 2 = v²) + 2 Ax= So
indeed V zky ( 2) kx² kky + 2kx2. That harmonic oscillator with frequencies:
15 = 2 and wy m
Wa
Wait, but our derivation assumed that the potential is separable in Cartesian coordinates, which is valid perform transformation
cos 0 y sin But that always valid? Let's double-check: The given potential is (rr, 8) _ kr² + kr² cos² 8 k y²) + k ( +2)) k=xx² + with x and degrees of freedom. So the energy eigenvalues are those of two independent harmonic
kyy + 2kxx. Yes that is indeed a separable potential in Cartesian coordinates
oscillators with frequencies </think>
V 4k 2 and wy v i
Adaptive Difficulty GFPO
<think>...
cos y rin Then cos2 =2 if + So then potential
becomes: V( y) 1/k((²² + +²) [j(22²²
+ + () =k((²² + + + k So rewriting:
V(x, v) kky + + 2 x² -,,ky² 2kx² Wait, check: x² + y² multiplied by factor k gives kx + ky² and then add "k gives total potential: V(x, y) = (ik+ + k) x² + kyy =
(2k)a² + kky So then the potential anisotropic harmonic oscillator in Cartesian coordinates with
frequencies: for We / 2 Ao2 and for y.: wy V
</think>
Comparison:
GRPO response recalculates the harmonic oscillator frequencies multiple times while the Adaptive Difficulty GFPO response only calculates this once.
19

---

the beam. But Published as a conference paper at ICLR 2026
GPQA: GRPO v/s Shortest 8/24 GFPO
User:
A light beam is propagating through glass with index of refraction n. The glass is moving at constant velocity the direction the beam and toward the observer in laboratory. What is the speed light glass relative the observer laboratory? Take speed light vacuum
GRPO
<think?... Let me derive from special relativity: speed light in a medium is c/n medium's 's rest frame. But when the medium is moving with velocity v. the speed of light in the lab frame is given by the relativistic velocity addition formula: n' T=(u))/22 Here, = c/n and e is the velocity of the medium in lab frame. But wait, is it v? But careful: The medium is moving with velocity in lab frame and the light moves in the medium frame with speed e/nu. But the medium's velocity relative lab is
velocity addition. But then the effective speed in lab c/n+v ? Wait, let recale: frame for light traveling in the same direction medium is: I+(ve/(nc2]) Actually, check: If 1/n then But wait, then answer is Linn? nt*
The formula is: u = T4w'e/o But e = 1, so u
1/n4v 140/(m)
1/m+= 1/n-v (K+6)/K (m+177n
l-nu
"+2""
But is that the Fizeau drag effect? The Fizeau drag coefficient is - 1/n".
Wait, I recall the formula
for the speed of light in moving medium in the direction of flow is: 1/n + (1 1/n )v.
Let me check with our expression:
1/nte 140/n (ntu)7n
1/n+v = - +v) _ 1+ne
Now expand that
at."
in Let be small. Write n+v m(1+u/m)
144//n Now expand denominator:
1+0/n 1 v/n + (u/r²² + ... we get (1 (1 v/n) = 1 + nw v/n now (v/n)) 1[1 ne n/n v²). Actually, let do it properly: (1+ nz) n/n) u/nnn+n-ii ne 0² So that equals nu v/nn 2². Dividing by gives 1/n w/n v²2 /n. So the coefficient of v <11 1/n"). So indeed, first order v, u 1/n 1/11-/nn) That is the Fizeau drag formula. So our expression is consistent with the known formula for the speed light a moving medium. But wait, are we using the correct sign? The medium is moving toward the observer, so the effective speed of light relative to the observer is increased relative to the medium's 's rest frame speed. 1/n4r which simplifies to 14me a+v
So the answer is Waab 1+0/n
</think>
Shortest 8/24 GFPO
<think>...
The transformation from glass frame to lab frame. where the speed of light is c/n in the glass frame and then add the velocity of glass, which But that the direction the the lab observer sees the beam speed + (1/n)? But wait not exactly, because the addition formula is: u'
1-+ But careful: Actually, we have an object moving at speed u (with respect to the glass) and the glass moving speed (with respect to lab) along same direction. The lab speed Wab of the light beam should be computed using relativistic addition formula: Wiah = "glos
w+e
But note that
I=ugloes V
Uglass is 1/n. So lab speed = 1/n+e T+(1/=j²' Multiply numerator and denominator by n:
1tne ...But ---
lab frame, the medium is moving, the speed of light in lab frame is: 1/n+v 1tme 1eze wit
I44/R
computed. Wait check: Multiply numerator and denominator by n., get </think>
Yes.
Comparison:
The GRPO response circles back and re-derives the velocity addition with Ugtass 1/n multiple times. while the Shortest 8/24 GFPO response does this with small recheck.
20

---

Published as a conference paper at ICLR 2026
A.3
EXTENDED ACCURACY AND RESPONSE LENGTH ANALYSIS
AIME 25
AIME 24
GPQA
Omni-MATH
LiveCodeBench
Avg Len
Avg Len
Avg Len
Avg Len
Avg
Acc
Acc
Acc
Acc
Acc
Len
SFT
64.2
10.9k
72.2
10.1k
67.0
6.6k
84.7
9.6k
57.7
10.3k
GRPO
72.4
14.8k
77.7
13.3k
67.5
10.7k
86.0
12.7k
56.7
13.9k
Dr. GRPO
66.8
13.1k
74.4
11.7k
66.9
8.1k
84.9
10.9k
57.6
13.6k
6 8
69.2
14.7k
79.6
13k
70.2
10.2k
88.3
12.9k
56.4
13.6k
8 of 16
70.2
13.9k
77.9
12.3k
70.0
9.7k
89.3
11.8k
59.8
12.6k
6 of 16
70.1
13.8k
76.9
12.2k
68.3
9.1k
87.8
11.4k
58.3
12.6k
4 of 16
69.7
13.3k
76.6
11.8k
68.6
8.8k
88.0
11.3k
57.2
12.3k
8 of 24
70.4
12.6k
75.1
11.6k
68.9
8.6k
87.5
11.1k
56.5
11.8k
6 of
68.5
13.1k
75.6
11.9k
70.2
8.7k
88.1
10.9k
58.7
12.4k
4 of
70.3
13k
76.5
11.3k
68.1
8.3k
87.6
10.5k
59.2
11.8k
Token Efficiency
69.5
12k
76.4
10.6k
68.5
7.5k
87.4
10.1k
57.0
11k
Adaptive
70.8
12.8k
76.6
11.6k
70.8
9k
88.9
11.6k
57.2
12.1k
Difficulty
Table 4: Pass @1 Accuracy and Average Response Lengths AIME 25, AIME 24, GPQA, Omni-MATH, and LiveCodeBench. GFPO variants substantially reduce response lengths while matching GRPO accuracy. We find no statistically significant differences in GFPO's accuracy under the Wilcoxon signed-rank test for any dataset. Dr. GRPO yields lower accuracy (66.6% vs 69.5% on AIME 25, 74.4% vs 76.4% on AIME 24, 66.7% vs 68.5% on GPQA) and substantially longer responses than GFPO (43.6% 70.9% len reduction on AIME 25, 48.5% vs 84.6% on AIME 24, 65.1% vs 79.7% on GPQA, and 7, 7.2% vs 79.7% on LiveCodeBench).
13000
6.
25000
Average Response Length
Efficiency
Average Response Length
Shortest 0/16 GFPO
Acieptive Diflisully
12500
20000
of 16
12000
of
10000
11500
24
of 24
5000
of
24
11000
0.2
0.3
0.4
0.5
0.6
0.7
Essy
Medium
Hard Diffcultty Quartile
Very Hard
Retained Fraction (k/G)
Figure 6: Average Response Length k/G. Reducing k/G, reduces average response length but beyond a point leads to diminish- ing returns.
Figure Average Response Length Across Prob- lem Difficulties. Response lengths rise with prob- lem difficulty for all methods, but GFPO reduces response length over all difficulty levels.
21

---

Published as a conference paper at ICLR 2026
Difficulty Bin
Method
Short
Mid-Low
Mid-High
Long
Hard
SFT
7298
9949
12576
18349
Hard
GRPO
12719
16292
19846
23834
Hard
Shortest 8/16
11292
13948
17897
22211
Hard
Shortest 8/24
10087
12839
15711
20837
Hard
Token Efficiency
8918
12044
15337
20815
Hard
Adaptive Difficulty
9593
12959
16126
21677
Very Hard
SFT GRPO
10707
15630
20875
25666
Very Hard
16728
22026
25309
27462
Very Hard
Shortest 8/16
15768
20786
24051
26935
Very Hard
Shortest 8/24
12657
18219
22671
25911
Very Hard
Token Efficiency
13034
18633
23223
26109
Very Hard
Adaptive Difficulty
13096
18625
22276
26279
Table 5: Average Response Length by Difficully and Length Bins for AIME 25. We bin each model's 's responses to hard and very hard problems into length quartiles (short, mid-low, mid-high. long) and report the average response lengths across length bins. We highlight the shortest average response length per response length quartile across the different RL methods.
Difficulty Bin
Method
Short
Mid-Low
Mid-High
Long
Hard
SFT GRPO
60.42
54.17
58.33
52.08
Hard
83.33
85.42
64.58
60.42
Hard
Shortest 8/16
77.08
79.17
70.83
52.08
Hard
Shortest 8/24
72.92
81.25
72.92
66.67
Hard
Token Efficiency
75.00
68.75
72.92
54.17
Hard
Adaptive Difficulty
62.50
64.58
72.92
56.25
Very Hard
SFT GRPO
21.88
14.06
12.50
12.50
Very Hard
32.81
31.25
25.00
17.19
Very Hard
Shortest 8/16
25.00
23.44
23.44
20.31
Very Hard
Shortest 8/24
35.94
25.00
18.75
10.94
Very Hard
Token Efficiency
26.56
29.69
17.19
15.63
Very Hard
Adaptive Difficulty
29.69
35.94
23.44
18.75
Table 6: Accuracy (%) by Difficulty and Length Bins for AIME 25. We bin each model's responses to hard and very hard problems into length quartiles (short, mid-low, mid-high, long) and report accuracies across length bins. We highlight the highest accuracy per response length quartile across the different RL. methods.
A.4 TRAIN TIME AND INFERENCE LATENCY ESTIMATION
Training step times. Each training run was executed for 100 steps. We report the average wall- clock step time by taking the mean over the 100 per-step times. This corresponds to end-to-end step duration including data loading, optimization, and logging. We train our models on 32xH100s, with global batch size of 64, and 32,768 context. See for more details.
Inference latency. Latencies are computed from evaluation runs on AIME 24, AIME 25, and GPQA. For AIME 24/25 we sample 32 responses per question and for GPQA we sample 5. For each response, the evaluation harness records a response_t time equal to wall-clock time from request submission to receipt of the final token. We report average latency over all evaluation responses on AIME 24, 25, and GPQA. All evaluations were served with vLLM on 8xH100 GPUs with bfloat16 and maximum context length of 32,768 We used single- GPU vLLM server per device with max_ _um bat ched_ tokens=65,55 All models were evaluated with identical decoding parameters (temperature, top-p. EOS/pad ids, and maximum length). See 3 for more details.
22

---

Published as a conference paper at ICLR 2026
A.5 TUNING AND SCALING RETENTION RATIO k/G
Our results demonstrate that the retention ratio k/G is critical hyperparameter for tuning the length reductions derived by GFPO. For practitioners, our findings translate into simple guidelines for choosing and G in new settings. We recommend starting with retention fraction k/G 25 33%, which consistently delivers large length reductions with limited accuracy loss across math, coding, and STEM tasks. Practitioners may start with the largest group size G that fits within their training budget, then choose & so that k/G lies in the 25-33% range (using higher ratio for harder datasets to allow for more exploration). Finally, while k/G remains a tunable hyperparameter, we find that strong rejection metrics such as token-efficiency filtering already provide substantial gains even at conventional ratios (e.g. k/G 50%), and in practice reduce sensitivity to the exact choice of k/G.
We explore scaling G from 8 + 16 - 24 and increasing k up to 8. As shown in Figure G average response length varies almost linearly with k/G across these settings. We do not experiment with k > 8 because this would exceed the effective group size of our GRPO baseline Shhi--Reassoning Plus, trained with G = 8), making comparisons unfair. Our findings indicate that the benefits of GFPO do not require scaling very large values of G:
Diminishing returns for large G. Once G is moderately large, further increases offer little additional gain. For example, 4/16 and 6/24 achieve nearly identical trace lengths.
k/G is the control knob. The same ratio can be obtained at multiple G values (e.g.. 4/ 16 6/24), allowing practitioners to fix G at moderate value (e.g.. 16) and tune k.
Better filtration beats bigger groups. Stronger selection metrics outperform brute-force scaling. For instance, Token-Efficiency 8/16 GFPO achieves larger length reductions than Shortest 8/24, despite using smaller group.
Moderate group sizes already provide stable advantages. We agree that, in principle, larger G and k reduce variance in advantage estimation. However, our empirical results- and prior work-show that G = 8 or 16 (Shao et al.) and even G = 4 (Qian et al.) already yield stable advantages and strong downstream performance.
A.6 ADAPTIVE DIFFICULTY ESTIMATES
Algorithm ADAPTIVE DIFFICULTY SAMPLING
Require: group G = toi...., ,o), t-digest tracker T, reward function R(-), prompt scoring metric(.), sort order (t/4)
1: PR + y E R,g,
2: T. (.UPDATT(MM))
3: if ...AAY() then
4:
(125, Too. 175) t- T. (25, 50, 75))
5:
if' UR < 925 then k <- kvery-hard
6:
elif PR < 950 then k khard
7:
elif P. < 475 then k +- kindd
8:
else + Reasy
9: else
<- kivery hard 11: end if
12: (S.m) + REJECTIONSAMPLE(G k; metric, order)
13: return S,
Algorithm| details Adaptive GFPO. We tested EMA-style smoothing for per-question difficulty, but ultimately opted for momentum-free approach. Given that difficulty evolves throughout training, meaningful adaptation requires a small momentum coefficient makiiinnttmaiinggggeeee the method effectively equivalent using the most recent estimate. Since method bins questions into difficulty categories, only boundary-aad boundary-adjacent adjacent questions are sensitive to noise in their difficulty estimates. We thus use the simpler momentum-free variant, which adapts faster and avoids storing per-question state.
23

---

Published as a conference paper at ICLR 2026
A.7 TRAINING DYNAMICS
0.55
Stayleed (CP
Stortest824 GPPO
24310
Adtodv: Dffiicu GPRO
Vrome Ettinerra GERO
0.30
0.25
5,0000
0.20
0.03
GAPC Sharhest 8:23 Gron
20300
Shomest Bx2 GROO
30825
Tiken Eiticenryy GYiP
0.000
2,0000
Edagrose tlitie city GO9e
20
60
3D
20
8D
100
80
Training Step
Tranning Soop
Tra ning Step
14000
Resporso Langth
Elfoweey GIIN
25 Accurec)
0.68
6,700
32000
Alleptien: Dificully GRRO
0.671
0.625
8000
0.60
0.000
GaPC
Sho tast 8/16 GFPO the ee apc GFPn
Taken Eflicinyy OPPO
Edberive fificityy GERC
60
0.50
80
40
60
120
Training Step
Thaining Step
Tra ning Step
Figure 8: GFPO GRPO Training Dynamies. We observe sudden KL explosions collapses during GFPO training-in fact GFPO's KL curves are often as smooth or smoother than GRPO's s. All GRPO and GFPO methods incur some reward oscillations over the course of training. But all methods monotonically converge towards the same reward, while GFPO substantially suppresses response length growth. We observe that GFPO tends to achieve its peak accuracy earlier than GRPO, while matching improving on GRPO's s best accuracy in almost all cases.
34
M Wrt
0.0005
Toben ency
0.00
GRPC
Tchan
0.0000
0.10
L,055
PPO
0,0002
00.
0.0000
0.2
100
150
Training
ning
GIFD
GRPO
Tohan LRf eney Grin
Iooen Lihoenne GP
Atoctive cullcullyy GAPO
GRRD
25003
Adiaplie Dificclts GPPO
GRP0
5.0
20003
Eroroyy
Respense
GRFD
10000
Toan Effrcnery
2-
Acoptive Difhouity
N.
Taining Step
Training Stop
ning Stop
Figure 9: GFPO vs Dr. GRPO Training Dynamics. We observe occasional spikes in gradient norm, KL, and clip fraction when training with Dr. GRPO. These arise when batch contains few very long trajectories with large (unnormalized) advantages: because Dr. GRPO drops both std normalization and per-response length normalization. such trajectories dominate the update and cause transient but sharp jumps in the PPO statistics which can cause training to diverge. Meanwhile GFPO inherits the stable learning curves of GRPO.
24


Reviews:
[{"review_id": "tA9FZ8xLo5", "reviewer": "Reviewer_rWEa", "summary": "The authors propose GFPO (group filtered policy optimization), a method to encourage concise responses in reasoning models by using both token efficiency and length as filters to select the best examples over a larger set of rollouts. Through this larger amount of oversampling they find better examples to optimize for token efficiency. It adds 7% to the training time but saves 90 seconds on average over long queries.\n\nThe adaptive version dynamically scales the number of samples taken from the model based on an estimate of the difficulty. If rollouts continue to fail, more are sampled until a maximum. There are a few insights guiding this: easy questions don’t need as much reinforcement, so we can stick to just giving a few short demonstrations. Harder questions may have different ways of expressing them. The authors claim theirs is the first RLVR method to do something like this.\n\nThey test using GFPO to RLVR Phi-4-reasoning against a baseline that used GRPO with 72k math problems. Their approach uses the same data setup as Phi-4-reasoning-plus, so it’s a fair comparison. All parameters are kept the same. GFPO delivers slightly better accuracy and significant improvements in length over GRPO.\n\nI’m not really the right person to review this as I don’t work on RL, so this is a low confidence review.", "questions": "N/a, sorry my review is late", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Simple, elegant, but novel (I think) idea\n- High-validity experiment by testing against an existing baseline with everything else held equal\n- Clear demonstration of improvement in performance\n- Lengthy and detailed analysis", "weaknesses": "I am pretty convinced by the results, but it would be nice to see the experiment replicated for at least one other model so we can know it’s not a fluke.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8nrzJ5E7ir", "reviewer": "Reviewer_9jp4", "summary": "GFPO (Group Filtered Policy Optimization), which curbs this length explosion by sampling larger groups per problem and only training on responses filtered by length and token efficiency (reward per token). This can make LLM generate shorter reasoning traces during inference.", "questions": "1. Is it possible to add some experiments on other sizes of models, besides 14 B?\n2. Is it possible to add some experiments on other model families, such as Qwen, DeepSeek, and Llama?\n3. Is it possible to include some analysis about the training stability?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The intuition of sampling more and using more computational resources on hard questions is straightforward and reasonable.\n2. The experiment shows the effectiveness of GFPO in different tasks, including mathematical, STEM, and coding reasoning.\n3. The method itself is simple and easy to plug into any other RL post-training framworks.", "weaknesses": "1. The evaluated model size and model family are limited, which only contain the 14B Phi-4 model.\n2. Considering this method is based on GRPO, maybe an analysis of training stability is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ksUAbbeFGk", "reviewer": "Reviewer_3LZ1", "summary": "This paper addresses the critical issue of length inflation in large language models trained with reinforcement learning, where models tend to produce excessively verbose outputs without corresponding accuracy gains. The authors propose Group Filtered Policy Optimization (GFPO), a simple yet powerful modification to the GRPO training process. The key idea is to sample a larger group of responses during training and selectively update the policy using only a filtered subset of the most desirable ones—based on metrics such as conciseness or token efficiency. This in-training filtering serves as an implicit form of reward shaping, effectively teaching the model to be more concise at inference time.", "questions": "1. How stable is the average-reward–based difficulty estimate throughout training? Did you explore smoothing or momentum-based approaches to mitigate early-stage noise?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.GFPO’s core idea of in-training filtering is clear and easy to implement. By sampling a larger pool of responses and selectively training on the best subset, it avoids the complexities of explicit reward engineering. Framing this as implicit reward shaping provides an intuitive and generalizable way to steer model behavior toward desirable attributes like conciseness.\n\n2. The paper demonstrates a rare and highly desirable outcome: improving inference efficiency while maintaining or even enhancing accuracy. The reported gains—up to an 85% reduction in excess length and a ~30% reduction in end-to-end latency are substantial and consistent across challenging STEM and coding benchmarks. \n\n3. The method generalizes well beyond the training domain. On the out-of-distribution LiveCodeBench benchmark, GFPO continues to reduce response length where GRPO inflates it, sometimes even improving accuracy.", "weaknesses": "1. The Adaptive Difficulty GFPO relies on the average reward of sampled responses to estimate problem difficulty. While conceptually clever, this heuristic can be unstable and noisy, especially early in training or on high-variance problems. Randomly poor rollouts may cause easy instances to be misclassified as “hard,” leading to inefficient resource allocation.\n\n2. The method’s success hinges on the retention fraction k/G, which determines how aggressively responses are filtered. Although the paper explores different ratios empirically, it offers limited theoretical or heuristic guidance for setting this parameter in new settings. This may require costly tuning for practitioners.\n\n3. GFPO’s core trade-off sample more to think less implies higher computational expense during training. While the authors show that a 7% increase in training time yields a 29% reduction in inference latency, this additional cost may still be significant in resource-constrained environments.\n\n4. Filtering based on conciseness or shortness may inadvertently penalize useful but verbose reasoning paths. This could suppress exploration and hinder the model from discovering complex problem-solving strategies that are initially long but later refinable into concise solutions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MN6V9yoGkD", "reviewer": "Reviewer_g7JB", "summary": "This paper introduces Group Filtered Policy Optimization, a simple modification to GRPO for RL with verifier rewards: for each question, sample a larger group of trajectories, rank by a target metric (e.g., length or reward-per-token), and update only on the top-k while zeroing advantages for the rest. Experimental results on Phi-4 reasoning demonstrate the effectiveness in reducing the generation length, latency, and training time.", "questions": "1. What is the performance of GFPO on the Qwen series model?\n\n2. Could you conduct an ablation study by scaling up K (e.g., to 64 or 128)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A simple yet effective approach to dynamically adjusting the group size can broaden the response pool, incorporating more candidates with desirable traits and thereby optimizing toward the desired response properties.\n\n2. The authors present comprehensive experimental results, evaluating not only mathematical tasks but also out-of-domain scenarios, and reporting both token-level efficiency and pass@1 accuracy.\n\n3. The authors demonstrate that GFPO reduces latency by generating shorter responses than GRPO, while the token-efficiency variant achieves this with nearly the same training cost.", "weaknesses": "1. There is a line of research on token-efficient GRPO methods [1–4]. Although the authors mention several of these works in the related work section, they do not provide a direct comparison in the paper. Including such comparisons would strengthen the paper’s results and claims.\n\n2. Additional training costs introduced during the process may limit the use of large values of G or K in GFPO.\n\n\n[1] Aggarwal, Pranjal, and Sean Welleck. \"L1: Controlling how long a reasoning model thinks with reinforcement learning.\" arXiv preprint arXiv:2503.04697 (2025).\n\n[2] Luo, Haotian, et al. \"O1-pruner: Length-harmonizing fine-tuning for o1-like reasoning pruning.\" arXiv preprint arXiv:2501.12570 (2025).\n\n[3] Huang, Chengyu, Zhengxin Zhang, and Claire Cardie. \"HAPO: Training Language Models to Reason Concisely via History-Aware Policy Optimization.\" arXiv preprint arXiv:2505.11225 (2025).\n\n[4] Arora, Daman, and Andrea Zanette. \"Training language models to reason efficiently.\" arXiv preprint arXiv:2502.04463 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)