Title: S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large language model

Reasoning model

Reinforcement learning

Post-training
Abstract: As Test-Time Scaling emerges as an active research focus in the large language model community, advanced post-training methods increasingly emphasize extending chain-of-thought (CoT) generation length, thereby enhancing reasoning capabilities to approach Deepseek R1-like reasoning models. 
However, recent studies reveal that reasoning models (even Qwen3) consistently exhibit excessive thought redundancy in CoT generation. This overthinking issue arises from the inherent limitations of conventional outcome-reward reinforcement learning, which systematically overlooks the regulation of intermediate reasoning processes. This paper introduces Serial-Group Decaying-Reward Policy Optimization (S-GRPO), a novel reinforcement learning paradigm that enables models to implicitly evaluate the sufficiency of intermediate reasoning steps, thereby facilitating early exit in CoT generation.
Unlike GRPO, which samples multiple possible reasoning paths in parallel (parallel group), S-GRPO only samples one reasoning path and serially selects multiple temporal positions from the path to exit thinking and directly generate answers (serial group). For correct answers within a serial group, rewards gradually decrease based on the exit positions along the reasoning path from front to back. This design encourages the model to produce more accurate and concise thoughts, while also incentivizing early thinking termination when appropriate. Empirical evaluations demonstrate that S-GRPO is compatible with state-of-the-art reasoning models, including Qwen3 and Deepseek-distill. Across diverse benchmarks such as GSM8K, AIME 2024, AMC 2023, MATH-500, and GPQA Diamond, S-GRPO achieves a substantial reduction in sequence length (40.4%～61.1%) while simultaneously improving accuracy (absolute 0.72%～3.92%).


Peer reviews:
comments: SUMMARY:
The authors propose two techniques to solve the long-answer problem in reasoning models: short responses generation and decaying reward assignment. The method enjoys shorter response length and better result.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The method is novel as far as I know. Inserting a termination sentence in the middle of the response is interesting.
2. The writing is crystal clear. I enjoy reading the paper.
3. The method is significant. Long response is a fatal problem for reasoning models.
4. The quality is good. The authors did solid ablation study to verify the contribution of each part. And the experimental result looks solid.

Weaknesses:
1. There is one more ablation study the authors may need to do. The whole process in the paper sounds generating the response twice: first getting the reasoning path and truncating it to insert the early termination sentence. This may have twice of the inference cost in training. More generations do have benefit in training.The authors should compare the results with double generation of the responses in training. 
2. It's unclear whether the prompt in termination sentence matter "Time is limited, stop thinking and start answering." Do the authors try different templates? What if the authors add this sentence to the end of normal training of reasoning models and ask the model to generate again?
3. Math is a relatively easy task. What's the method's performance on code related tasks?

QUESTIONS:
See Weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Serial-Group Decaying-Reward Policy Optimization (S-GRPO), a new RL training paradigm aimed at addressing the overthinking problem in current reasoning models during inference. \n\nInnovations of the Paper\n\n- Serial-Group Generation: Unlike GRPO, which parallelly samples multiple reasoning paths, S-GRPO forms a complete reasoning path. It then truncates this path at different positions to generate a series of intermediate results, constituting a \"serial group.\"\n\n- Decaying Reward Strategy: Rewards are assigned to the answers in the serial group based on their truncation positions, encouraging the model to exit as early as possible.\n\nExperimental results show that when combined with models such as Qwen3 and Deepseek-distill, S-GRPO achieves a sequence length reduction of 35.4% to 61.1% while increasing accuracy by 0.72% to 6.08% on benchmarks including GSM8K, AIME 2024, AMC 2023, MATH-500, and GPQA.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- Innovative Serial-Group and Decaying Reward Mechanism: By serializing intermediate reasoning steps and using decaying rewards, the model is guided to exit early when reasoning is sufficient.\n\n- Improved Efficiency and Accuracy: It significantly reduces reasoning length while enhancing accuracy, outperforming baseline methods (e.g., DEER, GRPO), with notable advantages in complex reasoning tasks (e.g, AIME 2024).\n\n- Adequate Experiments: Effectiveness is validated across multiple mainstream reasoning models (e.g, Qwen3, Deepseek-distill) and in both mathematical and scientific reasoning tasks.\n\nWeaknesses\n\n- Task Domain Limitation: Validation experiments are primarily focused on mathematical and scientific reasoning tasks; effectiveness in other domains remains untested.\n\n- Complex Hyperparameter Tuning: The reward function formula and sampling strategy for truncation positions are relatively complex, requiring separate tuning for different tasks.\n\nQUESTIONS:\nnone\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to implement an RL algorithm to make the LLM conduct reasoning efficiently. Based on the original group relative policy optimization (GRPO) algorithm, this paper proposes to form the trajectory group along the sequence length dimension, which only samples one reasoning path and serially selects multiple temporal positions from the path to exit thinking and directly generate answers. For correct answers within a serial group, rewards are gradually decreased when the sequence becomes longer. With S-GRPO, the reasoning models exhibit competitive reasoning performance while being much more efficient.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The method is simple yet effective.  The method simply conducts multiple early exits along one reasoning path, and uses exponential decay reward for correct answers to make it work. The experimental results show that the accuracies do not drop significantly with reasoning tokens are significantly decreased.\n2. The experimental\n\n## Weaknesses\n1. Inefficiency in implementation. For each query, S-GRPO uses the model to infer one response first, then conduct early exit from different parts of the trajectory. Intuitively, this will roughly double the inference time at each iteration step during training.\n2. The ablation study is a bit too simple. Intuitively, simply adding \"Time is limited, stop thinking and start answering.\\n</think>\\n\\n\" would make the model learns to output this sentence at each trajectory, and the truncation is conducted randomly, is there any more elegant strategy that worth study?\n\nQUESTIONS:\nFrom Figure 4, it seems that the model trained after S-GRPO will not output sentence like 'Time is limited, stop thinking and start answering.\\n</think>\\n\\nTo' why does this not happen if the model is trained with every correct trajectories that ended with this sentence during thinking?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper proposes Serial-Group Decaying-Reward Policy Optimization (S-GRPO), a novel reinforcement learning paradigm that incentivizes reasoning models to exit early during chain-of-thought (CoT) generation, addressing the overthinking problem in large language models. By introducing a two-phase rollout procedure and a decaying reward strategy, S-GRPO improves reasoning efficiency and accuracy. Empirical results show significant accuracy gains (0.72%–6.08%) and sequence length reductions (35.4%–61.1%) across diverse benchmarks. The method is compatible with state-of-the-art reasoning models and offers a practical solution for enhancing reasoning efficiency in post-training pipelines.

STRENGTHS AND WEAKNESSES:
Strengths：
Consistent accuracy improvements 
Ablation studies and detailed comparisons with existing methods 

Weaknesses:
Is it possible to directly apply a penalty for output length during the original GRPO training? Does the proposed method demonstrate superiority over this approach?


Early exiting is a common technique in dynamic neural networks and will be discussed in the related works.
For example, 
[1]  Dynamic neural networks: A survey (small modles) 
[2]LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (language models)
[3] A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (multi-model mdoels)

How many GPUs were used for training?

QUESTIONS:
see weakness

LIMITATIONS:
see weakness
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-05