DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization
Authors: Gang Li
Yan Chen
Ming Lin
Tianbao Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Reasoning Model
Efficient Reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a novel method which integrates an optimized positive data distribution under a KL regularization into a discriminative objective to encourage efficient reasoning with minimal effect on performance.

Abstract: Recent large reasoning models (LRMs) driven by reinforcement learning algorithms (e.g., GRPO) have achieved remarkable performance on challenging reasoning tasks. However, these models suffer from overthinking, generating unnecessarily long and redundant reasoning even for simple questions, which substantially increases computational cost and response latency.  While existing methods incorporate length rewards to GRPO to promote concise reasoning, they incur significant performance degradation. We identify the root cause: when rewards for correct but long rollouts are penalized, GRPO's group-relative advantage function can assign them negative advantages, actively discouraging valid reasoning. To overcome this, we propose Decoupled Reward Policy Optimization (DRPO), a novel framework that decouples the length-based learning signal of correct rollouts from incorrect ones. DRPO ensures that reward signals for correct rollouts are normalized solely within the positive group, shielding them from interference by negative samples. The DRPO's objective is grounded in integrating an optimized positive data distribution, which maximizes length-based rewards under a KL regularization, into a discriminative objective. We derive a closed-form solution for this distribution, enabling efficient computation of the objective and its gradients using only on-policy data and importance weighting. Of independent interest, this formulation is general and can incorporate other preference rewards of positive data beyond length. Experiments on mathematical reasoning tasks demonstrate DRPO's significant superiority over six efficient reasoning baselines. Notably, with a 1.5B model, our method achieves 77\% length reduction with only 1.1\% performance loss on simple questions like GSM8k dataset, while the follow-up baseline sacrifices 4.3\% for 68\% length reduction.

Published as a conference paper at ICLR 2026
DRPO: EFFICIENT REASONING VIA DECOUPLED RE- WARD POLICY OPTIMIZATION
Gang Li
Yan Chen Ming Lin
Tianbao Yang'
1 Texas A&M University
The University of Virginia
gang- amu edu, jnkeviiiig edu 1imming04egmai com, cianbao-yangetamu. edu
ABSTRACT
Recent large reasoning models (LRMs) driven by reinforcement learning algo- rithms (c.g.. GRPO) have achieved remarkable performance on challenging rea- soning tasks. However, these models suffer from overthinking, generating unnec- essarily long and redundant reasoning even for simple questions, which substan- tially increases computational cost and response latency. While existing methods incorporate length rewards to GRPO to promote concise reasoning, they incur sig- nificant performance degradation. We identify the root cause: when rewards for correct but long rollouts are penalized, GRPO's group-relative advantage function can assign them negative advantages, actively discouraging valid reasoning. To overcome this, we propose Decoupled Reward Policy Optimization (DRPO), a novel framework that decouples the length-based learning signal of correct roll- outs from incorrect ones. DRPO ensures that reward signals for correct rollouts are normalized solely within the positive group, shielding them from interference by negative samples. The DRPO's objective is grounded in integrating an opti- mized positive data distribution, which maximizes length-based rewards under a KL regularization, into a discriminative objective. We derive closed-fore -ored-fommm lution for this distribution, enabling efficient computation of the objective and its gradients using only on-policy data and importance weighting. Of independent interest, this formulation is general and can incorporate other preference rewards of positive data beyond length. Experiments on mathematical reasoning tasks demonstrate DRPO's significant superiority over six efficient reasoning baselines. Notably, with a 1.5B model, our method achieves 77% length reduction with only 1.1% performance loss on simple questions like GSM8k dataset, while the follow- up baseline sacrifices 4.3% for 68% length reduction. The code available
com Optimipiimmm zat ion AT/DRPO
INTRODUCTION
Recently, large reasoning models (LRMs) driven by Reinforcement Learning (RL. (Guo al. 2025 Team et al. 2025b) have demonstrated remarkable performance complex reasoning tasks like mathematics, coding, and scientific problem-solving. Unlike conventional language models that focus direct thoughts and solutions, LRMs improve performance by generating extended chain- of-thought paths (Wei et al.) (022), allowing them to revisit intermediate steps, correct errors, and even explore alternative reasoning paths. This approach equips LRMs with stronger reasoning abil- ities and has become a standard paradigm to develop models capable of solving complex tasks.
However, existing LRMs suffer from overthinking with lengthy and redundant reasoning paths. As demonstrated by Chen et al. (2024), reasoning models like DeepSeek-RI (Guo et al., (205)) need to generate about 1.000 tokens answer "what the answer of 2 plus 3", while only around 10 tokens are needed for non-reasoning models. Such overly generated reasoning paths raise significant issues, leading to substantially increased computational cost and longer inference time. Numerous studies have been conducted to explore ways to eliminate redundant reasoning and improve the reasoning efficiency. A popular strategy to introduce explicit reward shaping with length penalties in RL to
"Equal Contribution

---

Published as a conference paper at ICLR 2026
guide the model toward concise reasoning (Arora & Zanette Huang al.) Xiang et al. Aggarwal & Welleck). e.g., penalizing rewards of correct answers based on reasoning length to encourage shorter reasoning. Nevertheless, almost all existing methods fall short in preserving performance while shortening reasoning, causing substantial performance loss. This raises a key question: how guide for efficient reasoning with minimal performance drop?
We identify the root cause underlying the insufficiency of existing RL-based efficient reasoning methods. The recent advancement of training efficient reasoning models has been largely built upon the Group Relative Policy Optimization (GRPO) framework, due to its groundbreaking per- formance (Guo et al., 2025). GRPO's effectiveness hinges on its group-relative advantage function, which normalizes a rollout's 's reward against the group average to create a learning signal that dis- tinguishes positive from negative examples. Yet, this very strength becomes greatest weakness when moving beyond simple correctness. We demonstrate that the framework is fundamentally ill- suited for composite rewards. Incorporating length penalty reduces the reward for correct but long answers, often pushing their group-relative advantage below zero. Consequently, GRPO is misled into interpreting verbose correct answers as negative examples, discouraging valid reasoning and creating significant optimization barrier (Figure IT.
An effective mechanism must not only distinguish right from wrong but also efficient right from inefficient Sibht-assigning strong positive signal to concise answers and weaker positive signal to verbose ones, all while suppressing incorrect reasoning. To this end, we introduce Decoupled Re- ward Policy Optimization (DRPO), novel RL framework that fundamentally rethinks how learning signals are constructed. DRPO's ' core innovation is the decoupling of the learning signal calculation: normalizes rewards for correct rollout only against other correct rollouts, completely insulating them from the negative examples that corrupt GRPO's signal. This ensures that length penalty pro- portionally reduces the positive signal of long correct answer but never pushes it into negative ter- ritory, thereby achieving more favorable trade-off between efficiency and accuracy. We formalize this intuition by deriving a generalized objective a discriminative RI. framework. This objective integrates perturbed version of the on-policy positive data distribution, where the perturbation is designed explicitly to maximize a length- based reward. We derive a closed-form solution of the perturbed distribution, which allows us to efficiently optimize the objective without any additional data collection, using only on- on-policy samples via importance weighting.
Our contributions are four-fold:
We diagnose a critical, previously overlooked deficiency in the widely-adopted GRPO framework: its group-relative advantage function is ill-suited for correctness-lentth composite rewards and actively harms learning when incentivizing efficiency.
We propose Decoupled Reward Policy Optimization (DRPO), new paradigm that decouples learning signals for positive and negative data. DRPO provides consistent, uncorrupted policy gradients for multi-reward optimization (e.g.. correctness and length).
We derive a rigorous formulation for DRPO by integrating a goward-mxximzzing perturbed pos- itive data distribution directly into a discriminative objective. We obtain a tractable closed-form solution to the perturbed distribution, yielding a practical algorithm requiring only on-policy data with no overhead.
We conduct experiments demonstrate the superiority of DRPO in training efficient reasoning models, substantially outperforming strong baselines across different model sizes and various mathematical reasoning benchmarks.
2 RELATED WORK
Large Reasoning Models. Earlier structured prompting approaches such as Chain-of-Thouge (CoT) (Wei et al. 2022). , Tree-of-Thought (ToT) (Yao et al. 2023). and Craph-of-Thougee (GoT) (Besta et al. 2024 demonstrated the importance of decomposing complex problems into intermediate steps. However, these methods rely heavily on prompting and search heuristics, lack- ing a unified learning framework to optimize reasoning efficiency and robustness.
The breakthrough came with DeepSeek-R1, which revealed that large reasoning models (LRMs) trained via large-scale reinforcement learning, particularly GRPO (Shao et al., (024). can au- tonomously acquire advanced reasoning behaviors such as branching, verification, and backtracking.
2

---

Published as a conference paper at ICLR 2026
This success inspired numerous follow-up up studies; some aimed at reproducing the effectiveness of GRPO (Wen et al. 2025 Luo et al. 2025b; He et al. 2025), while others investigated its limitations
and proposed refinements to further enhance reasoming performance et [2025] [ et al.) 2202a
Chen et al., 2025 |025| et al., 2025). In parallel, many open-weight reasoning models have adopted GRPO, including Qwen-3 Yang al.] 2025a), GLM-4.55 (Team et al. 2025a), K2-Think (Cheng) iet al., 2025). and Goedel-solver (Lin et al. 2025), among others. Despite these advances, existing LRMs often suffer from overthinking produuccing unnecessarily long and redundant reasoning even for simple problems. This work aims to directly address this issue by training LRMs to reason both efficiently and effectively.
Efficient Reasoning in LRMs. To address the issue of overthinking in LRMs, variety of tech- niques have been proposed (Sui et al. 2025; Yue et al., 2025). including (1) training-free methods,
which shorten the reasoning paths via prompt (Aytes et al., 2025 Han et al. 2024) or manipulating
the decoding process (Yang et al., 2025b; Yong et al. 2025| Wang et al. 2025a Liu et al. 2025a
Wang et al. (025c): (2) Supervised Fine-tuning (SFT) methods, which rely compressed reason-
ing datasets for linetuning. These datasets are curated via token-level selection (Yuan et al.
2025
Xia et al. 2025 Zhuang et al. 2025), step-level selection (Xiao et al. 2025 Cui et al. 2025 Wang
et al.] 2025b). path-level selection (Munkhbat et al.) 2025 Ghosal et al.
[022 (3) RL-based meth-
ods, which carefully design reward mechanisms to guide the model to reason efficiently (Hou al.
Aggarwal & Welleck Arora & Zanette Xiang et al. Luo et al. Huang et al.) Yi et al. 2025
ct al. 2025b Xu et al., 2025: Fang et al. 2025 Li ct al. 2025b).
Among these techniques, RL-based methods have been demonstrated to be one of the most effective approaches due their scalability and flexibility. Specifically, LI (Aggarwal & Welleck) integrates a length constraint into the reward function to encourage higher performance while meeting the length goal specified in the prompt. Arora & Zanette employs online RL with a length penalty based on the distribution of correct answers, penalizing correct responses longer than the average while encouraging those shorter than the average. AL.P (Xiang et al.) adaptively adjusts length penalties according to problem difficulty, measured by pass rate, assigning stronger penalties to high pass-rate problems to discourage overlong reasoning. HAPO (Huang et al.) keeps track of the minimum length of correct responses for each question, penalizing outputs that exceed this length while rewarding those that are shorter. Nevertheless, these methods all suffer from misleading learning signals and fall short in preserving performance while shortening reasoning, due to the limitation of their adopted relative advantage function.
Discriminative Learning for LRMss Discriminative learning is a classical paradigm applied widely to traditional tasks like classifications (Bishop & Nasrabadi 2006 Yang & Ying 2022] and rank- ings (Burges et al.] 2005; Cao et al. 2007). These methods follow the principle of raising scores for positive (correct) samples while lowering scores for negative (incorrect) ones. Recently, sev- eral works have explored applying the principle of discriminative learning to LRM training, For example, Li et al. (2025a) proposed a discriminative constrained RL framework with verifiable bi- nary rewards finetune LRMs. Lyu et al., (2025) and |Bai et al. (2025) utilize discriminative loss for behavior cloning on positive samples and policy gradient on negative samples. [S et al. (2025) employs a discriminative learning approach based on positive-negative pairs for reasoning tasks. However, these methods are limited to binary accuracy rewards and don't 't address the challenge of overthinking in LRMs.
3 LIMITATION OF INCORPORATING LENGTH PENALTY INTO GRPO
Notations. We study the fine-tuning of generative reasoning model ne parameterized by 0. At each learning step, the previous model is denoted by Told, which is responsible for generating answers to a given set of questions. For a question q @ Σ' (including its prompt). the output o E Σ' is sampled from Tola (-lg(, consisting of both reasoning traces and the final answer, where Σ- denotes the space of all sequences of tokens with arbitrary length. More concretely, o generated sequentially at the token level: for a given question and its corresponding answer in the output o is verified by either matching the extracted answer against the ground-true answer or a formal verification tool (Guo et al. 2025 Zhang et al. 2025). Let क (+1g) denote the conditional distribution of outputs when the reward is one (i.e., correct answers) and (-19) denote the conditional distribution of outputs when the reward is zero (i.e., incorrect answers). Let [[] max(-, 0) denote the hinge function.
Tota--lg, Oct), for /. = 1....., lo]. The correctness reward r=(oll) E (1.0)
3

---

Published as a conference paper at ICLR 2026
Generation
Length
Advantage (A:)
Objective for Rollout
1.00 1 1.00
3+2+7
GRPO
incorrect
Orrectnecs
min + Y A, clip(r;)]
Length
Advantage (A,)
"efor It)
Langth
Reat
GRPO
Incod...
-0.17
Sqgune....
correct
Oorectonss
X
Length
Learning Signal (w:)
Lthik.....
0.33
W/+ (o:
DRPO
0.05
correct
incorrect
Brrestweik
Figure 1: Illustration of the limitation of GRPO with length penalty and the benefit of our approach.
Suppose [1, 1, 1, O, O, 0] are the accuracy rewards for 6 responses, and [0.73, 0.6, 0.2, 0, O, 0) are the rewards after applying the length penalty to correct answers. Using the group-relative advantage calculation of GRPO, the advantages for the third response shift from 1 (without length penalty) to -0.17 (with length penalty added), inadvertently penalizing the third correct response, which may substantially harm performance. In contrast, our proposed DRPO reduces the learning signal for lengthy and correct responses but never pushes them to the negative territory.
Following the success of DeepSeek-R1, Group Relative Policy Optimization (GRPO) is widely adopted in existing RL--based efficient reasoning methods estimate the relative advantage from group rewards instead of the critic model. In the following, we illustrate the limitations of incorpo- rating length penalty into GRPO for promoting efficient reasoning, We note that similar limitations occur to other RL methods, such as RLOO (Ahmadian et al.) 2024), and other REINFORCE-bar methods (Hu et al.) 2025] Chu et al. 2025). which couple rewards for correct and incorrect answers to compute advantages for learning.
The GRPO objective for maximization is given by:
JGRPO (8)
(o:) (o. (
loal
min (iiplllal).ciip(raa1----
- 3B (Tell [|Tre))
where ri, =au(or._1..1...)) Tet is a frozen reference model, DKL (..-) denotes the KL divergence between two distributions, and A(o;(q) denotes the relative advantage of output o, which quantifies how much better the reward of o, denoted by r(o; 4) compared to the group average reward and guides the learning direction. Specifically, A(o:/g) is computed by
A(o.1g) r(o.lg) mwwa(r(o)))))oo ((2))))))) ((ooll))))
(2)
std(r(oilq),
.r(ollg))
Let us consider incorporating length penalty into GRPO. Existing RL methods with length con- trol reveal shared principle: penalizing rewards of correct answers based on reasoning length to encourage shorter reasoning, e.g., r q) re(olg)= - r)(ogg) where rı(-) is length-based cost or reward function (refer to Table 2 for detailed formulations). The consequence is that the re- ward of correct with long output will be shifted down relatively. Incorporating this reward into GRPO's group relative advantage calculation may bias the intended effect, misleading the learning process. Let consider illustrative example in Figure Suppose there six generated outputs with different lengths, whose correctness rewards are [1, I, 1, 0, 0, 0] and corresponding lengths are [2000, 2500, 4000, 2800, 3800, 3200). After combining the length reward with correctness reward,

---

Published as a conference paper at ICLR 2026
So far, it looks like that all the designs work well since short correct answers have larger positive rewards and longer correct answers have smaller positive rewards while incorrect answers have zero rewards. However, when computing group relative advantage A(olg) in GRPO (i.e., Eqn. (2)), the advantage for the third correct response shifts from 1 (without length penalty) -0.17 (with length penalty added). This negative signal will discourage valid reasoning and create a significant optimization barrier, which may substantially harm performance.
the combined reward of each answer becomes [0.73, 0.6, 0.2,
4 DRPO: DECOUPLED REWARD POLICY OPTIMIZATION
The main reason of getting a negative learning signal in GRPO for a verbose correct answer that its reshaped reward could become less than the mean reward of all samples including positive and negative ones. Our approach to avoid this issue to decouple the rewarding of positive and negative samples that the length rewards are only normalized within the positive group. To this end, we develop our approach based on a recent work (Li et al. 2025a), which proposes a discriminative optimization framework (DisCO) that directly increases the generative likelihood of positive answers and decrease that of negative answers. Below, we first introduce DisCO and then present our novel approach of integrating length rewards into DisCO's objective.
4.1 DISCRIMINATIVE CONSTRAINED POLICY OPTIMIZATION (DISCO)
DisCO was proposed to address several inherent limitations of GRPO, including difficulty bias and clipping operations. Let so 9) be scoring function, which measures the generative likelihood of answer o given the input q. In this paper, we will consider so q) = -o Σ log To(oz (g oct), which is effective as demonstrated in (Li et al. 2025a). The objective of DisCO is formulated as:
(s
max Ø
Eyg o (1)
- log E (or-a(()))) exp
s.t. DKL (Tolal (T)) S 8,
where > is a hyper- hyper-parameter. The intuition behind this formulation is straightforward: increases the scores of positive responses sponses, aggregated through a log-sum-exp function. The log-sum-exp has its roots in discriminative learning, appearing in losses such as cross-entropy and contrastive loss, and naturally emphasizes hard negatives by assigning them larger learning signal. The constraint IDxL. (Toul) (T) S 6, inspired by TRPO (Schulman al. 2015), is added to ensure the stability of training.
(- while decreasing the scores of negative
While DisCO demonstrates impressive gains in reasoning performance over GRPO, the length of its reasoning is uncontrolled, leaving the challenge of enhancing reasoning efficiency unresolved. Moreover, the above objective is derived under a binary reward setting, which does not accept flexible reward design. In the following, we discuss how incorporate length rewards into the framework to encourage efficient reasoning.
4.2 DECOUPLED REWARD POLICY OPTIMIZATION
Jol for any correct response o, where C' is constant denoting maximum response length. An intuitive idea is to assign a weight to positive answers before their scores (p q) in (3) such that shorter answer is assigned with a larger weight than longer answer. Below, we formalize this idea by proposing a principled objective.
We consider simple length reward (o)
Our goal is maximize the score of correct outputs with high length rewards while penalizing those of wrong outputs regardless of their lengths. Suppose we have a distribution Pr which specifies a distribution of correct outputs with high length rewards given a question q. Then, we can modify the objective in B as
so(o' 9)
max Egg
Eo s0 g) log E (B exp
(4)
'These values are calculated with the formula proposed in (Arora & Zanette). We note that various reward combination designs in the literature lead to the same issue. Refer Appendix A.3 for detailed discussion.
5

---

Published as a conference paper at ICLR 2026
This be explained that if we have off-policy data distribution Pa of correct outputs with high length rewards, we can use its sampled data to steer the model training such that generates correct outputs with high length rewards more likely. However, the issue is that P is not readily available. Although a naive solution is to curate such data manually, as in SFT-based efficient reasoning methods, it requires substantial human effort and lacks scalability. To address this, draw insights from reinforcement learning from human feedback (RLHF) (Bai et al. 2002) Ouyang al. 2022 Ziegler et al. 2019) by finding an optimal policy lg that maximizes the length reward with a KL regularization:
P arg max E pri(o) - AD (P, Ja (-14))
(5)
PEP
where À > 0 is regularization parameter, P denotes the set of all probability measures / on correct
data given g, which are absolutely continuous with respect to (-19), i.e., Toid (olg) = 0 indicates P(o) 0.
However, unlike RLHF that uses a LLM to learn P", we derive its closed analytical solution similar to (Rafailov et al. 2023) (See Appendix A. for complete derivation):
Tala (olg) exp(r((0)/A)
(0)
(6)
E B exp(r:(o)/A)
As result, we have
(olg) exp(((())))
Eo so(o
o (1)) exp(ri(o)/A) so sa(o.g)
exp(r((0\AA)
E (olg)) E (1 exp(ri(o)A)) sp(o.g). q).
Plugging this formulation back into we obtain the final objective function:
onra (11) exp(ri(0)/A) so(o,9) - Flog ( (1)) exp (sa(d',g) T
exp(ri(0)/A)
max
Egg E Eown talle) (1)
s.t. DKL (T) < 8.
It is notable that the final objective only relies the on-policy data, and it has explanation that (x)(rr(o)/A) each positive data assigned with weight (olg) o exp(re(o]/A) informed by its length
but normalized only within the positive data. It notable that when +00, then w(olg) 1 and the above objective reduces to that of DisCO in (3)
We solve the optimization problem 7 similarly (Li al. 2025a). In particular, the expectations are replaced by empirical averages and the KL divergence is estimated by using sampled data, and
the constraint handled by adding a penalty function [ ( [ 6]2_ to the objective, where 3o is a penalty constant. For completeness, we present a full algorithm for solving (7) in Algorithm in the Appendix. We refer to this method as Decoupled Reward Policy Optimization (DRPO).
EXPERIMENTS
Datasets. We validate our method on mathematical reasoning tasks. Specifically, we train mod- els on the PeepScalRR-Preview-. question-ansee answer pairs sourced from AIME problems from 1984 2023, AMC problems before 2023, Omni-MATH (Gao et al., 2024) and Still (Min et al. 2024) datasets. We evaluate all the mod- els on math problems with different levels of difficulty, including (a) easy level: GSM8K (Cobbel cet al., 2021 (b) medium level: MATH-500 (Hendrycks et al. 2021), (c) hard level: Olympiad- Bench (He et al. 2024), and (d) very hard level: AIME (aggregating 2024 and 2025). To verify the generalizability of our method to non-mathematical reasoning tasks, also conducted experiments on K&K logic puzzle dataset (Xie et al. 2024), which is included in Appendix A..4
(Luo et al.) 2025c), which consists of approximately 40.3k
Models. We adopt three reasoning models our base models: DeepSeck-RR RI- (Distill-Qwen-i model. DeepSeek-RR MeaSSeek-R--Disttee --eeek-R--Distll--eeeeee Disttill-QQenn-BB, and DeepSeck-BR ---ekk------sill DDstilllLlamaal -BB, and conduct RL fine- tuning from them.

---

Published as a conference paper at ICLR 2026
Training Dynamics (1.5B)
Training Dynamics (7B)
6.45
1/7
5307
5.40
4302
Possigg
DAFO Siimal
paroo (oed 5)
2.00
Eiinng 1/we) 5)
2000
20
2000
E.000 (1-00 21
+52
4:33
1r62
Steps
Steps
Steps
Steps
Figure 2: Training dynamics of DRPO with different regularization weights X. The left two plots are for fine-tuning the 1.5B model, and the right two are for fine-tuning the 7B model. À to0 denotes the reference method DisCO, which does not incorporate length rewards in training.
Baselines. We compare our methods with six of the most recent state-of-the-ar efficient reason- ing methods, including (1) the method in (Arora & Zanette), which integrates length reshaped re- wards into the RLOO advantage function, referred to as RLOO-LP: (2) ALP (Xiang al.), which uses a length penalty GRPO that the length scaled by the solving rate of each question; (3) HAPO (Huang et al.),, which penalizes the responses longer than the shortest correct answer in the history while rewarding those that are shorter; (4) L1-max (Aggarwal & Welleck). which is a rea- soning language model that produces outputs satisfying maximum length constraint given in the prompt; (5) ShorterBetter (SB) (Yi et al.] 2025), which aims to match Sample Optimal Length de- fined as the shortest correct response among multiple generations; (6) LASER-D (Liu et al. 2025b), which employs a step length reward function based on difficulty- dynamic target length. We train models for methods (1)-(3) using the experimental settings described below. For methods (4)-(6), we evaluate the models provided in their original works. All the compared models were finetuned from the same base models on the DeepScaleR-Prevics dataset, except LI-max, which was trained on DeepScaleR-1.511 1eepScaleR-1.5---Peeiee Preview. We summarize different reward designs of the above baselines in Table 2 in Appendix.
Training Details. For all the training, we employ the Adam optimizer with weight decay of 0.01 and set the learning rate to a constant 2e 6 for 1.5B model, 1e for 7B model, and 5e -7 for 8B model, following Li et al. (2025a). We set the batch size to 128 for each step of RL, the mini-batch size to 32 for each iteration of model update, and sample 8 responses per question for training. For RLOO-LP, tune their weight parameter weight 3 E (le-M, 1e- le le-").
f0.05, 0.1, 0.2). For ALP, tune their penalty For HAPO, we tune their weight parameter w E {0.01, 0.1, 1]. For the proposed method, we tune À {0.5, 0.2, 0.13. These parameters serve the same role that controls the tradeoff' between efficiency and accuracy. For all other hyperparameters. we follow the default values from their official papers. Details are provided in Appendix A.2 The generation budget is limited to 8k tokens for both training and evaluation.
Evaluation. We use @ass@ averaged over the 16 generated answers per prompt as the performance metric and use the averaged number of tokens as the length metric. For all methods, we train the model for 1000 RL steps enable convergence and conduct evaluation every 200 steps. The models with the best pass@1 @1 are reported, as we aim to enhance reasoning efficiency with minimal performance reduction. For models that are trained by us, we set temperature 0.6 and 0.955, consistent with the training setup. For LI-MAX and LASER-D, we also use temperature = 0.6 and top-p = 0.95, while we use temperature 0.9 and top-p = 0.9 for SB, following the original paper.
In addition to pass@ @ and reasoning length, following (Luo al. 2025a Yi et al. 2025) also adopt Accuracy Efficiency Score (AES) as a supplementary metric. AES integrates performance and reasoning length into single measure, directly quantifying the trade-off between accuracy and computational cost. The AES is computed by:
(V Alnggh + 3 AAcel
if A 2 0, Length 7 A Accels if AAc < 0.
AES
where a, 8, 7 > Lenggh Length -Length and AAce Accpota where the quantities with Length
Accwt
subscript "ref" means each method's 's baseline that does not consider length reward, and that with subscript "model" means the model for evaluation. In experiments, use the default values a = 1, 3 = 3 same as (Luo et al., 2025a), but set y = 10 to emphasize the importance minimizing performance degradation.
7

---

Published as a conference paper at ICLR 2026
1.58 Model
78 Model
88 Model
0.49
Pessia (loe.
Perspp2 (Aree
0.33
AL3
ORPO
0.4-
0.90
LiceN
0.35
2006
1240
2003
3530
3300
3556
044
2030
1340
2003
2000
3003
0.43
3530
2006
2533
2000
3500
Longthh
Length
Length
trade-off. Left is for fine-tuning 1.5B model, mid- dle is for fine-tuning 7B model and right is for fine-tuning 8B model. Grey lines represent the base model performance before finetuning, with generation length of 4698 for 1.5B model, 4119 for 7B model, and 4325 for 8B model. Squares denote models trained with reference methods without length penalties, i.c., x=+00 (corresponding to DisCO) for DRPO, = 0 for RLOO-LP, 3 0 (corresponding to GRPO) for ALP, w for HAPO. Triangle, star, and rhombus markers represent models trained by other works.
Figure 3: Comparison of performance--efiiicnee
5.1 VISUALIZATION OF LEARNING PROCESS
To directly verify the effectiveness of the proposed method, we present training dynamics of DRPO with different regularization weights À in Figure 2 where λ=+00 corresponds the reference method DisCO, which does not employ length reward during training. In terms of performance (first and third figures in Figure 2)
that DRPO with smaller À values exhibits marginally worse or comparable performance compared with DisCO, while the second and fourth figures show that smaller À values lead to substantial reductions in response length, with À = 0.1 reducing length by over 50% relative to >=+x (DisCO). These observations demonstrate the effectiveness of DRPO achieve more concise reasoning while maintaining nearly unchanged training performance. In the following section, we will evaluate the generalization of DRPO to test datasets, compared with other strong efficient reasoning baselines.
5.2 COMPARISON WITH BASELINES
In this part, we evaluate the effectiveness of the proposed method on test datasets, compared with existing efficient reasoning baselines.
Trade-off between performance and efficiency. We present the trade-off between performance and efficiency for various methods in Figure where the averaged pass@ 1 over four math datasets of different difficulty is reported. We observe that our proposed DRPO consistently achieves sig- nificantly better performance-efficiencyy trade-off than all baselines on finetuning 1.5B, 7B and 8B models, including the models trained by other work. Notably, relative to the reference learning method (square marker), the proposed DRPO finetuning 7B model in Figure 3 (Middle) effi- ciently reduces reasoning length from 3053 to 1502 (51% length reduction) with only 2.6% loss of performance via varying À control the tradeoff, demonstrating the effectiveness of DRPO duce reasoning length while preserving the reasoning capability. In contrast, all the efficient reason- ing baselines suffer from severe performance degradation when the reasoning length is reduced. For example, RLOO-LP reduces reasoning length from 2975 to 1841 (38% length reduction) but incurs 7.1 % loss in performance on finetuning 7B model, with ALP showing a similar trend. Compared with DRPO, these methods trade off more performance for less reduction in length, highlighting the superiority of our method.
Evaluating Trade-off via AES. Additionally, we directly quantify the effectiveness of our proposed method in enhancing reasoning efficiency with minimal performance drop, using the Accuracy- Efficiency Score (AES). AES is positive when the model reduces output length while maintaining or enhancing accuracy, and negative when accuracy deteriorates. For fair comparison, reference model in AES is each method's counterpart without length reward, i.e., d=+oo for DRPO, = 0 for RLOO-LP, 3 = for ALP, w = for HAPO. We present the best AES score for each method in Table I and defer detailed results in Appendix A.6 From Table I observe that almost all the baseline methods exhibit negative AES scores across 1.5B, 7B, and 8B models, indicating the inefficiency of existing methods in reducing reasoning length while preserving performance. In
8

---

Published as a conference paper at ICLR 2026
Table Accuracy Efficiency Score (AES) Comparison with Baselines. The best AES score for each method is presented.
Method
Pass@1
Length
AES
1.5B Model
RLOO-LP
0.567
2531
-0.129
ALP
0.606
3494
-0.387
HAPO
0.534
1791
-0.519
DRPO
0.624
1527
0.178
7B Model
RLOO-LP
0.692
2649
-0.033
ALP
0.679
2170
-0.134
HАРO
0.596
1717
-1.080
DRPO
0.714
1502
0.249
8B Model
RLOO-LP
0.619
2249
0.251
ALP
0.637
2358
-0.01
HAPO
0.595
1981
-0.366
DRPO
0.690
1758
0.297
GSMBK
MATH500
OlympladBench
AIME
0.50
0.35
0.80
0.85
0.25
DR70
0.80
0.40
0.20
0.70
ALP
RFDO-LF
0.75
5.15
NARO
0.30
0.10
500
1000
1500
1030
1300
2000
2300
2000
3000
4000
3000
4000
5000
6000
Length
Length
Length
Length
(a) Model
GSMBK
MATH500
OlympiadBench
AJME
0.50
0.50
Pass/01
0.30
0.85
0.55
0.45
0.55
0.40
0.75
0.50
DSPO
ALP
RLOO4
0.80
0.75
0.50
M6PO
0.70
0.70
250
500
750
1000
500
1000
1500
2000
2000
1003
3000
4000 Length
5000
Length
Length
Length
(b) Model
GSMBK
MATH500
OlympiadBench
AIME
0.45
0.9
0.30
Pa5s:91
0.85
0.53
0.50
3.50
DXPO
RLCOLP
0.75
2.45
ALP
5.40
HAP0
0.55
0.35
0.20
500
1000
1500
1000
1590
2000
2500
2000
5000
4000
5000
4000
5000
5000
Length
Length
Length
Length
(c) 8B Model
tradeoff individual datasets with increasing difficulty levels from left to right. (a) is for finetuning 1.5B model, (b) is for finetuning 7B model, (c) is for finetuning 8B model. Squares denote models trained with reference methods without length penalties (i.e., d=+00 (corresponding to DisCO) for DRPO, a = 0 for RIOOOLPP 3 = 0 (corresponding to GRPO) for ALP, = 0 for HAPO).
Figure 4: Performance-efficin
contrast, DRPO consistently achieves a positive AES score for finetuning 1.5B, 7B, and 8B models, highlighting its capability of improving reasoning efficiency while maintaining performance.

---

Published as a conference paper at ICLR 2026
5.3 LENGTH REDUCTION FOR DIFFERENT PROBLEM DIFFICULTY
In this part, we study the impact of length reduction on questions with varying difficulties. In Figure we present the performance of different methods across four math datasets of increasing difficulty. from GSM8K (easy) to AIME (very hard). As shown in Figure 4 we observe that: (1) on relatively easy questions like GSM8K, DRPO significantly reduces generation length from 1563 to 356 (77.2% length reduction) for 1.5B model and from 969 to 261 (73.1% 1% length reduction) for 7B model with negligible performance drop (-1.1% for 1.5B model and -0.6% for 7B model). In contrast, all the baselines still bring dramatic performance degradation. indicated by steep slope. (2) With question difficulty increasing from left to right in Figure 4 all the models exhibit longer reasoning length and introduce increasing performance drop, which is reasonable since difficult problems inherently require longer reasoning paths to solve. (3) When comparing Figure 4a) and (b), with two models from the same LLM family, we can see that 7B models generally demand shorter reasoning length than 1.5B models. Besides, models with larger sizes, e.g. 7B and 8B models, more resilient length reduction introduced by DRPO, evidenced by flatter slopes than 1.5B models. (4) Across various difficulty levels, DRPO consistently achieves a better trade-off' between performance and length than all the baselines, demonstrating its effectiveness in guiding efficient reasoning.
In Appendix A.5 we conducted an ablation study on the effect of different length reward designs in DRPO, comparing (1) linear length reward. (2) concave length reward and (3) cosine length reward. The results are summarized in Figure 6 which shows that the linear reward provides the broadest trade-off spectrum, while the concave and cosine rewards tend to yield higher accuracy at the cost of longer reasoning lengths.
We conduct further case studies in Appendix A.7 to analyze the efficient reasoning behavior after DRPO training, compared with DisCO. It is shown that models trained by DRPO preserve existing reflection capability while effectively reducing redundant back-and-forth reasoning.
6 CONCLUSION AND DISCUSSIONS
In this work, we revealed key limitation of existing RL-based efficient reasoning methods, that their reliance on group-relative advantages can mislead learning when length rewards are included. To address this, we proposed DRPO, which decouples learning signals of correct and incorrect answers so that length penalties reduce the positive signals for correct reasoning but never reverse them. Our method integrates an optimized positive data distribution under KI. regularization into discrimi- native objective, for which we derived closed-form solution, enabling efficient computation using only on-policy data with importance weighting. Experiments on math reasoning tasks demonstrate significant superiority of our approach, compared with existing efficient reasoning methods.
While DRPO is designed for efficient reasoning, its formulation is general and can be extended to incorporate other rewards on positive data beyond length, such as process rewards or other prefer- ence rewards. Extending the proposed approach to broader reasoning tasks remains interesting direction. Another interesting direction is to incorporate mechanisms to adapt À to the difficulty level of questions, larger X for more difficult questions and smaller À for easier questions implied by our results in Figure Al which we leave for future investigation.
REFERENCES
Pranjal Aggarwal and Sean Welleck. LI: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning. URL//ttp://arxxv.orog absrgg/bb//55000 04697
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pictquin, Ahmet Üstün, and Sara Hooker. Back basics: Revisiting reinforce style optimization for learn- ing from human feedback in llms. arXiv preprint arXiv: /4740, 2024.
Daman Arora and Andrea Zanette. Training Language Models to Reason Efficiently. URI. URI.http: //axxiv.org/abs//// 04463
Simon A Aytes, Jinheon Back, and Sung Ju Hwang. thotch-of-thought: Efficient Ilm reasoning with adaptive cognitive-inspired sketching. arXiv preprint arXiv:2503.05/79, 2025.
10

---

Published as a conference paper at ICLR 2026
Lei Bai, Zhongrui Cai, Yuhang Cao, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, et al. Intern-sl: A scientific multimodal foundation model. arXiv preprint arXiv:2508. 2025.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michat Podstawski, Lukas Giani- nazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoe- fler. Graph of thoughts: solving elaborate problems with large language models. In Proceedings of the Thirty-Eightt AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Ad- vances in Artificial Intelligence, AAATT 57735-887-9, doi: v38 6. 29720
AAAI Press, 2024. ISBN 978-1- URL btRtstts::// wwtp://....../.o.././/./
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hul- lender. Learning to rank using gradient descent. In Proceedings of the 22nd international confer- ence on Machine learning, pp. 89-96, 2005.
Zhe Cao, Tao Qin, Tie Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach listwise approach. In Proceedings of the 24th international conference on Machine learning, pp.
2007.
Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-ml: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506. 2025.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, al. Do not think that much for 2+ 3=? on the overthinking of ol-like llms. arXiv preprint arXiv:2412.21188 2024.
Zhoujun Cheng, Richard Fan, Shibo Hao, Taylor W. Killian, Haonan Li, Suqi Sun, Hector Ren, Alexander Moreno, Daqian Zhang, Tianjun Zhong, Yuxin Xiong, Yuanzhe Hu, Yutao Xie, Xudong Han, Yuqi Wang, Varad Pimpalkhute, Yonghao Zhuang, Aaryamonvikram Singh, Xuezhi Liang, Anze Xie, Jianshu She, Desai Fan, Chengqian Gao, Liqun Ma. Mikhail Yurochkin, John Maggs, Xuezhe Ma, Guowei He, Zhiting Hu, Zhengzhong Liu, and Eric P. Xing. K2-think: A parameter-efficient reasoning system, 2025. URI. 07604
org. /0tt:///w...///////0.....
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14102 2021.
Yingqian Cui, Pengfei He, Jingying Zeng, Hui Liu, Xianfeng Tang, Zhenwei Dai, Yan Han, Chen Luo, Jing Huang, Zhen Li, al. Stepwise perplexity-gui thought reasoning in large language models. arXiv preprint arXiv:2502. 13260, 2025.
refinement for efficient chain-of-
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Thinkless: Llm learns when to think. arXiv preprint orXiv:250S.13379 2025.
Bofei Gao. Feifan Song, Zhe Yang. Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv22410.07985, 2024.

---

Published as a conference paper at ICLR 2026
Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Yifu Lu, Mengdi Wang, Dinesh Manocha, Furong Huang, Mohammad Ghavamzadeh, and Amrit Singh Bedi. Does think- ing more always help? understanding test-time scaling in reasoning models. arXiv preprint arXiv:2506.042 042000 10, 2025.
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Mare Dymet- man. Aligning language models with preferences through f-divergence minimization. arXiv preprint arXiv:23022.02215, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token- budget-aware IIm reasoning. arXiv preprini arXiv:24/2.18547, 2024.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xi- aoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tian- wen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner series.
hetpesr/ccerrrcooooseevveeeeeeer aictiooo..s.. site/ hatp::/c....aaaaa...eooooooaaaaa
2025. Notion Blog.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint 1rXiv:2103.03874 2021.
Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang- of LLMs via Reinforcement Learning. URL
ThinkPrune: Pruning Long Chain-of-
httt:://axxia org/ wwtgg/ab////// 2254 .1225044112966
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03266 2025.
Chengyu Huang, Zhengxin Zhang, and Claire Cardie. HAPO: Training Language Models to Reason
Concisely via History-Awar Policy Optimization. URL http::/aax.v..rr//bb////////////
11225
Gang Li, Ming Lin, Tomer Galanti, Zhengzhong Tu, and Tianbao Yang. Disco: Reinforcing large reasoning models with discriminative constrained optimization. arXiv preprint arXiv:2505.12366 2025a.
Zheng Li, Qingxiu Dong, Jingyuan Ma, Di Zhang, and Zhifang Sui. Selfbudgeter: Adaptive token allocation for efficient ml reasoning. arXiv preprint arXiv:2505. 11244, 2025b.
Yong Lin, Shange Tang, Bohan Lyu, Ziran Yang. Jui-Hui Chung, Haoyu Zhao, Lai Jiang, Yihan Geng, Jiawei Ge. Jingruo Sun, Jiayun Wu, Jini Gesi, Ximing Lu, David Acuna, Kaiyu Yang, Hongzhou Lin, Yejin Choi, Danqi Chen, Sanjeev Arora, and Chi Jin. Goedel--rover-v2: Scaling formal theorem proving with scaffolded data synthesis and self-correction, 2025. https:
7/arxiv. org. abs 2508 03613
Kaiyuan Liu, Chen Shen, Zhanwei Zhang, Junjie Liu, Xiaosong Yuan, et al. Efficient reasoning through suppression of self-affirmation reflections in large reasoning models. arXiv preprint arXiv:2506.. /2353, 2025a.
Wei Liu, Ruochen Zhou, Yiyun Deng, Yuzhen Huang, Junteng Liu, Yuntian Deng, Yizhe Zhang, and Junxian He. Learn to reason efficiently with adaptive length-based reward shaping. arXiv preprint arXiv:2505.1 /56/2, 2025b.
12

---

Published as a conference paper at ICLR 2026
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-Prumer: Length- Hurmonizing Tine-n for O1-Like Reasoning Pruning. URL http::ttp/aaat /ttp://aavvvvv/am org/ absa 2501. 12570
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-pruner: Length-harmonizing fine-tuning for ol-like reasoning pruning. arXiv preprini MXXi::55011157000 2025a.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol -peeview with a 1.5b model by scaling rl, 2025b. Notion Blog.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffirey Luo, Tianjun Zhang. Li Erran Li, et al. Deepscaler: Surpassing ol ol-preview with a 1.5 b model by scaling rl. Notion Blog, 2025c.
Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao. Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv preprint arXiv:2502.0678. 2025.
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report on alow-thinking reasoning systems. arXiv preprint arXiv:2412.09411. 2024.
Tergel Munkhbat, Namgyu Ho, Seo Hyun Kim, Yongin Yang, Yujin Kim, and Se- Young Yun. Self- training elicits concise reasoning in large language models. arXiv preprint 2011 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 36:53728-5377 2023.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1889-1897. PMLR, 2015.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Xuerui Su. Shufang Xie, Guoqing Liu, Yingee Xia, Renqian Luo, Peiran Jin, Zhiming Ma, Yue Wang, Zun Wang, and Yuting Liu. Trust region preference approximation: A simple and stable reinforcement learning algorithm for Ilm reasoning. arXiv preprint arXiv:2504.0455 2025.
Yang Sui, Yu-Neng Chuang. Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503. 2025.
5 Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie. Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, Bowen Xu, Can Huang. Casey Zhao, Changpeng Cai, Chao Yu, Chen Li, Chendi Ge. Chenghua Huang, Chenhui Zhang, Chenxi Xu, Chenzheng Zhu, Chuang Li, Congfeng Yin, Daoyan Lin, Dayong Yang, Dazhi Jiang, Ding Ai, Erle Zhu, Fei Wang. Gengzheng Pan, Guo Wang, Hailong Sun, Haitao Li, Haiyang Li, Haiyi Hu, Hanyu Zhang, Hao Peng. Hao Tai, Haoke Zhang, Haoran Wang, Haoyu Yang, He Liu, He Zhao, Hongwei Liu, Hongxi Yan. Huan Liu, Hui- long Chen, Ji Li, Jiajing Zhao, Jiamin Ren, Jian Jiao, Jiani Zhao, Jianyang Yan, Jiaqi Wang, Jiayi
13

---

Published as a conference paper at ICLR 2026
Gui, Jiayue Zhao, Jie Liu, Jijie Li, Jing Li, Jing Lu, Jingsen Wang, Jingwei Yuan, Jingxuan Li, Jingzhao Du, Jinhua Du, Jinxin Liu, Junkai Zhi, Junli Gao, Ke Wang. Lekang Yang, Liang Xu, Lin Fan. Lindong Wu, Lintao Ding, Lu Wang, Man Zhang. Minghao Li, Minghuan Xu, Mingming Zhao, Mingshu Zhai, Pengfan Du, Qian Dong, Shangde Lei, Shangqing Tu, Shangtong Yang, Shaoyou Lu, Shijie Li, Shuang Li, Shuang-Li, Shuxun Yang, Sibo Yi, Tianshu Yu, Wei Tian, Weihan Wang, Wenbo Yu, Weng Lam Tam, Wenjie Liang, Wentao Liu, Xiao Wang, Xiaohan Jia, Xiaotao Gu, Xiaoying Ling, Xin Wang, Xing Fan, Xingru Pan, Xinyuan Zhang, Xinze Zhang, Xiuqing Fu, Xunkai Zhang, Yabo Xu, Yandong Wu, Yida Lu, Yidong Wang, Yilin Zhou, Yiming Pan, Ying Zhang, Yingli Wang. Yingru Li, Yinpei Su, Yipeng Geng, Yitong Zhu, Yongkun Yang, Yuhang Li, Yuhao Wu, Yujiang Li, Yunan Liu, Yunqing Wang. Yuntao Li, Yuxuan Zhang, Zezhen Liu, Zhen Yang. Zhengda Zhou, Zhongpei Qiao, Zhuoer Feng, Zhuorui Liu, Zichen Zhang, Zi- han Wang, Zijun Yao, Zikang Wang, Ziqiang Liu, Ziwei Chai, Zixuan Li, Zuodong Zhao, Wen- guang Chen, Jidong Zhai, Bin Xu, Minlie Huang, Hongming Wang. Juanzi Li, Yuxiao Dong, and Jie Tang. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025a. URL aittps://arx.s http://///aaiii.oo////////////// ottg:b//////// 06471
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501. 2025b.
Chenlong Wang, Yuanning Feng. Dongping Chen, Zhaoyang Chu, Ranjay Krishna, and Tianyi Zhou. Wait, we don't need to" wait"! removing thinking tokens improves reasoning efficiency. 2025a.
arXiv preprint arXiv:2506.08844
Yibo Wang, Li Shen, Huanjin Yao, Tiansheng Huang, Rui Liu, Naiqiang Tan, Jiaxing Huang, Kai Zhang, and Dacheng Tao. RI RI-compress: Long chain-of-thought compression via chunk compres- sion and search. arXiv preprint 250. 2025b.
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. Thoughts are all over the place: On the underthinking of ol-like lmss. arXiv preprint arXiv:2501. 188888, 2025c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. th-hout prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He. Qi An. Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lx, et al. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:2503. 10460, 2025.
Heming Xia, Chak Tou Leong, Wenjie Wang, Yongqi Li, and Wenjie Li. Tokenskip: Controllable chain-of-thought compression in 1lm. arXiv preprint arXiv:2502. 12067, 2025.
Violet Xiang, Chase Blagden, Rafael Rafailov, Nathan Lile, Sang Truong, Chelsea Finn, and Nick Haber. Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning. UtL httttp:://axiv.orggg. abs 2506.55066..0 05256
Yang Xiao, Jiashuo Wang, Ruifeng Yuan, Chunpu Xu, Kaishuai Xu, Wenjie Li, and Pengfei Liu. Limopro: Reasoning refinement for efficient and effective test-time scaling. arXiv preprint 2rXiv:2505.19187 2025.
Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. arXiv preprint Xiiv:24100231233 2024.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong. Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14768 2025.
Yuhui Xu, Hanze Dong. Lei Wang, Doyen Sahoo, Junnan Li, and Caiming Xiong. Scalable chain of thoughts via elastic reasoning. arXiv preprint arXiv:2505.05//5, 2025.
14

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang. Wenbiao Yin. Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025a. URL https: arxiv. ozq. ab./
C9388
Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Qiaowei Li, Zheng Lin, Li Cao, and Weiping Wang. Dynamic early exit in reasoning models. arXiv preprint arXiv:2504.15895 2025b.
Tianbao Yang and Yiming Ying. Auc maximization in the era of big data and ai: A survey. ACM computing surveys, 55(8):1-37, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In seirty- Conference on Neural Information Processing Systems, 2023. URL https:
//openzeview nwt. form?? ddxcmmme1xxxccexxx111
Jingyang Yi, Jiazheng Wang, and Sida Li. Shorterbetter: Guiding reasoning models to find optimal inference length for efficient reasoning. arXiv preprint arXiv:2504.21370, 2025.
Xixian Yong, Xiao Zhou, Yingying Zhang, Jinlin Li, Yefeng Zheng, and Xian Wu. Think or not? exploring thinking efficiency in large reasoning models an information-theone lens. arXiv preprint wrXivv2505.18235 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Hang Yuan, Bin Yu, Haotian Li, Shijum Yang, Christina Dan Wang, Zhou Yu, Xueyin Xu, Weizhen Qi, and Kai Chen. Not all tokens are what you need in thinking. arXiv preprint arXiv:2505. 17827, 2025.
Linan Yue, Yichao Du, Yizhi Wang, Weibo Gao, Fangzhou Yao, Li Wang, Ye Liu, Ziyu Xu, Qi Liu, Shimin Di, et al. Don't overthink it: A survey of efficient rl rl-style large reasoning models. arXiv preprint arXiv:2508.02111
2025.
Chong Zhang, Yue Deng, Xiang Lin, Bin Wang, Dianwen Ng, Hai Ye, Xingxuan Li, Yao Xiao, Zhanfeng Mo, Qi Zhang, al. 100 days after deepseck-rl: A survey replication studies and more directions for reasoning language models. arXiv preprint arXiv:2505.0055// 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071 2025.
Ren Zhuang, Ben Wang, and Shuifa Sun. Accelerating chain-of-thought reasoning: When goal- gradient importance meets dynamic skipping. arXiv preprint arXiv:2505.08339. 2025.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv 2019.
preprint
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DERIVING THE OPTIMUM OF THE KI-CONSTRAINED REWARD MAXIMIZATION OBJECTIVE
In this part, we derive the optimal policy Pg that maximizes the length reward with KL regular- ization:
max E Pri(o) XIDxL (P. (-19), PEP
where À > 0 is a regularization parameter, ro( denotes length reward of response P denotes set of all probability measures on correct data given q. which are absolutely continuous with respect to (-1)) i.e., -iaa (olg) = 0 indicates P(o) 0.
Following prior work Rafailov et al. 2023);
;: Go et al. (2023), we have
max E Pri(o) xL (P, ( (11))
PEP
= max (o)pr(() AR log P(olg)
PEP
Told (olg)
P(o[
-max BowN ri(() Alog
PEP
oold
(0
P(olg)
min BowP log
r1(0)/A)
PEP
"old (
P(olg)
PEP min Bo-P llog
Ø(q) Toua (olg) exp(ri(o)/A) log Z(q))
(9)
where Z(g) = E. T (olg) exp(ri(o)/A) is partition function, which doesn't depend on P. Let's first detine P(olg) )o exp(ri(o)/A). Since P(olg) > 0 for all and
Σ. P(olg) = 1, P(olg) is a a valid probability distribution. Thus, we can reformulate 9 as:
P(o]g)
min Bo-p log - PEP
Z(q) Tola (olg) exp(r(()/A)) - log Z(q)
-in PEP
(P, log Z(4)
(10)
Since Zg)) doesn't depend on P, the minimum 10 achieved by minimizing the first KL term. With Gibbs' inequality that KL-divergence is minimized at O if and only if the two distributions are identical. Therefore, we have the optimal solution:
(0) P(olg) Z(q) Tola (olg) exp(ri(o)\))
(11)
t (ol()) (x)(2) E Em cxp(r(()//A)
(12)
A.2 DETAILED HYPERPARAMETER SETTING
In this section, we provide detailed hyperparameter settings used in our experiments.
For all the methods, we employ the AdamW optimizer with weight decay of 0.01 and set constant learning rate 2e for 1.5B model and 1e for 7B model, following Li et al. (2025a). We set the batch size to 128 for each step of RL, the mini-batch size to 32 for each iteration of model update, and sample 8 responses per question for training. The generation budget is limited to 8k tokens for both training and evaluation. The temperature is set to 0.6 for training.
RLOO advantage estimator and clip ratio 0.2. We tune their weight parameter & E {0.05, 0.1, 0.2). Following their paper, we normalize its loss by the length of the response.
For RLOO-LP,
16

---

Published as a conference paper at ICLR 2026
Table 2: Various reward designs for efficient reasoning, where re(olg) 1l(o is correct) E {1, denotes correctness reward.
Method
r(a.g) r.fo, -a a ralo, + of li massi walar (oua))1))
RLOO-LP
Arora & Zanetto
Xinng et al
Pe(o.g) B [o] max (r.(9, K-))
HAPO
Huang et al.
refo.g) I w max costming 3 le
Mii mi), rafo.
min confmin( & nig o)),0(1 rc(a.2))
it
LI-MAX
Aggarwal & Welleck)
rufo. g) clip(aiZy - [ol] 6).0.
SB Yi al. 2025
a ralo.g) - [8 abu (lo Lssolti).
where (ooz(o) min[|o]],ra60o. 9) - 13 if lenst one response is comret, mean( [a.]] ocherwise
LASER-D D Liu al. 20256
ro oflle] s La)
Table 3: Illustration of negative learning signal of correct outputs of existing reward designs.
Mothad
Hyperparamter
Length
Cureetrae
Reward
Autoantage
RL.0-LP
1200, 2200,
0.55, 0.8,3
[0:25, 0.21.
2000, 6400,
1200)
D.
ALP
3-0,000
[1500. 1200, 1900, 2200,
L.
[Q8K, 08, 0,85, 0,82, 0.78,
[0:74, 0,65, 0:46, 0.32, -,99,
2000, 6-000,
1200)
0.
0.71, 0.1.
HAPO
u-l, 7,777 201200
[1500. 1280, 1900, 2200, 7800:
I,
[062, 1, 03300,3, 0.3.
[6:99, 2.28 -0.1, -0.1.
2000, 5600, 6400), 1800, 12000
1, 1,0.0
0,,003,03,, 40,,333,1113 -0.]
20.1,0.1, 1, 21.13]
L.-MAX
o=0000
o=00000,7y5400000 f:0.5
[1 500. 1700, 1900, 2000, 7800,
1,
1...1.11.0000 0.86 1, 0.00
[0.8, 0.8. 0.8, 0.S. 0.65.
2000 56000, 6400) 1800, 12000
1, 1, 0..0
0.8, -0.06 -1.45, -1,48, -1.48)
SB
0-2, 040,00
[1500L 1200. 1900, 2200, 2800,
[1. 1, L,
[12.22,133, 1. DA,,, 1.2. 0.4, 3.2 0.1.
0.992, 1.14, 0.64, 0.43, 0.01.
2000, 36000 64000, 1300, 1200)
3,
0.57, 0.56, 2.53, 0.35, 0.24)
LASER-D
0-0.5 1,00000
[1500, 1200, 1900, 3200, 7800,
[1. 1.
[0.59, 0,59,
0.59.
2000 9600), 6400, 1900, 1.000)
1, 0.
15.15.1,,,00)
0.59,0.80 4.9.0.99..0 -1.94
For ALP, we follow their paper to use GRPO advantage estimator and normalize its loss by the total number of tokens. The KL Coefficient is set to 0.001. We tune their penalty weight B E {le-9 le 8 1e- and and clip ratio e 0.2
For HAPO, we follow their paper to use GRPO advantage estimator, set the cutoff c = -077, KL Coefficient to 0, and clip ratio
0.2. The loss normalized by the length of the response. We tune their weight parameter w E (0.01,0.1,14
For the proposed DRPO, we follow Li et al. (2025a) to set constraint value 8 14-4, penalty constant 30 = 1e T = 10. we tune regularization parameter X E {0.5, 0.2, 0.1}.
A.3 LIMITATION OF INCORPORATING LENGTH REWARD WITH GROUP ADVANTAGE
In this part, we summarize different reward designs of existing baselines, which incorporate length reward to encourage efficient reasoning, in Table 2 To illustrate the inherent limitation of incorporating length reward with group advantage, in Table 3 we provide detailed examples of how these reward designs fail to work with group advantage, resulting in mislcading learn- ing signals. Specifically, we follow the hyperparameters used in their paper to calculate the ad- vantage with RLOO advantage estimator for RLOO-LP method and GRPO advantage estima- tor (i.e., Eqn. 22) for other methods. RLOO advantage estimator is calculated as A(oslg) = r((gl(())))))))) (q). As indicated by red values in Ta- bleB all reward designs produce varying amounts of mislcading learning signals. We see that HAPO suffers most, yiclding incorrect learning directions in 6 out of 10 cases. This could help explain why HAPO exhibits larger performance degradation than other baselines in our experiments.
r(o;/g) mean(r(oi Iu),... r(o--
A.4 EXPERIMENTS ON NON- WEN--MMTHEAAICA REASONING TASK
To evaluate DRPO's generalization experiments on the logic puzzle reasoning task. Following Xie et al. (2025); Suu et al. (2025), the training dataset is limited to 3 to 7-person logic puzzles with K&K logic puzzle dataset (Xie et al.) 2024) and the test dataset contains 2 to 8-person puzzles. We train 1.5B models for all methods for 400 steps and conduct evaluation every 80 steps. As shown in Figure 5 DRPO method still exhibits a substantially better trade-off than all other baselines, demonstrating the generalizability of the proposed method other tasks. Notably, DRPO significantly reduces the generation length
non-mathematical reasoning tasks, we conducted additional
17

---

Published as a conference paper at ICLR 2026
1.5B Model
1.0
A=
A=0.07
-=0.2
Pass@1 (Ave.)
0.9
0=0.2
8-1e-8,
a=0.2
8=
B=1e-7
0.7
W=0.1
0=0.01
RLDO-LP
w-1
HAPO
1500
2000
2500
3000
Length
Figure 5: Comparison of performance-efficieney trade-off on logical puzzle reasoning task. Squares denote models trained with reference methods without length penalties, i.e., λ=+oo (corresponding to DisCO) for DRPO, a = for RLOO-LP, 3 (corresponding to GRPO) for ALP, w = 0 for HАРO. Triangles denote the models trained by other works.
0.66
Length Reward
Pass (Ave.)
0.64
0.6
0.62
0.4
Linear
DRPO+Linea DRPO+Concave
0.2
Concave
0.58
Cosine
DRPO+Cosine
0.0
0.56
0.0
0.2
0.4
0.6
0.8
1.0
1500
1750
2000
2250
2500
2750
3000 3250
3500
Jo//C
Length
Figure 6: Ablation studies different reward designs. Left is the length reward values with respect lol/C; Right the serformance-efficieee trade-offs for different reward designs on mathematical reasoning task. Square denotes the model trained without length penalties, i.e., A=+0o (DisCO).
from 2095 1400 (33.2% length reduction) without any performance drop (from an accuracy of 0.972 to 0.974).
A.5 ABLATION STUDY LENGTH REWARD DESIGN
C where C is stant denoting maximum response length. To study the effect of different length reward designs in DRPO, conduct experiments mathmatical reasoning task with (1) concave length reward (i.e..7(o) = 1 (19) )²) and (2) cosine length reward (i.e., 0.5+0.5 cos( (g))), using a 1.5B model. For all reward design choices, tune X in {0.5, 0.2, 0.1). The results are summarized in Figure 6 We observe that all reward designs achieve competitive performance. The linear reward provides the broadest trade-off spectrum, while the concave and cosine rewards tend to yield higher accuracy at the cost of longer reasoning lengths. These findings suggest that developing more sophisticated length rewards is promising direction for further improving DRPO.
In the main experiments, adopted simple length reward ri(o) 1 lo|
18

---

Published as a conference paper at ICLR 2026
Table 4: Detailed AES performance for 1.5B models.
Method
GSM8K
MATH500
OlympiadBench
AIME
Ave.
RLOO-LP (a=0.1)
0.078
0.021
-0.423
-0.738
-0.172
RLOO-LP (a=0.2)
0.246
0.325
-0.279
-0.694
-0.129
RLOO-LP (a=0.4)
0.026
-0.13
-0.454
-1.18
-0.412
ALP (3=1e-9)
0.06
0.016
-0.603
-1.571
-0.387
ALP (3=1e-8)
-0.189
-0.088
-0.91
-2.229
-0.602
ALP (3=le-7)
-1.019
-0.518
-1.887
-3.774
-1.4
HAPO (w=0.01)
-0.209
-0.348
-0.879
0.202
-0.519
HAPO (w=0.1)
-0.74
-0.811
-1.066
-0.457
-0.969
HAPO (w=1)
-1.05
-1.127
-0.841
-0.182
-1.063
DRPO (A=0.5)
0.296
0.21
0.152
-0.143
0.093
DRPO (A=0.2)
0.614
0.296
0.184
-0.597
0.164
DRPO (A=0.1)
0.662
0.332
0.098
-0.729
0.178
Table Detailed AES performance for 7B models.
Method
GSM8K
MATH500
OlympiadBench
AIME
Ave.
RLOO-LP (a=0.1)
0.391
0.23
-0.051
-0.532
-0.033
RLOO-LP (a=0.2)
0.211
0.284
-0.046
-0.829
-0.122
RLOO-LP (a=0.4)
0.045
-0.04
-0.103
-0.709
-0.331
ALP (8=le-9)
0.451
0.265
-0.297
-0.984
-0.134
ALP ff1e--)
0.161
-0.351
-1.048
-1.798
-0.68
ALP (5=le-7)
-0.923
-1.211
-1.745
22679
-1.573
HAPO (w=0.01)
-0.105
11.01
-1.318
-16776
-1.08
HAPO (w=0.1)
-0,999
-1.433
-1.252
-11997
-1.42
HAPO (w=1)
-1,483
-1.407
-1.344
-1.503
-1.6
DRPO (A=0.5)
0.053
0.106
-0.126
-0.155
-0.007
DRPO (A=0.2)
0.439
0.303
0.015
-0.33
0.115
DRPO (A=0.1)
0.672
0.514
0.254
-0.455
0.249
A.6 DETAILED AES PERFORMANCE
In this part, we present detailed AES performance for each method in Table 5 and where bold values denote the best AES performance for each method. We observe that all baseline methods yield negative AES scores for almost all settings, underscoring their inefficiency in preserving per- formance while reducing reasoning length. In contrast, DRPO consistently achieves positive AES scores for most cases, demonstrating its effectiveness in improving reasoning efficiency without sacrificing performance.
A.7 CASE STUDY
We analyze the reasoning path of DRPO, compared with DisCO, which corresponds to DRPO with A +x. Figure shows the reasoning paths on an easy prompt, where DRPO reaches the correct answer with clear reasoning in only 89 tokens, achieving 6x reduction compared to the 526 tokens required by DisCO. Although DisCO also produces the correct answer, its reasoning is highly dundant, including four occurrences of reflection words (c.g., wait) and three repeated answers. By contrast, DRPO's reasoning involves only single reflection and one correct answer.
We further evaluate the reasoning efficiency on a more challenging prompt in Figure X and Figure As shown in Figure 8 DisCO requires 4497 tokens, about 10 times longer than the 455 tokens of DRPO, to solve the question. Moreover, the reasoning of DisCO contains 78 reflection words and 19 answers, six of which incorrect. In comparison, DRPO produces only correct answers and
19

---

Published as a conference paper at ICLR 2026
Table Detailed AES performance for 8B models.
Method
GSM8K
MATH500
OlympiadBench
AIME
Ave.
RLOO-LP (a=0.1)
0.378
0.198
0.282
0.295
0.212
RLOO-LP (a=0.2)
0.515
0.203
0.313
0.315
0.251
RLOO-LP (a=0.4)
0.064
-0.126
0.127
0.392
-0.119
ALP (3=le-9)
0.476
0.215
0.053
-0.67
-0.01
ALP 3=1e-8)
0.22
-0.252
-0.686
-2.405
-0.584
ALP (3=le-7)
-2.139
-1.773
-1.81
-3.733
-2.289
HAPO (w=0.01)
-0.021
-0.17
-0.251
-0.755
-0.413
HAPO (w=0.1)
0.197
-0.264
-0.445
-0.673
-0.366
HAPO (w=1)
-0.23
-0.667
-0.542
-1.579
-0.775
DRPO (A=0.5)
0.29
0.172
0,098
0.091
0.122
DRPO (A=0.2)
0.607
0,435
0.313
0.068
0.296
DRPO (A=0.1)
0.624
0.449
0.292
0.04
0.297
Algorithm Decoupled Reward Policy Optimization (DRPO)
1: Input: Initial policy model Too reward function question set D, hyperparameter 6, X.
2: Policy model TT = To
do 4: Sample a batch of questions B from
3: for Step 1,
5: Update the old policy model Fold my
6: For each question E sample responses {o ~ (ala-)1)) denoted by S, and partition
it into S and S based on correctness rewards r(:igg) E {0,
7: for minibatch Bm E B do
8:
Compute KL divergence estimator by
log Te (oel..cc)))))
(KK(() Excem Locs,
Σ
9E oES, aml
9:
Compute gradient estimator of the objective in Eqn. by
G1
Σ
Σ
exp(r(()/A) Zotest exp(ri(0')A))
Vso(o, ((T log
Σ exp( sold'-1)
B
BEBm
(oES,
d'ES,
10:
Compute gradient estimator of penalty function of the constraint by G2 23r (0) 8]4 vl)
11:
Update Ty with Adam-W using the gradient estimator G = G + G2
end for
13: end for
five reflection words as seen in Figure 9 It is worth noticing that reasoning from DisCO explicitly says that 'but I'm not 100% sure' before concluding final answer. It suggests that excessive back-and-forth in the reasoning, indicated by frequent use of reflection words, may result in less confident answers.
LLM USAGE
We use LLM only to help correct grammar emrors and polish writing.
20

---

Published as a conference paper at ICLR 2026
Prompt
A robe talkes 2 bolts blue fiber that much white fiber. How many bolts total does take? Let's think by step and output the final answer within Mboxed[].
Answer from DisCO (DRPO A +oo) 526 tokens
Answer from DRPO 0.1)
Okay,
problem
Okay, so
figure
many
bolts of blue fiber and half that much white fiber.
robe takes. says the robe takes bolts of blue fiber and half that much white fiber. Wail, so blue bolts, white total is +1=3 bolts, That seems right. </thinks>
need to figure out how many bolts in total Hmm,
this down step by step.
blue,
First, the problem says the robe uses bolts of That's straightforward. Now, it also takes half that much white fiber. So, need to find out how much white fiber is that.
The total number of bolts needed is
$1bowed(3j$
Let write this out. Blue fiber is botts. White fiberia is half tft So, can calculate the white fiber as:
White fiber (1/2) Blue fiber White fiber (1/2)* 2
Him,, let me do that that math. Half of is white fiber bott.
Now, the total number of bolts is the surn of blue white
would
Total Total botts Blue fiber White fiber bolts
Which equals 3. So, the total number of bolts needed
Wait, let me just make sure I didd''t skip any steps any mistalees. fiber is half of that, divided is indeed Adding them together, + 3. Yeah, that seems right.
don't think need to cansider any other factors here, Like mixing them something. The problem just says takes 2 boits blue and half that much white. So, blue and white, totaling
Wait, another way think about half that much white is bolt, then blue plus whits is total, Yeah, that's correct.
don't think need to overcomplicate this, It's pretty straightforward. So, the total number of bolts is
**Final Answer**
The number of bolts needed is \bowed(3),
</think>
The bolts blue fiber. The white fiber required is half of that, which is:
Whiit fiber) (frac(1]2) (2) \times 2 \text! bott)
\]
Adding the blue and white fibers together:
Total botts) -2 3
The total number of bolts needed is \boked(3),
Figure 7:
Example reasoning for Prompt from DisCo (DRPO = +0o) and DRPO (A = 0.1). Words with green color are answers, and with blue color are reflection words. DRPO reaches the correct answer with clear reasoning in only 89 tokens, achieving a 6x reduction compared to the 526 tokens required by DisCO.
21

---

Published as a conference paper at ICLR 2026
Prompt 2
Jash decides to try tlipping a house. He buys a house for $80,000 and then puts in $60,000 in repairs. This increased the value of the house by 150%. How much profit did he make? Let's think step by step and output the final answer within Wooked]).
Answer from DisCO (DRPO with A + 4497 tokens
Oleey, losh bought house 580,000. Then he spent $10, 0000000
Atter that, the value of the house increased by 150%,. need steps Fine, the orggina the house was $90 000, He 55n onn 5130,000
fgum out how much profit he made., Lee me brenk this down
increase purshase price? the house 150%," So, ino sase 160% the orignnal house piice. Wall, no. original value 880,000, but he's stready invosted $130,200. Walt, maybe Ihhuld clantyy problem says bought the house for $50,000, then spent $50,000 on ropalis, 20 total invesoment is 11300,0000 20,0000 Then, value the house incresseed by 159%s think 150% of that. Wall, let's represent this muthematically.. Let me denase: Original cost $80, 000 Repains: $50,000 Total cost $80,000 + $50, $1 so, ono The vnlun increased by 150% Sn, the incrusss is 150% $ 1150, 2007000 Or is 150% of the on original house price? Wists the prociem sespe "increseed the valiae of the house by 150%, Su, the incressw is 150% of the original valua belons repairs? Wall, no. It's probablyy the incressse 1500s of the total meveiment. Welt, lat's med the problem agin: 'lush buys a house for $80,000 and that puts in 550 $50,200 in reswirs. This inc moreased the vatue of the house by 150% Hew mush proftt did he make?? So. he spant 580k and then 500s, so total S1 1.K Then the value incrossed 155%,
repalis, so inveitmnnnt becomes: $90.000 jpurchaset+ 1,0,000
but the problem
value 100% So, nned caloulate the vatus house atter the increase. Himm, the increase 160% investment repaits?
the
again:
incressw is 150% top the total invostmment. Bi, the new value
total investmeett plux
the viluee 150k 150% of 1200? War. that can't be. West, maytw the incross 1500
sommthing slow. Whil,, partaga it' bettee to think that the intresse value 150% of the migiinal purchese price. Well, no Wist, the probblem maybe total increased value 100% So, the
price plus 150%, the orig $80k, then the nowvollue $50k- 150% SoOk? Walt, would BBck 680K BBOK $120k $2001. But spoms high, Walt, porhaps inorease 150% the socal invostment.
intreased volue the
150%,""
Walt, the probiem
So, total invostment was $1301. It increased by 150%, so increase 1.5 120k S18Bk. So, the new volue 81
$195k 6325k.
clarily: Originall price: 880,000 Repairs: $50,000, so total investment: 81 30,000 After repains, the value increased by 1504. maybel noed to 150% the total investment.
Thon, total proftt would $325k 580k (originoli SS0K (rpaliss) $226k 1100k $185k, But soems too high.
increase
increase is 130,000 196,000 So, the new value 130,000 195,000 525, 225,000. Then, his total cutfiow was 80, 000 50,000 - 150,000, Hix tonal inflow in 395, .on. So. his profit 325, 0000000000000 130, 000000000 - 195, 200 Wer. but that seems flo a loe. Allurasiindl, maybe the incressse is 150% on top of the original purchsoe price. Si. origi ad purchee was $50,000, incrrssoed by 150% would be 50,000 *1 1.5 - 120,00 So, the new valur is 80 000 + 120,000 200,000 But he alsuady put in 50k, so his total cuiflow is 150k. Sa. profit 2006 - 1300k 70k, Weit, this is contlctingg Whst let's read the problem statement again: losh buys a house for 500 000 and then puts make?? So, Vint formalize: Let the volue 50k rppairs total outloy: 130k. value incressed 150%, V 160% of
$60.000
incroased value
150% How proftt
$80.000
Ropair: $60,000 Volue increased 100% So, the
150% totol after
repairs, He bought for B0k,
newrvalue 160% than the orig purchase price? Walt, the probiem says "tho increased the value the house 160% the original value acfore repairs was B0k, then after repairs, 80k 100% B0K? Walt, that would be Value after repairs 50k (1.5 5010 50K 120k- 2006 50k on repair,, ap total cutlay 60k
Wan, think need model differently The value the
150%
hac avrady spent 1300 profii, Wari, but the problem saya "hii incrvased
130k.
total join
the value of house by 160% So, the increase is 150% the ginal value? Or 150% of current value? Walt, perhapa the incresse
160% top the purchase price, so: Original purchose: B0k Ropairs: 50k So, tosal outlay: 130k Value after repairs: 130k (150% 1801) 190k 196k- 5254 Well, but that swemss too high. A Alseraliveyy maybe the incresse is 150% of the or glrna walun, an 50k 1.5 128k. So the raw valuee is 150k (after repar)) + 120k - 250k? Vasi, I'm gatling confused. Whll, lat's by another apoeowch, Lat me denata V us the new value after repairs. The incresse is 150% of somatting ing The probiem saya "tha incressed the value of the house by 150% So. the increase is 1595s of the original house price, which is M0k. So, the incresse in 1.5-B0e-1P00 So, the new valian in Bok 17m But iosh slready put in 50t on ropaiis, his total outloy is +5kk--50k-300 10k his
2009 180k-70k Walt, that's the same bofore. Wait, another prtaptetaiion: value 120k 195k-325kk.
The increase 100%
His investment 130k,
increase 1,5-120x-1900 1966,
the original value? Or of its curnent value? Wanlt the problem is ambigguuu that
Phonit tn 325k 1200c 186k, Wait,
problem
"thie incressed
150%) that the rotal
value up by 150%
Lat' this
in 150%
the original purchess pricw, So, original oriow is Bik, incresss 150% of 100- so new valus is +120k-000k Then, his ouillyy vacc s0k+-0lu- 1806, su proft 2009 1100 Allarnatival. the incrussu 150% of the current valus. Cument value afler repairs 50k- BOK. Bo, increase is 150% of 190k-1955, new value would 1906 1951-825k Profi would be 525k -100k 195k. Wat, but the problem says "his incressed the value the house by 150%. So, the value incroased by 150%, meaning that the new value is 150% more than the origindd Whit, the original value is d0t, so filk 1150% of 801-B0k +120k-0800. Welt that's w woother way. So the intresse is 150% of the criginsi valua, not the curaen valaa, So. in this case, the new value is 50k 1500% of 200- Has total ouiley vas B0k +50k 15kk. So, profit 2006 305--701, Wait, out the problem orye "increased the value of the house by 150% So, perhope the volun went up by 150%, meaning the new oolun is 1001e 11000-2250% of the original, Whin that can't be because the original was Iilk, and he spent 500 on masins Wast, I'm gatting targled hame. Wert, mayhe iet's model it stop by stap: 1. Insh higs the to for Stilk. He than spants $50t on mppirn, so total ieparddiuue 8 130k. The value of house incressee 100%, so the increase 150% of orighna purchase price? or the tonsl expenditurs? Wait. the prockem saye "increased the value
100% So, the incresse 160% current
War, think 2009. 700 Wall, the problem says "Increased
Or the orginall
prozkeem that the increase 1.5%% of 90c-500-1801 So, his
purchase
So, the
150% $t0k, which $1 $120k. So, the
s0k- 1120k-200k His expenditure the house by 150%." So.
oridinal value was B0K, then 150% of that 120k, so the now value B0k +120k-200k. So, profit is 200k 1800-700k Walt, another way: Maybe incressu is 150%
total excendiure, So, 150% of B130k $1951, so new value 130k B55k-825K. Profis is 325k B3ke Walt, that soems high. think the probiem is that the increase 160% of the origina. purchase price, so 150% of B0ken 120k making the new value B0k 120k Honoo, proftt 2009 180k 701. Wall, but the problem says Increased the value of the house by 100% So, the new value is 100% of the onginal. Walt, no. Walt, mayoe the value went up by 150%, meaning the new valur is 100% - 10% 2200% of the original So. 250% of 801- 2001. came as betors, Henoe prote 2003 1000e Ti, Walt, but 150% increase on the velun after repairs rn. Witti, this gerting too tangled Parhspp, the cornect aporosch in Lee V be the value afher repairs. The increase 18000,, an e 1.8V - total outlay?? Wart nn Velll, the total nutley was 80k 150b-1 13kk The walun whter repaice intr by 150%, no V- 155% of Ve originat? Waill, I'm not getting this Whil, partapa the protiam is that the valun incressedd by 150% of the original purchess price. So the increase 100% of 12k- check stops: Buy 2009 1801 Hence, made profi of $70,000. Walt, but original
value is 20k-200kk Hence, prott 200k 1000-701 Wart that seems reasonable. Let BKk is 150% 150% of onig valus, 150% of 12k- So, new value 80c +1 200-2000. Henos, profis is 2008 -00--701. Henos, answeer $70, 970.000. Wail, but me check once more. Original purchase: B0k Repair:: 50K, total: 130k, Value incressed by 150%: So, the volue becomes 190k 80% of 180 13k- 825k War, that's another way. Wall, so the increase is 150% of the current saluo, which la 160%% 130km B5K, so new volue +1 1255-1125 Hemow profit is 3254- 50
Resein: 50k, spent 10kk. Volue
80 1201,
-1200-2000k
the onig
value. Wort, the problem says incressed thevelue
by 150%.
Walt but which interprnaiion on is conmett? The problem anys "Incwwaedd the veluoe the house by 100% So, It's ambiguous whether it's 150% of the original or 150% ohehe curent.
Wan, the 180% of the original, ih's 120k incressn new velue 200k prate 70k, ihit' 180% of the current it's 185k increase, new valuae 325k, profit 108k. Glven that the probise anys "this inconned the valus of the hmuse by 150% mink ihe 1500 of the original purchees price, So. 1800 of 10k- new whls 200k, Hence, profit bo -1 301-77i, Wall,, but wnlll, another thought: The "ieceaoon" is 150% of the totall 130k - 323k-326k proftt 325k 300e- 188k. 1hhnk need maybe probiom that the mncressd 150% the ginal purch prics, which BdK 1.5 120k, value 80k+ 120k-200k, Hence, but the proskem statoment. think that's the 1bo0ed)70000)
investment, Wait, total investment was 120k, 150% that 1861, new suttle
200k (80k-55kk-200 700. Hence, anower 870,000, but I'm 100% "Final Answer**
<tthink Josh bouaght a house the new value is 80 poo e 10% 10,0000 - 200,000. The profi ix catrolatte se the mov vallue minus the tonall investment:: 300,000- 130, .0000000 - 20,000. Trus, Insh made a profit Woooe///000)).
and spent $50, 000 repairs, socating 000. The value of the house increased by 160%, moaning
Figure& 8: Example reasoning for Prompt 2 from DisCo (DRPO A = +oo). Words in green/red color are correct and incorrect answers, respectively. Blue words are reflection words. The reasoning of DisCO contains 78 reflection words and 19 answers, six of which are incorrect.
22

---

Published as a conference paper at ICLR 2026
Prompt2
puts $50, repairs. increased the value the house by 150%. How much profit did he make? Let's think step by step and
decides try Ilipping a house. buys
output the final answer within Wboked|).
Answer prompt from DRPO
455 tokens
Okay, Josh buys house for $80, and spends another 150,000 repairs. So total is 80k +50k-1
Then the house's s value increased by 150%, Wait, increased by 150% means it's now 100% 150% the original value.
Wsit, original post was 880,000, after repairs, the total investment $130,000, right? Whit,, maybe I should clarify: when they say the house's value increased by 150%, based original purchase price total cost?
Hmm, usually, such problems consider increase based the cost basis. So the increased by 150%, that's 150% of the original purchase price, which was $80,000.
So 150% of BOK 1.5*80k-120k. So the value went up 120K, nelvalue 80k- 120k-200k.
Wail, spent repairs. 130k=70k.
profit
Wsit,
Purchase price:80k
Repairs:50k
Vatue increased by150% so 150% of 90k-120K, so new -0
He sold for 200k, proftt-200k-130k-700
sa profit $70,000.
</think>
prafit $70,000.
The answer 1boxed(70000).
Example reasoning for Prompt 2 from DRPO (A 0.1). Words in green/red color are correct and incorrect answers, respectively. Blue words are reflection words. DisCO uses 4497 tokens, about 10 times longer than the 455 tokens DRPO.
Figure 9:
23


Reviews:
[{"review_id": "XjRYRPE7WU", "reviewer": "Reviewer_tZfR", "summary": "This paper focuses on the \"overthinking\" problem in Large Reasoning Models. Existing reinforcement learning-based solutions (e.g., GRPO) generally lead to performance degradation, which the paper attributes to the fact that current methods incorporating length rewards set the advantage value of correct but verbose reasoning to negative due to GRPO's group-relative advantage function, resulting in inverse optimization. To address this issue, the paper proposes the Decoupled Reward Policy Optimization (DRPO) framework, which is core-based on the DisCO discriminative optimization framework. It decouples the learning signals of correct and incorrect reasoning, normalizes length rewards only within the positive sample group, and integrates the optimized positive data distribution that maximizes length rewards under KL regularization, enabling efficient computation using only on-policy data. The paper validates the framework on multiple mathematical reasoning tasks: the 1.5B model achieves a 77% reduction in length with only a 1.1% performance loss on GSM8k, while the 7B model achieves a 51% reduction in length with only a 2.6% performance loss. These results are significantly superior to baseline methods, with a positive Efficiency-Accuracy Score.", "questions": "- Does the author consider some operations on the length optimization for the negative sample? At present, there is only the part for the positive sample in the design of the paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The method design is elegant. By decoupling the learning signals of correct and incorrect reasoning and normalizing length rewards only within the group of correct reasoning samples, it ensures that the rewards for correct but verbose reasoning are only proportionally reduced without becoming negative. The derivation of the optimal positive data distribution is particularly elegant.\n- The performance is significant. The experiments demonstrate excellent performance across datasets of varying difficulty levels, achieving significant reductions in output length while maintaining nearly unchanged performance, outperforming baselines.", "weaknesses": "- The premise for DRPO to work is that positive and negative samples can be optimized separately, which strongly relies on the DISCO algorithm. This somewhat weakens the contributions of this paper.\n- Missing references. It is recommended that the authors add discussions on the following works [1-4] in the 'large reasoning models' part of related work.\n- The presentation of this paper is not very user-friendly, and some parts are hard to follow.\n\n[1] Lyu C, Gao S, Gu Y, et al. Exploring the limit of outcome reward for learning mathematical reasoning[J]. arXiv preprint arXiv:2502.06781, 2025.\n\n[2] Zheng C, Liu S, Li M, et al. Group sequence policy optimization[J]. arXiv preprint arXiv:2507.18071, 2025.\n\n[3] Chen A, Li A, Gong B, et al. MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention[J]. arXiv preprint arXiv:2506.13585, 2025.\n\n[4] Bai L, Cai Z, Cao Y, et al. Intern-s1: A scientific multimodal foundation model[J]. arXiv preprint arXiv:2508.15763, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Iig7dG9FDZ", "reviewer": "Reviewer_ziRG", "summary": "The paper proposes DRPO, a reinforcement learning framework to improve reasoning efficiency. It identifies a flaw in GRPO where length penalties distort advantage estimation and introduces a decoupled reward formulation that separates positive and negative samples. Experiments on multiple reasoning benchmarks show that DRPO significantly shortens reasoning chains with minimal accuracy loss.", "questions": "1. Could the decoupling idea generalize to other auxiliary rewards?\n2. Could λ be learned dynamically during training (e.g., based on question difficulty or model confidence) rather than fixed manually?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Identifies and formalizes a core issue in GRPO under multi-objective rewards.\n2. Proposes a principled decoupling mechanism with a closed-form optimization solution.\n3. Demonstrates substantial efficiency gains with competitive accuracy across multiple benchmarks.", "weaknesses": "1. Evaluation focuses only on math reasoning; unclear applicability to other domains (e.g., coding, logic).\n2. The evaluation relies mainly on **Pass@1**, which can be unstable for reasoning tasks; additional metrics (e.g., Pass@k) would strengthen the empirical validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JNBTbIgJnM", "reviewer": "Reviewer_Z7wp", "summary": "This paper proposes a novel RL objective to enhance reasoning efficiency in LLMs by encouraging shorter reasoning paths. The authors observe that applying a length penalty reward may cause long but correct responses to receive negative advantages, inadvertently promoting unnecessarily long reasoning trajectories. To address this issue, they propose a decoupled advantage computation that separately handles positive and negative responses, ensuring positive advantages for high-quality responses. Empirical studies on DeepSeek-R1-Distill-Qwen-1.5B and 7B models across multiple mathematical reasoning benchmarks verify the effectiveness of the proposed approach in reducing reasoning length while maintaining accuracy.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and clearly organized.\n\n2. The analysis of positive responses receiving negative advantages is insightful and leads naturally to the proposed DRPO formulation.\n\n3. The proposed DRPO method outperforms other efficient reasoning approaches by a substantial margin.", "weaknesses": "1. The authors claim that redundant reasoning arises from positive responses with negative advantages. However, even the vanilla GRPO without a length penalty still exhibits redundant reasoning, despite all positive responses receiving positive advantages. More in-depth analysis is needed to substantiate the claim that “positive responses with negative advantages greatly harm reasoning efficiency.”\n\n\n2. The experiments lack a comparison against the vanilla GRPO baseline. In the abstract, the authors mention a 1.1% performance drop of DRPO compared to the baseline. If DRPO exhibits a larger accuracy decline than GRPO, it would weaken the claimed significance of the proposed efficiency improvement.\n\n\n3. In my opinion, measuring reasoning efficiency solely by token count is not comprehensive. An efficient response should be logical, step-by-step, and free from redundant reasoning, rather than simply shorter in length.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "16kYjG5Nvv", "reviewer": "Reviewer_Wot1", "summary": "This paper proposes DRPO. \nDRPO tackles overthinking in large reasoning models by decoupling length-based rewards for correct answers from incorrect ones, preventing GRPO’s group-relative advantage from misclassifying verbose correct solutions as negative.\nIt introduces a KL-regularized, closed-form reweighting of on-policy correct samples that favors concise reasoning without ever flipping their learning signal to negative.\nOn four mathematical benchmarks, DRPO trims output length up to 77 % while losing only ~1 % accuracy, outperforming six strong baselines that sacrifice 4–7 % for smaller reductions.", "questions": "* How does the performance of DRPO compare to the original DisCo algorithm? Adding relevant comparisons in Figure 3 would facilitate better comparison.\n* Is DRPO still effective for step-wise rewards (i.e., process-level reward)?\n* Could DRPO generalize to different families of LLMs?\n* As mentioned in weakness, how DRPO performs on non-mathematical reasoning tasks?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The paper identifies a subtle but critical flaw in GRPO’s group-relative advantage when length penalties are added, and proposes the first principled solution that decouples the normalization of correct and incorrect samples, a conceptual advance that is both novel and generalizable to other composite rewards.\n* Extensive comparisons on four standard math-reasoning datasets show DRPO consistently dominates six recent baselines across 1.5 B and 7 B scales; ablations, AES metrics, and per-difficulty breakdowns confirm that the gains hold under rigorous evaluation, while a closed-form derivation guarantees computational efficiency without extra data.\n* Good writing and easy to follow", "weaknesses": "* All experiments stop at 7 B parameters and four math domains; without data on >=30 B models or non-mathematical reasoning (code, science, commonsense).\n* The length reward $r_l(o)=1−\\frac{|o|}{C}$ is essentially linear; ablations with concave, step-wise or difficulty-aware penalties (e.g. LASER-D) are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)