PEAR: Phase Entropy Aware Reward for Efficient Reasoning
Authors: Chen Huang
Wei Lu
Wenxuan Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Reasoning Models
Large Language Models
Efficient Reasoning
Primary area: generative models
TL;DR: We propose Phase Entropy Aware Reward (PEAR), which leverages phase-dependent entropy to adaptively shorten reasoning traces in Large Reasoning Models, reducing response length while preserving accuracy and demonstrating strong OOD robustness.

Abstract: Large Reasoning Models (LRMs) have achieved impressive performance on complex reasoning tasks by generating detailed chain-of-thought (CoT) explanations. However, these responses are often excessively long, containing redundant reasoning steps that inflate inference cost and reduce usability. Controlling the length of generated reasoning without sacrificing accuracy remains an open challenge. 
Through a systematic empirical analysis, we reveal a consistent positive correlation between model entropy and response length at different reasoning stages across diverse LRMs: the thinking phase exhibits higher entropy, reflecting exploratory behavior of longer responses, while the final answer phase shows lower entropy, indicating a more deterministic solution. This observation suggests that entropy at different reasoning stages can serve as a control knob for balancing conciseness and performance. Based on this insight, this paper introduces Phase Entropy Aware Reward (PEAR), a reward mechanism that incorporating phase-dependent entropy into the reward design. Instead of treating all tokens uniformly, PEAR penalize excessive entropy during the thinking phase and allowing moderate exploration at the final answer phase, which encourages models to generate concise reasoning traces that retain sufficient flexibility to solve the task correctly. This enables adaptive control of response length without relying on explicit length targets or rigid truncation rules. 
Extensive experiments across six benchmarks demonstrate that PEAR consistently reduces response length while sustaining competitive accuracy across model scales. In addition, PEAR demonstrates strong out-of-distribution (OOD) robustness beyond the training distribution.Our code is available at: https://github.com/iNLP-Lab/PEAR.

Published as a conference paper at ICLR 2026
PEAR: PHASE ENTROPY AWARE REWARD FOR EFFI- CIENT REASONING
Chen Huang' Wei Lu² Wenxuan Zhang! 'Singapore University of Technology and Design NNayyang Technological University chen huannguungmmam il. sutd. edu.sg sg wei. www.1uunntueeee edu..g wxzhangesutd. edw. .g
ABSTRACT
Large Reasoning Models (LRMs) have achieved impressive performance on com- plex reasoning tasks by generating detailed chain-of-thought (CoT) explanations. However, these responses are often excessively long, containing redundant reason- ing steps that inflate inference cost and reduce usability. Controlling the length of generated reasoning without sacrificing accuracy remains an open challenge. Through a systematic empirical analysis,
reveal a consistent positive corre- lation between model entropy and response length at different reasoning stages across diverse LRMs: the thinking phase exhibits higher entropy, reflecting ex- ploratory behavior of longer responses, while the final answer phase shows lower entropy, indicating a more deterministic solution. This observation suggests that entropy at different reasoning stages can serve as control knob for balancing conciseness and performance. Based on this insight, this paper introduces Phase Entropy Awar Reward (PEAR), reward mechanism that incorporates phase- dependent entropy into the reward design. Instead treating all tokens uni- formly, PEAR penalize excessive entropy during the thinking phase, allowing moderate exploration at the final answer phase, which encourages models to gen- erate concise reasoning traces that retain sufficient flexibility solve the task cor- rectly. This enables adaptive control of response length without relying on explicit length targets or rigid truncation rules. Extensive experiments across six bench- marks demonstrate that PEAR consistently reduces response length while sustain- ing competitive accuracy across model scales. addition, PEAR demonstrates strong out-of -ifl--f-dittitttionn (OOD) robustness beyond the training distribution. Our code is available at: https:://gtt http:::///gthhb..////// thub. com.:oo.o/// NNP- Lab/pPaabbPPA
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities, particularly when employing techniques like Chain-of-Thought (COT) prompting (Wei et al. 2022). Building on this, recent Large Reasoning Models (LRMs) (Jaech et al.) 2024 Guo et al. 2025 Yang et al,,
2025a Team et al. 2025 Team 2025) encourage an explicit thinking phase via special tokens before generating the final answer, further improving models' complex problem-solving capability. However, LRMs tend to generate excessively long chain-of-thought responses (Chen et al. 2024
Yue et al. 2025), the models often produce redundant calculations or verbose explanations, which
leads bloated outputs and reduces inference efficiency (Hassid et al.) [025)] Kuo al.. 2025).
Consequently, key challenge is to enable models to think less while preserving the performance.
Recent works have attempted to address this issue by enforcing efficiency through further training on filtered concise data (Yue et al. 2025 Qu et al.) 2025 Sui et al.) (205)). The common paradigm to modify the training corpus so that the model is exposed primarily to shorter reasoning traces Yuaa et al. 2025 An al. 2025 Zhao et al. 2025b). By strictly constraining the supervision signal, the model often struggles to adapt to novel reasoning styles or out-oo-distribution (OOD) problems where the optimal length of reasoning may differ (Aggarwal & Welleck] 2025). Moreover, such methods risk discarding valuable intermediate reasoning that could improve accuracy. This motivates the need for more adaptive and model-driven approach efficient reasoning.

---

Published as a conference paper at ICLR 2026
LRMs
Thinking Phase
Final Answer Phase
<thinl>>
</thinks
Reasoning
Henink Exploratory
Hanswar Deterministic
Long responses with unnecessary steps
High Entropy
Entropy
Phase Entropy Aware Reward (PEAR)
Discourage Hthink
Encourage H answer
Eatropy
<think-
</think>
Reasening Path
Hehink
Hanswer
More concise response
Figure 1:
PEAR reduces the response length by penalizing excessive entropy during the thinking phase while allowing moderate exploration at the final answer phase.
Concurrently, there has been growing interest in understanding how token-level uncertainty. mea- sured by entropy, influences model behavior (Lei et al., 2025; Cheng et al.) 2025al Zhang et al.] 20256). Entropy captures the spread of the predictive distribution: high-entropy segments often correspond to exploratory reasoning steps where the model searches for a correct path, while low- entropy segments capture more deterministic computations or final answer generation (Wang al.) 2025c Zhang et al. 2025f). Therefore, recent works have begun to exploit these signals for im- proving calibration or enhancing reasoning robustness (Zhang et al., 2025c Wang et al., 20256). However, the connection between entropy and efficient reasoning has been largely overlooked.
Intuitively, model that operates at consistently high entropy may explore too broadly and thus pro- duce unnecessarily long reasoning chains, while model biased toward low entropy may commit earlier to a determined reasoning path with more concise outputs. Motivated by this hypothesis, we first conduct empirical analysis, and observe a consistent positive correlation between average token-level entropy and response length across model scales and benchmarks. Interestingly, this relationship is not uniform across reasoning stages: the "thinking" portion of the output exhibits substantially higher entropy than the "final answer" portion, highlighting distinct roles of explo- ration and commitment in different stages of reasoning. Moreover, when we filter out high entropy tokens, models' performance will not be affected within a certain ratio, suggesting that excessive entropy be pruned without harming reasoning quality. Based on these observations, Phase Entropy Aware Reward (PEAR), a reward mechanism that explicitly decomposes entropy into thinking and final answer phases and integrates both components into the training objective. As illustrated in Figure M by penalizing excessive entropy during the thinking phase while moderating entropy in the final answer phase, PEAR encourages models produce more concise reasoning traces, providing soft and adaptive mechanism for balancing exploration with efficiency.
propose
We evaluate PEAR on MATH500 (Hendrycks et al. 2021b), AIME24 et et al. 2024). AMC23 (Li al. 2024), GPQA Di- amond (Rein et al.| (024) and MMLU (Hendrycks al. 2021a). Across models of different scales, PEAR achieves substantial reductions in response length, ranging from 32.4% to 56.6%. while pre- serving accuracy with decreases of less than 1%. By incorporating both phases of a model's response into the reward calculation, PEAR eliminates the need for manual data curation and generalizes ef- fectively to -o-mann questions through its broadly applicable training objective.
widely used benchmarks: GSM8K (Cobbe et al. 2021
To summarize, work makes the following key contributions:
We empirically establish and validate positive correlation between model entropy and response length in LRMs, and show that the thinking phase exhibits substantially higher entropy than the final answer phase.
We introduce Phase Entropy Aware Reward (PEAR), reward mechanism that leverages this property to adaptively promote concise reasoning traces without depending on curated datasets explicit length constraints.
2

---

Published as a conference paper at ICLR 2026
Accuracy Scale
Thinking phase Entropy
DS-1.58
DS Series
0.35
phase Entropy
QwQ Series
QwQ-328
DSL-8B
Series
0.30
Accuracy: 64.4%
Entrog
0.35
DS-7B
Accuracy: 76.9%
Entropy
Qwen3-4 34B
DS-328
DS-148
Owen3-1.78
@wen3-88
Qwen3-328
Qwen3
0.25
3000
4000
5000
6000
7000
8000
9000
Tokens
Qwen3 Series Models
Figure 2: (a) Relationship between average entropy and response length across different models. The dot size indicates accuracy. DS(L) represents MipSeek-R1--isttee Ri---issill--wee/lllaae Qwen/Llama, (b) Comparison of average entropy between the thinking phase and the final answer phase.
We provide extensive experimental evidence GSM8K, MATH500, AIME24, AMC23, GPQA and MMLU, showing that our method achieves substantial reductions in re- sponse length while preserving accuracy, with strong generalization capability out-of- distribution tasks.
PRELIMINARY ANALYSIS
In this section, we present empirical observations that motivate our approach. We first examine the relationship between entropy and response length, showing how higher entropy is associated with longer reasoning traces. Next, we differentiate the roles of entropy in the thinking phase versus the final answer phase, highlighting distinct patterns across stages. Finally, conduct entropy- filtering experiments to demonstrate the robustness of low-entropy reasoning traces. All analyses performed GSM8K, MATH500, AIME24, and AMC23, where report average accuracy, response length (in tokens), and entropy.
2.1 ENTROPY AND RESPONSE LENGTH
We begin by analyzing the correlation between response entropy and length across diverse set of LRMs.. For each model, we measure the average entropy of the predictive distribution across all generated tokens and compare it against the total number of tokens produced during inference.
The entropy of the predictive distribution at each token position t is defined
M
H.
Σ
(0) Pt
log (1)
H
T
where vta denotes the predicted probability of token i at position I, |V| is the vocabulary size, T is the total response length, and is the average entropy across the entire response.
Figure 2 aaa) shows consistent positive correlation between average entropy and response length across all examined model families and benchmarks. Responses with higher entropy are typically longer and more exploratory. while lower entropy corresponds to shorter and more concise traces. This pattern is especially evident within individual model series, where models of different scales exhibit clear alignment between entropy levels and response characteristics.
These findings suggest that the entropy-length relationship is a fundamental property of large rea- soning models. Longer responses naturally reflect higher uncertainty diversity in token predic-
3

---

Published as a conference paper at ICLR 2026
tions, captured by increased entropy. This makes entropy interpretable internal signal for shaping model behavior. By integrating entropy into the reward design, we can provide models with principled mechanism balance thorough reasoning with efficient generation, enabling finer control over response length without relying on explicit constraints.
2.2 PHASE-DEPENDENT ENTROPY ANALYSIS
To further investigate the role of entropy in model responses, analyze how entropy is distributed across different stages of generation. As shown in Figure (bb), a clear distinction emerges between the thinking phase (before the </think> token) and the final answer phase (after the </think> token). The thinking phase exhibits consistently higher entropy, reflecting exploratory behavior as the model searches through multiple potential reasoning paths and generates longer, more diverse traces. In contrast, the final answer phase shows much lower entropy, indicating a more confident and deterministic commitment to specific solution. These results indicate that the two phases serve complementary functions of exploration versus conclusion and should therefore be optimized differently. Phase-specific reward mechanisms can leverage this distinction, reducing unnecessary exploration during reasoning while preserving confidence and clarity in final answers.
2.3 ENTROPY FILTERING EXPERIMENTS
To assess how high-entropy tokens influence model reasoning and whether pruning them im- pacts reasoning quality, we conduct a system- atic filtering experiment, shown in Figure Our procedure consists of two stages: first, we generate complete reasoning traces and com-
100
87.1%
10000
881.1%
-80
Average Tokens
8000
8360
69.3%
Accuracy (%))
6000
pute token-level entropy within the thinking phase. Second, we preserve only specified percentage of tokens with the lowest entropy and discard the remaining high-entropy tokens, resulting in filtered reasoning traces consist of only low-entropy tokens. These filtered traces then fed back to the model to produce final answers, enabling us to directly examine how entropy-based filtering influences both reason- ing efficiency and task accuracy. Results for more models can be found Appendix
3088
8001
20
100%
80%
60%
40%
20%
Filter Setting
Thinking
Final Answer
Accuracy
Figure 3: Accuracy and average response length in the entropy filtering experiments on Qwen3-4B.
B
When retaining 80% or 60% of low-entropy tokens, accuracy remains stable or even improves com- pared to the unfiltered baseline. This indicates that the high-entropy tokens being removed mainly drive excessive exploration rather than contributing to correct reasoning, and their absence reduces noise the reasoning process. Performance degradation only emerges under more aggressive fil- tering: retaining 40% or fewer low-entropy tokens leads to a sharp drop in accuracy, showing that essential reasoning steps are lost when the trace is compressed too heavily. Notably, the length of the final answer phase remains relatively unchanged across filtering levels, reinforcing that redun- dancy is concentrated in the thinking phase, where high-entropy tokens leads to over-elaboration and inflates response length without improving outcomes.
3 METHOD
3.1 GROUP RELATIVE POLICY OPTIMIZATION (GRPO)
We begin with brief introduction to the Group Relative Policy Optimization (GRPO) algorithm
Shao et al. 2024). Unlike standard PPO (Schulman et al. 2017). GRPO eliminates the need for a cnitic model by estimating advantages through reward normalization across group of sampled responses the same prompt. Specifically, for prompt y with G responses and corresponding
4

---

Published as a conference paper at ICLR 2026
rewards {r,),=1- the group group- normalized advantage is defined as:
ALt r3 mcan (fry
std({r,)]=1)
This normalization emphasizes the differences among candidate outputs for the same question, which improves the stability of the gradient signal even under sparse reward settings. GRPO also incorporates a KL divergence term that regularizes the learned policy against a reference policy. The overall surrogate objective be written as:
(G) (0) = PI
(3)
min (r,e(0)Ars, clip (r),,) -, +e 3Dxx medd
==
where
(0) To(oi, q. 0i,c))))
(4)
Tena(os, g, 0,0,,)))
and B hyperparameters, and DxL. denotes the KL divergence between the learned policy my and a reference policy Tef
3.2 PHASE ENTROPY AWARE REWARD (PEAR)
In the original GRPO algorithm, the reward r is typically defined a rule-based manner, assigning value of correct responses and 0 to incorrect ones. While simple and effective, this binary scheme overlooks richer characteristics of the response, such as the degree of cxploration or reflec- tion embedded in the reasoning trajectory. As result, it provides no guidance on how the model should balance exploratory reasoning with concise and reliable answer generation.
Building on the observed correlation between model entropy and response length in Section 2 introduce Phase Entropy Aware Reward (PEAR) that leverages entropy as guidance to train mod- els to reason more efficiently. Let sampled response the token sequence y = (y)..... yr) that contains thinking segment between <think> and </think> followed by the final answer. Let denote the index the closing token </think> in y. We compute token entropies with respect to the old policy T
H. - Temm( Vee)) log Te.. Vee),
t= 1,
T.
(5)
vEV
We then average entropies for the thinking phase and final answer phase (excluding the </think> token itself):
Hahink
H
Hanswer
H
T
k 1=k+1
The phase reward P integrates entropy from both the thinking and final answer phases, defined as:
P(y) max(0,
Hiiik - a H. Hanswrr)
The max(0..) ensures that the entropy- based term never becomes negative, keeping the combined reward within the canonical [0, 1] range used GRPO and avoiding pathological scaling of advan- tages. The policy update remains the same changes the per-sample scalar reward. Thus, the optimum under PEAR differs from standard GRPO only in preferring correct trajectories with lower excessive thinking-phase entropy where correctness remains the dominant signal, and the entropy term acts as a second-order preference among correct responses. The coefficient is tunable hyperparameter that adjusts the contribution of the final answer phase entropy, enabling flexible control over the balance between reasoning exploration and final answer confidence.
objective as in GRPO, PEAR only

---

Published as a conference paper at ICLR 2026
As discussed in Section 2.2 the reasoning process exhibits distinet entropy pattems: the thinking phase is characterized by higher entropy with exploratory behavior, while the final answer phase reflects lower entropy associated with deterministic solutions. To promote more efficient reasoning, we therefore aim to reduce entropy during the thinking phase to mitigate unnecessary exploration. Although the final answer phase already exhibits low entropy, the role of the anH term is not to encourage uncertainty in the answer. Instead, it makes the entropy penalty phase-relative rather than absolute. Subtracting final-answer entropy normalizes the thinking-phase entropy against the natural entropy level required to articulate a correct final answer. This prevents degenerate reward gaming through indiscriminate entropy collapse and ensures that the penalty activates primarily when thinking-phen entropy is disproportionately large.
Given a base score E (0, 1] for correct final answer and a format score Time E [0,1) for mal- formed/incorrect answers, the phase-aware entropy-inclusive reward for response is:
min(L 1, s - P(y)), if the extracted answer equals the ground truth,
r(y)
Trint:
otherwise.
Finally, replace To in Eq. 2 by rly)) and keep the same GRPO advantage normalization:
A r(y) mean ((r())))))))
Edge cases. If </think> token is absent we set k = T and use Hpust = 0 (i.c.. only thinking phase entropy contributes); if the answer cannot be parsed, we assign r(y) = rimt- This fallback does not cause such responses to systematically receive higher rewards. PEAR applies entropy- based shaping only when the extracted final answer correct. In practice, responses missing a closing </think> token are overwhelmingly malformed or incomplete and therefore fall into the incorrect category, receiving the fixed reward Fint independent entropy.
With PEAR, the model is guided not only by final answer correctness but also by the quality of its reasoning behavior. The component for the thinking phase discourages excessive exploration, high-entropy reasoning yields lower reward, thereby encouraging the model generate more focused and efficient reasoning traces. Meanwhile, the component for the final answer phase helps stabilize and structure the concluding steps, ensuring that the model produces complete and coherent answers without sacrificing accuracy.
4 RESULTS
4.1 EXPERIMENT SETTING
Baseline Methods. GRPO (Group Relative Policy Optimization) (Shao et al. 2024 is a rein-
forcement learning framework that eliminates the need for critic model by estimating advantages through reward normalization within a group of responses to the same prompt. Step Entropy (Li al. 2025) adopts a two-stage training strategy that enables LLMs generate compressed chain-of- thought (CoT) reasoning at inference time by strategically inserting [SKIP] tokens. LCPO (Length- Controlled Policy Optimization) (Aggarwal Welleck 2023) reinforcement learning method designed to jointly optimize for accuracy and compliance with user-specified length constraints.
Baseline Models. We evaluate our method on widely used LRMs, including DeepSeek -R RI-Distill- Qwen- 1 1.5B (Guo et al. 2005), Qwen3-4B, and Qwen3-8B (Yang et al. (025a), which are commonly adopted in prior works. For fair comparison, we also report results on these baseline models across different model scales. Detailed implementation settings for all bascline methods provided in Appendix
Training and Evaluation Sctup. We conduct training using the open-source verl framework (Sheng et al. 2025). with 7,473 samples from GSM8K (Cobbe et al. (022) as the training dataset for all models. The dataset consists of grade school math word problems, which designed to evaluate question answering on basic mathematics that requires multi-step reasoning. The training configuration uses batch size of 128 and learning rate of 1 x 10-6 We set the coefficient

---

Published as a conference paper at ICLR 2026
Table 1: Acc@ I results on mathematical reasoning benchmarks across LRMs. indicates the rela- tive change with respect to the Original row of each model. PEAR consistently achieves the largest reduction in token usage across model scales, while maintaining comparable accuracy.
Method
GSMSK
MATH500
ATME24
AMC23
GPQA
MMLU
Average
Acc
Tok
Acc
Tok
Ace
Tuk
Aec
Tokk
Acc
Tok
Aze
Tok
Acc
Tok
DeepSeek-Ri
Original
85.97
1496
75.00
3620
8843
36.36
7878
40.38
1739
55.73
GRPO
87.86
1493
76.80
3132
78399
67.50
40.90
8288
41.09
1868
57.91
4587 (4.555)
Step Entropy
85.59
1629
76.80
3298
5640
70.00
4911
38.88
7095
40.67
1649
56.43
4037
(- 16.0%)
LCPO
87.11
2149
76.00
2895
26.66
5358
70.00
3324
37.87
6371
38.55
1450
56.03
25,3%))
PEAR
87.94
624
77.20
2358
23.33
5379
70.00
3705
38.38
6193
41.87
1242
56.45
3250 (, 32.4%)
Qwen3-4B
Original
94.69
2634
85.40
5795
56.66
16792
87.50
9234
52.52
8184
72.32
1930
74.85
7428
GRPO
94.38
2321
84,80
5434
63.33
1406]
90.00
8568
46.96
8244
73.01
1919
75.41
6758 (4 (1.9.0%)
Step Entropy
94.84
2261
85,
4704
60.00
9467
87.50
7317
53.53
7527
71.71
1889
75.50
5528 (, 25.6%)
LCPO
93.47
1846
84,20
3569
63.33
8528
85.00
6518
47.97
5073
73.94
1373
74.65
4485
(9.6%)
PEAR
94.01
1439
84.00
2695
56.66
5685
87.50
4173
49.49
4177
73.96
1154
74.27
3221
(2 56.6%)
@wen3-8B
Original
96.13
2335
86.60
5532
63.33
14977
90.00
8161
55.55
8572
73.28
1495
77.48
6845
GRPO
95.83
1999
85.20
5375
66.66
13195
90.00
7881
52.02
8745
75.54
1368
77.54
6427 (4 6.1%)
Step Enirupy
95.14
2087
86.00
4658
60.00
6816
90.00
7352
57.07
6063
72.89
1454
76.85
4738
(L.30.8%)
LCPO
94.54
1645
85.00
4234
63.33
7173
82.50
6961
46.46
5229
73.23
1165
74.18
4401
(5.35.7%)
PEAR
94.54
1092
85.40
2664
60.00
6104
92.50
4045
54.04
4333
78.86
961
77.56
3200 (3.53.3%)
for the final answer phase reward calculation as To evaluate the effectiveness and generalizabil-
ity of our compression method, we conduct experiments on four standard mathematical reasoning benchmarks: GSMSK test set (Cobbe et al., [202T), MATH500 (Hendrycks et al.) [021b]. AIME24
(Li et al. 2024). and AMC23 (Li et al., 2024). We also evaluate on two knowledge benchmarks: GPQA Diamond (Rein al., 2024) and MMLU (Hendrycks et al., 2021a) to further demonstrate
the out-of-domain capability of our method beyond mathematical reasoning. Descriptions of these
benchmarks are provided in Appendix D
Performance is measured along two dimensions: Accuracy (Acc) and the number of Generated Tokens (Tok), with a generation length cap of 16,384 tokens. Following the evaluation protocol of Guo et al. (2025), we adopt sampling with temperature to 0.6 and top-p set to 0.95. Answer extraction and verification are carried out following the methodology of Yang et al. (202)).
4.2 EFFECTIVENESS OF PEAR
As shown in Table I PEAR achieves the most substantial reduction in response length across all benchmarks and evaluated models, while maintaining accuracy at level comparable to original models. Compared to original reasoning models, PEAR achieves an average response length reduc- tion of 32.4% to 56.6%, while preserving the same performance with the decrease of only 0.58% in accuracy. This indicates that encouraging models to lower entropy level at the thinking phase during training provides an effective mechanism for eliminating redundant reasoning steps, thereby producing more concise outputs without compromising correctness.
Compared the 1.5B model, the results for the 4B and 8B models suggest that larger models, which are prone to verbose reasoning, benefit more from PEAR by achieving over 50% reduction in response length. This supports the intuition that bigger models tend to "over-explain", creating greater opportunities for efficiency gains. Moreover, PEAR delivers superior efficiency-accuracy trade-off on larger models relative other baselines. In the case of Qwen3-8B, while Step Entropy and LCPO enforce shorter responses, they incur larger accuracy drops of 3.3%. In contrast, PEAR achieves even greater compression while remain the performance unchanged. This underscores PEAR's adaptive nature, enabling it to compress reasoning traces aggressively without compromis- ing accuracy.
In addition, the benefits of PEAR extend beyond the training distribution, demonstrating strong OOD robustness. Although trained solely on the GSM8k training set, our method yields consistent efficiency improvements across all six benchmarks. For example, on Qwen3-4B, PEAR matches the

---

Published as a conference paper at ICLR 2026
0.5
Overall (After)
(Before)
300
Final Answer Phase
Thinking Phase
Thinking Phase (Before)
Answer (Bafore)
250
Avg Takens/Step (Boffore
Avg Tokens/Step (After)
Avg: 50.1
0.4
0.40
Thinking Phase (After)
Final Answer
Number of Steps
per Step
0.33
200
40
Entropy
Avg: 35.7
0.3
6.29
0-25
300
0.2
6.18
20
0.1
50
10
Owen3-48 Models
Qwen3-88
GSMBK
MATH500
AIME24
AMC23
Benchmarks
Figure 4: (a) Entropy changes before and after training with PEAR across thinking and final answer phases. (b) Changes in the number of reasoning steps and average tokens per step for Qwen3-4B. PEAR reduces both the number of reasoning steps and the average tokens per step.
vanilla model's s accuracy on AIME24 and AMC23 while consuming only 34% and 45% of the origi- nal reasoning budget, respectively. Furthermore, on non-math reasoning benchmarks such GPQA and MMLU, PEAR still achieves high efficiency without performance degradation, further demon- strating OOD robustness across tasks and domains. These results highlight that phase-dependent entropy serves as universal, domain-agnostic signal for controlling reasoning efficiency, enabling our approach to generalize effectively across diverse tasks.
Overall, these results validate the central hypothesis of our work: incorporating phase-dependent en- tropy into the reward design enables LRMs to generate shorter and more efficient reasoning trajecto- ries, while preserving accuracy and demonstrating strong generalization across domains. Extended experiments on additional model series and scales are Appendix
4.3 How PEAR AFFECTS REASONING
We further analyze how PEAR influences model reasoning across different phases, focusing on changes in entropy, number of reasoning steps, and average tokens per step after training with PEAR.
As shown in Figure a), PEAR consistently reduces the overall entropy across all evaluated mod- els. Crucially, the largest reduction occurs in the thinking phase, where excessive exploration had previously contributed to unnecessarily long reasoning traces. This demonstrates that our reward effectively steers models toward more confident and focused reasoning, eliminating redundant ploratory steps in the thinking process. In contrast, the final answer phase shows a slight increase in entropy, indicating that the model retains flexibility when articulating its conclusions. Such phase- specific adjustments highlight PEAR's 's ability to suppress over-exploration during reasoning while still supporting diversity and completeness in the final answer through the control towards entropy.
Figure 4b) illustrates the changes in the number of reasoning steps and tokens per step for the Qwen3-4B model across all benchmarks before and after applying PEAR. The results show that PEAR not only reduces the total number of reasoning steps but also decreases the average tokens per step, reflecting shift toward more deterministic and efficient reasoning. Importantly, the re- duction is concentrated in the thinking phase, consistent with PEAR's objective of discouraging excessive exploration while maintaining entropy in the final answer phase. This effect especially pronounced more challenging datasets such AIME24, where the number of thinking steps is reduced by more than half. These results further validate the effectiveness of PEAR in producing concise reasoning trajectories without compromising solution quality.
Crucially, these findings explain why PEAR achieves substantial reductions in response length with- out sacrificing accuracy, highlighting phase-dependent entropy powerful control signal for bal- ancing efficiency and performance in large reasoning models. We also include a case study in Appendix to provide qualitative comparison.
8

---

Published as a conference paper at ICLR 2026
4.4 HYPERPARAMETER STUDY
A central hyperparameter in reward design is the coefficient for the final answer phase's entropy. This parameter directly controls the extent to which the model is encouraged for higher en- tropy in the final answer phase. Figure the impact of the hyperparameter on Qwen3-4B across four benchmarks. By default, @ is set to a positive value in order to avoid "reward gaming", where the model drives entropy down indiscriminately degraded performance. Conceptually, although PEAR does not introduce a separate coefficient ex- plicitly multiplying Htiikk the hyperparameter a implicitly controls the effective strength of the thinking-phass entropy penalty. According to Eq. ( decreasing increases the relative weight of Huink by enlarging the gap Humikk a answer while increasing a offsets the thinking- -haase penalty through the final-answer entropy term. Thus, single hyperparameter jointly regulates the entropy penalties on both phases and determines how aggressively exploratory reasoning is suppressed.
maximize reward, which often leads to
The experiments confirm this hypothesis. When a 0. post-thinking entropy ignored, and the model is optimized solely to mini- mize entropy in the thinking phase. While ef- ficient, this strict reduction harms accuracy, the model loses the flexibility needed in the an- phase refine adjust its predictions. The problem becomes even more pronounced when 1. where both the reasoning and answer phases are simultaneously penalized for entropy. In this setting, the model is overly con- strained, producing shorter but less reliable re- sponses and further degrading performance.
5000
4500
77.4
78.1
79.7
79.4
79.2
26.3
80
Average Tokens
4000
73.5
3758
Accuracy (%)
3612
70
35000
3500
3098
31.43
3000
2043
2500
2307
Average Tokens
55
Accuracy
2000
-1.0
0.5
1.0
1.5
2.0
As a increases, the penalty on post- thinkiiin en- tropy becomes stronger. This relaxes the re- strictive effect on the answer phase, allowing the model to preserve higher entropy where needed and thereby improving accuracy. At moderate values a (e.g.. 1), we observe a favorable balance: the model reduces redundancy in its reasoning while maintaining strong performance. However, when is set too high, the penalty ef- fect becomes negligible, and the model's behavior converges toward the baseline, producing longer responses and diminishing the efficiency gains. Overall, these results reveal clear behavioral tran- sition governed by a. Small a values lead to overly aggressive suppression of thinking-phan tropy, causing premature commitment and accuracy degradation. Large a values weaken the entropy penalty and recover baseline behavior with limited efficiency gains. The intermediate regime around As 1 consistently emerges as a stable operating point across benchmarks and model sizes, where PEAR selectively suppresses excessive, redundant exploration while preserving the core reasoning required for correctness. This explains why a Rs 1 serves as a robust default choice practice.
Hyperparameter Alpha
Figure 5: Average accuracy and response length of Qwen3-4B trained with different
en-
RELATED WORK
5.1 EFFICIENT REASONING
A growing body of research has focused on improving the efficiency of LRMs. Early exit stops model dynamically once certain criteria has been reached (Liao al.) 2025). Typical methods in- clude designing stopping rules based on internal reasoning state Yang et al. 2025b Qiao et al.) Zhu et al. 2025 Xu et al., (025), generation behavior (Wang et al.) 2025a d Liu & Wang. or without relying on pre-defined triggers (Dai et al. (025). Another complementary re- search direction focuses on compressing chain-of-thought reasoning traces, such as parallel thinking compression (Munkhbat et al. 2025 Ghosal et al./ (025), filtering or summarizing intermediate rea- soning tokens and steps YYu et al. 2025] Luo et al. 2025a Yuan et al. 2025 Xia et al. 2025 Zhao et al. 2025a), and compression reward mechanisms (Cheng et al. 2025b Zeng al. 2025 ). No-
2025
2025
tably. Li et al. (2023) introduce step entropy for quantifying the informational contribution of each reasoning step within CoT trajectories, enabling selective removal of low-entropy steps. Besides,

---

Published as a conference paper at ICLR 2026
adaptive reasoning methods attempt to dynamically adjust the depth length of reasoning depend- ing on the difficulty of the input, this includes carefully designed reward (Jiang et al. 2025 Wang
et al. 2025e Luo et al. 2025b) and reasoning mode switching (Zhang et al.] 2025d Huang al. 2025 Zhang et al. 2025a). . For example, ICPO (Aggarwal & Welleck 2025) ) include user-specified
length constraint into the training reward guide the model toward answering within the constraint.
However, such methods discard valuable intermediate reasoning that could improve accuracy. In contrast, our method utilizes the intrinsic phase-dependent entropy as reward signal, making an adaptive and model-driver approach to helps the model reason more efficiently.
5.2 REASONING THROUGH ENTROPY CONTROL
With the increasing research focus on Reinforcement Learning with Verifiable Rewards (RLVR), model entropy (Shannon 1948) has emerged
powerful internal signal for shaping reasoning behaviors in large language models. Recent work has investigated how policy entropy evolves dur- ing reinforcement learning- -based post-training of reasoning models. [Zhang al. (2025f) reveal the correlation between entropy collapse and performance saturation as well as subsequent degradation. wwrobabilityhighh-aaaattee
Cui et al. (2025) further shows how
updates systematically reduce entropy. Another complementary direction treats entropy minimization itself as supervision by di- rectly minimizing token-level entropy via finetuning or using negative entropy as the sole reward in RL (Agarwal et al. 2025 Prabhudesai et al. 2025). Besides, recent work has explored augmenting reinforcement learning approaches by incorporating entropy- bbsed mechanisms encourage ex- ploration in reasoning chains (Zhang al. 2025e] Cheng al. 2025a). Furthermore, Wang et al.) (025c) reveal that the effectiveness of RLVR stems primarily from optimizing high-entropy to- kens that determine critical reasoning directions. Selectively targeting these high-entropy minority tokens during optimization can substantially enhance reasoning capabilities while improving com- putational efficiency. While most existing studies leverage entropy to improve reasoning capability, our approach uses entropy as a control signal for efficiency, enabling adaptive length control with- explicit token budgets while preserving accuracy. This reframes entropy not only as a tool for capability shaping but also as a principled knob for controlling the cost of reasoning.
6 CONCLUSION
In this work, we conduct empirical analysis and observed the consistent positive relation between entropy and response length across reasoning stages: the thinking phase exhibits higher entropy, reflecting exploratory behavior of longer responses, while the final answer phase shows lower en- tropy, indicating more deterministic solution. Based on this finding. we address the challenge of efficient reasoning by introducing Phase Entropy Aware Reward (PEAR), a reward mechanism that distinguishes entropy between the thinking phase and the final answer phase during training. By dis- couraging entropy in the thinking phase while preserving flexibility in the final answer phase, PEAR enables adaptive control response length without requiring explicit length targets or rigid trun- cation rules. Extensive experiments across six benchmarks have demonstrated that PEAR reduces token redundancy by a large percentage of 32.4% to 56.6% while preserving accuracy with less than 1% degradation. Additionally, PEAR demonstrates strong generalization out-of-distributio tasks on both math reasoning and general knowledge benchmarks.
ACKNOWLEDGEMENTS
We would like to thank the anonymous reviewers and area chairs for their constructive and helpful comments this work. This research/project is supported by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative. (AISG Award No: AISG- NMLP-2024-005).
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics Our research focuses on improving the efficiency of Large Reasoning Models (LRMs) through chase-dependent entropy reward design. No human sub- jects, personally identifiable information, or sensitive user data were used in this study. All datasets employed (GSM8K, MATH500, AIME24, AMC23, GPQA and MMLU) are publicly available benchmarks designed for evaluating mathematical reasoning tasks and general tasks. The meth- ods proposed in this paper aim reduce computational overhead by shortening reasoning traces, which contributes to lowering energy consumption and improving the sustainability of large-scale model deployment. We do not anticipate direct harmful applications: however, as with all advances in language modeling, there exists risk of misuse in generating misleading harmful reasoning traces. We encourage responsible use and recommend that future work continue to consider faireess, transparency, and accountability in the deployment of reasoning models. No conflicts of interest or external sponsorships influenced this work.
REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of our work. All datasets used in experiments (GSM8K, MATH500, AIME24, AMC23, GPQA and MMLU) are publicly available and referenced in Section 4. and Appendix D Detailed descriptions of our training setup, hyper- parameters, and evaluation protocol are provided in Section 4.1 and Appendix [ For baselines, we follow official implementations and cite the corresponding repositories ensure faithful comparison in Appendix 2025). and we will release the complete source code and training scripts in the future to facilitate replication of results. Together, these resources provide clear pathway for reproducing both the training process and reported results.
C Our method is implemented using the open-sourced verl framework (Sheng et al.)
REFERENCES
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effec- tiveness of entropy minimization in I1m reasoning. arXiv preprint arXiv:2505.15734, 2025. URL
https://arxiz: org/ abs abs/2505. 15134
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04699, 2025. /tLlhttss://axxiv..
abs 0bs/2503.0469
Sohyun An, Ruochen Wang, Tianyi Zhou, and Cho-Jui Hsieh. Don't think longer, think wisely: Optimizing thinking dynamics for large reasoning models, arXiv preprint arXiv:2505.21765, 2025. URtpptt:s///axxi
org attaaa////222222 21765
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, al. Do not think that much for 2+ 3=? on the overthinking of olke llms. arXiv preprint arXiv:2412.21187, 2024.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. 14758,
2025a. URL/httss://arxt wwwpr//www.rrg/b////5/222 14758
Zhengxiang Cheng, Dongping Chen, Mingyang Fu, and Tianyi Zhou. Optimizing length com-
pression in large reasoning models. arXiv preprint arXiv:2506. 14755, 2025b. URL https:
14755
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arX11.2110.14168 2021. htLpstts::////
org/

---

Published as a conference paper at ICLR 2026
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.226111 2025. org/abs/ 2505 250...2615.2.7722
Muzhi Dai, Chenxu Yang, and Qingyi Si. S-grpo: Early exit via reinforcementt learning in reasoning models. arXiv preprint arXiv:2505.07666, 2025. URL
07686
Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Yifu Lu, Mengdi Wang, Dinesh Manocha, Furong Huang, Mohammad Ghavamzadeh, and Amrit Singh Bedi. Does think- ing more always help? understanding test-time scaling reasoning models. arXiv preprint arXiv:2506.042111
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025. httpps:::///axxt
12948
Michael Hassid, Gabriel Synnaeve. Yossi Adi, and Roy Schwartz. Don't overthink it. preferring shorter thinking chains for improved Ilm reasoning. arXiv preprint arXiv:2505.17813. 2025.
URL htttps://azxi...ott abs/2505. 17813
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. Ad- vances in Neural Information Processing Systems (NeurIPS), Track on Datasets and Benchmarks, 2021b.
Shijue Huang. Hongru Wang, Wanjun Zhong, Zhaochen Su, Jiazhan Feng, Bowen Cao, and Yi R Fung. Adactrl: Towards adaptive and controllable reasoning via difficulty-awwre budgeting. arXiv preprint ar arXiv:2505. 2025. https:/:://aaxi org/ abs /205. 18822
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024. URL
www.:iii..org.....oo 2412.16722
Lingjie Jiang, Xun Wu, Shaohan Huang, Qingxiu Dong. Zewen Chi, Li Dong, Xingxing Zhang, Tengchao Lv, Lei Cui, and Furu Wei. Think only when you need with large hebrii-reasoning models. arXiv preprint arXiv:2505.1466 2025, URL 14631
Martin Kuo, Jianyi Zhang, Aolin Ding, Qinsi Wang, Louis DiValentin, Yujia Bao, Wei Wei, Hai Li, and Yiran Chen. H-cot: Hijacking the chain-of thouhht safety reasoning mechanism to jailbreak large reasoning models, including openai ol/o3, deepseek-rl, and gemini 2.0 flash thinking. arXiv preprint arXiv:2502. 12893, 2025. URL httptttsss//xii...t.cm //tp:i/aaiivv.v. ort. /2022 12893
Shiye Lei, Zhihao Cheng, Kai Jia, and Dacheng Tao. Revisiting 11m reasoning via information bot- tleneck. arXiv preprint Stiv:25077.8391 2025. URL BRRL/ttteer//aaxi.oorr///////// 18391
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024.
Zeju Li, Jianyuan Zhong, Ziyang Zheng, Xiangyu Wen, Zhijian Xu, Yingying Cheng, Fan Zhang, and Qiang Xu. Compressing chain thought in 1mms via step entropy. arXiv preprint arXiv:2508 03346, 2025. URL //apiii//////a/////aa/raa wttp:/wwwwww///////////////////a
12

---

Published as a conference paper at ICLR 2026
Baohao Liao, Hanze Dong, Yuhui Xu, Doyen Sahoo, Christof Monz, Junnan Li, and Caiming Xiong.
Fractured chain-of-thoughh reasoning. arXiv preprint 2rXiv:2505.12992 2025. URL https:
//arxiv.org/ 2ttpaa/255..1233
Xin Liu and Lu Wang. Answer convergence as signal for early stopping in reasoning. arXiv preprint arXiv:2506.02536 2025, URL httthttps///aat wtwpz/iw..or//ab//555eee ////////t//////a...rrr///////..0 02536
Feng Luo, Yu-Neng Chuang, Guanchu Wang, Hoang Anh Duy Le, Shaochen Zhong, Hongyi Liu, Jiayi Yuan, Yang Sui, Vladimir Braverman, Vipin Chaudhary, et al. Autol2s: Auto long-short reasoning for efficient large language models. arXiv preprint arXiv:2505.22662, 2025a. URL
attps://arxts org/ abs//bb///// 2505 22662
Haotian Luo, Haiying He, Yibo Wang, Jinluan Yang, Rui Liu, Naiqiang Tan, Xiaochun Cao, Dacheng Tao, and Li Shen. Adarl: From long-cot to hybrid-cot via bi-level adaptive reasoning optimization. arXiv preprint arXiv:2504.21659 2025b. URL 21649
Tergel Munkhbat, Namgyu Ho, Seo Hyun Kim, Yongjin Yang. Yujin Kim, and SouYo Yun. Self- training elicits concise reasoning in large language models, arXiv preprint arXiv:2502.20122, 2025.
20122
Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning, arXiv preprint arXiv:2505.22660, 2025. URL
22660
Ziqing Qiao, Yongheng Deng, Jiali Zeng, Dong Wang, Lai Wei, Fandong Meng, Jie Zhou, Ju Ren, and Yaoxue Zhang. Concise: seffiiennce-uuided compression in step-by-step efficient reasoning. arXiv preprint arXiv:2505.04880 04881, 2025. BtRphttps://aat /ettps://arxivv. org. http:///aaawv..caaa///////////
04861
Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, et al. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond. arXiv preprint wwXiv:2503.2/6111 2025. URL https:
f/arxiv.org abs 2503 21614
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. net/fooum? id=Tidiin 67584498
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. URL https://azas
org/abs/1707 06347
Claude Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):399-233, 1948.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. arXiv preprint arXiv:2402.03300 2(3):5, 2024. URL https::/aaxxie org/ abs 2402 03300
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference Computer Systems, pp. 1279-1297, 2025.
Yang Sui, Yu-Neng Chuang. Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503. 16411 2025. URL https:
Vlarxiv.or//ae abs 2503. 1641
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with 1lss. arXiv preprint arXiv:2501.12599 2025. URL https: axxiv. org/ wtt.xiv.org//b//550000 abs/2501.
12599
13

---

Published as a conference paper at ICLR 2026
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL atttps://wwenmm.ce github io/bloq/qwag
Chenlong Wang, Yuanning Feng. Dongping Chen, Zhaoyang Chu, Ranjay Krishna, and Tianyi Zhou. Wait, we don't need to" wait"! removing thinking tokens improves reasoning efficiency.
arXiv preprint 200.::25000003333,,, 2025a. URL URLbttts: /ttp://awwv..o///a///555.00.00
org/ hacp//bs///0 httpa/aas//0...../ 08343
Jiakang Wang, Runze I. Liu, Fuzheng Zhang, Xiu Li, and Guorui Zhou. Stabilizing knowledge, pro- moting reasoning: Dual-token constraints for rlvr. arXiv preprint arX(i:2500..5777, 2025b. URL.
https://arxi:s org. http:/aa//25000 2177
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive reinfrrree learning for 11m reasoning. arXiv preprint arXiv:2506.01939 2025c. URL https://arxis: org/ abs /506 01939
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. Thoughts are over the place: On the underthinking of ol-like 1lms. arXiv preprint arXiv:2501.18558 2025d. URL htttss:///axiiii orw.
2501.18588
Yunhao Wang. Yuhao Zhang, Tinghao Yu, Can Xu, Feng Zhang, and Fengzong Lian. Adaptive deep reasoning: Triggering deep thinking when needed. arXiv preprint arXiv:2505.20101, 2025e. URL.
https://arxisss org/ wwwp/////////////////////ass /2005 20101
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24882 2022.
Heming Xia, Chak Tou Leong, Wenjie Wang, Yongqi Li, and Wenjie Li. Tokenskip: Controllable chain-of-thought compression in llms. arXiv preprint arXiv:2502.12067, 2025. URL https:
12067
Yuhui Xu, Hanze Dong, Lei Wang, Doyen Sahoo, Junnan Li, and Caiming Xiong. Scalable chain
of thoughts via elastic reasoming. arXiv preprint arXiv:2505.05315, 2025. URL https://
arxiv. wwt.:vorggabboo...
www.:bs.g/bbbboe..e htw.aa////55555555 05315
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. bttps::::/// wttp:///wwr//a////...00000
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a. URL htlhtttps//aaxtte wwg.abs///5....... C2388
Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Qiaowei Li, Zheng Lin, Li Cao, and Weiping Wang. Dynamic early exit in reasoning models. arXiv preprint arXiv:2504. 15895, 2025b. URL httph::tt////tttt
org/ abs/ 20lss/2504.5 2204 15895
Bin Yu, Hang Yuan, Haotian Li, Xueyin Xu, Yuliang Wei, Bailing Wang, Weizhen Qi, and Kai Chen. Long-short cfain-of-thought mixture supervised fine-tuning eliciting efficient reasoning in large language models. arXiv preprint arXiv:2505.03440 2025, WRRLbttss://axxiitn
abs 2505. 03469
Hang Yuan, Bin Yu, Haotian Li, Shijun Yang, Christina Dan Wang, Zhou Yu, Xueyin Xu, Weizhen Qi, and Kai Chen. Not all tokens are what you need in thinking. arXiv preprint 2505:::2005.7822 2025. URL URL htttp://aarx wwtp:iiw.wg/a///////eeaa
17827
Linan Yue, Yichao Du, Yizhi Wang, Weibo Gao, Fangzhou Yao, Li Wang, Ye Liu, Ziyu Xu, Qi Liu, Shimin Di, et al. Don't overthink it: survey of efficient rl -sylle large reasoning models. arXiv preprint ar0iv:2508 02 /20, 2025. URL https: httppss::///rv....//mm org wwwp////////////// 02120
14

---

Published as a conference paper at ICLR 2026
Zihao Zeng, Xuyao Huang, Boxiu Li, Hao Zhang, and Zhijie Deng. Done is better than per- feet: Unlocking efficient reasoning by structured multi-turn decomposition. arXiv preprint arXiv:2505.19788, 2025. /ttp:sts///az http://ww.rrraaa//5...555325
Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, and Juanzi Li. Adaptthnnk Reasoning models can learn when to think. arXiv preprint arXiv:2505. 1344177755134,,, 2025a. URL https://arxi>> org/
abs/2505.
Jinghan Zhang, Xiting Wang, Fengran Mo, Yeyang Zhou, Wanfu Gao, and Kunpeng Liu. Entropy-
based exploration conduction for multi-step reasoning. In Findings of the Association for Compu- tational Linguistice:: ACL 2025, pp- 3895-3906, 2025b, doi: 10. 18653/11//22222 1tw.://////222.....nnng//////// findings-acl.201.
URL
http:/loyo.oo/g/oooo/ 2025. ftef5...nniinng acl.nngsssalll..... findings-acl.
2017
Qingyang Zhang. Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised Ilm reasoning incentivization. arXiv:2504.058// 2025c. URL /tt//
arXiv preprint
05812
Shengjia Zhang, Junjie Wu, Jiawei Chen, Changwang Zhang, Xingyu Lou, Wangchunshu Zhou, Sheng Zhou, Can Wang, and Jun Wang. Othink-rl: Intrinsic fast/slow thinking mode switch- ing for over-reasoning mitigation. arXiv preprint arXiv:2506.02397, 2025d. URL https:
barxii..org/abs/2566..
Xingjian Zhang, Siwei Wen, Wenjun Wu, and Lei Huang. Edge-grpo: Entropy-driven grpo with guided emrrr correction for advantage diversity. arXiv preprint MxXiv25577221853 2025e. URL
https://arxiv: wttp://wwwwiiioooooooorrro//ooaa
org/ abw.g////s/// 2507 21848
Yanzhi Zhang. Zhaoxi Zhang, Haoxiang Guan, Yilin Cheng, Yitong Duan, Chen Wang, Yue Wang, Shuxin Zheng, and Jiyan He. No free lunch: Rethinking internal feedback for Ilm reasoning. org/ abs 2506 17219
arXiv preprint arxiv:2506.122 17219, 2025f. htttss:t:://arr
Haoran Zhao, Yuchen Yan, Yongliang Shen, Haolei Xu, Wenqi Zhang, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, and Yueting Zhuang. Let Ilms break free from overthinking via self- braking tuning. arXiv preprint arXiv:2505. 2025a. URL httttttss///aiiii wrw. abs/ 2565. 14604
Shangziqi Zhao, Jiahao Yuan, Guisong Yang, and Usman Naseem. Can pruning improve reason- ing? revisiting long-cot compression with capability in mind for better reasoning. arXiv preprint 1rXi::2505.14582. 2025b. URL / abs
14582
Zihao Zhu, Hongbao Zhang, Ruotong Wang, Ke Xu, Siwei Lyu, and Baoyuan Wu. To think or not to think: Exploring the unthinking vulnerability in large reasoning models. arXiv preprint 122:222022.2222.. 2025. Utthttps:: http:///xxxxv.../.aabaa. aarg/abs//50// 12202
15

---

Published as a conference paper at ICLR 2026
100
89.5%
87.5%
10000
81.3%
80
Average Tokens
8000
Accuracy (%)
6949
6481
60
6000
4073
40
4000
3279
6203
5736
1966
20
2000
1293
100%
80%
60%
40%
20%
Filter Setting
Thinking Tokens
Final Answer Tokens
Accuracy
Figure 6: Accuracy and average response length the entropy filtering experiments on Qwen3-8B.
A THE USE OF LARGE LANGUAGE MODELS (LLMs)
Large Language Models (LLMs) were used solely for polishing the writing and improving the clarity of presentation. They were NOT involved in research ideation. methodology design, experiments, analysis, or any other substantive aspect of this work. All scientific contributions, results, and con- clusions are the sole responsibility of the authors.
B ENTROPY FILTERING EXPERIMENTS FOR QWEN3-8B
Figure 6 demonstrates the entropy filtering experiment result on Qwen3-8B. The results reveal similar trend as Qwen3-4B discussed in Section 2.3 When retaining 80% or 60% of low-entropy tokens, accuracy remains stable or even improves compared to the unfiltered baseline. Performance degradation only emerges under more aggressive filtering: retaining 40% or fewer low-entropy to- kens leads to a sharp drop in accuracy, showing that essential reasoning steps are lost when the trace is compressed too heavily. Notably, the length of the final answer phase also remains relatively unchanged across filtering levels, reinforcing that redundancy concentrated in the thinking phase.
This result further supports the conclusion that the high-entropy tokens being removed mainly drive excessive exploration rather than contributing to correct reasoning, and their absence reduces noise in the reasoning process.
C EXPERIMENT DETAILS FOR BASELINE METHODS
We evaluate three baseline methods: GRPO (Group Relative Policy Optimization) (Shao al., 2024). Step Entropy (Li et al. 2025), and LCPO (Length-Controlled Policy Optimization) (Ag- garwal Welleck 2025) using the GSM8K training set (Cobbe et al. 202T). Experiments are conducted across three model sizes: DeepSeck-RI-Die -ist-----Diilll--e------ 1.5B (Guo et al. (022). Qwen3-
4B, and Qwen3-8B (Yang et al. 2025a). The implementation details for each baseline are provided below.
For GRPO (Shao al. 2024). use the open-source verl framework (Sheng et al. 2025 with the original rule-based reward, which assigns a reward of I for correct answers and O otherwise. We set the rollout number to and the KL penalty coefficient x 10-3
For Step Entropy (Li et al. 2025), we use the official implementation provided by the author The method follows a two-stage training strategy: Supervised Fine-Tuning (SFT) with pruned CoT data,
/
staoom///////yt///mmmmm/mmmmmm/t ove/ COT wwwp_osssmeeeooooceoo wtt___a_tteeeetrrr_ppp
16

---

Published as a conference paper at ICLR 2026
followed by Reinforcement Learning (RL) with GRPO. During the SFT stage, training is performed with mixed precision (FP16), a learning rate of 2 x 10 and a weight decay of 0.01. In the RL stage, the learning rate is set to 1 10 and the KL penalty is fixed at 0.1.
For LCPO (Aggarwal & Welleck 2025), we use the official codebase provided by the authors and follow the L1-Exact setup. Training is performed with GRPO under length control and maximum length constraint. We set the learning rate to x 10-6 with a batch size of 64, and restrict the context length to 4K tokens during training. Rollout number is fixed with a sampling temperature of 0.6, and the KL penalty coefficient is set to x 10-3
D EVALUATION BENCHMARKS
To evaluate the effectiveness and generalizability of our compression method, we benchmark on four standard mathematical reasoning datasets and two knowledge benchmarks.
GSM8K test set (Cobbe et al. 2021) is a carefully designed benchmark comprising 1,319 grade- school mathematics word problems. Each question typically requires two eight sequential rea- soning steps, primarily involving basic arithmetic operations applied across multiple intermediate stages. MATH500 (Hendrycks al. 2021b) contains subset of 500 problems drawn from high school mathematics competitions. We follow the evaluation setup of OpenAl by adopting the same curated subset. AIME24 (Li et al. 2024) features 30 problems from the 2024 American Invitational Mathematics Examination (AIME). As one of the most prestigious secondary-level competitions, AIME problems demand sophisticated reasoning across diverse topics, including algebra, combina- torics, geometry, number theory, and probability. AMC23 (Li et al. 2024) consists of 40 problems taken from the 2023 American Mathematics Competition (AMC). The dataset covers core high school mathematics domains such algebra, geometry, combinatorics, and number theory. provid- ing a broad yet rigorous evaluation of mathematical reasoning ability. GPQA (Rein et al.) 2024) carefully constructed benchmark of graduate-level questions spanning physics, chemistry, and biology. In our experiments we use the GPQA Diamond subset, which comprises 198 questions selected for highest quality. MMLU (Hendrycks et al. 2021a) is large multitask benchmark of multiple-choice questions drawn from 57 subjects across the humanities, social sciences, STEM, and other areas (e.g., elementary mathematics, US history, computer science, and law). Strong performance on MMLU requires broad world knowledge and problem-solving ability.
E EXTENDED EXPERIMENTS
To validate that PEAR achieves robust efficiency across recent models, different base architectures, and model scales, we conducted additional experiments presented in Table
Recent models. Following the release of DeepSeek RI 0528 and its improved distilled variants, we evaluated PEAR using PrepSeek-R1-4e 101-0528-Qwen33 and stronger chain-of-thought capabilities newer I.Mss. Across all five benchmarks, PEAR con- tinues to demonstrate its effectiveness on this more advanced distilled model: it achieves 41.3% duction in reasoning length on average while maintaining or slightly improving accuracy compared with both the base model and the GRPO-trained model. Notably, PEAR improves performance on AIME24 and GPQA Diamond, two challenging benchmarks that typically benefit from deeper rea- soning, indicating that reducing redundant high-entropy exploration does not harm, and may even enhance, the model's reasoning robustness. These results confirm that PEAR generalizes well to recent models with different thinking patterns and that the phase-dependent entropy reward remains effective for models with evolving reasoning styles.
which reflects the updated reasoning style
Different architectures. To address whether PEAR generalizes beyond Qwen-based models, we further evaluated it on with DeepSeek-RI reasoning traces. We used the same four math benchmarks as in the main paper and additionally GPQA Diamond. Across all benchmarks, PEAR again achieves substantial reduc- tions in reasoning length while maintaining competitive accuracy relative to the base model. These results indicate that PEAR is model- aggossic and can be applied effectively different LLM fam-
model from the Llama family trained
https://githube wwt.tttw...oooeeoooooooooooeoo cmu- 13/11
17

---

Published as a conference paper at ICLR 2026
Table 2: @cc@ results on five benchmarks across three LRMs different size and model series. + 4/t indicates the relative change in average token usage with respect to the Original row of each model.
GSM8K
MATH500
AIME24
AMC23
GPQA
Average
Method
Acc
Tok
Acc
Tok
Acc
Tok
Acc
Tok
Acc
Tok
Acc
Tok
DeepSeek-R1-OS20
Original
93.55
1969
80.6
5560
53.33
11768
87,50
8287
52.02
5260
73.40
6569
GRPO
95.75
2371
81.20
5854
60.00
13354
92.50
8414
57.57
5628
77.40
7124 (1 8.45%)
PEAR
95.07
689
81,00
2560
66.67
8052
92.50
5318
59.09
2669
78.87
3858 (4. 41.28%)
DeepSeek-RT-Distiee
I-8B
Original
83.32
1272
80.60
3303
46.66
5673
90.00
4586
50.00
4255
70.
3818
GRPO
91.58
1586
79.00
3530
43.33
5585
87.50
4568
48.98
4384
70.08
3931 (1 2.96%)
PEAR
88.70
1056
77.60
2419
40.00
4270
87.50
4356
47.97
2868
68.35
2994 (L (1.58%)
Qwen3-14B1
Original
96.05
1721
85,80
4279
73.33
10181
95.00
6693
55.55
3946
81.15
5364
GRPO
96.13
1700
83.80
4023
73.33
9677
97.50
6491
56.56
3618
81.46
5101
4.89%)
PEAR
96.66
1196
84.40
2772
76.66
7650
97.50
4150
54.54
2772
81.95
3708 (L 30.87%)
ilies, including Alama-based reasoning models, without relying on model-specific inductive biases or training pipelines.
Larger scale. To demonstrate that PEAR is model-size agnostic, we evaluated on Owen3-14B, extending our analysis beyond the 1.5B 8B models reported in the main paper. The results show that PEAR continues to deliver substantial reductions in reasoning length (over 30% on average) while preserving slightly improving accuracy across all benchmarks. This consistent pattern at 14B confirms that PEAR scales effectively with model capacity and maintains its afficiency-accuraa benefits as the underlying LRM becomes larger. These results integrated into the evaluation section and discussed the context of scalability, demonstrating PEAR's 's robustness across model sizes.
F CASE STUDY: QUALITATIVE COMPARISON OF REASONING TRACES
study comparing the complete reasoning processes of the original Qwen3-8B model and the PEAR-trained Qwen3-8B model (a = 1) on the same mathematical problem in F This example is representative of pattern consistently observed across bench- marks. Both models arrive at the correct solution and employ the same core deductive steps: recall- ing the definition of polar coordinates, computing the radius, determining the angle, and verifying the quadrant. However, their reasoning traces differ substantially in structure and efficiency. The baseline model repeatedly revisits the same checks, including angle verification and formula vali- dation, leading to long reflection segments that do not contribute new information. In contrast, the PEAR-trained model follows the same logical path but avoids repeated velf-verification and meta- cognitive loops. As result, the reasoning trace becomes shorter and more focused.
We present qualitative
Importantly, the reduction length does not degrade reasoning correctness. Across qualitative in- spection of multiple tasks, we do not observe cases where PEAR increases step-level errors for problems whose final answers are correct. This behavior directly follows from PEAR's reward design: entropy-basee shaping is applied only to already-correct responses, the model is not in- centivized to shorten incorrect or incomplete reasoning. Thus, PEAR improves reasoning efficiency by removing redundant exploration while preserving the essential logical structure of the solution.
18

---

Published as a conference paper at ICLR 2026
Example Qwen3-8B: Baseline vs. PEAR
Question
Convert the point (0,3) in rectangular coordinates polar coordinates.
Baseline Model (3195 tokens)
the rectangular coordinates (0, 3) to polar coordinates. Hmm, me recall how to this. From what remember, polar coordinates are represented as (r, 8), where r is the distance from the origin to the point, and / is the angle measured from the positive uraxiis
<think> Okay, need
First, think need find
[450 tokens]
However, in this case, x 0 and y is 3, Let me think. (506 tokens]
But wait, let me make sure there isn't another angle that could also work.
[434 tokens]
Wait, me verify with the formula.
[309 tokens]
Wait, let me make sure there isn't 't a mistake here.
[425 tokens]
Therefore, think my answer correct. The polar coordinates for (0,3) are (3, x/2).
But just to be thorough, let me check with another method.
[342 tokens]
Therefore, the final answer is (3, x/2).
think that's all. considered different approaches, checked for quadrant issues, and verified using multiple methods. Seems solid. </think>
convert the rectangular coordinates (0, 3) polar coordinates, we use the standard formulas for converting from rectangular to polar coordinates:
[467 tokens]
Final Answer. Thus, the polar coordinates of the point (0, 3) are: (3, x/2)
PEAR-Traine Model a 1 (2317 tokens)
<think> Okay, need convert the rectangular coordinates (0, to polar coordinates. Hmm. remember how that works. Polar coordinates are given (r, 8), where is the distance from the origin to the point, and is the angle made with the positive æ-axis.
First, lct me recall the formulas for converting from rectangular (x, 3/ to polar (r, 9).
[368 tokens]
But let make Let think about the coordinates.
[286 tokens]
Wait, let me check again. Let me visualize the point (0,3). It's units up the y-axis.
[439 tokens]
don't see any mistakes here. Let check with another method.
[302 tokens]
Therefore, the polar coordinates are (3, m/2). </think>
[408 tokens]
Final Answer. Putting all together, the polar coordinates corresponding to the rectangular point 3)
T/2)
Figure 7: Qualitative comparison of reasoning traces produced by the original Qwen3-8B model and the PEAR-trained model on the same problem. PEAR significantly reduces redundant reflective reasoning while preserving correctness.
19


Reviews:
[{"review_id": "8TOlwXDZaN", "reviewer": "Reviewer_6Dkj", "summary": "This paper proposes a phase entropy aware reward mechanism which penalizes excessive entropy during the thinking phase and encourages moderate exploration at the final answer phase. This reward design encourages models to generate concise reasoning traces while retaining the original accuracy.\n\nHowever, the vanilla GRPO training simultaneously improves accuracy and reduces the generation tokens. Compared with the simple GRPO baseline, the proposed PEAR seems to reduce more tokens with the cost of losing the RL training accuracy gain. More experiments using new and larger base models and comparison with more baselines should be conducted to validate the effectiveness.", "questions": "1. Does the proposed reward scale with the model size? \n\n2. How is the performance on general tasks after training with PEAR and other baseline methods?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper proposes a simple but effective reward for RL training to improve the reasoning efficiency. By penalizing the entropy in the thinking process, the method effectively reduce the generation tokens  37.8%~59.4% compared with original model while achieving similar reasoning accuracy on 4 math datasets and 3 LRMs. The authors also further analyze the changes in entropy, number of reasoning steps, and average tokens per step after training to validate the effectiveness of PEAR.", "weaknesses": "1. There are several previous related works about RL based training and reward shaping for efficient reasoning [1, 2]. In addition, there are also several SFT and training-free based baselines [3] should be compared to thoroughly validate the effectiveness of the proposed reward design. \n\n2. Only 3 LRMs are compared. With the release of Deepseek R1 0528, the thinking process of LRMs already changed. Could you please use the recent Deepseek R1 distilled LRMs such as deepseek-ai/DeepSeek-R1-0528-Qwen3-8B to validate the effectiveness on the improved and effective reasoning models? \n\n[1] Chen, X., Xu, J., Liang, T., He, Z., Pang, J., Yu, D., Song, L., Liu, Q., Zhou, M., Zhang, Z. and Wang, R., 2024. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187.\n\n[2] Liu, W., Zhou, R., Deng, Y., Huang, Y., Liu, J., Deng, Y., Zhang, Y. and He, J., 2025. Learn to reason efficiently with adaptive length-based reward shaping. arXiv preprint arXiv:2505.15612.\n\n[3] Lin, W., Li, X., Yang, Z., Fu, X., Zhen, H.L., Wang, Y., Yu, X., Liu, W., Li, X. and Yuan, M., 2025. TrimR: Verifier-based Training-Free Thinking Compression for Efficient Test-Time Scaling. arXiv preprint arXiv:2505.17155.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eJNONTrO6o", "reviewer": "Reviewer_Ddwk", "summary": "This paper investigates the problem of large-scale inference models generating excessively long inference chains. Through systematic empirical analysis, the authors find a positive correlation between the model's token-level entropy and response length, exhibiting different entropy patterns at different stages of inference: higher entropy in the thinking stage and lower entropy in the final answer stage. Based on this observation, the paper proposes the PEAR method, a GRPO-based reward mechanism that encourages the model to generate more concise inference trajectories by penalizing excessive entropy in the thinking stage while maintaining moderate flexibility in the final answer stage. Experiments on four mathematical inference benchmarks including GSM8K, MATH500, AIME24, and AMC23 show that PEAR achieves a response length reduction of 37.8% to 59.4% across different model sizes, with an accuracy decrease of less than 1%, and demonstrates good out-of-distribution generalization ability.", "questions": "Q1. In Equation 5, entropy is calculated using the old policy `πθ_old`, but `πθ_old` is continuously updated during training. Does this mean that the same response will yield different entropy values ​​and rewards in different training iterations? If so, will this non-stationarity of the reward affect training stability?\n\nQ2. In the edge case, if the `think` tag is missing, the paper sets k=T and H_answer=0, but is this reasonable? Would this lead to such responses always receiving higher rewards?\n\nQ3. Could you provide some qualitative examples demonstrating the complete reasoning process of the original model and the PEAR-trained model on the same problem? This will help us determine whether conciseness comes at the expense of clarity. Also, for mathematical reasoning, the correctness of each step is important. Does PEAR increase the error rate in the reasoning steps, even if the final answer is correct?\n\nQ4. From Figure 2(b), the entropy is already low in the final answer stage. Why then subtract `α * H_answer` from the reward? What is the intuitive reasoning behind this? Is it to encourage the model to maintain a certain degree of uncertainty and flexibility when providing the answer? However, in mathematical reasoning, the final answer should be deterministic. Why encourage entropy?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The authors established the relationship between entropy and response length through systematic empirical analysis. They not only demonstrated the overall correlation but also decomposed the entropy patterns at different inference stages, providing a solid empirical foundation for subsequent methods. Furthermore, the entropy filtering experiment cleverly proved that high-entropy tokens primarily contribute to redundant exploration rather than correct inference, directly supporting the design philosophy of PEAR.\n\nS2. The experimental results are very convincing. On all tested models and benchmarks, PEAR achieved significant length reductions (37.8%-59.4%), while the accuracy decline was minimal (less than 1% on average).\n\nS3. PEAR's design is simple and elegant, requiring only the addition of a phase entropy-based penalty term to the GRPO reward function. This simplicity makes the method easy to understand and implement, and also facilitates integration into existing training workflows.\n\nS4. The experimental section of the paper is very comprehensive.", "weaknesses": "W1. While the experimental results are promising, it remains unclear why PEAR maintains accuracy while significantly reducing length. Are there any theoretical guarantees? What is the theoretical basis for the design of the reward function, especially the max operation in Equation (7)? The paper does not provide a convergence analysis, nor does it discuss the relationship between the optimal solution of PEAR optimization and the standard GRPO.\n\nW2. The paper only evaluated PEAR on mathematical reasoning tasks, but it is unclear whether PEAR is applicable to other types of reasoning tasks (such as common sense reasoning, code generation, scientific reasoning, etc.).\n\nW3. The paper mentions that Step Entropy uses two-stage training (SFT+RL) with inserted `[SKIP]` tags, while PEAR uses end-to-end RL training, but it doesn't delve into the impact of this difference. A similar comparison with LCPO is made; specifically, LCPO uses explicit length constraints, while PEAR uses entropy as an implicit constraint, but under what circumstances do these two approaches have their advantages? Furthermore, the paper doesn't discuss whether PEAR can be combined with other methods to achieve better results.\n\nW4. Although the paper analyzes the impact of the hyperparameter `α` in Section 4.4, only a limited number of values ​​were tested. The optimal `α` may vary greatly for different types of tasks or different models, but the paper does not provide guidelines on how to choose `α`.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rarxFllI6P", "reviewer": "Reviewer_g2a5", "summary": "This paper aims to address the redundancy in thinking steps in Large Reasoning models. The authors observe a consistant positive correlation between model entropy and response length. Inspired by the observation, this paper introduces Phase Entropy Aware Reward (PEAR) to punish entropy in thinking phase and encourage entropy in final answers. The model trained with GRPO shows much reduced response length with comparable performance.", "questions": "1. The ablation study only shows how the coefficient in $H_{answer}$ changes the performance. I'm curious about how the coefficient $H_{think}$ changes model's behavior? My understanding is the reduction in reasoning length comes from reduced exploration in model's thinking, which sacrifices reasoning performance. I wonder given a fixed $alpha$, what is a \"sweet point\" for the exploration-efficiency trade-off. \n2. For table 1, can you provide the standard deviation?\n3. Does the proposed method work for other series of models (e.g. llama series)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper studies an interesting and important problem of efficient reasoning. \n2. The trained model shows consistent reduction in reasoning lengths across datasets, while largely preserving the performance.", "weaknesses": "1. The claimed OOD robustness is only supported by model's performance on other math datasets. Whether current method generalizes to non-math tasks (coding, general reasoning...) is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bIJNLQBD9d", "reviewer": "Reviewer_NdYN", "summary": "This paper reveals a correlation between model entropy and response length at different reasoning stages in reasoning models, based on which the author introduce Phase Entropy Aware Reward (PEAR) in the new reward design. PEAR penalize excessive entropy during the thinking phase and allowing moderate exploration at the final answer phase, which encourages models to generate concise reasoning traces that retain sufficient flexibility to solve the task correctly. This enables adaptive control of response length without relying on explicit\nlength targets or rigid truncation rules. Extensive experiments across four benchmarks demonstrate that PEAR consistently reduces response length while sustaining competitive accuracy across model scales. The experiments in the paper demonstrate the validity of this claim and also compare it with some recent works.", "questions": "Considering the weakness, more experiments and discussions are needed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "A new reward design and some experiments show the positive results.  PEAR achieves the most substantial reduction in response length across all benchmarks and evaluated models, while maintaining accuracy at a level comparable to original models.  The author further analyze how PEAR influences model reasoning across different phases, focusing on changes in entropy, number of reasoning steps, and average tokens per step after training with PEAR", "weaknesses": "1. The risk of over-penalizing necessary complexity.  The central assumption is that lower entropy during the thinking phase is desirable because it correlates with conciseness. However, this may not hold true for all problems.\n2. More discussions are need for the hyperparameter sensitivity of \"Excessive Entropy\"\n3. The experiments are conducted on models up to 8B parameters. While this demonstrates a trend, the behavior of language models can change dramatically at larger scales.\n4. If the benchmarks are all in similar domains (e.g., mathematical or logical reasoning), the method's effectiveness might be confined to tasks with structured, verifiable reasoning paths. It is unclear how PEAR would perform on tasks requiring commonsense reasoning, planning, or creative generation, where the notion of a \"correct trace\" is more fluid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)