DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains
Authors: Tian Liang
Wenxiang Jiao
Zhiwei He
Jiahao Xu
Haitao Mi
Dong Yu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Reasoning Models
Reasoning Efficiency
Reinforcement Learning
Primary area: reinforcement learning
TL;DR: This paper introduces DeepCompress, a dual reward strategy that simultaneously enhances both the accuracy and efficiency of large reasoning models.

Abstract: Large Reasoning Models (LRMs) have demonstrated impressive capabilities but suffer from cognitive inefficiencies like ''overthinking'' simple problems and ''underthinking'' complex ones. While existing methods that use supervised fine-tuning (SFT) or reinforcement learning (RL) with token-length rewards can improve efficiency, they often do so at the cost of accuracy. This paper introduces DeepCompress, a novel framework that simultaneously enhances both the accuracy and efficiency of LRMs. We challenge the prevailing approach of consistently favoring shorter reasoning paths, showing that longer responses can contain a broader range of correct solutions for difficult problems. DeepCompress employs an adaptive length reward mechanism that dynamically classifies problems as "Simple" or "Hard" in real-time based on the model's evolving capability. It encourages shorter, more efficient reasoning for "Simple" problems while promoting longer, more exploratory thought chains for "Hard" problems. This dual-reward strategy enables the model to autonomously adjust its Chain-of-Thought (CoT) length, compressing reasoning for well-mastered problems and extending it for those it finds challenging. Experimental results on challenging mathematical benchmarks show that DeepCompress consistently outperforms baseline methods, achieving superior accuracy while significantly improving token efficiency.

Published as a conference paper at ICLR 2026
DEEPCOMPRESS: A DUAL REWARD STRATEGY FOR DYNAMICALLY EXPLORING AND COMPRESSING REA- SONING CHAINS
Tian Liang*.1 Wenxiang Jiao Zhiwei He-
Jiahao Xu-
Haitao Mi
Dong Yu-
Tencent AT Lab
ABSTRACT
Large Reasoning Models (LRMs) have demonstrated impressive capabilities but suffer from cognitive inefficiencies like "overthinking" simple problems and "un- derthinking" complex ones. While existing methods that use supervised fine- tuning (SFT) or reinforcement learning (RL) with token-length rewards can im- prove efficiency, they often do so at the cost of accuracy. This paper introduces DeepCompress, novel framework that simultaneously enhances both the accu- racy and efficiency LRMs. We challenge the prevailing approach of consis- tently favoring shorter reasoning paths, showing that longer responses contain a broader range of correct solutions for difficult problems. DeepCompress em- ploys adaptive length reward mechanism that dynamically classifies problems as "Simple" or "Hard" in real-time based on the model's evolving capability. It encourages shorter, more efficient reasoning for "Simple" problems while pro- moting longer, more exploratory thought chains for "Hard" problems. This dual- reward strategy enables the model to autonomously adjust its Chain-of-Thour (CoT) length, compressing reasoning for rell-mastered problems and extending it for those finds challenging. Experimental results on challenging mathematical benchmarks show that DeepCompress consistently outperforms baseline methods, achieving superior accuracy while significantly improving token efficiency.
INTRODUCTION
Large Reasoning Models (LRMs) have demonstrated significant advancements, exemplified by Ope- nAl's ol series (OpenAI, 2024), DeepSeek's 's RI (Guo et al., 2025), Google's Gemini 2.5 (Google, 2025), and Anthropic's Claude 3.7 (Anthropic, 2025). These models exhibit remarkable capabil- ities across diverse complex reasoning tasks. Key characteristics of LRMs include their ability to perform self-verification, engage in reflection, and generate extended Chain-of-Thought (CoT) rea- soning, leading to improved accuracy. However, recent research reveals inherent inefficiencies in LRM cognition. These include overthinking (Chen al., 2024), characterized by excessive inter- mediate step generation for simple problems, and underthinking (Wang et al., 2025b), manifesting as frequent, unstable thought shifts during complex problem-solving. These findings underscore the necessity for adaptive strategies to enhance both the efficiency and accuracy of current LRMss.
Recent studies have explored various strategies to enhance the reasoning efficiency of LRMs. One line of research leverages Supervised Fine-Tuning (SFT) curated datasets of shortened CoT ex- emplars (Chen et al., 2024; Kang et al., 2025; Yu et al., 2025b). This approach trains LRMs to infer correct answers using fewer intermediate reasoning steps. Conversely, another line of work incor- porates token-length reward functions into Reinforcement Learning (RL) frameworks (Team et al., 2025; Luo et al., 2025; Aggarwal & Welleck, 2025; Liu al., 2025). These methods explicitly optimize for shorter reasoning paths while penalizing unnecessarily verbose ones. Although these compression techniques achieve significant efficiency gains, they often accompanied by slight accuracy trade-offs. Therefore, the fundamental challenge remains in simultancously achieving both superior accuracy and computational efficiency.
e *Correpoondngg ttianliang @ tencent.com.

---

Published as a conference paper at ICLR 2026
In this paper, propose DeepCompress, novel framework that incorporates an adaptive length reward mechanism which dynamically adjusts the preference for shorter or longer responses based on the problem difficulty perceived in real-time by the LRMs. Specifically, we first reveal that longer responses contain a wider coverage of potentially correct solutions than shorter ones for the same problems. In other words, current methods that constantly optimize shorter responses in RL pro- cesses may constrain the problem-solving capacity of LRMs and restrict their reasoning boundary. At the meantime, it is infeasible to encourage LRMs to always favor longer responses in RL pro- cesses considering the efficiency for both training and inference. Our DeepCompress addresses this challenge by first dividing the problems into "Simple" or "Hard" classes and then applying different length reward modes to them, respectively, during training. With Group Relative Policy Optimiza- tion (GRPO, Shao et al., 2024) as our basic RL algorithm, we consider problem as "Simple" when its group pass ratio (i.e., the proportion of correct samples among its G generated responses) exceeds the batch pass ratio (i.e., average of group pass ratio in the batch), and "Hard" when otherwise. Then, DeepCompress encourages the LRMs to favor shorter responses of the "Simple" problems, but longer responses of the "Hard" problems. Through this mechanism, DeepCompress dynami- cally adapts the reasoning chain length- autonomously compressing lengthy CoT for well-mastered problems while extending CoT for under-learned cases.
The contributions of this paper are summarized below:
We propose DeepCompress, which incorporates a model-aware mechanism to dynamically classify questions by difficulty, and dual length reward to adaptively explore longer responses for "Hard" questions and favor shorter responses for "Simple" ones.
Experimental results on challenging mathematical benchmarks demonstrate the capability of DeepCompress in achieving superior performance consistently over baseline methods while also improving the token efficiency significantly.
Our in-depth analysis reveals that DeepCompress fosters a more effective learning process by encouraging high policy entropy. This promotes efficient exploration and reflection, leading to superior performance, particularly on challenging problems.
2 RELATED WORK
Manipulating Reasoning Length through Prompt Engineering Research on reasoning length in LLMs presents a central trade-off. Some studies show that longer reasoning paths can improve task performance (Jin et al., 2024). whereas others advocate for conciseness to boost inference ef- ficiency, using strategies such as Constrained-Col (CCoT) (Nayab et al., 2024). To navigate this complexity, several methods have been proposed to optimize or adapt the reasoning process. Recog- nizing that excessive length in Chain-of-Thought (CoT) can impair performance, Yang et al. (2025) developed the Soinking-Optinan Scaling strategy to find an ideal length by filtering for the shortest correct reasoning paths. Other approaches focus on dynamic adaptation to the specific problem. Adaption-of. of-Though (ADOT), for example, addresses the mismatch between question difficulty and prompting complexity (Xu et al., 2024), while TALE directly manages token overhead by dy- namically tuning the number of reasoning tokens via the prompt (Han et al., 2024).
Post- Dost-Training for Reasoning Efficiency A significant body of work improves LLM reasoning efficiency through post-training, primarily via Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). SFFT-aaed methods train models on datasets of curated, concise reasoning exem- plars, which can be generated by stronger LLMs (Chen et al., 2024; Kang et al., 2025) or used within a weighted objective that adapts the reasoning budget to question difficulty (Yu et al., 2025b). The majority of approaches, however, utilize RL to penalize excessive length. In its direct form, this in- volves a simple length-based reward to encourage brevity (Team et al., 2025; Luo al., 2025; Arora & Zanette, 2025). More advanced methods employ dynamic reward-shaping which calibrates the length penalty based on task difficulty (Cheng et al., 2025), response correctness (Yuan et al., 2025), self-supervised optimal length signals (Yi et al., 2025; Liu et al., 2025), explicit user constraints (Aggarwal & Welleck, 2025). Innovations also extend to the training architecture itself, through methods like auxiliary reflection models (Deng et al., 2025) and iterative pruning (Hou et al., 2025). These approaches achieve notable efficiency gains, yet they offer limited accuracy improvements and occasionally incur minor performance losses
2

---

Published as a conference paper at ICLR 2026
MATH 500
Olympiad Beneh
Minerva MATH
AIME 25
1.00
0.70
0.65
0.40
DeepMath-Ze---
0.90
0.60
0.55
0.30
Paceg 52
Pasegl
0.60
0.30
0.25
0.00
3
Length
1.00
0.30
0.500
DespMath-Zero.11
0.95
0.70
0.65
0.40
0.90
0.00
0.55
0.30
0.85
0.50
0.45
0.20
0.80
0.40
0.35
0.30
Figure 1: Relationship between standardized response length (z) and mathematical reasoning per- formance (pass@k) (ossskk. Pass@) score decreases with increasing length, while Pass @ @3 generally in- creases.
3 PRELIMINARIES
Existing work has made significant efforts to reduce model response length, but this has concurrently led to degradation in performance. In this section, designed preliminary experiments to further analyze the relationship between response length and performance.
3.1
EXPERIMENTAL SETUP
Data To assess the mathematical performance of our models, we followed Zeng et al. (2025) and evaluated them four challenging benchmarks: MATH-500 (Hendrycks et al., 2021b), Olympiad- Bench (He et al., 2024), Minerva Math (Lewkowycz et al., 2022) and AIME 2025 (MAA, a).
Model We conducted experiments on DeepMath-Zero- which are created by finetuning Qwen models on Deep:atth1105 103K (He et al., 2025) dataset via Zero RI. These well-trained model has achieved state-of-the-art results on many challenging math benchmarks and demonstrates prominent "aha moment" phenomenon (e.g. longer response length and more cognitive behaviors).
and NeepMath-Zero-72
Metric Following Deepscek-RI (Guo et al., 2025), define rule-based outcome verifier and report the pass@k score (Chen et al., 2021). We set the maximum generation length to 32,768 tokens. For each problem, we generate samples (n k) using sampling temperature of 0.6 and a top-p value of 0.95. Let e be the number of correct samples among the n generated samples, then Pass @k calculated as:
R
RR-0 &
pass @k 1
Evaluation To better understand how response length impacts performance, developed fined evaluation strategy. We standardized the lengths of all sampled 8, responses for given problem (refer Section 4.2) and sorted them accordingly. These responses were then uniformly divided into 16 bins based on their standardized lengths. For each bin, we calculated both the aver- age response length and the average pass@k score. Specifically, we reported pass@) I (our general test-time metric) and pass @ 32 (the sampling group size used in our later training settings).
htttps://huggingfacce
zwhe99 htt.://wwwhee//eetttttteeeeeeeee
hhttps://huggiggff co/ 29he99 saaeeeeg//eeMaath-eeee.
3

---

Published as a conference paper at ICLR 2026
3.2 RESULTS
Figure/ plots the results of pass @k with respect to the standardized response length. For test-time metries pass@1 I, shorter responses exhibit better performance compared to the longer ones. How- ever, when it comes to pass@k, longer responses surprisingly catch up and surpass their shorter counterparts, except for SeepMath-Zero-ae conclusion still hold with larger k value (e.g,, k=64). In prevelant RL algorithms like GRPO, we usually sample multiple solutions for single question (c.g., 32) and optimize the policy by leveraging relative comparisons between solutions. Therefore, the trend of pass @k score be critical guidance for these RL algorithms. On one hand, it suggests that longer responses contain a wider coverage of potentially correct solutions, thereby providing critical positive reward signals necessary for effective RL training. On the other hand, current length reduction strategies (Team al., 2025) that constantly optimize for shorter responses, while seemingly improving efficiency, may inadvertently constrain the problem-solving capacity of LRMs, especially for complex prob- lems requiring extended reasoning. However, it is inpractical to encourage LRMs always favor longer responses in RL training processes considering the efficiency for both training and inference, indicating the need for an adaptive strategy.
on the challenging AIME25, where we found the
4 DEEPCOMPRESS
We propose DeepCompress, a novel framework that dynamically adjust the preference of LRMs for longer or shorter responses, in order to achieve superior performance and efficiency simultane- ously. Our method enhances the Zero RL by introducing two core innovations: Dual Length Reward and 2) Model-Aware Difficulty.
4.1 ZERO RL
In our study, we follow the zero RL training recipe from He et al. (2025) and utilize DAPO (Yu et al., 2025a) as our RL algorithm. Let To denote the large language model policy. Given a training set D (1r,, y:)} comprising question-answer pairs where x; question and yi is its ground-truth answer, the language model ma samples a group of outputs ( 07 ...,, ...,9 (. for each question 2; where ý is the predicted answer and G the group size. We adopt rule-based verifier V to judge each answer, and use its final accuracy as the outcome reward. This binary reward Ro is computed as:
Ro(ū.v) v) +1, if the extracted final answer is exactly correct.
otherwise.
4.2 DUAL LENGTH REWARD
Our primary objective is to train models generate correct solutions using the minimal number of tokens, thereby maximizing response efficiency. To simultaneously maintain the models' capability for deep exploration when addressing complex problems, design distinet length reward modes for "Simple" and "Hard" questions, respectively.
Specifically, for a set of G generated responses ( compute the response length mean μr and standard deviation o,. The standardized length z; is then obtained as:
J==1 corresponding to a given question, we
P4
2
O, +
where e a small constant introduced for numerical stability to avoid division by zero. The length reward R2 utilizes sigmoid function for nonlinear transformation:
1
(. = sigmoid( (Bano) = + "
(4)
where (3 is a hyperparameter controlling the steepness of the sigmoid function, thereby modulating R. Rz (ū.8)'s (8's sensitivity to token length deviations.
4

---

Published as a conference paper at ICLR 2026
1.1
1.1
Reward Value for Simple Question
Reward value for Hard Questions
1.0
Ra (Carrect) (Incorrec)
0.4
Ro (Correct)
8 0.44
(Incorrect)
B -066
0.a
-0.0
0.8
0.2
1.0
-0.2
g -1.0
1.0
1.0
-4
-4
-2
(a) Reward for Simple Questions
(b) Reward for Hard Questions
Figure 2: Reward values for our DeepCompress method. Subfigure (a) illustrates the reward for Simple Questions, and (b) for Hard Questions. For both, Blue indicates correct responses and Red indicates incorrect responses. The dashed line denotes the baseline outcome reward (R,). while the solid line represents our final combined reward (R R. + Rj). effectively showcasing how our Dual Length Reward (Rt) dynamically modulates the reward signal based on standardized response length (z) and question difficulty (B).
By configuring the sign of B, we enable two operational modes illustrated in Figure For simple questions, 3 > 0 yields higher rewards for shorter responses. 2) For complex (hard) ques- tions, 0 encourages longer responses, facilitating more extensive exploration. As discussed in Section 3, this strategy increases the probability of generating at least one correct solution for diffi- cult problems. Furthermore, we introduce hyperparameter a scale the magnitude of R R2(0, B), resulting in the final length reward:
Rí ax R2(0. B).
(5)
4.3 MODEL-AWARE DIFFICULTY
A core challenge in implementing the dual length reward is the dynamic assessment of question difficulty. While public datasets like MATH (Hendrycks et al., 202 and DeepMath (He et al., 2025) offer curated difficulty labels, this approach incurs additional annotation costs and fails to adapt to the model's evolving capabilities during training.
To address this limitation, propose model-aware difficulty mechanism that dynamically clas- sifies each question as "Simple" or "Hard"' During RL training, we compute two key metrics: the group pass ratio per question and the batch pass ratio. The latter provides real-time indicator of the model's current overall capability.
Specifically, for each question I; within batch, define its group pass ratio Py(x)) the pro- portion of correct responses among its G generated outputs:
Pa(x,) Σg TI( (R. =
(6)
where I(.) denotes the indicator function and R, represents the outcome reward. Concurrently, the batch pass ratio Ps for a batch size B is computed as:
(r().)
P.
B
Here, P. quantifies the model's current global performance, while P, (xx) reflects the difficulty of each question v; for that model state. We then determine the difficulty label by assigning the

---

Published as a conference paper at ICLR 2026
following bias term, and obtain the corresponding length reward:
8 P,(x,) - P. E e-1,
R (..
positive 3, (i.e., P,(x;) P₂) indicates that the question relatively easier for the current model ("Simple"). Conversely, negative Bg (i.e., Pz(2,) < Pb) signifies that the model finds the question comparatively challenging ("Hard"), This mechanism prioritizes extended reasoning paths for the most challenging questions per batch, thereby enhancing solution coverage and ultimately improving overall performance.
Finally, the reward for RL optimization integrates both outcome reward and length reward:
R. Rg.
Mechanism of B in DeepCompress
Here's breakdown of how 3 (i.e., B (T,) Pb) influences the length reward:
Mode Control: The sign (#) of 3 controls the length reward modes: 1) 8 > 0 designates
Simple questions, activating short-response prioritization; 2) 0 flags Hard questions, triggering extended-reasooing mode.
Policy Intensity: The absolute value (13) scales reward pressure, such that larger [ leads stronger preference to shorter or longer responses.
4.4 ENHANCING ROBUSTNESS
Building upon the DeepCompress framework, we introduce two enhancements ensure robust exploration: 1) Correctness-Cnndiiione Lentgh Reward; 2) Smoothed Batch Pass Ratio.
Cerrectnes-Coodditione Length Reward In DeepCompress, the dual length reward applies uni- formly to all generated responses, indenpendent of their correctness. This unconditional application risks creating reward hacking scenario, where models may prioritize length optimization over lution accuracy. potentially favoring incorrect responses. To address this issue, we refine the length reward mechanism by restricting its application exclusively responses that produce correct solu- tions (i.e., those with Rs 1). The overall reward then becomes:
Ro + R₁, solution ys is correct, Ra,
(10)
otherwise.
Smoothed Batch Pass Ratio In DeepCompress,
the batch pass ratio P, quantify the model's current global performance. However, this choice may impact the training stability. First, the batch pass ratio reflects only one-sided performance of the model and can fluatuate noticeably accross the batches. Second, the models often exhibit weak performance at the early steps of RL. training, resulting in low batch pass ratio. Consequently, questions may be inadvertently mis- judged simple, with undesirably large 3 values (derived from P3 Pt). This phenomenon can prematurely constrain response length, impeding critical exploration the solution space.
To enhance the training robustness, we smooth the batch ratio by tracking its historical values with an exponential moving average (EMA). Specfically, the smoothed batch pass ratio Pb, at each training step t is updated as:
Po. A Pa,-1 + (1 -)) pfrue
(11)
where ptrue denotes the batch pass ratio, and [0. 1J) the EMA parameter. Then, Po, is used in Equation determine the effective for length modulation. This updating rule avoids the bias by pirue and gives a more stable estimation of model's current global performance. Besides, we initialize Po,, with 1.0, which ensures that Po,, gradually adapts from an optimistic initial state, preventing premature penepaaizizzionn due low true pirue in early training.

---

Published as a conference paper at ICLR 2026
Table 1: Math reasoning performance. "DeepCompress" denotes models trained with our novel DeepCompress approach, which significantly improves the reasoning accuracy.
MATH
AMC
Olympiad Bench
Minerva Math
AIME
AIME
Poly Math
Avg
Model
500
23
24
25
Acc
Qwen-2.5-3B
50.4
24.2
21.2
20.4
1.5
21.9
20.5
Qwen-2.5-3B-Instruuc
66.0
42.5
29.4
28.9
5.4
2.5
27.3
28.9
DbepMath-ZZro--33
72.8
48.0
38.0
30.8
11.5
6.9
34.1
34.6
DeepCompress-Zeroo Zero-BB
75.3
49.4
39.3
32.7
16.7
7.1
35.8
36.6
Qwen-2.5-7B
54.8
35.3
27.8
16.2
7.7
5.4
28.1
25.0
Open-Reasonerr7erro-77
81.8
58.9
47.9
38.4
15.6
14.4
40.7
42.5
2.ve--2.--B-SRL-222
77.0
55.8
41.0
41.2
15.6
8.7
33.1
38.9
SeepMath-Zero-171
85.6
64.7
51.3
45.4
19.4
13.1
42.6
46.0
DeepCompress-Zeve Zero-7
85.6
67.8
53.3
47.4
23.5
19.6
44.0
48.7
5 EXPERIMENTS
This section details the experimental setup and presents comprehensive evaluation across suite of challenging mathematical benchmarks. particular, we aim to investigate how DeepCompress improves both the performance and efficiency of models simultaneously.
5.1 EXPERIMENTAL SETUP
RL Training Our models fine-tuned following the RL training recipe from He al. (2025), which has produced state-of-the-art reasoning models (e.g., DeepMath-Zes -erotth--eer7BB 7B). We applied dynamic sampling policy optimization (DAPO) algorithm from Yu al. (2025a), and trained Qwen2.5 5- 5-3', Qwen2 5-7 with a rule-based reward Ro (as defined in Equation 2). Fol- lowing Hu al. (2025), we adjusted the chat template of the Qwen model. Further details on the training settings can be found in Appendix B.
Evaluation We comprehensively evaluate model performance on seven challenging mathematical benchmarks: MATH-500 (Hendrycks et al., 2021b), AMC 2023 (MAA, b), OlympiadBench (He al., 2024), Minerva Math (Lewkowyez et al., 2022), AIME 2024-2025 (MAA, a), and the English subset of PolyMath (Wang et al., 2025a). As primary metrics, we samples 16 responses for each question and report the pass@1 accuracy. We construct a validation set, which consists of 60 ques- tions from MATH and 60 from AIME 2022-2023, to select the checkpoint with the highest pass @ score for evaluation. We utilized vLLM (Kwon et al., 2023) for efficient batch inference, and fixed the decoding parameters temperature=0.6, top-p=0.95, and max tokens=32,766 To ensure fair comparison and eliminate variance from evaluation scripts, we re-evaluate baseline models under our precise evaluation settings.
5.2 MAIN RESULTS
DeepCompress exhibits stronger reasoning capabilities Table 1 presents the main experimen- tal results. Our proposed DeepCompress consistently outperforms all existing Zero RL base- lines across all seven mathematical reasoning benchmarks, establishing new state-of-the-art (SOTA). Compared to the previous SOTA model, DeepMath-Zero, DeepCompress achieves an av- erage absolute improvement of +2.0 points with the 3B model and +2.7 points with the 7B model. Notably, DeepCompress demonstrates substantial gains on challenging problems. For example, DeepCompress- Zero-7B surpasses by +6.5 on AIME 25. These results highlight the robustness of DeepCompress in enhancing LIM reasoning through deeper exploration, particularly on complex tasks that require extended reasoning paths to push the boundaries of performance.
SaapMath--Zer--11 by +4.1 absolute points on AIME 24 and
weoo//wen/ee Qwen2. 5- 5-
co/ Qweq Qwen2, 7-

---

Published as a conference paper at ICLR 2026
10000
DeepMath-Zens Zero-
16000
DeepMall-Zno-788
DespC 7enerss-722n--11
DecpCanpnos
12500
Responac Longth
11.022
Response Longth
9640
9600
4003
6400
4,00
ATM
3200
MATH
AMC33
Oyapied
Mosrra
AIMES
ABME25
PolyMiad
MATH
AMC23
Olympiad
Miseria
AIME24
ATME23
PolyMadh
Figure 3: Average Response Length across mathematical benchmarks. DeepCompress- Zero models achieve significantly shorter average outputs compared to DeepMath-Zero models.
Table 2: Performance on the DiQA--Damood, MMLU-STEM and Big-Bench Hard.
GPQA-Diamond
MMLU
Big-Bench
Model
Biology
Physics
Chemistry
Overall
STEM
Hard
Qwen-2.5-3B
29.9
19.8
20.3
21.0
7.5
48.1
Qwen-2.5-3B --sstrucctt
45.1
26.1
30.7
29.9
48.7
71.3
DeepMath-7en-3B
45.1
25.3
32.2
30.2
54.7
71.6
DeepCompress-7enn--1
44.1
25.5
35.7
31.7
56.0
73.7
Qwen-2.5-7B
33.6
21.4
27.8
25.3
12.1
41.5
Open-Reesoner-Zero--
50.3
26.7
47.8
38.1
47.0
83.2
31.9
22.6
37.9
30.2
15.0
74.9
SeepMath-Zero-777
58.6
29.5
53.2
42.6
72.7
85.0
DeepCompress/Zeoo-77
57.6
31.2
58.2
43.9
75.5
85.7
DeepCompress demonstrates higher inference efficiency As shown Figure 3, DeepCompress generates significantly more concise responses compared to DeepMath-Zero models across all the evaluated benchmarks. On average, DeepCompress compresses the response length by 57.9% with the 3B model and 16.6% with the 7B model. Particularly AIME 24, DeepCompress- -eapprrss-------BBB Zero-3B uses 37.6% less tokens to achieves +5.2 absolute improvement, and DeepCompress-Zerte Zero-7B uses 35.2% less tokens to gain +4.1 improvement (see Table 1). These results establish that dynamically adjusting the preference for shorter or longer responses is truly an effective strategy for advancing reasoning boundaries while minimizing inference costs.
Generalizable reasoning beyond mathematics To validate that our method generalizes beyond mathematical reasoning, added evaluations on -Diamond (Rein al., 2024) (biology, physics and chemistry), Big Bench Hard (Hendrycks et al., 2021a) (complex multi-step reasoning), and MMLU-STEM (Suzgun et al., 2023) (science and engineering reasoning). As shown in Table 2, DeepCompress models consistently outperform other baselines, demonstrating significant improve- ments across all three benchmarks. These findings suggest that the dynamic strategy of exploration and efficiency facilitated by DeepCompress does not merely optimize for mathematical tasks, but rather fosters a more robust reasoning capability that generalizes across diverse scientific domains.
5.3 IMPACT OF LENGTH REWARD
Experimental Settings To understand how DeepCompress's length reward contributes to perfor- mance gains, we conducted further ablation experiments beyond the main results. Specifically, we analyzed variants trained with fixed parameter 3: Length Penalty (B 1) designed purely to reduce response length, and Length Bonus (3 = -1) intended encourage longer outputs. Their behaviors were then compared against DeepCompress method, alongside DeepMath-Zeri Zero--B. We observed and analyzed policy entropy, response length, and pass scores across different steps, presented in Figure
8

---

Published as a conference paper at ICLR 2026
1.6
DeepMath Zero
Length Penalty
Policy Enaropy
1.2
Longth
Resporse Length
62
0.5
48
0.4
2X
100
300
400
500
600
Steps
Steps
(a) Policy entropy (training)
(b) Response length (training)
16k
Response Longtb
12kk
Pasi@d Scure
8k
4k
0k
600
200
400
Steps
Steps
Pass@1 score (test)
(d) Response length (test)
Figure 4: Training dynamics and evaluation results of DeepCompress. (a) Policy entropy during training. (b) Average response length on training batches. (c) Average pass@) score (%) on test
sets. (d) Average response length on test sets.
Results and Analysis As shown in Figure 4a, the policy entropy dynamics reveal interesting pat- terns. Models trained with length bonus exhibit higher policy entropy during training, while length penalty consistently maintains a remarkably stable and low entropy. Regarding response length as shown in Figure 4d, the models' behaviors align perfectly with our length reward design: those trained with length bonus consistently generate longer outputs, while length penalty variants pro- duce remarkably shorter responses. Meanwhile, these additional reasoning overheads contribute to stronger mathematical reasoning capabilities. Figure 4c shows that length bonus variant exhibits higher performance compared other methods.
In contrast, our DeepCompress method demonstrates more adaptive balancing act. As depicted, DeepCompress's policy entropy initially increases, reflecting a phase of broad exploration, then gradually stabilizes as the model converges. Similarly, its average response length shows an ini- tial increase (for exploration) followed by a controlled reduction (for efficiency). Throughout this dynamic process, DeepCompress's performance on test sets exhibits continuous growth. This il- lustrates how DeepCompress intelligently and automatically balances exploration with efficiency, achieving simultaneous optimality in both dimensions and continuously improving performance.
5.4 QUANTIFYING EMERGENCE OF REASONING BEHAVIORS
To further investigate the mechanisms behind DeepCompress's high policy entropy and superior performance, we conducted an analysis on a targeted set of challenging problems. Specifically, we constructed this hard problem set from instances where our baseline models (Qwen2. 5-3B and Qwen2. 5-)B), failed to produce correct solution. On each set, we follow Zeng et al. (2025) to track the emergence of four cognitive behaviors described Gandhi al. (2025), with the prompt shown in Appendix D. The manifestation of these behaviors suggests reproduction of the "aha moment" phenomenon observed RI (Guo al., 2025).

---

Published as a conference paper at ICLR 2026
As shown in Table 3, DeepCompress reflects more often than the baseline models. Intrigu- ingly, despite this higher reflection frequency, its average response length remains shorter. This indicates that DeepCompress has learned a more efficient reflection process, enabling more concise and targeted attempts at a solu- tion. This mechanism also proves highly effec- tive, as evidenced by DeepCompress' 's stronger pass @1 I score. Therefore, DeepCompress does not just encourage more thinking, but rather smarter thinking, turning each reflective act into a productive step towards the solution.
Table Reflection Frequency on hard questions. We use GPT-4o to extract and track "aha moment" behaviors, with the prompt shown in Appendix D.
Model
Reflect
Length
Pass @
DeepMath-Zero-31
2.45
11,222
7.21
DeepCompress-Zeno-1B
2.73
4,853
8.72
DeepMath-Zero-75
2.59
7,180
11.35
DeepCompres-7een-777
2.64
5,942
13.81
w/ Length Penalty
2.20
2,520
9.94
w/ Length Bonus
2.87
13,575
11.89
5.5 HYPERPARAMETER ANALYSIS
Reward Weight a and EMA Parameter À As illustrated in Figure 5, our method exhibits remarkable robustmess with respect to the re- ward weight a, with all configurations con- sistently outperforming the previous baseline, -heMattt--er-B-- Additionally, empiri- cal results indicate that a larger smoothing fac- tor (A 0.99) is crucial for achieving optimal performance. By incentivizing longer reason- ing trajectories and deeper exploration during the early training phase, this elevated factor ef- fectively empowers the model to break through existing performance bottlenecks.
DespConpres-Zen--88
DeepMath-Zeo-38
37
37
Average 1
Average Pass
36
35
35
34
34
A
(4) 0.99
(b)
Figure 5: Ablation studies on a and The dashed line represents the DeepMath-Zero-31 baseline.
CONCLUSION
This paper introduces DeepCompress, novel framework that enhances both the accuracy and ef- ficiency of Large Reasoning Models. By incorporating a model-aware difficulty mechanism and a dual length reward, DeepCompress dynamically adapts its reasoning strategy encouraging concise solutions for simple problems while promoting deeper exploration for hard ones. Our experiments on challenging mathematical benchmarks show that DeepCompress achieves performance while simultaneously making significant gains token efficiency. Further analysis reveals that method fosters more effective learning process by encouraging high policy en- tropy for exploration, leading to more frequent yet more effective reflection behaviors. By enabling models to intelligently allocate their reasoning efforts, DeepCompress represents promising step toward developing more powerful and efficient autonomous reasoners.
state-of-the-re art
A limitation of this work is that method's effectiveness relies sufficient length variation among responses sampled within the RL group. Furthermore, to ensure training efficiency, we capped the maximum generation length at 10k tokens. This constraint may have restricted the model's ability to explore more complex or longer-form solutions.
7 ETHICS STATEMENT
The authors of this work have read and adhere to the ICLR Code of Ethics. Our research focuses on developing a novel reinforcement learning framework, DeepCompress, aimed at enhancing the reasoning capabilities and computational efficiency of Large Reasoning Models. The primary goal of our work is to advance the scientific understanding of Al reasoning and to develop models that are both more effective and more efficient, which believe is positive step towards sustainable and accessible AI research. Our work is foundational and does not introduce societal harms. All code and models will be released publicly promote open research and reproducibility.
10

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY STATEMENT
To ensure full reproducibility, all our code, training scripts, and final model weights will be made publicly available. Detailed descriptions of our training setup, including all hyperparameters, soft- ware versions, and implementation specifics, are provided in Appendix B. This will allow re- searchers to verify our results, build upon our framework, and further explore adaptive training strategies for large reasoning models.
REFERENCES
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503,0400 2025.
Anthropic. Claude 3.7 sonnet and claude code. 2025.
Daman Arora and Andrea Zanette. Training language models to reason efficiently. arXiv preprint arXiv:2502.0446. 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint vrXiv:2107.03374 2021.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like lims, 2024. URL https://arxii org/abs/ 221.2 21187.
Xiaoxue Cheng, Junyi Li, Zhenduo Zhang, Xinyu Tang, Wayne Xin Zhao, Xinyu Kong, and Zhiqiang Zhang. Incentivizing dual process thinking for efficient large language model reasoning. arXiv preprint arXiv:2505. 16315, 2025.
Hexuan Deng, Wenxiang Jiao, Xuebo Liu, Jun Rao, and Min Zhang. Rea-rl: Reflection-aware online reinforcement learning for efficient large reasoning models. arXiv preprint arXiv:2505. /9862, 2025.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah Goodman. Cogni- tive behaviors that enable self- self-improving reasoners, four habits of highly effective stars. arXiv preprint (rXiiv2503300300 307, 2025.
Google. Gemini 2.5: Our most intelligent ai model. 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token- budget-aware IIm reasoning. arXiv preprint arXiv:2412.188477 2024.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual mul- timodal scientific problems, 2024.
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath- 103k: A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504. / 1456, 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference Learning Representations, 2021a.
11

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt, Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long shiin-o-thoughh of Ilms via reinforcement learning. arXiv preprint 1xXiv:2504.01296 2025.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung- Yeung Shum Xiangyu Zhang. Open- reasoner-zero: An open source approach to scaling reinforcement learning on the base model. Reaseeeesooee-eerrrrr Opeen Reasoner- Zero, 2025.
httpp://gittie
Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengman Du. The impact of reasoning step length on large language models. ACL (Findings), 2024.
Yu Kang. Xianghui Sun, Liangyu Chen, and Wei Zou. C3ot: Generating shorter chain-of-thought without compromising effectiveness. Proceedings of the AAAI Conference on Artificial Intelli- gence, 2025.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Prin- ciples, pp. 61 116-226,50 2023.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guage models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 3843-3857. Curran Asso- ciates, Inc., 2022. URL http:ttttpoococeee..iiteee wwwpeeddiiiinnnnnnnppereeeee neurips. wwtppppepppppeeeeeeeeeeeeeeeeeee files//le//ee hLLe//aabbeEEceee00tt0322seeeee3 Satprpaceeeeeellaaaaaaeaeaaaeeee Cowf-rnnneeeeeocee pdf.
Wei Liu, Ruochen Zhou, Yiyun Deng, Yuzhen Huang, Junteng Liu, Yuntian Deng, Yizhe Zhang. and Junxian He. Learn to reason efficiently with adaptive length-based reward shaping. arXiv preprint ar 2rXv:25555.1662, 2025.
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. Ol-pruner: Length-harmonizing fine-tuning for ol-like reasoning pruning. arXiv preprint arXiv:2501.125700 2025.
MAA. American invitational mathematics examination (AIME). Mathematics Competition Series, n.d.a. URL https:
MAA. American mathematics competitions (AMC 10/12). Mathematics Competition Series, n.d.b. URL httppp:://aaa.aae/ wwwg/athghhhao...o..o wwwp:tthhhcmmmtttiiiioiiii ame.
Sania Nayab, Giulio Rossolini, Marco Simoni, Andrea Saracino. Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli. Concise thoughts: Impact of output length on Ilm reasoning and cost. arXiv preprint arXiv:2407. 19825 2024.
OpenAl. Learning reason with 11ms. 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-prood q&a bench- mark. In First conference on language modeling, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li. et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.n 0330000 2024.
12

---

Published as a conference paper at ICLR 2026
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery. Quoc Le, Ed Chi, Denny Zhou, et al. Challenging big bench tasks and whether chain-of-though guistics: ACI. 2023, pp. 13003-13051, 2023.
solve them. In Findings of the Association for Computational Lin-
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint (rXiv:2501.12555 2025.
Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, Qiqian Cang, Yichang Zhang, Fei Huang, Junyang Lin, Fei Huang, and Jingren Zhou. Polymath: Evaluating mathematical reasoning in multilingual contexts. arXiv preprint arXiv:2504.18428, 2025a. URL https:/ http///xxvvvv//////////// orgg abs/2504. 18428.
Yue Wang. Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Thoughts are all over the place: On the underthinking of ol-like Ilms, 2025b. URL attps://arxiv.ot abs / 2201 1585.
Mayi Xu, Yongqi Li, Ke Sun, and Tieyun Qian. tfhption-of-thought: Learning question difficulty improves large language models for reasoning. In Proceedings of the 2024 Conference on Empir- ical Methods in Natural Language Processing, pp. 5468-5465455 0488-549.2222 2024.
Wenkai Yang, Shuming Ma, Yankai Lin, and Furu Wei. Towards thinking-optinn scaling of test- time compute for IIm reasoning. arXiv preprint arXiv:2502. 18080, 2025.
Jingyang Yi, Jiazheng Wang, and Sida Li. Shorterbetter: Guiding reasoning models to find optimal inference length for efficient reasoning. arXiv preprint arXivv2504.21370, 2025.
Qiying Yu, Zheng Zhang, Ruofci Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503.
2025a.
Zishun Yu, Tengyu Xu, Di Jin, Karthik Abinav Sankararaman, Yun He, Wenxuan Zhou, Zhouhao Zeng, Eryk Helenowski. Chen Zhu Sinong Wang, et al. Think smarter not harder: Adaptive reasoning with inference aware optimization. arXiv preprint arXiv:2501.17974 2025b.
Danlong Yuan, Tian Xie, Shaohan Huang, Zhuocheng Gong, Huishuai Zhang, Chong Luo, Furu Wei, and Dongyan Zhao. Efficient training for reasoning models via length-aware optimization. arXiv preprint arXiv:2505.12284 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simpleri- 20o: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar 2503. 18892, 2025.
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
In the preparation this manuscript, we utilized Large Language Model as a writing assistant. Its role aid in polishing the text, including improving grammar and sentence structure. The core research ideas, experimental design, and analysis presented in this paper are entirely our own.
B TRAINING DETAILS
We use verl as the training framework'. Configurations are listed in Table 4.
Table 4: Configurations for training DeepCompress series models.
Config
DeepCompress-Zzno-33
DeepComprres/Zero-77B
Ir kl_coef
le-6
le-6
0.0
0.0
max wwr-ppropp_eeeggeee
2K
2K
max mexxresponse.uee
rength
10K
10K
sein_batch_sin size
512
512
poo_mini_batch_tie
32
32
clip_ratio
0.20
0.20
hiipratio_high
0.28
0.28
temperature
1.0
1.0
rollout.n
32
overlong.buffee beffer.len
2K
total training steps
600
reward weight
0,2
EMA, parameter EMA.parameter
0.99
0.99
C
BATCH PASS RATIO
The mechanism of DeepCompress relies the batch pass ratio (Pr) to judge problem difficulty. We recorded the changes in P, throughout the training process, and as shown in Figure 6, the model's P. exhibits stable growth, indicating very low noise in the difficulty judgment process.
DeeppCompres-/zno--Bk DespCompress-zao--B
Batch Pass Ratio
0.5
0.25
100
200
300
400
500
600
Steps
Figure 6: Batch pass ratio during training.
'https: cow.oooooooooocoooooooooooo cengi verr
14

---

Published as a conference paper at ICLR 2026
D REASONING BEHAVIOR PROMPT
Prompt for Identifying and Analyzing Reasoning Behaviors
Below is chain-of-reasoning generated by Language Model when attempting solve math problem. Evaluate this chain-of-reasoning to determine whether it demonstrates beneficial problem-solving behaviors that deviate from typical linear, monotonic reasoning patterns commonly observed in language models.
<sttart of reasoning>>
(input) rendoonnrrasonnngg>>
Specifically, actively identify and emphasize beneficial behaviors such (1) Backtracking: Explicitly revising approaches upon identifying errors or dead ends (e.g., "This approach won't work because...").
(2) Verification: Systematically checking intermediate results reasoning steps (e.g., "Let's verify this result by...").
(3) Subgoal Setting: Breaking down complex problems into smaller, manageable steps (e.g.. "To solve this, we first need to...").
(4) Enumeration: Solving problems by exhaustively considering multiple cases or possibilities.
Additionally, remain attentive to and encourage the identification of other beneficial behaviors not explicitly listed here, such creative analogies, abstraction simpler cases, insightful generalizations.
Important:
Clearly specify each beneficial behavior you identify. Provide explicit examples from the reasoning chain. no beneficial behaviors are observed, explicitly return an empty list. Provide your evaluation clearly, formatted as follows:
"'son
"behaviour";
"example":
15


Reviews:
[{"review_id": "BDvcjFg6x5", "reviewer": "Reviewer_kWZ8", "summary": "This paper introduces DeepCompress, a new framework designed to enhance both the accuracy and efficiency of Large Reasoning Models. The authors identify that LRMs often \"overthink\" simple problems and \"underthink\" complex ones, and that current methods to shorten reasoning paths often sacrifice accuracy.\n\nDeepCompress addresses this with a dual reward strategy in an RL setup. It dynamically classifies problems as \"Simple\" or \"Hard\" based on the model's real-time performance. For \"Simple\" problems, it rewards shorter, more efficient reasoning chains. For \"Hard\" problems, it encourages longer, more exploratory thought processes.\n\nExperimental results on challenging mathematical benchmarks show that models trained with DeepCompress consistently outperform baselines, achieving higher accuracy while simultaneously generating more concise responses, thus improving token efficiency.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper introduces a method that rewards shorter reasoning for simple problems and encourages longer, exploratory reasoning for difficult ones. This challenges the standard approach of always favoring conciseness.\n\n- This paper demonstrates that DeepCompress models achieve state-of-the-art results, outperforming strong baseline models across multiple challenging mathematical reasoning benchmarks, especially on difficult problems like AIME.\n\n- The research provides a thorough analysis showing that DeepCompress encourages a more effective learning process by balancing exploration (via high policy entropy) and efficiency, leading to more frequent and effective \"reflection\" behaviors.", "weaknesses": "- The core mechanism hinges on classifying a problem as \"Simple\" or \"Hard\" based on whether its group pass ratio (Pg) is above or below the batch pass ratio (Pb). This is a noisy and relative metric. A problem isn't inherently \"Hard\"; it's just harder than the batch average. This could lead to suboptimal rewards, especially in batches with skewed difficulty distributions. \n\n- The paper proposes conditioning the length reward on the correctness of a solution to prevent reward hacking. However, this creates a paradox for the most challenging problems. If a problem is so hard that the model fails to generate any correct solutions in a group (Pg = 0), the length reward mechanism (which is designed to encourage longer, exploratory paths for hard problems) will never be triggered. The model receives no signal to try harder, precisely when it needs it most.\n\n- The authors themselves acknowledge that the method's effectiveness relies on sufficient length variation among the sampled responses. If the model's sampling process (governed by temperature and top-p) produces responses of very similar lengths for a given problem, the standard deviation (σi) will be close to zero, making the standardized length (zi) unstable and the resulting length reward signal meaningless. This makes the framework's success sensitive to the initial state of the model and the choice of decoding hyperparameters.\n\n- The preliminary analysis (Figure 1) shows that pass@32 (a proxy for the training objective) generally improves with longer responses, while pass@1 (the primary evaluation metric) decreases. The framework is trained to find at least one correct answer within a group, which favors exploration. However, it is evaluated on its ability to produce a correct answer on the first attempt (pass@1), which favors exploitation and conciseness. While the paper shows success on both, this fundamental tension between the training signal and the desired inference behavior is a subtle but significant weakness.\n\n- The main comparison is against DeepMath-Zero, a vanilla reinforcement learning approach. The ablation studies use fixed \"Length Penalty\" and \"Length Bonus\" strategies, which are arguably strawman arguments. The paper would be stronger if it compared DeepCompress against other adaptive length-reward strategies mentioned in its own related work section, which would provide a more rigorous test of its claimed superiority.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G2d4FOuPyN", "reviewer": "Reviewer_EuSL", "summary": "This paper presents an an adaptive length reward mechanism that trade-off the efficiency and performance for the large reasoning models (LRM). Different from the previous length-penality method, this method introcue the length reward according to the “difficulty” of problem at hand. However, while the core idea has merit, there are several concerns regarding experimental rigor, theoretical justification, and presentation clarity that need to be addressed.", "questions": "+ How does DeepCompress perform on problems of known, fixed difficulty (e.g., using MATH's difficulty labels)? This would separate the \"model-aware\" aspect from actual difficulty.\n\n    + Or use similar difficulty assessment similar to [1]\n\n        [1] Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning. Zihe Liu, et al.\n\n+ Have you tested on non-mathematical reasoning tasks? (e.g., LiveCodeBench)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Observation: The observation that Pass@1 decreases while Pass@32 increases with length (Figure 1) is genuinely insightful.\n\n+ The experiments is fully evaluated acorss different becnhmark (7 benchmark) and is hoslitc enough for the math problem.\n\n+ The paper presents a coherent and compelling research: clear motivation, principled method design, and  in-depth analysis. The writing clearly connects empirical findings to design choices to performance outcomes, making the contribution well-motivated.", "weaknesses": "1. The authors observe that Pass@1 diminishes with length increase, while Pass@32 generally increases. The paper attributes this to \"longer responses contain a wider coverage of potentially correct solutions.\" However, this interpretation requires deeper scrutiny:\n- **Is this truly about difficulty?** Or is it simply a statistical artifact? When you have 32 samples, longer responses might just have more opportunities to \"stumble upon\" correct solutions through increased exploration, not necessarily because the problem is inherently harder.\n- What is the correlation between problem difficulty (ground truth labels from MATH dataset) and average response length? This would validate whether length truly reflects difficulty or model capability.\n\n2. The classification depends on batch composition. A problem could be \"Simple\" in one batch and \"Hard\" in another depending on what other problems are sampled. This inconsistency could lead to unstable training signals. And moreover, there is a lot of recent study on dynamic batching and rollout filtering, the usage of batch and group-level metric for defining “difficulty” may be somewhat unrealiable.\n\n3. Experimetns, since the author demonstrate the adaptive length rewards, I think at least should compare with some popular length-penality such as those in Kimi-1.5 or over-long filtering method. But in current experiments, the author just experimenet with several baseline models.\n\n4. Moreover, I would recommand using Qwen-3 series for further validation since Qwen-3 is more prone to think longer comapred to Qwen-2.5.\n\n5. I will be glad to see the ablation experiment accordong to different hyper-parameters such as $\\beta$ in Eq.4, the reward weight $\\alpha$.\n\n6. In Figure 4, Length Bonus baseline shows higher entropy and longer responses but lower final performance than DeepCompress. This contradicts the paper's claim that longer responses help with hard problems. Why doesn't pure length bonus work better?\n\n7. Typo\n\n    + Lien 58: infeasbile -> infeasible\n    + Line 59: effciency -> efficiency", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aAcSFCCmca", "reviewer": "Reviewer_GJ2N", "summary": "This paper proposes DeepCompress, a reinforcement learning (RL) framework for large reasoning models (LRMs) that introduces a dual reward strategy—encouraging shorter reasoning for “simple” problems and longer reasoning for “hard” ones. The method dynamically classifies questions based on the group pass ratio within RL batches, and adjusts a sigmoid-shaped length reward accordingly. The authors claim improved accuracy and token efficiency across mathematical reasoning benchmarks such as MATH-500, AIME, and OlympiadBench.", "questions": "1. How does the model perform if we train only with the model-aware difficulty component but no explicit length reward?\n2. Does the adaptive β term (Eq. 8) introduce additional variance into the policy gradient estimator? If so, how do you ensure stable convergence?\n3. How much of the gain comes from model-aware difficulty versus dual reward?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The problem setup—balancing reasoning efficiency and accuracy—is interesting and timely.\n\n- The adaptive length reward (shorter for easy, longer for hard) is conceptually reasonable and aligns with observed overthinking/underthinking phenomena in LLMs.\n\n- The paper provides extensive experimental evaluation on multiple math reasoning benchmarks.\n\n- Figures and tables (e.g., Table 1, Fig. 4) show measurable improvements over baseline RL models like DeepMath-Zero.", "weaknesses": "1. Missing ablation and causal analysis: Although the authors attribute improvements to the dual reward, there are no clear ablations isolating which component (dual reward vs. model-aware difficulty) contributes most to the gain. The provided variants (length bonus / penalty) are simplistic and insufficient to explain why the accuracy improves. There’s also no sensitivity study on α, β, or λ—key hyperparameters controlling reward magnitude.\n\n2. It seems the results contradict the motivation on length vs. accuracy: Figure 1 and Table 4(c) still show that pass@1 accuracy decreases with increasing reasoning length. This raises doubts about the claim that longer reasoning improves correctness for hard problems. The authors interpret longer reasoning as beneficial in the RL phase, but the empirical results suggest diminishing returns for long outputs.\n\n3. Ambiguity in difficulty classification: The definition of “simple” vs. “hard” via group pass ratio is circular—model competence determines the classification, which in turn affects training rewards. This may cause instability or reinforce existing biases rather than lead to genuine adaptivity.\n\n4. Unclear explanation of policy gradients and learning signal: It is not clear how the “dual reward” integrates into the policy gradient objective beyond heuristic addition. The paper repeatedly mentions DAPO and GRPO but doesn’t formally show how the modified reward affects gradient updates or stability. It reads more as an empirical tweak than a well-motivated RL formulation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9uZQvakfkU", "reviewer": "Reviewer_t6ev", "summary": "This paper introduces DeepCompress, a novel framework that enhances Large Reasoning Models (LRMs) by dynamically classifying problems as \"Simple\" or \"Hard\" based on the model's real-time performance. By applying a dual-reward strategy that encourages shorter reasoning for \"Simple\" problems and longer exploration for \"Hard\" ones, DeepCompress achieves improved accuracy while simultaneously improving token efficiency on multiple mathematical benchmarks.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The problem studied in this paper is critical.\n2. The paper is easy to follow and understand.\n3. Experimental results show that the performance of the proposed method improves consistently over base models.", "weaknesses": "1. The paper lacks important baselines to compare with other RL-based reasoning chain compression works, for example [1, 2, 3], and does not position the proposed method among them.\n2. The difficulty measure completely relies on batches, which may lead to instability. In a batch containing only simple problems, a problem that is simply \"not that simple\" may also be incorrectly marked as \"difficult\" because $P_g $is lower than the extremely high $P_b $. Vice versa. This mechanism cannot enable the model to learn a stable and absolute internal representation of the difficulty of the problem, but instead causes the reward signal to fluctuate dramatically with the random fluctuations of batch data.\n3. It is unclear how the hyperparameters are determined, e.g., $\\beta$ is set to 1 and -1 without explanation or investigation.\n4. How will the performance change with a different group size $G$?\n\nReferences:\n\n[1] Tu S, Lin J, Zhang Q, et al. Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL[J]. arXiv preprint arXiv:2505.10832, 2025.\n\n[2] Shrivastava V, Awadallah A, Balachandran V, et al. Sample more to think less: Group filtered policy optimization for concise reasoning[J]. arXiv preprint arXiv:2508.09726, 2025.\n\n[3] Yuan D, Xie T, Huang S, et al. Efficient RL Training for Reasoning Models via Length-Aware Optimization[J]. arXiv preprint arXiv:2505.12284, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)