Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy
Authors: Zeju Li
Jianyuan Zhong
Ziyang Zheng
Xiangyu Wen
Zhijian Xu
Yingying Cheng
Fan Zhang
Qiang Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient LLM
CoT compression
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Models (LLMs) using Chain-of-Thought (CoT) prompting excel at complex reasoning but generate verbose thought processes with considerable redundancy, leading to increased inference costs and reduced efficiency. We introduce a novel CoT compression framework based on step entropy, a metric that quantifies the informational contribution of individual reasoning steps to identify redundancy. Through theoretical analysis and extensive empirical validation on mathematical reasoning benchmarks, we demonstrate that steps with low entropy are indeed highly redundant. Our experiments reveal that an astonishing 80% of low-entropy intermediate steps can be pruned without significant degradation in the final answer accuracy across DeepSeek-R1-7B, 14B and Qwen3-8B. This finding sharply contrasts with random or high-entropy pruning, which severely impairs reasoning performance. Building on this, we propose a novel two-stage training strategy combining Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) reinforcement learning. This approach enables LLMs to autonomously learn to generate compressed COTs during inference by strategically incorporating [SKIP] tokens. Our method significantly enhances LLM inference efficiency while rigorously preserving accuracy, offering profound implications for practical LLM deployment and a deeper understanding of reasoning structures.

Published as a conference paper at ICLR 2026
MAKING SLOW THINKING FASTER: COMPRESSING LLM CHAIN-OF-THOUGHT VIA STEP ENTROPY
Zeju Li', Jianyuan Zhong', Ziyang Zheng', Xiangyu Wen', Zhijian Xu', Yingying Cheng', Fan Zhang", Qiang Xu 1.3+
The Chinese University of Hong Kong
2 Huawei Technologies Co., Ltd 3 Shenzhen Loop Area Institute
(zj1124, jyzhong, zyzheng23,, xywen22, zixu21, qxulecse buhk.edu... hk (cheng.yingy yingyingl, zhang faan] Jan2))nuuwei.com
ABSTRACT
Large Language Models (LLMs) using Chain-of- Thin-ht (CoT) prompting excel at complex reasoning but generate verbose thought processes with considerable redundancy, leading to increased inference costs and reduced efficiency. We in- troduce novel CoT compression framework based on step entropy, metric that quantifies the informational contribution of individual reasoning steps to identify redundancy. Through theoretical analysis and extensive empirical validation on mathematical reasoning benchmarks, we demonstrate that steps with low entropy are indeed highly redundant. Our experiments reveal that an astonishing 80% low-entropy intermediate steps can be pruned with minor degradation in the fi- nal answer accuracy across DeepSeek-RI-7B,, 14B and Qwen3-8B. This finding sharply contrasts with random high-entropy pruning, which severely impairs reasoning performance. Building on this, we propose a novel two-stage train- ing strategy combining Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) reinforcement learning. This approach enables LLMs to autonomously learn to generate compressed COTs during inference by strategi- cally incorporating [SKIP] tokens. Our method significantly improves LLM in- ference efficiency while preserving accuracy. paving the way for more scalable LLM deployments and better understanding of their internal reasoning, The code and data are released in tpt com/ staymylove/Con Compresst ion_oonviiiii Sa____vaa_____________ cnteep_entrooy
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, particularly when employing techniques like Chain-of-Thought (COT) (Wei et al. 2022 By generating explicit intermediate reasoning steps, often referred
"slow thinking' Large Reasoning Model (LRM) such the DeepSeck-RI (Guo et al.] 2025) Series and Qwen3 Yang al. 2025) significantly enhance performance on multi-step problems domains like mathematics, coding and symbolic logic. This process allows the model to break down complex problems into more manageable components, leading to more reliable and accurate outcomes.
However, notable drawback of current slow thinking COT implementations is the considerable redundancy often present within generated thought processes (Deng et al., 2023 Zhong et al.) 2025a). These verbose reasoning paths, while thorough, lead to increased inference latency, higher computational costs, and diminished overall efficiency. As models become larger and are deployed at scale, these inefficiencies present significant bottleneck for practical applications.
To mitigate this, prior research has explored several compression strategics. One prominent direction focuses making the CoT process implicit or latent, finetuning the model to internalize reasoning
"Corresponding Author

---

Published as a conference paper at ICLR 2026
steps without verbalizing them (Deng al. 2024 Hao al. 2024) dynamically compressing them in latent space (Tan et al, 2025). Other work has focused on compressing the reasoning chain different granularities, from pruning tokens in the input context (Li et al. 2023), enabling controllable token-level skipping during generation (Xia et al. 2025), to chunk-based compression (Wang et al. 2025a). While these methods improve efficiency, they do not offer principled way to identify and remove entire reasoning steps that are semantically redundant.
Intuitively, when humans tackle complex problems, they record only key milestones, omitting obvi- ous thoughts. Recent work has sought to teach LLMs a similar ability to skip steps (Liu et al. 2024] Jiang et al., 2025) or tune for ength-compressiine CoTs (Ma et al. 2025b). However, a fundamental question persists: how can we systematically identify which steps in reasoning chain are crucial versus superfluous?
In this paper, we propose novel, entropy-based method to identify and quantify the significance of each step within an LLM's Chain-of- Chain-of-Thought. We introduce the concept of step entropy. metric that measures the informational contribution of individual reasoning steps by aggregating token- level entropy during generation. Our core hypothesis is that steps with lower entropy represent more predictable, and therefore less informative, parts of the reasoning chain that can be safely pruned without compromising accuracy.
To validate this approach, conduct systematic empirical analysis by calculating step entropy for reasoning trajectories and investigating the impact of pruning varying proportions steps (10% to 100%) using three strategies: low-entropy pruning, high-entropy pruning, and random pruning. Our findings, as shown Figure reveal that pruning up to 80% of low-entropy steps maintains accu- racy while achieving substantial token reductions (16-45% across multiple benchmarks), whereas high-entropy step removal causes immediate performance degradation. Cross-model validation on DeepSeek-RI (7B, 14B) and Qwen3-8B demonstrates the universality of our entropy-baud barrepybbased ap- proach across different architectures. To demonstrate the superiority of step-level pruning, we com- pare our method with direct token-level pruning in the experimental section.
Building on this validation, we introduce a two-stage training strategy combining Supervised Fine- Tuning (SFT) with Group Relative Policy Optimization (GRPO) (Shao et al. 2024) that enables models to autonomously generate compressed reasoning trajectories. The SFT stage teaches models to predict when to use [SKIP] tokens based on entropy-compresse training data, while GRPO optimizes composite reward function balancing accuracy, compression ratio, and response length. Our trained models achieve 35-57% token reductions while maintaining or improving accuracy, demonstrating that LLMs can learn to perform efficient reasoning without sacrificing quality.
The main contributions of our work are summarized as follows:
We introduce step entropy as a principled metric for quantifying the contribution of each step in the Seuin-of--hhuggee thinking trajectory.
We provide strong empirical evidence that low-entropy steps largely redundant and can be pruned up to 80% without significant loss of accuracy.
We propose two-stage training strategy that enables LLM learn the efficient com- pressed reasoning policy, significantly improving inference efficiency while maintaining performance.
RELATED WORK
2.1 LLM REASONING WITH REINFORCEMENT LEARNING
Reinforcement Learning (RL) has emerged as powerful paradigm for enhancing the reasoning
capabilities of Large Language Models (LLMs). Recent advancements, such as those demonstrated
in Shao et al. 2024) and Xie et al.] (2025), showcase RL's efficacy in refining LLMs' ability to
tackle complex reasoning tasks. Furthermore, strategies involving "long COT" Yeo et al. (2025) and "slow thinking"
BaankinggZhonggggg Zhhng al. (2025b) (which involves extending inference time) Comanici al.
(uo
Guo et (225) OpenAl (2025) have been shown to significantly improve LLM reasoning
performance by allowing for more elaborate and deliberate thought processes.
2

---

Published as a conference paper at ICLR 2026
(a) Accuracy Mask DeepScaleR
Accuracy vs. Usage benchmarks
80
0.8
Toleess Usage Rabio
Accuracy
70
Accuracy (ACC)
60
0.6
40
0
10
20
30
40
30
60
70 80758885900 95100
Math500
AIME 2024
AIME 2025
Mask Ratlo (%)
Figure 1: Comprehensive Performance of COT Compression via Step Entropy. (a) Accuracy vs. Mask Ratio on 50 samples from DeepScaleR. This plot illustrates the impact of different pruning strategies (Random, High-Entropy Steps, Low-Entropy Steps) final answer accuracy as the mask ratio of intermediate COT steps increases. Note that pruning up to 80% low-entropy steps main- tains Complete COT accuracy (b) Accuracy vs. Tokens Usage Ratio on other benchmarks. This plot compares the accuracy and token usage ratio of the Full COT against our Compressed COT (80% low-entropy steps pruning) across Math500, AIME 2024, and AIME 2025 DeepSeck-R1-7B.
However, the increased length and computational overhead associated with these verbose COTs have led to concerns regarding efficiency. [Wang et al. (2025b); Cuaddoo et al. (025); SSi et al. (2025) highlight the phenomenon of "overthinking." where excessively long COT's can paradoxically lead to diminished efficiency without proportional gains in accuracy. This emphasizes the need for methods that can optimize the length and content of reasoning trajectories.
2.2 COT COMPRESSION AND LATENT REASONING
To address the inefficiency of verbose reasoning, researchers have pursued two main avenues: com- pressing the explicit CoT and making the reasoning process entirely implicit.
Explicit compression methods aim to shorten the generated text at various granularities. At finest level, some works enable controllable token-level skipping (Xia al. 2025 explore the information- - theoretic minimum number of tokens required for a solution (Lee et al. (025). At a coarser grain, R1e.,,,-Coooo R1-Compress introduces chunk-hased compression and search (Wang al. 2025a). Other strategies use length- congth-constrained tuning, integrating penalties into RL reward functions (Shen al.) 2025 Hou al., (025) or developing specific architectures for length-compressible CoTs like Valve (Ma et al., 2025b). Our work advances this line by proposing a more semantically grounded approach: we operate the step level, arguing it better mimics human cognition (skipping entire thoughts, not words). Furthermore, our method is guided by an explicit iiormmttion-hhorede signal- step entropy-teachien the model not just be shorter, but to selectively discard what verifiably uninformative.
alternative, more radical approach is to make reasoning implicit or latent. Methods like iCOT (Deng et al.] (024) and COCONUT (Hao et al. (024) fine-tune models to internalize reasoning steps, while others use knowledge distillation to embed the process the model's hidden states Deng al. 2023). More recently, dynamic latent compression performs reasoning entirely within these hidden states, avoiding explicit generation altogether (Tan et al. 2025). While these latent strategies offer maximum efficiency, they sacrifice the critical interpretability and verifiability of an explicit COT. Our work carves a distinct path by focusing on optimizing the explicit reasoning chain, preserving its benefits while drastically improving its efficiency.
3

---

Published as a conference paper at ICLR 2026
3 COT COMPRESSION VIA STEP ENTROPY
This section details our framework for CoT compression, which is built on a novel, entropy-based metric. We begin by formally defining step entropy and providing its theoretical justification measure of a reasoning step's importance. We then deseribe our primary contribution: the low- entropy steps pruning strategy and the process for performing LLM inference with the compressed COT.
3.1
STEP ENTROPY
The foundational premise of our work that all steps in CoT contribute equally the final answer. To formalize this, we introduce step entropy as a measure of the informational contribution of each reasoning step. We hypothesize that steps generated with high confidence (low uncertainty) by the model are more likely to be redundant, Information entropy provides natural way to quantify this uncertainty.
Given CoT sequence generated by LRM, we first segment it into series of distinct steps, C= (S,,S2.. ...., Sw), where each step S; is a sequence of tokens, S; = (4,,1, .............. ) During autoregressive generation, for each token ti.j. the model produces probability distribution p(-1c1,) over its entire vocabulary V, where C,2 is the context consisting of the input prompt and all previously generated tokens.
The entropy of this distribution, which represents the model's uncertainty at that generation step, is calculated using the standard Shannon entropy formula:
((tag)asg) =- Σ p(wcey,) loga p(w) ca.))
WEV
We define the step entropy (())) as the sum of token-level entropy across all tokens within that step:
M.
HSSSS Σ ((tiylesg)) = H (t.... M,
A high step entropy (S,) indicates that the model was, on average, highly uncertain when gener- ating step S,, while a low step entropy indicates the deterministic generation.
Lemma 1 (Entropy-Bounr Bounded Information). Given a reasoning process where a sequence of steps C= $2....,Sy leads to final answer A, the conditional mutual information A(S)) between the step and the answer, conditioned on all other steps S = C' {S is bounded by the step entropy (5)) of the step itself. Specifically:
A() S H(S))S<s)
The formal proof is provided in Appendix B.
This result demonstrate that when the entropy of step Sy is low, the conditional mutual information of Sy and the final answer A, i.c. the relation of Sy and A is minor.
Theorem 1 (Entropy-Boumin Bounded Information on Subset). For any subset of K + steps $ (Sho: Skys..., Skax) C. C, the conditional information contribution of this subset to the final answer bounded by the sum of the step entropies of the steps within it:
K
(15:AIC) S) ([SK,,Sek))
1=0
The formal proof is provided in Appendix B.2 The result denotes that steps Sk,, Sk. Sky could have minor relation to the final solution A. This observation implies that, a step with low entropy suggesting the deterministic thinking, which has minor relation to the finally solution, thus denotes such step could be less informative, and potentially redundant.
While step entropy theoretically serves upper bound on the mutual information between the CoT steps and the response, in practice, it suffers from bias introduced by step length. To mitigate this issue, we propose to use the length-normalized step entropy, defined in Definition
4

---

Published as a conference paper at ICLR 2026
Definition (Length-normalle Step Entropy). Given reasoning step S; consisting of M, tokens and the preceding context Ses. we define the nengtiznormalize step entropy as:
M,
H(S Sea)
Σ
H(tis (s,))
(3)
M4 5=1
where big represents the j-th token of step and Ca.j represents the context up to that token.
Inference with Compressed
dhaskoy istt need calculate the sum of the number: 991. 993, 995, 997, 999.
Slep!
Thinking COT
These-sumbers close 1300,-00,--- eex.-eeesseesccccceeeeeoee Fumber-as 1000 minus SXIP)
smail value
Slep2
User Query Problem
Noxt sum these exprestions:p55000 Step3
+3]
<think>
Now; the smal--abbsst//-- 1- 6 20 24in24 -2 [SXIP)
Step
"S1, [SKIP], Sa, [SKIP], [SKIP]. SN
Sub-tituing lig back ilto ------iiooinnn000000---0 266- 497g-
[SXIP]
Vially; Ill set up equetion to solve-lor 25-m4995 55000-0
[SXIP]
</think>
Sleph
Skp?
Subtracting 2975 from both sides gives/Inti 25'r
Final Answer Generation
001
Step Entropy
(h) The compressed CoT sequence Scchinko..sthhiing is concate- nated with the user query the prompt context.
(a) The case I Low-Enttopy steps pruning strategy for COT com- pression, and replacing each selected low-entropy step with a spe- cial "[SKI]" token.
3.2 LOW-ENTROPY STEPS PRUNING STRATEGY FOR COT COMPRESSION
Based on the observation established above, we propose a practical CoT compression approach that selectively removes low-entropy steps while preserving the essential reasoning structure. Our method operates on the principle that steps with low entropy are more likely to be redundant and can be safely pruned without significantly impacting the final answer quality.
For each problem instance æ, we use DeepSeck-R--Distil---e--- to generate complete CoT trajectory, the response format is <think> C </think> final answer The reasoning steps S1, ,,,.,.,,,y, delimited by \n)nn, are ex- tracted from thinking content C between the <think> and </think> tags, C S2,..,,,
Generate Full CoT:
2. Calculate Step Entropy: For each step S, C, compute its step entropy H(.)) using Equation 2
3. Entropy-Based Pruning:
We rank all steps in ascending order of their entropy scores and identify the x N lowest-entropy steps for pruning, where is the pruning ratio hyperparameter. The compressed CoT C" is constructed by replacing each selected low- entropy step with a special "[SKIP]' token, while preserving high-entropy steps in their original form. The example of this process is shown in Figure 2a
4. Inference with Compressed CoT: The compressed sequence C" is concatenated with the user query and the </think> delimiter to prompt the model to generate only the final answer, as illustrated in Figure 2h
This approach allows us to systematically compress CoT sequences while maintaining reasoning coherence. The pruning ratio provides flexible control mechanism for balancing compression efficiency and answer quality, with optimal values determined empirically across different datasets and problem types. The upper bound of steps pruning ratio x will be discussed in Experiment 4.1
3.3 AUTONOMOUS COMPRESSION VIA TWO-STAGE TRAINING
While our entropy-basee pruning strategy effectively compresses existing CoT sequences, enabling models to autonomously generate compressed reasoning trajectories during inference represents
5

---

Published as a conference paper at ICLR 2026
more practical advancement. Our two-stage training methodology successfully achieves this goal by teaching models to balance accuracy with efficiency through learning when to skip redundant reasoning steps.
Stage 1: Supervised Fine-Tuning (SFT) We first train the model on (problem, compressed CoT, answer) pairs of the dataset in the Experiment 4.2 using our low entropy- -based pruning strategy. The model learns to predict compressed reasoning paths and generate [SKIP] tokens by minimizing cross-entropy loss, providing robust initialization for reinforcement learning.
Stage 2: Group Relative Policy Optimization (GRPO) While SFT teaches static imitation of compressed traces, does not explicitly optimize the accuracy- trade-off. We employ Group Relative Policy Optimization (GRPO) al. 2024) to further optimize this behavior through reward-drie driven learning.
For each input prompt, sample group of K completions. The model's goal is to learn policy Ty that maximizes a composite reward function R(C) for each generated completion C. The total reward is the sum of four components designed balance correctness with efficiency:
R(C)- [Rorrectness; Rskip ratio, Rakip num; Bresporse length
Let Thhink (C) be the thinking content within the completion C. The reward components are defined follows:
Correctness Reward (oorreetnees::: large positive reward for generating the correct final answer. Let Aextmated (C) be the answer extracted from completion C' and A" be the ground truth.
2.0 if = Aextraced
Reomectness (², =
(5)
0.00 otherwise
2. Skip Ratio Reward (Rskip.ratio): tiered reward for achieving a high ratio of skipped steps. encouraging compression. Let Nokip be the count of '[SKIP]' tokens and Nsups be the total number of steps in Tihink (C). The skip ratio is Ratio = Natpp max (1, Naeps).
1.0
if Ratio Khiig
Rakip (C)
0.5 if Kiow &Ratioskp < Khigh
(6)
0.0
otherwise
3. Skip Number Penalty (Rsa): Penalty -1.0 when [SKIP] tokens exceed Tskip,aum to pre- vent degenerate behavior.
4. Response Length Penalty (R,)): Penalty -1.0 for responses exceeding Nength tokens to encourage conciseness.
This two-stage process trains the model strategically decide when perform detailed reasoning versus when to skip steps, achieving efficient reasoning while preserving accuracy. The experiments of training and evaluation results are presented in Section 4.3 with an ablation study of the reward components available in Appendix
4 EXPERIMENTS
This section presents experiments validating our entropy-based CoT compression method. We first conduct controlled studies to establish an optimal pruning ratio. We then demonstrate the effective- ness and generalizability of this strategy across multiple benchmarks and model sizes, justifying our step-level approach over token-level alternatives. Finally, we evaluate a two-stage training method that enables a model to autonomously generate compressed reasoning trajectories.
4.1 DETERMINING THE OPTIMAL PRUNING RATIO
To identify the safe threshold for step pruning, we conduct a controlled experiment using 50 samples from DeepScaleR (Luo al. 2025) dataset SrepSeek-R1-7B And investigate the impact

---

Published as a conference paper at ICLR 2026
Table 1: Comparing the full COT baseline with our proposed step-entropy based pruning (Our)
method, which prunes 80% of the lowest-entropy steps for DeepSeek-R1-BB 14B and Qwen-8B. We conduct experiments to get the @ass@1 (cCCracy(ACC)()) and the number of Average Thinking Tokens per answer (contains the Unicode characters) during the inference on GSM8K, Math500, AIME2024 and AIME2025.
Metbod
GSMBK
Math500
AIME 2024
AIME 2025
ACC(S)
Aug Tokens
(CC(%)
Avg Tokens
ACC (%)
Aug Tokens
(OC(
Aug Tokens
DeepScek-R1-2
78.54
298.33
88.17
5703.83
63.33
15843,43
35.71
18203.23
DeepScek-R1- R1-7B (Our)
80.82
294.29
88.17
2604.23 (199-15)
56.67
10092.80 (11605))
35.71
11471.17
DeepScok-R1-145
82.4 64
283.63
84.37
2853,73
65.52
15414,83
58.62
18000 10
DeepSeek-R1-Ie (Our)
(110000)
1980.97
8708.57
(148.00)
51.92
10842.07 (120885)
Oweat-8B
94.46
2053.67
91.37
7138.49
79.31
20936.57
76.92
19902.43
Qwen3 8B (Our)
94.39
2557.47 (16695)
91.13
5209.
24 (177.095)
81.48
11533,57
76.00
11716.63
final answer accuracy by pruning steps using three distinct strategies with a mask ratio varying from 10% to 100% ("no-thinking" mode (Ma et al. (202a))), shown in Figure[[ In our methodology, pruned steps are replaced with a special "[SKIP]" token. This choice is informed by an ablation study detailed in Appendix H which confirms that explieit placeholder is more robust high compression ratios than alternatives like direct removal.
Low-Entropy Steps Pruning: Steps with the lowest entropy are replaced by "[SKIP]""
High-Entropy Steps Pruning: Steps with the highest entropy are replaced by "[SKIP]".
Random Steps Pruning: Steps are replaced by "[SKIP]" at random, serving as a control.
The results, illustrated in Figure D strongly validate our hypothesis. With Low-Entropy Steps Pruning, we observe that final answer accuracy remains stable and unaffected even when up to 80% of the lowest-entropy steps are masked. Beyond this 80% threshold, accuracy begins to decline, eventually converging to the accuracy of the "no-thinking" mode. This provides powerful evidence that vast majority of low-entropy steps are indeed redundant. We found that with the best pruning strategy, we can prune up to 80% lowest-entropy steps (40% tokens redundancy) when not affecting the accuracy, shown in Figure (right)).
Conversely, with High-Entropy Steps Pruning, accuracy degrades immediately upon masking even small fraction of steps. When the mask ratio exceeds 40%, performance drops below that of the "no-thinking" mode, indicating that removing these critical, high-information steps is more detri- mental than providing no reasoming at all. The Random Steps Pruning strategy's performance falls between the two, beginning to decline at a 40% ratio.
Based on these findings, we establish our core strategy: pruning 80% of steps with the lowest entropy (K = 0,8), replacing them with [SKIP] tokens while preserving the remaining high-entropy steps. Moreover, we validate this strategy also work on Deepseek-Ri Deepseek-R11144 R1-14B and Qwen-8B.
4.2 VALIDATING LOW-ENTROPY STEPS PRUNING STRATEGY
With the 80% threshold established, conduct extensive experiments to validate our strategy's effectiveness, generalizability, and scalability across different models and datasets.
Models and Datasets. We use models from the DeepSeek-RI series (7B and 14B) and Qwen3- 8B (Yang et al. 2025), which are open-source Large Reasoning Models with strong mathematical reasoning capabilities. To generate the initial CoT trajectories for our training data, we merged 40k samples from the DeepScaleR dataset (Luo et al.) (2025) with 90k samples from the 1-Mtth dataset (OpenR1 - 2025), creating a final composite dataset of 130k instances. To test the effectiveness and generalizability of our compression method, we evaluate performance on several standard mathematical benchmarks: GSM8k (Cobbe et al. 2021)). Math500 (Lightman et al. 2023
AIME2024 (dataset card AIME 2024) and AIME2025 (dataset card AIME 2025).
Performance and Generalizability Benchmarks. To test the broader effectiveness of our strategy, we applied the 80% low-entropy steps pruning strategy to multiple benchmarks across both Heepseek-R1-775 14B and Qwen3-8B models.
7

---

Published as a conference paper at ICLR 2026
Table preeents comprehensive results comparing our compressed CoT method against full CoT baselines. Our approach consistently achieves substantial efficiency gains while maintaining or improving accuracy across all models. The DeepSeck-RI series shows remarkable token ductions: 29.7-37.0% for the 7B model and 30.6-43.5% for the 14B model across mathemati- cal benchmarks, with GSM8k showing slight aceuracy improvements for both sizes. Notably, Qwen3-8B demonstrates the strongest performance with impressive token reductions of 16.2- 44.9% while maintaining competitive accuracy and even achieving slight improvements on AIME 2024
..8%). The arosttrrc consistency- Sppaminnnn--nannnnggggg both DeepSeek-RR R1 and Qwen3 model families demonstrates that step entropy is a robust and generalizable principle for identifying redundancy, independent of model architecture, size. For a detailed analysis of our method's performance across different domains, we conducted experiments on the MMLU bench- mark (Hendrycks et al.). The results, presented in Appendix G demonstrate that step entropy is generalizable metric for identifying redundancy beyond mathematical reasoning.
Scalability and Dataset Creation. To ensure our method scales beyond controlled experiments, we further validated that the 80% low-entropy pruning strategy holds at a much larger scale. To validate this strategy at scale, we applied this compression pipeline to the entire DeepScaler (40k) and OpenR 1eenR--Mathe datasets (90k). The results in Table cablleeoonfimm that even across these tens of thousands of examples, the accuracy of the statically compressed Co'Ts remains almost identical to that of the full CoT's This large-scale valida- tion not only proves the robustness of our method but also serves as the direct procedure for Dataset Creation. The final dataset (130K), consisting of (problem, compressed CoT, answer) pairs our training strategy.
Table 2: Comparing the Accuracy (%) of Full CoT (Full Chain of Thought) against Our Compressed CoT, based DeepSeek-RI-7B on two large datasets: DeepScaleR (40K) and OpenR I-Math (90K).
Model
DeepScaleR
-PatRI
CoT
Compressed COT
62.17
47.31
Discussion of Our Method v.s. Directly Mask- ing Tokens A crucial aspect of our methodol- ogy is the decision to prune entire reasoning steps rather than individual tokens. To justify this, we compared our step-based pruning approach against a token-based pruning baseline, where we remove the lowest-entropy tokens from the think- ing trace irrespective of the steps they belong to. The results, shown in Figure 3 are unequivocal. While our step-pruning method maintains base- line accuracy even after removing up 40% of the total thinking tokens, the token-pruning ap- proach leads to sharp and immediate decline in performance. Accuracy drops significantly af- ter just a 20% token mask ratio. This demon- strates that reasoning step is the correct semantic unit for compression. Removing individual low- entropy tokens (e.g., common words or operators) can break the syntactic and semantic integrity of a critical reasoning step, rendering it incomprehen- sible to the model. In contrast, removing an entire low-entropy step preserves the structure of the maining, more important steps, leading to a much more robust compression strategy.
Accuracy vs. Thinking Tokens Mask Ratio
75
70
33
40
50
To Tolens Mask Ratio (%)
etep-entropyl teken--etropp
Figure 3: Comparing the accuracy of Our Method (via step-entropy) and Directly Masking Tokens (via token-entropy) across various thinking token mask ratios, with Full COT serving as the baseline of Deepseek- R1-14B on DeepScaleR dataset.
4.3 TRAINING AND EVALUATION FOR AUTONOMOUS COMPRESSION
We evaluate our two-stage training framework (detailed in Section 3.3) for autonomous CoT com- pression. We analyze the trained model's balance of accuracy and efficiency, and benchmark its performance against advanced compression baselines to demonstrate its state-of-the-art capabilities.
8

---

Published as a conference paper at ICLR 2026
Experimental Setup We use ----seek-R--issill--Ne- (DeepScaleR, OpenRI preprocessed using 80% entropy- based pruning, yielding 70k training samples after filtering sequences exceeding 4096 tokens. Stage 1 (SFT): 3 epochs on 70k samples using DeepSpeed Stage and LoRA (r=16, a=16). Stage 2 (GRPO): 10k samples with reward parameters Thigh 0.8, Tlaw 0.5, Takip 100, Tlenghh 3500. Training uses DeepSpeed Stage 3, LoRA (r=16, a=32), AdamW, G=14, KL=0.04 on 8x80GB GPUs. More details and the descriptions of baselines can be found Appendix C
with 130k mathematical problems
Training Experimental Analysis The results of our two-stage training process, presented in Table 3 demonstrate the effectiveness of our approach in creating efficient yet powerful LRM.
The initial SFT stage provides a strong foundation by teaching the model to imitate compressed reasoning paths, achieving significant token reductions (e.g., 43% on GSM8K). On AIME 2024, achieves a 42% token reduction with a moderate drop in accuracy. The subsequent GRPO stage optimizes this behavior through reward-drived learning, teaching the model to intelligently bal- ance efficiency and correctness. The final "SFT + GRPO" model achieves impressive efficiency gains across all benchmarks-
44% token reduction on GSM8K, 35% on Math500, 57% on AIME 2024, and 41 % on AIME 2025-while maintaining or even improving accuracy (e.g., GSM8K: 78.54% 79. 15%155)) This result demonstrates that the model learns a nuanced, adaptive policy that goes beyond static imitation (see Appendix Foor for reward components ablation).
Finally, comparing our trained model (SFT + GRPO) to the results static pruning inference strat- egy from Table T reveals the key ad advantage of our training methodology. While static pruning is effective, our trained model learns a more dynamic and ultimately superior compression policy. On the most complex benchmark, AIME 2024, our trained model not only achieves far greater token reduction (a 57.0% drop vs. 36.3% for static pruning) but also slightly improves accuracy (57.145 56.67%) This indicates that by training the model, it learns more sophisticated, context-aware policy that can more effectively identify and prune redundancy in complex reasoning chains than fixed, static rule. While static pruning preserves slightly higher accuracy on simpler benchmarks like Math500, the trained model's ability to achieve state-of-the-an compression on challenging problems makes it more robust and powerful solution for practical deployment.
Table 3: Comparison of Pass @1 Accuracy (ACC %) and Average Thinking Tokens per answer across baseline (DeepSeek-R1-22)) SFT, and SFT+GRPO training results on GSM8k, Math500, AIME2024, and AIME2025 benchmarks.
Method
GSM8k
Math500
AIME 2024
AIME 2025
ACC (%)
Avg Tokens
ACC (%)
Avg Tokens
(CC(%)
Avg Tokens
ACC (%)
Avg Tokens
DeepSeck-RLe R1-77
78.54
298 33
88.17
3703.83
63.33
15843.43
35.71
18203,25 23
SFT
78.47
169 (148))
85.92
2776.48 (1068))
56.67
9231.40 (L485)
30.00
11771.80 (1995)
SFT+ GRPO
79.15
168.05 (1445))
85.00
2439.42 (1888)
57.14
6839.33 (U6795)
33.33
10750.03 (14175
Table 4: Comparative analysis of various Chain-of-Thouglh (CoT) compression methods on Math500 and AIMH2024 benchmarks. All values represent the percentage change in accuracy (ACC) and generated token count relative to an uncompressed Full-CoT baseline.
Method
Math500
AIME 2024
ACC
Tokens
ACC
Tokens
Full-CoT (Baseline)
Token-efficient Prompts Lee et al.
2025
16.4%
111.1%
10.0
17.2%
LC-Prompt (ia et al.
2025
11.6%
.9.8%
12.6%
166.5
CoT-Valve Ma al.
2025b
110.6%
148.4%
115.0%
134.6%
TokenSkip X et al.
2025
15.2%
111.1%
112.3%
127.5%
RI -l-Coopress Wang ct al,
2025a
13.2%
420.3%
16.2%
112.9%
Our Method (SFT RL)
13.2%
135.0%
46.2%
55..0%
Comparison with Baselines To contextualize performance, compare our final trained model against several tfate-of-the-ar CoT compression methods in Table While most existing ap- proaches offer some token savings, they often come at the cost of significant drop in accuracy. For

---

Published as a conference paper at ICLR 2026
instance, CoT- Valve achieves 48.4% token reduction Math500 but with steep 10.6% accuracy penalty. Our method, however, establishes a new state-of- -f---h---tttr---e art in the accuracy-efficienny trade-off. On Math500, model matches the accuracy drop of the strongest baseline, RI-Compress (13.2%%), while delivering nearly double the token savings (135.0% vs. J20.3%). The advantage is even more pronounced on AIME 2024. Our method again matches the aceuracy performance of R1 R1-Compress (16.2%) but achieves a massive .7.0% reduction in tokens-more than four times the savings of- fered by RI-Compress (112.9%). This robust performance across benchmarks validates that our entropy-gun guropy-guided, two-stage training strategy is highly effective at producing models that reason both accurately and efficiently, outperforming existing methods.
CONCLUSION AND FUTURE WORK
Conclusion We introduced a novel framework using step entropy to identify and compress redun- dant steps in LLM Chain-of-Thoughhe reasoning. Our empirical validation, confirmed across multiple model architectures, demonstrates that pruning up to 80% of low-entropy steps maintains accuracy while achieving substantial token reductions (16-57%). Furthermore, our two-stage training strat- egy successfully enables models to autonomously generate these compressed reasoning trajectories, offering significant implications for efficient LLM deployment.
Limitations and Future Work Our work has several limitations that suggest avenues for future research. First, our method's reliance on newline characters (n))n) generated by LLM segment reasoning steps and its validation primarily on mathematical and select MMLU domains may limit its generalizability. Future work should explore more robust semantic segmentation and test the step entropy framework a broader range of open-ended and multimodal reasoning tasks. Second, the fixed 80% pruning threshold, while empirically effective, may not be universally optimal across all model architectures and problem types. Developing adaptive compression strategies that dynami- cally adjust the pruning ratio based on task complexity or model characteristics presents a promising direction. A more detailed discussion of these points is available in Appendix H
ACKNOWLEDGMENTS
This work supported in part by the CUHK Research Matching Scheme under Grant No. 7106937 and 8601 130.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We have adhered to the ICLR Code of Ethics throughout the development of this research. Our work focuses on improving the inference efficiency of large language models on publicly available mathematical and multitask benchmarks (GSM8K, Math500, AIME, MMLU, etc.). We recognize the dual-use potential of highly efficient reasoning models; while they can serve as valuable tools for education and research, they could also be misused for tasks like automated academic dishonesty. The primary goal of our research to enhance the practical deployment of LLMs by reducing their computational cost, making powerful reasoning tools more accessible and sustainable, and to contribute to the scientific understanding of redundancy in machine- -echineegenerated thought processes. Our research utilizes publicly available pre-trained models. While our two-stage training process fine-tunes these models, we have made no effort to alter or remove their inherent safety mechanisms. We believe our work creating more efficient and transparent reasoning systems aligns with the principles of responsible Al development.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility our research. All experiments were conducted using publicly available large language models (DeepSeek-RI, Qwen3) and standard academic benchmarks, the specifics of which are detailed in Section + and the Appendix G To facilitate the full reproduction of findings, will make source code and training configurations publicly available upon publication. This release will include: (1) the implementation for calculating step entropy and performing static CoT pruning, (2) the scripts used generate the compressed dataset for training, and (3) the complete code for both the Supervised Fine-Tuning (SFT) and Group Rel- ative Policy Optimization (GRPO) training stages. Key hyperparameters and experimental settings, such as the pruning ratio ( k=0.8), LoRA configurations, and the specific reward components for GRPO, are described in detail in our Experimental Setup of Section 4.3 and the Training Details in the Appendix C
REFERENCES
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14111 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.066: 2025.
Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. The danger of overthinking: Examin- ing the reasoning-action dilemma in agentic tasks. arXiv preprint 25X:022502..08235, 2025.
dataset card AIME. Url http::/wwennngeeeeeaaaaaaaeninen 2024. URL
dataset card AIME. Url hhtp://ngginggaccooodddssssooppp
2025. URL https://
wwwgingface.co./////ooooo htt.g///c....oetttaaaaaeeeerrrao
Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stu- art Shieber. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.0146/0 2023.
Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to inter- nalize cot step by step. arXiv preprint arXiv:2405..48838 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint ar0iv:2501. 12000....94....... 2025.

---

Published as a conference paper at ICLR 2026
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations.
Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of-thought of Ilms via reinforcement learning. arXiv preprint arXiv:2504.01296. 2025.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Yuxuan Jiang, Dawei Li, and Frank Ferraro. Drp:
Distilled reasoning pruning with swill-aware step decomposition for efficient large reasoning models, 2025. URL
abs/
22555.1977
Ayeong Lee, Ethan Che, and Tianyi Peng. How well do llms compress their own ohhain-of-thought a token complexity approach, 2025. Utt/attt..//aaaav....rr///22... 01141
Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance inference efficiency of large language models. arXiv preprint crXiv:2310.00201, 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman. Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
Tengxiao Liu, Qipeng Guo, Xiangkun Hu, Cheng Jiayang, Yue Zhang, Xipeng Qiu, and Zheng Zhang. Can language models learn to skip steps?, 2024. URL https::/aaxiv. abs/
241 018555
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1..55:17 2017.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deep- scaler: Surpassing ol-preview with 1.5b model by scaling rl. httss//pretty-rrado Bee--Moe--by--Saaiinee
b75. 1968 11681192c14660 4680555e10000000 Ba4e, 2025. Notion Blog.
Wenjie Ma, Jingxuan Hc, Charlic Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effecttive without thinking. arXiv preprint wwxiv:2504.098555 2025a.
Xinyin Ma. Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. Cot-valve: Length-
compressible chain-of-thought tuning, 2025b. URL
09601
OpenAl. Openai o3 and 64-mini system card, 2025. cdn cdn.opene
URL
https://
o3-and- o3-aad-oo--ini---yytemmm--o www.ssyysee-cmma....... -0--
Dataset OpenR1 Openr1-math-2200k dataset, 2025. URL (atasets/ppen-22/ppalll -Math- 220k
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.033300 2024.
Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, Zhaoxiang Liu, and Shiguo Lian. Dast: Difficulty-daaptve slow-thinking for large reason- ing models. arXiv preprint arXiv:2503.04477 2025.
12

---

Published as a conference paper at ICLR 2026
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang. Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXhv25503.1611, 2025.
Wenhui Tan, Jiaze Li, Jianzhong Ju, Zhenbo Luo, Jian Luan, and Ruihua Song. Think silently, think fast: Dynamic latent compression of I1m reasoning chains, 2025. URL httpps:s//aaiiitt org/
Abbs/2505.1655
Yibo Wang, Li Shen, Huanjin Yao, Tiansheng Huang, Rui Liu, Naiqiang Tan, Jiaxing Huang, Kai Zhang, and Dacheng Tao. R -coomprsss: Long chain-of-thought compression via chunk compres- sion and search, 2025a.
wttp:iii.orgaaooo abs 16838
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang. Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. Thoughts are all over the place: On the underthinking of ol-like 1ms. arXiv preprint arXiv:2501. 2025b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 01224844248879 2022.
Heming Xia, Chak Tou Leong, Wenjie Wang. Yongqi Li, and Wenjie Li. Tokenskip: Control- lable chain-of-thought compression in Ilms, 2025. URL wtppaaxxivvv..o
12067
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rulc-based reinforcement learning, arXiv preprint arXiv:2502.14766. 2025.
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Edward Yeo, Yuxuan Tong. Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain- of-thought reasoning in llms. arXiv preprint arXiv:2502.03337. 2025.
Jianyuan Zhong, Zeju Li, Zhijian Xu, Xiangyu Wen, Kezhi Li, and Qiang Xu. Solve-detect-verffy: Inference-ting time scaling with flexible generative verifier. arXiv preprint arXiv:2505.11966 2025a.
Jianyuan Zhong, Zeju Li, Zhijian Xu, Xiangyu Wen, and Qiang Xu. Dyve: Thinking fast and slow for dynamic process verification. arXiv preprint 17X7: 2025b.
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
We use Large Language Models (LLMs), including ChatGPT and Gemini, solely for the purpose of editing and polishing the writing this paper.
B PROOFS OF THEORETICAL RESULTS
B. PROOF OF LEMMA 1
Proof. Now assume that the entropy of the step S, is low, i... (S))S<<)) is low, we want to explore the relation of S and final solution A. We consider the conditional mutual information
1(5;;
S,-,, Stt Sx)
(7)
=H,))))) (SS)SS,,
(8)
S (H())),)) (()
(9)
=(())S<<)) [(S);
(10)
Since (551; 2 O, have I(S;; A(S,) H(S))Se))
B.2 PROOF THEBRRM
Proof. Assume that there are K+1 steps, S Sho Sk. Skic and we have
A| Σ T(Sk;; Al(C//(Sk Sk,l))
(11)
Without loss of generality, assume the indices h; arranged in descending order, i.e., Ki k3-1- Therefore, we could split the sequence C/[Sk...... Sk.] into Sck, and Syk, /SSko ..., Skmnl Now consider the item with k, according to Lemma we have:
[[SL::|[[C/Stto Sk.)) S [(5 Sk.1)
(12)
= =S(Sk, [8ck, /S>x// (Skg. ())
(13)
< (S (Sck.)
(14)
Therefore, we conclude that
I(Š; ([S:AI(C/) (C/5)) Σ I(Su.; Al(C/[Skon Sx.)))
(15)
K
< (SSk,1Sck))
(16)
1=0
C TRAINING DETAILS
Our experiments utilize DeepSeek-R1-7B as the base Large Reasoning Model (LRM). For data preparation, began with initial dataset of 130k (DeepScaleR and OpenRI-Math) mathematical problems, which were pre-processed by masking 80% of their low-entropy steps. After filtering out sequences exceeding 4096 tokens, obtained refined dataset of 70k samples for the initial training stage.
Supervised Fine- Fine-Tuning (SFT). The 70k pre-processed samples used for SFT. This stage was conducted for epochs using DeepSpeed Stage for distributed training and LoRA PEFT Hu al. (2022) with parameters 1=16 and o=16.
Stage 1:
14

---

Published as a conference paper at ICLR 2026
Stage 2: Reinforcement Learning (RL). For the RL phase, subset of 10k data samples domly selected from the 70k SFT-trained samples. We employed Group Relative Policy Optimiza- tion (GRPO) to further train the SFT-initialized model. This stage also utilized DeepSpeed Stage for distributed training. The optimization objective involved a composite reward function designed to balance accuracy, [SKIP] token ratio, [SKIP] token number, and overall response length. LoRA PEFT was applied with parameters r=16 and a=32, and the Adam W (017)) was used. Key GRPO parameters included G=14 samples per imput and KL coefficient of 0.04. All experiments were performed on a cluster of 8 GPUs, each has 80GB RAM and over 300 TFLOPS of BF16 compute performance.
& Hutter
D BASELINES
To rigorously evaluate our proposed method, we compare it against several representative baselines spanning different Chain-of-Thought (CoT) compression strategies. These baselines include zero- shot prompting techniques that require no additional training, as well as more advanced, state-of- the-art methods that involve fine-tuning or specialized architectures.
Prompt-BeConcise: A zero-shot prompting technique, as explored in Lee et al. (025 we select the prompt of Concise to encourage the model generate more succinct reasoning chain naturally.
LC-Prompt (Lennghh-Contool Prompt): Another zero-shot approach, adapted from Xia et al. (2025), that directly instructs the model to reduce the length of its reasoning by a fixed proportion, using the prompt Please reduce about 50% of the words in the Chain-of Thought process.
Advanced Compression Methods: We also compare against recent, more sophisticated techniques:
Col-Valve (a ct al. 2025by: A method that fine-tunes a model to generate CoTs
controllable length for adjusting reasoning verbosity.
TokenSkip (Xia al. 2025): An approach that enables controllable, token-level skipping during the generation process, allowing for fine-grained compression of the reasoning path.
RI-Compress (Wang et al. (025a): This method performs compression at coarser gran- ularity by operating on "chunks" of the reasoning chain, combining compression with search algorithm maintain logical coherence.
E ABLATION STUDY ON THE STATIC PRUNING STRATEGY
We conduct an ablation study to identify the optimal replacement strategy for pruned reasoning steps, as simply deleting them may disrupt the model's logical flow. We compare four strategies aggressive pruning ratios (80% 85%, and 90%) on DeepSeck-R1-785
Replace with "[SKIP]": Our proposed method, using a distinct special token as a place- holder.
Replace with "[MASK]"; generic mask token.
Replace with "and then":: A natural language filler phrase.
Directly Remove Steps: Deleting the pruned step entirely from the context.
methods maintain full CoT accuracy (74%) at an 80% pruning ratio. However, compression increases, directly removing steps using "[MASK]" leads to faster accuracy degradation (68% at 90% pruning) compared to using explicit placeholders like "[SKIP]" or "and then" (70% 90% pruning).
As shown in Figure
This result indicates that explicitly signaling an omitted step is critical for reasoning coherence under high compression. We selected "[SKIP]" as primary strategy due to its superior token efficiency (a single token) and its unambiguous semantic role as a placeholder, which avoids introducing po- tentially confounding natural language.
15

---

Published as a conference paper at ICLR 2026
Impact of Pruning Strategy on Final Accuracy
74
Rinal Answer Accuracy (80
72-
71-
70-
69
Replace
Replace with "[MASK]"
Replace with "and then'
67-
Directly Remove Steps
Full CoT Accuracy (74%)
66
Thinking Accuracy (66%)
Pruning Ratio (%)
Figure 4: Ablation study different replacement strategies for pruned low-entropy steps. The experiment is conducted on DeepSeck-Ri with the same sampled data of DeepScaleR in Fig (left), comparing four methods for handling pruned steps high pruning ratios (80% 85%, 90%).
F REWARD COMPONENTS ABLATION STUDY
Table 5: Ablation experiment for different rewards in GRPO on GSM8K and Math500 benchmarks.
Method
GSMSk
Math500
ACC (%)
Avg Tokens
ACC (%)
Avg Tokens
DeepSeek-R1-77
78.54
296.09
88.17
3703.83
R. + Hor
75.93
557.17
51.00
4082.70
R. Ror + Pmn
78.70
458.27
85.00
2693.91
R. + Rer + R.n + Rei
79.15
168.05
85.00
2439.42
Table 5 demonstrates the necessity of our multi-component reward design through systematic abla- tion. Using only correctness and skip ratio rewards (R, Rsr) yields catastrophic results, severely Math500: 17-11.- -0.0%) while paradoxically increasing token usage by 88.2% and 10.2% respectively. This indicates that naive skip optimiza- tion without constraints leads to degenerate policies generating excessive low-quality [SKIP] tokens. Adding the skip number penalty (RRn) restores competitive accuracy but token usage remains sub- optimal. The complete reward function (Ra Rgr + Rg,. + Rri) achieves optimal balance, maintain- ing near-baseline accuracy while delivering substantial efficiency gains: 43.3% token reduction on GSM8K and 34. % on Math500. These results underscore that effective CoT compression requires carefully balanced multi-objective optimization, where each component addresses specific failure modes to enable robust compression without sacrificing reasoning quality.
degrading performance (GSM8k: 78.54-75..93
G EXTENDED EXPERIMENTS
Model-aware Experiments To validate the generalizability and robustness of our step entropy- based compression method across different model architectures and scales, we conducted compre- hensive model-aware experiments on four distinct Large Reasoning Models: DeepSeek-R1-1 -1-7B, DeepSeek-RR PinoSeck-RI-14P R1-44B Qwen3-8B, and QwQ-32B, across four mathematical reasoning benchmarks. The results in Table 6 demonstrate remarkable consistency across diverse model architectures, with our method showing consistent performance across three different model families families-DeepSeel. Qwen3, and QwQ. This cross-architecture consisteney indicates that step entropy captures funda- mental properties reasoning redundancy that transcend specific architectural choices or training methodologies, making it generalizable metric for identifying redundant reasoning steps. The
16

---

Published as a conference paper at ICLR 2026
compression benefits scale effectively across model sizes ranging from 7B to 32B parameters, with token reduction percentages remaining relatively consistent within model families while absolute to- ken savings increase with model scale. DeepSeek-RI RI models achieve token reductions from 0.6% to 43.5% while maintaining or improving accuracy, with the 14B variant showing particularly impres- sive performance gains GSM8k (82.64% (62.64%--84.00%) Qwen3-8B exhibits the most aggressive compression capabilities with reductions ranging from 16.2% to 44.9%, even achieving accuracy improvements on AIME 2024 the highest compression potential with up to 55.1% 1% token reduction on AIME 2024, indicating that larger models generate proportionally more redundant reasoning steps. Benchmark -speciffiisssssccccceeececicccie analysis reveals distinct patterns: GSM8k shows the smallest token reductions but maintains accuracy, sug- gesting elementary problems require fewer redundant steps; AIME benchmarks consistently show the highest compression ratios (36.3% to 55.1 () across all models, indicating complex competition- level problems generate the most redundancy; and Math500 demonstrates balanced performance with 27.0-33.2% token reductions while maintaining high accuracy. The consistency of compres- sion patterns across fundamentally different training methodologies provides strong evidence that step entropy captures universal properties of reasoning redundancy rather than model-specific arti- facts, making our compression framework broadly applicable to current Large Reasoning Models while delivering substantial computational efficiency gains for practical deployment scenarios.
QwQ-32B, as the largest model, demonstrates
Table 6: Comparing the full COT baseline with our proposed step-entropy based pruning (Our) method, which prunes 80% of the lowest-entropy steps for DeepSeek-R1-7B, 14B, Qwen-8B and QwQ-32B. We conduct experiments to get the Pass@1 Accuracy(ACC)(5) and the number of Av- erage Thinking Tokens Per Answer (contains the Unicode characters) during the inference on GSM8K, Math500, AIME2024 and AIME2025.
Method
GSMBk
Math500
AIME 2024
AIME 2025
(S)
Asg Tokens
ACC (%)
Avg Tokens
ACC (%)
Avg Tokens
ACC
Aug Tokens
DeepSock-R--7B
78.51
298.33
88.17
3703.83
63.33
15813.43
35.71
18203 23
DeepScol-R1-7 RR--8
82
294.29
2604..2 (130.7%)
10092 80 (136.394)
35.71
11471.17 (272
DespScck-R1-B5
82.60
84.37
2853.73
15414.83
$8.62
18000.10
DeepSeck-R1-1e R1-B4 (Our)
84.00
278.29
121.990
82.16
1980.97
58.62
8705.57
(148.8))
$1.72
10842.07 (U0083)
Owen3-8B
94.46
3053.67
91.37
7138.49
79.31
20936.57
76.92
19902.45 43
Owee3-8B 8B (Our)
94.39
2557.47 (100.29))
91.13
5209. 24
(227
81.48
9544.13 (14499))
11533,57
76.00
11716.63
(AL118)
OwQ-32B
94:09
1978.91
92.35
5955,
79.13
21243.93
66.67
23711.00
QwQ-32B (Our)
93.56
1629.03
91.75
3978.06
74.07
(L56 (9))
66.52
13434.17
(190
Table 7: Comparing the full COT and No Thinking baseline with our proposed step-entropy based pruming method, which prunes 80% and 90% of the lowest-entropy steps for QwQ-32B. We con- duct experiments to get the Pass @1 I Accuracy(ACC)( and the number of Average Thinking To- kens Per Answer (contains the Unicode characters) during the inference on GSM8K, Math500, AIME2024 and AIME2025.
Method
GSMSK
Math500
AIME 2024
AIME 2025
ACC (S))
Avg Tokens
ACC (%)
Avg Tokens
(%C)
Avg Tokens
(CC(
Avg Tokens
QwQ-328 (Pull COT)
94.00
1978.1
92.35
1955, 30
79.13
21343.93
66.67
23711.60
QwQ-32B (80%)
93.
1629.03 (137.74)
91.75
3978 06 cusalar)
74.07
9544.15 (U55.1%)
65.52
13434.17
(44448)
QwQ-32B (90S)
93. 56
1524,03 (122008))
90.95
3011.36 (440.49)
70.37
7523.70
(104.89)
66.67
6459.73
(172893)
QwQ-32B (No thinking)
93.86
88.53
62.07
56.67
Domain-aware Experiments To evaluate the domain generalizability of our step entropy-based compression method beyond mathematical reasoning, we conducted experiments on MMLU (Mas- sive Multitask Language Understanding) benchmarks, specifically focusing on College Medicine present results for DeepSeek-R1-7B and QwQ-32B models respectively, comparing different compression levels (80%, 90% low-entropy step pruning) against full CoT and no-thinking baselines. The results reveal domain-specific compression char- acteristics that differ significantly from mathematical reasoning tasks, with both models showing varying degrees of compression tolerance across the two MMLU domains.
and High School History tasks. Tables and X
DeepSeek-RI.s R1-7B demonstrates robust performance on MMLU tasks with our compression method, and High School His- while reducing token usage by 18.6% and 7.1% % respectively at 80% com- pression. Notably, the model maintains improves accuracy even at 90% compression levels,
achieving accuracy improvements on College Medicine (61.73%-66..34))
tory
17

---

Published as a conference paper at ICLR 2026
suggesting that knowledge-basd baseedeee-asddd reasoning tasks contain substantial redundancy that can be effec- tively identified through step entropy. The dramatic performance drop in the no-thinking baseline (52.46% and 47.82%) emphasizes the critical importance of maintaining some reasoning structure, validating our selective pruning approach over complete reasoning elimination.
QwQ-32B exhibits even stronger compression capabilitiie on MMLU benchmarks, maintaining per- fect accuracy preservation on High School History (92.83%) across all compression levels while achieving substantial token reductions of up to 20.1 1 % 90% compression. On College Medicine, the model shows minimal accuracy degradation 13% -9%.974 with significant efficiency gains (15.0-20.1 1 % token reduction). The domain-specific patterns- where History tasks show higher compression tolerance than Medicine tasks- that factual recall and historical reasoning contain more redundant steps than medical reasoning. which may require more careful step-by-step analysis. These results demonstrate that our step entropy method successfully generalizes beyond mathematical domains while revealing important domain-specing characteristics that could inform adaptive compression strategies.
Table 8: QwQ-32B performance on MMLU-College Medicine and MMLU-High School History datasets showing accuracy and average tokens per answer across different pruning levels. Comparing the full COT and No Thinking baseline with our proposed step-entropy based pruning method, which prunes 80% and 90% of the lowest-entropy steps of per answer thinking tokens reduction percentages.
Method
MMLU-College Medicine
MMLU--Hgg School History
ACC (%)
Avg Tokens Per Answer
ACC (%)
Avg Tokens Per Answer
QwQ-32B (Full COT)
86.13
2912.3
92.83
1703.9
QwQ-32B (80%)
84.97
2475.9 (L15.0%)
92.83
1683.9 (11.2%)
QwQ-32B (90%) QwQ-32B (No Thinking)
84.97
2326.4 (L20.1%)
92.83
1683.9 (11.2%)
84.30
91.14
Table 9: PeepSeck-R1-77 performance on MMLU-College Medicine and MMLU-High School History datasets showing accuracy and average tokens per answer across different pruning levels. Comparing the full COT and No Thinking baseline with our proposed step-entropy based pruning method, which prunes 80% and 90% of the lowest-entropy steps of per answer thinking tokens reduction percentages.
Method
MMLU-Collegee Medicine
MMLU-High School History
ACC (%)
Avg Tokens Per Answer
ACC (%)
Avg Tokens Per Answer
DeepSeck-R11788 (Full COT)
61.73
2612.7
61.74
2054.3
DeepSeck-R11788 (80%)
62,34
2127.9 (118.6%)
64.32
1907.5 (L7.19)
DeepSeck-R1-78 (90%)
62,34
2069.4 (120.8%)
61.74
1936.2 (.5.7%)
DeepSeck-R1-7B (No Thinking)
52.46
47.82
Key Findings and Analysis Our comprehensive experimental evaluation reveals several critical insights about Chain-of-Thouge compression via step entropy. The most significant finding is that 80% of low-entropy reasoning steps can be safely pruned without accuracy degradation across mul- tiple model architectures and reasoning domains. This substantial redundancy indicates that current Large Reasoning Models generate highly verbose thought processes, with the majority of steps con- tributing minimal informational value to final answer quality. The cross- cross-architccturul consistency of our results spannsnng DeepSeek-Ri (7B, 14B), Qwen3-8B, and QwQ-32B demonstrates that step entropy captures fundamental properties of reasoning redundancy that transcend specific model de- signs. Token reductions ranging from 16.2% to 55.1% % across mathematical benchmarks, combined with maintained or improved accuracy, provide strong evidence that our entropy-basee metric suc- cessfully identifies genuinely redundant reasoning components rather than model-specific artifacts. Nomain-specinn compression patterns emerge from our MMLU experiments, revealing that factual reasoning tasks (High School History) tolerate higher compression rates than analytical reasoning tasks (College Medicine). QwQ-32B maintains perfect accuracy on History tasks while achieving
18

---

Published as a conference paper at ICLR 2026
20.1% token reduction, whereas medical reasoning shows more sensitivity to aggressive compres- sion. This suggests that different cognitive processes exhibit varying degrees of redundancy, opening avenues for adaptive, domain- compression strategies.
H EXTENDED DISCUSSION ON LIMITATIONS AND FUTURE WORK
Our framework provides robust method for CoT compression, but it also highlights several impor- tant areas for future investigation.
Sensitivity to Step Segmentation core methodological choice our work the definition of a "reasoning step," which we segment based on the ((n) delimiters generated by the LRM. This approach is practical and effective, but the entropy patterns we identify might be sensitive this specific segmentation strategy. Different step boundaries- for instance, segmenting by sentences or logical clauses-con could potentially alter the calculated entropies and the resulting compression per- formance. Future work should investigate the robustness of our findings to alternative segmentation methods and explore more semantically grounded techniques for automatically identifying discrete reasoning steps.
Domain and Modality Generalizability Our validation focuses primarily on mathematical rea- soning and select knowledge-based tasks from MMLU. While we demonstrate strong performance in these areas, the transferability of step entropy as universal metric for redundancy is not yet established for reasoning types. The nature of redundancy in more open-ended, creative, or commonsense reasoning tasks may differ significantly. Therefore, an important direction for future work is to apply and adapt our framework this method to multimodal reasoning, where steps may involve interpreting images or data visual- izations, represents a challenging but valuable next frontier.
broader range of domains. Furthermore, extending
Adaptive and Model-Aware Compression Our experiments establish an 80% low-entropy prun- ing threshold that works remarkably well across several models. However, we acknowledge that this fixed threshold may not be universally optimal. As shown in our experiments, different models or problem domains can exhibit unique redundancy patterns. This suggests need for more dynamic approaches. A promising area for future work the development of an adaptive compression strat- egy. Such a system could dynamically adjust the pruning ratio based on the model's 's architecture, the specific problem's complexity, or the reasoning domain, moving from static threshold more intelligent, context-aware compression policy.
I BROADER IMPACT
Our work addresses the critical challenge of reasoning efficiency in practical LLM deployment, where verbose Chain-of-Thought processes create significant computational bottlenecks. By pro- viding principled method for identifying and removing redundant reasoning steps, we enable more sustainable and accessible deployment Large Reasoning Models. The interpretability benefits of maintaining explicit reasoning chains while achieving substantial compression offer advantages over latent reasoning approaches. Practitioners can retain the transparency and verifiability of explicit Chain-of-1 Thouhht while significantly reducing computational overhead. Our findings contribute the theoretical understanding of reasoning structures in Large Language Models, revealing that cur- rent models generate substantial redundancy in their thought processes. This insight informs future model design and training methodologies, potentially leading to more efficient reasoning architec- tures.
19


Reviews:
[{"review_id": "SN0EdK5IBS", "reviewer": "Reviewer_S8MC", "summary": "The paper proposes step entropy as a principled, per-step information measure for CoT, shows that pruning low-entropy steps preserves accuracy while cutting tokens, and trains models to self-compress via SFT + GRPO with an explicit [SKIP] token. Theory upper-bounds each step’s information by its entropy (Lemma/Theorem 1), and experiments across DeepSeek-R1 and Qwen families show strong accuracy–efficiency trade-offs.", "questions": "1. Entropy normalization:\nDid the authors try normalizing entropy by step length (e.g., average or log-length scaling)? If so, how did this affect correlation with information contribution?\n\n2. Alternative to entropy-based labeling:\nInstead of relying purely on entropy, have the authors tried using an LLM itself to label which steps are informational or non-trivial (e.g., “steps that advance reasoning” vs. “repetitive or obvious steps”)? Such annotations could provide a complementary supervision signal for training or validating entropy thresholds.\n\n3. Fine-tuning stability:\nDuring GRPO fine-tuning, how sensitive are results to the [SKIP] penalty coefficient? Does the model ever collapse to always skipping or never skipping?\n\n4. Adaptive threshold:\nCould the target entropy ratio κ be dynamically chosen based on per-question entropy distribution?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Using entropy as information proxy is clean. The paper defines step entropy by summing token entropies within a step and proves the conditional information contribution is bounded by this entropy, offering clear intuition for “skip low-entropy steps.” This is simple, transparent, and theoretically motivated.\n\n2. Theorem 1 provides usable intuition. Bounding the information of any subset of steps by the sum of their entropies gives a direct justification for step-level pruning rather than token-level pruning. The token-vs-step ablation empirically supports this semantic unit choice.\n\n3. Fine-tuning setup is sensible and practical. The two-stage SFT → GRPO pipeline, rewarding correctness and compression while discouraging degenerate [SKIP] flooding, is clear and leads to better compression than static pruning on hard sets (e.g., AIME 2024).\n\n4. Empirical results are good across models/benchmarks. They show ~30–55% token reductions with minimal accuracy loss; on some tasks accuracy even improves. Cross-architecture results (DeepSeek-R1-7B/14B, Qwen3-8B) and comparisons to recent compression methods are solid.", "weaknesses": "1. Step segmentation heuristic.\nSteps are segmented using simple newline heuristics. While this works reasonably, it can blur step boundaries or merge logically distinct thoughts. A sensitivity study with sentence-based or LLM-predicted segmentation would improve robustness.\n\n2. Fixed 80% pruning threshold.\nThe global 80% rule is justified empirically but may not generalize across datasets or reasoning styles. An adaptive or learned κ could better reflect per-problem difficulty.\n\n3. Unnormalized entropy may bias toward longer steps.\nThe paper uses total (unnormalized) entropy per step. While this matches the theoretical bound, longer steps automatically accumulate more entropy even when per-token uncertainty is low, potentially biasing the pruning policy. A length-normalized or mixed variant could help disambiguate whether information or verbosity drives retention.\n\n4. Scope of benchmarks.\nMost experiments center on math and logic tasks. Including one additional open-ended domain (commonsense, code, or writing) would broaden the evidence that entropy-guided compression generalizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ukipou2aVy", "reviewer": "Reviewer_1Kz9", "summary": "This paper proposes a method to prune out extra reasoning steps in reasoning models, and then SFT+RL with the result to allow for more concise chains of thought.", "questions": "None", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Clarity: The paper is generally written pretty clearly, and the method is not very complex which is nice. I would say that the writing is a little repetitive (saying the same thing many times), but not to the point where it makes the paper hard to understand.\n2. Significance: It seems that the method, while simple, is reasonably effective. It greatly compresses the resulting CoT at a reasonable loss in accuracy.", "weaknesses": "1. To be honest, the method feels rather \"hacky\" to me, inserting skip tokens based on heuristics. My feeling is that the community in general is trying to move towards methods that perform end-to-end RL in a more principled way rather than these sorts of processes.\n2. Relatedly, while this paper proposes methods to compress chains of thought, there are other methods to directly control the length of chains of thought such as L1 (Aggarwal and Welleck). These seem simpler, more elegant, and can also be retro-fitted onto existing models. I was surprised that there was no discussion of this work, and it seems like it would be a competitive baseline.\n\nL1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning\nPranjal Aggarwal, Sean Welleck", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ROlKAh6eZz", "reviewer": "Reviewer_brNR", "summary": "This paper introduces step entropy, an information-theoretic metric to quantify each reasoning step’s contribution in LLM Chain-of-Thought. By pruning up to 80% of low-entropy steps, the method reduces tokens by 16–57% with minimal accuracy loss across DeepSeek-R1 and Qwen3 models. A two-stage training framework combining SFT and GRPO further enables models to autonomously skip redundant steps via [SKIP] tokens. The approach outperforms prior CoT compression methods, offering an efficient and interpretable way to accelerate LLM reasoning.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The method is simple and easy to implement, requiring only entropy calculation and pruning based on low-information steps to construct the pruned CoT. It achieves strong performance, such as maintaining accuracy on Math500 even with a 30% compression ratio, demonstrating both effectiveness and efficiency.\n2. The introduction of the SFT+RL framework makes the approach more practical. By allowing the model to learn when to skip redundant steps automatically, it extends the static compression method into a trainable and deployable solution.", "weaknesses": "1. The segmentation and granularity of reasoning steps are not rigorously defined. The approach relies on manually designed delimiters like \\n\\n, which may not generalize well across datasets or model architectures.\n2. The definition of step entropy as the sum rather than the average of token entropies could bias the metric toward longer steps, potentially misrepresenting their true informativeness.\n3. The presentation of Table 1 is not so good. A clearer organization would make the results easier to interpret.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rbuJGaVdOV", "reviewer": "Reviewer_Fb8Y", "summary": "The paper proposes **step entropy** as a signal to identify and prune redundant segments within Chain-of-Thought (CoT) traces, aiming to make slow “deliberate” reasoning faster without sacrificing accuracy. Concretely, the step-level entropy for the \\(i\\)-th step \\(S_i\\) is defined as the sum of token entropies conditioned on the prior context, \\(H(S_i \\mid S_{<i})=\\sum_j H(t_{i,j}\\mid c_{i,j})\\). The core hypothesis is that **low-entropy steps contribute little information** to the final answer and can be safely skipped. The authors present (i) an information-theoretic motivation; (ii) a pruning recipe that removes the lowest-entropy steps and replaces them with a special token (e.g., [SKIP]); and (iii) a training pipeline (SFT and GRPO) to teach models to emit compressed CoTs during inference. Empirically, they report substantial token reductions (often 16–57%) with modest accuracy degradation on math-style reasoning benchmarks.", "questions": "1. **Causal necessity at inference:** Can you run **inference-time interventions** that compress the low-entropy steps while holding other decoding parameters fixed, and report accuracy relative to full-CoT? This would directly test whether those steps are unnecessary *causally* rather than *post hoc*.  \n2. **Fair baselines on identical data/compute:** Train strong baselines (e.g., token/chunk compression, rule-based CoT pruning) on the same ~130k instances with the same training budget. Do your gains persist under these controls?  \n3. **Difficulty-aware adaptivity:** Can the compression ratio be predicted per instance (e.g., via a learned controller that thresholds entropy or estimates a target depth \\( \\kappa(x) \\))? \n4. **Segmentation robustness & granularity:** How exactly are “steps” defined and validated? Have you analyzed token-entropy distribution and how aggregation affects pruning decisions?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **Conceptual clarity:** A clear, information-theoretic criterion (step entropy) with an intuitive link to redundancy.  \n- **Granularity choice:** Results indicate step-level pruning is more effective than naïve token-entropy pruning, suggesting the *step* is a useful unit.  \n- **Practical interface:** Using a placeholder like [SKIP] makes the compression operationally simple; ablations on replacement strategies are helpful.", "weaknesses": "1. **Autoregressive dependency not directly addressed**: Evidence is largely post-hoc (compress after generating a full CoT). In AR decoding, earlier “redundant” content can steer later tokens; removing it afterward does not prove it can be skipped causally during generation. \n2. **Low-Entropy Steps Pruning shows no pre-training practical gain; current use is post hoc.** As implemented, “Inference with Compressed CoT” is applied **after generating the full CoT**, so it yields no acceleration and provides limited practical value before additional training. To make this genuinely efficient without extensive post-training, the method should be reframed as an inference-time control。\n3. **Attribution vs. training data/compute:** The main contribution is both a compression rule and a data-construction pipeline (e.g., ~130k compressed pairs). Without baselines trained on identical data with matched optimization budgets, it’s hard to attribute gains to step entropy rather than more/better post-training.  \n4. **Fixed compression ratio:** A static global pruning rate (e.g., “up to 80%”) ignores that redundancy varies by instance difficulty, dataset, and model size; no mechanism adapts compression per-instance.  \n5. **Step segmentation heuristic:** Steps defined by formatting (e.g., `\\n\\n`) can be brittle. The paper does not validate segmentation accuracy or analyze sensitivity to finer/coarser granularity or token-entropy sparsity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)