Compressing Chain-of-Thought in LLMs via Step Entropy
Authors: name: Zeju Li; name: Jianyuan Zhong; name: Ziyang Zheng; name: Xiangyu Wen; name: Zhijian Xu; name: Yingying Cheng; name: Fan Zhang; name: Qiang Xu
Published: 2025-08-05 11:48:18

Abstract: Large Language Models (LLMs) using Chain-of-Thought (CoT) prompting excel at
complex reasoning but generate verbose thought processes with considerable
redundancy, leading to increased inference costs and reduced efficiency. We
introduce a novel CoT compression framework based on step entropy, a metric
that quantifies the informational contribution of individual reasoning steps to
identify redundancy. Through theoretical analysis and extensive empirical
validation on mathematical reasoning benchmarks, we demonstrate that steps with
low entropy are indeed highly redundant. Our experiments reveal that an
astonishing 80\% of low-entropy intermediate steps can be pruned with minor
degradation in the final answer accuracy across DeepSeek-R1-7B, 14B and
Qwen3-8B. This finding sharply contrasts with random or high-entropy pruning,
which severely impairs reasoning performance. Building on this, we propose a
novel two-stage training strategy combining Supervised Fine-Tuning (SFT) and
Group Relative Policy Optimization (GRPO) reinforcement learning. This approach
enables LLMs to autonomously learn to generate compressed COTs during inference
by strategically incorporating [SKIP] tokens. Our method significantly enhances
LLM inference efficiency while rigorously preserving accuracy, offering
profound implications for practical LLM deployment and a deeper understanding
of reasoning structures.