Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction
Authors: Ryan Lucas
Kayhan Behdin
Zhipeng Wang
Qingquan Song
Shao Tang
Rahul Mazumder
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Compression
Pruning
Reasoning
Chain-of-Thought
Primary area: optimization
TL;DR: Standard pruning causes large accuracy losses in reasoning LLMs. Reconstructing chain-of-thought traces during calibration preserves performance, maintaining ~95% accuracy even at 50% sparsity.

Abstract: Reasoning language models such as DeepSeek-R1 produce long chain-of-thought traces during inference time which make them costly to deploy at scale. We show that using compression techniques such as neural network pruning produces greater performance loss than in typical language modeling tasks, and in some cases can make the model slower since they cause the model to produce more thinking tokens but with worse performance. We show that this is partly due to the fact that standard LLM pruning methods often focus on input reconstruction, whereas reasoning is a decode-dominated task.  We introduce a simple, drop-in fix: during pruning we jointly reconstruct activations from the input and the model’s on-policy chain-of-thought traces. This “Reasoning-Aware Compression” (RAC) integrates seamlessly into existing pruning workflows such as SparseGPT, and boosts their performance significantly. Anonymized code can be found at: https://github.com/RyanLucas3/Reasoning-Aware-Compression

Published as a conference paper at ICLR 2026
REASONING MODELS CAN BE ACCURATELY PRUNED VIA CHAIN-OF-THOUGHT RECONSTRUCTION
Ryan Lucas*
Kayhan Behdin Linkedln, Sunnyvale, CA
LinkedIn, Sunnyvale, CA
Massachusetts Institute of Technology, Cambridge, MA
Zhipeng Wang" LinkedIn, Sunnyvale, CA
Shao Tang LinkedIn, Sunnyvale, CA
Qingquan Song LinkedIn, Sunnyvale, CA
Rahul Mazumder
LinkedIn, Sunnyvale, CA Massachusetts Institute of Technology, Cambridge, MA
ABSTRACT
Reasoning language models such as DeepSeek-RI produce long chain-of- thought traces during inference time which make them costly to deploy at scale. We show that using compression techniques such as neural network pruning pro- duces greater performance loss than in typical language modeling tasks, and in some cases can make the model slower since they cause the model to pro- duce more thinking tokens but with worse performance. We show that this is partly due to the fact that standard LLM pruning methods often focus on input reconstruction, whereas reasoning is decode-dominated task. We in- troduce a simple, drop-in fix: during pruning we jointly reconstruct activa- tions from the input and the model's I'ss on-policy chain-of- traces. This "Reasoning-Awan Compression" (RAC) integrates seamlessly into existing prun- ing workflows such as SparseGPT, and boosts their performance significantly. Anonymized code can found at: https::/gitaii //gpt/httb///////////////t com www////m///a/// / Reasoning- ceesoonngg-wwareee Compression
INTRODUCTION
Large Language Models (LLMs) with step-by-step reasoning abilities have become essential for solving complex, multi-step tasks in domains such as mathematics, coding, and logical reason- ing (Wei et al., 2022). Reasoning models generate explicit chains-of-thought (intermediate reason- ing steps) that significantly improve accuracy challenging benchmarks, but at the cost of pro- ducing very long outputs for each query. For example, the DeepSeck-R. R1 model (671B parameters) (DeepSeek-AA et al., 2025) achieves strong reasoning performance but must output lengthy explana- tion traces, making it extremely resource-intensive to deploy at scale (Guo et al., 2025; Zhang et al., 2025a).
To reduce the cost of serving LLMs, recent years have seen a surge of interest in LLM compres- sion techniques. Model pruning (LeCun al., 1989; Hassibi & Stork, 1992) is popular model compression technique where the goal is to remove redundant weights or neurons from the model, while ensuring the model quality remains high. This can lead to models with smaller memory and compute footprint, which can be more resource efficient. Pruning has been particularly successful when applied to LLMSs (Frantar & Alistarh, 2023; Meng et al., 2024; Sun et al., 2024). Given the computational requirement of reasoning L.Mss. model pruning has appeared as an attractive propo- sition for efficient reasoning. However, the application existing pruning methods to reasoning models can often result in significant accuracy loss (Zhang et al., 2025b), limiting the application of such compression techniques in practice where model quality of high importance.
"Correspondence: ryan't wccuerspeon.. @ggaa com, zhi pwang@1 i inkei i. com

---

Published as a conference paper at ICLR 2026
As an example, in Figure 1, we prune the DeepSeck-RI RI--iiDiiillIOUA -well-BeeeeBBBB (DeepSeck-Aletaa al., 2025) checkpoint with SparseGPT using a single-pase pass C4 calibration set of 1M tokens. At sparsity levels of 30%, 50%, and 70%, evaluate each pruned model on the MATH-500 benchmark (zero-shot, 32k max tokens). As sparsity increases, MATH-500 accuracy falls while total evaluation time grows sharply. Counter-intuitivelyy heavy pruning actually slows down inference because the model pro- duces much longer chains of thought, rambling more yet answering less accurately. Since the goal of compression to maintain accuracy while reducing inference time, this is obviously not desirable.
In this work, we focus on one-shot pruning of reasoning LLMs (that is, we do not conduct any re- training after pruning). One-shot pruming is well motiivated for this use-case, since while DeepSeck models are open-soured (i.e., open- ween-weights), the complete training and distillation pipeline required to retrain these models to full accuracy is not fully available (HuggingFace 2025). Moreover, re- training is generally expensive and requires a large cluster of several GPUs. In contrast we perform all of our one-shot pruning experiments on a single H100 GPU.
We propose new model pruning approach that bet- ter preserves the reasoning capabilities of LLMs. Our approach explicitly aligns the pruning-time reconstruc- tion problem with the activations the model computes at inference-time (in its CoT) to generate its response. This is in contrast to existing approaches such as (Zhang al., 2025b) that do not make use of model's s CoT when pruning, Our work demonstrates that by reconstructing the COT, reasoning LLMs can be pruned up 50% sparsity in one-shot accurately, maintaining up to 95% of dense model's accuracy on math and coding tasks. Ad- ditionally, our proposed method improves accuracy of pruned models on math and coding tasks by up to 17% compared to existing pruning approaches.
Accuracy vS. Dvalustion
ecross Sparsity Levels
1.0
Eveluton me
0.-
0.2
0.-
36
Sparidy (M1
Figure 1: Pruning hurts both accuracy and runtime on MATH-500.
2 BACKGROUND
Reasoning models. Conventional LLMs are trained, with parameters 0, to maximise the condi- tional likelihood no (Wo.r.-1 x) 11 Te(Ms I Ix,,<) of an output sequence y E V of length L from the vocabulary given a prompt æ. A reasoning model instead produces:
(Co:T-1, Vo:L-1),
with Co:T E VT, 10.L-1 E V""
where c is a thain-of-thought (CoTT) of length T and y is the final answer e.g. a single numeric value, complete proof, or a code block. While a conventional LLM can be prompted to emit CoT c before producing its answer reasoning models are explicitly trained so that the generated chain and answer yield a verifiable task reward R(z, v) E [o. 1], for example, exact-match grading on math, unit-test passes for code, or logical consistency checks for formal proofs. This has recently become popularized by the DEEPSEEK-H R1 reasoning model, which is optimized via Group- Optimization (GRPO) (DeepSeek-Al et al., 2025). The combined CoT and answer distribution is modeled To (c.y y x) I Te(c; xx, Cet) Tely | Iz cu:r-1). Given pre-trained LLM with parameters 0, GRPO fine-tunes 0 by drawing K full trajectories {(c()), ) for the same prompt x, evaluating a scalar task reward rh R(x, ((),y(()) and computing the clipped policy- gradient loss:
Policy
K clip pm (0), -
K
1
(ORPO =
K
-,, 1, +e) (rk F),
==
K
rk-
xo(clA)) (b) (t)
where (0) =
Teaul (okk) [o())_g((+)))) is ratio describing the probability of response k in the new policy
9 (being optimized) relative to the current policy faad Here, when F > F i.e. the k-th response generated higher reward than its group mean, the policy will be updated in favor of this response,
See Wei et al. (2022) for evidence that even noisy CoT traces hoost reasoning accuracy.
2

---

Published as a conference paper at ICLR 2026
and vice versa. This process of generating long thought chains before outputting answer be viewed as training the model to perform search i.. to try out multiple paths towards a solution before deciding answer. This is made explicit by generalizations of CoT such rre-oo-thoughtt which explicitly enforce a search-like procedure (Yao et al., 2023). Consequently, a longer search process (or higher number of inference tokens), has been associated with more accurate answers (OpenAI, 2024). While this makes these models more effective, it also means they incur substantial inference latency (Wei al., 2022; Guo et al., 2025).
LLM pruning methods. Modern language models contain billions of parameters, so model prun- ing is widely used for reducing GPU memory footprint, inference latency, and energy cost while preserving most of the model's accuracy. A popular approach to model compression is model prun- objective (Frantar & Alistarh, 2022). Suppose a pretrained LLM with L layers
ing via a layer-win is given, with layer weights We E R for & E [L], and de, Pe denote the input and output sizes of layer f, respectively. We also let X, E Rdde denote the input activations to layer gathered on a calibration dataset of N tokens. Layer-wise LLM pruning methods find compressed weights W by solving, independently for each layer:
min W.X: - W. W.Xx s.t. Weelo S W
(1)
where || |l denotes the number of non-zero coordinates of a matrix and 5 is the desired number of non-zero coordinates. Numerous algorithms have been proposed for layer- pruning of LLMs via solving (1) (Frantar & Alistarh, 2023; Meng et al., 2024; Sun et al., 2024). For structured or semi-structured pruning, the constraint can be modified comprise the set of block-sparse matrices or N:M sparse matrices, and the problem can be solved with similar algorithms.
The calibration dataset. In Equation 1, X is the so-called calibration data, and for LLMs a text corpus of size N tokens comprises the column dimension of X,- The calibration data is typically chosen to mimic the general distribution of natural language, e.g. the C4 dataset (Raffel et al., 2020) is a common choice. For standard I.MMs, the calibration data is typically derived from set of inputs (or prompts) æ. Concretely, let No:N-1 be a batch of N prompt tokens from the calibration corpus and let E E Rdx[V] denote the embedding matrix. Define the layer-wise hidden states for each token by:
(0) X
ez2
X A
fe(x E
where ft is the th transformer layer (including attention, MLP, residual connections, etc.), and ea. the embedding for token The states are stacked column-wise to obtain the calibration activation matrix:
(e-1),
xi (t-1)
(=-1 E RdxN
X
[x0
so the t-th token embedding (processed after transformer layers) is exactly the vector that will enter layer when the dense model processes token 24. This X, is what standard pruning algorithms use to measure the reconstruction error of the compressed weight matrix W.. This setup aligns with typical LLM workloads where |xx
>> |yl (long context, short reply). By contrast, in reasoning L.MMs often observe |c| + ly] >>>>> |ar| as the CoT dominates the token budget. Consequently, calibrating solely on prompts risks an inference-time distribution shift, where activations processed primarily CoT tokens rather than input tokens.
3 RELATED WORK
Benchmarking compressed reasoning L.Mss. Concurrently with our work, Zhang al. (2025b) run an extensive evaluation of compressed DeepSeek-RI variants. They apply SparseGPT to stu- dent models distilled from Qwen and LLaMA, but follow the default C4-style calibration pipeline. Their results are similar to what observe: when using generic dataset, accuracy complex reasoning tasks drops sharply with sparsity, the CoT become repetitive or degenerate, and longer post-compression outputs correlate with lower task accuracy. Unlike their empirical study, our RAC method modifies the calibration distribution itself by injecting on on-policy CoT activations, and thereby mitigates the same performance loss during pruning.
3

---

Published as a conference paper at ICLR 2026
Calibration data for post-training pruning. Most single-pass pruning algorithms (e.g- Pp WANDA (Sun et al., 2024), ALPS (Meng et al., 2024)) rely on small calibration set such as C4 to minimise layer-wise reconstruction loss. A recent paper reports that this choice far from optimal. Bandari et al. (2024) compare seven pre-training and downstream datasets and finding large accuracy differences after pruning, and that across tasks C4 is rarely the optimal choice of calibration dataset. PPC-GPT (Fan et al., 2025) distils pruned student models with synthetic CoT traces, but still computes pruning scores on standard C4 activations. In contrast, inject CoT acti- vations directly into the pruning objective, eliminating the separate distillation stage. A related idea ours self-calibration (Williams et al., 2025). While the mechanism is superficially similar to RAC in that both use model-generated text calibration data, the underlying setting is quite differ- ent. self-calibration, the model is prompted only with beginning-of-sequenn -e-uueeceee token and allowed to generate free-form text until end-of-sequence token or a length limit is reached. The resulting calibration distribution is therefore meant mimic the model's pre-trate pre-training text distribution and does not condition task prompts prune reasoning traces.
4 REASONING-AWARE COMPRESSION
The core insight of RAC is to align the pruning-tine timing-time reconstruction problem with the activations that the model actually computes at inference-tmme in order to generate the response. As such, we first recap the inference process for reasoning LLM, and then outline procedure for aligning the activations encountered during decoding with those used for calibration.
Inference in autoregressive reasoning model. Let (Zo..... ........ ... TTi TTm--) E >iin be prompt to an autoregressive reasoning model, which could be problem of math, coding, etc., and let denote the dense model's conditional distribution over the vocabulary V. At inference, reasoning model processes and generates completed sequence:
20:T+ = (No-- -T,-1 CT CT, yr+1,-... UT++))
which includes the prompt æ, the ch (CoT) tokens c, and the final answer tokens y. We partition the indices into p {O..... Tin }} being the prompt indices and D [Tim...... L} being the decode indices. Each generated token 2, (either o or yt) drawn autoregressively.
24 To( 10:t11,
te D.
To compute the generated token, for each step the model first produces embedding for the current tokens as:
x (0) Eez, E IR LEPUD
and processes them through L transformer layers computing:
x (2) = ((x((-))))) )Ts:)
1= Lb LEPUD
(4)
where again fe is the 4t transformer layer.
The hidden states that required for decoding thus:
x fE L}, tEPUD}
That is, crucially, to generated the complete sequence, the model relics on the activations that are computed the input, but also activations that arise from its own self-generated tokens. Once the final hidden state x! (L) computed, mapped to vocabulary logits via the output projection Wont E xlvxxdz given by y: Wont x!_) BR which gives the next token distribution ITa:
20.r) = softmax(y.))
Aligning pruning with decoding during offline calibration. Suppose have M calibration prompts (m) J with corresponding prompt index sets P.,. and decode index sets Dm. Standard post-training compression collects activations only for t E Pm. Lee only from the fixed prompt tokens in each sequence. For typical LLM applications. this is natural since in most settings PPm [Dml (long context, short reply). so the majority of inference-tima activations come from the prompt, and the few decode-time activations that exist often just noisy continuations that act proxies
4

---

Published as a conference paper at ICLR 2026
for the original prompt distribution, for example, in autocomplete tasks where the completion closely mirrors the previous context. In RAC. since in reasoning tasks typically [Dml >> iPmll we modify this procedure self-generating tokens during calibration simulate the activations observed at decoding time. At each decode step i, the model's own prediction is immediately re-used as the next input:
(m) r+1
Tol. ((m)
Tol. =(m)) coftmax(Wonn Wott x3 m
Dm
(5)
x-+1
(0.m)
E (
(f.) 4+1 = Se ((x (x((-1)mm) Jrset1), l= e=
(6)
which gives a new set of hidden states for reconstruction (x {xf+) The resulting layer-l input (8-1,m) is then appended new column to the decode-time activation matrix: Xit1
(f-1,m) ].
X1+1
so that after all decode steps, the activation matrix:
x
x
(1-1,,m 1,m)
LED
E RdexND
contains the same sequence of activations that the model will encounter during generation, where The full calibration matrix concatenates both prompt and decode activations
ND = Em=1 [Dmll
XBAC X xP [dox(N++NN)), with Np Σm [Pml. The RAC calibration loss for
layer is then:
M I(W, W.) x; I.m)
I(W - We) X 2
Σ Σ MELIEP
The key difference between standard prompt-only calibration and RAC is that RAC's calibration set covers "x! (f-1mmm)
1, , L. E Pm U m = I.....M, M }. i.. all activations used during the full on-policy rollout. This set is generated by the same autoregressive mechanism (2)-(4) used at inference, so the empirical distribution over these activations during calibration matches the inference- time distribution. Since the activations are collected under the dense model's own policy, the procedure is analogous to on-policy distillation in reinforcement learning (Agarwal et al., 2024).
Algorithm 1: Reasoning-Awarr Compression (RAC)
Input: {o (m)) ( M==:
calibration prompts; fe: dense LM with L layers; Tmax: max decode length; S:
target sparsity
Output: f: compressed model
Phase Activation collection
for m I,...,M M do
// Prompt phase
for t E P. do compute (0.m) (f.m) via (3)-(4); append a 1.m)
x1
to x,
// Decode phase for E Dn do sample (m) no(. 2 via (2); compute x! (0.m) (t,m)) via (3)-(4) append x; (6-1,m) to X
x
Phase II: Layer-wise compression
for
1. 1...., .., L do
X RAC +- [XP X
10
Wr - PRUNE Wt X RAC (A)
// e.g., SparseGFT, MANDA
return f (
Wz}
5

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
5.1 ONE-SHOT PRUNING
Experimental Setup. To test the effectiveness of RAC, perform one-shot pruning on two fam- ilies of open-source reasoning models: (i) DeepSeek-RI distilled Qwen variants (1.5B, 7B, 14B, 32B) released by DeepSeek-Al al. (2025), and (ii) the Qwen3 reasoning models (1.7B, 8B, 14B). Each model is pruned in one-shot with SparseGPT at layer-wise unstructured sparsity of 20%. 30%, 40% and 50% using 1M calibration tokens from: (i) the standard English-web C4 corpus. (ii) OPEN- RI-MATH-222 220K (Hugging 2025) or CODEFORCES (Shanghaoran Quan, 2025) problem state- ments without answers or reasoning traces ("prompt only"), and (iii) those prompts augmented with up to Tax = 8192 on on-policy chain of thouught tokens collected from each corresponding dense model, as described in Algorithm 1 ("RAC").
For mathematical reasoning we use MATH500 and report acc@ 1:1: the percentage of problems for which the model's single most-confident prediction exactly matches the ground-truth answer (Top-1 accuracy). For code generation we use the CodeGin evaluation harness and report acc@ 1:16, i.e. the percentage of cases in which the correct solution appears anywhere within the model's top-16 predictions, irrespective of rank (Top-16 16 accuracy). All evaluations are zero-shot with no additional few-shot examples, and a 32k output token budget. This follows the evaluation pipeline used by DeepSeek-AI et al. (2025) and the open-source replication from HuggingFace (2025).
Table DeepSeek-RI Qwen MATH500 ace@ 1:1 under one-shot pruning. Accuracy with standard error (SE)
on the left, total evaluation time in minutes on the right. Best accuracy or fastest runtime in green.
Model
Sparsity
Accuracy + SE
Runtime (min)
C4
Prompt: Only
RAC
C4
Prompt
RAC
Dense
0,832
0,832
0.832
22.6
22.6
22.6
20%
0.822 (0.017)
0.840 (0.016)
0.832 (0.017)
25.6
24.4
22.6
1.5B
30%
0.762 (0.019)
(0.080)
0.822 (0.017)
31.6
49.9
25.8
40%
0.658 (0.021)
(0.020)
0,774 (0.019)
57.7
65.9
32.1
0.356 (0.021)
(0.922)))
0.664 (0.021)
156.5
154.8
Dense
0,936
0.936
0.936
23.3
23.3
23.3
20%
0.902 (0.013)
(0.01))
0.934 (0.011)
23.5
23.5
21.7
7B
30%
0.904 (0.013)
(0.012)
0,934 (0.011)
27.4
26.8
25.2
(0.014)
(0.014)
0,912
38.3
37.4
50%
0.744 (0.020)
(0.01)))
0.900 (0.013)
135.0
115.6
35.3
Dense
0,941
0,941
0,941
50,3
50.3
50.3
20%
0.952 (0.010)
0,954 (0.009)
0.962 (0.009)
58.4
56.5
54.7
14B
30%
0.936 ()
(0.30()
0.936 (0.011)
67.2
64.9
58.7
40%
(0.003)
(0.012)
0,942 (0.010)
97.9
73.1
66.3
0.878 (0.015)
(0.015)
(0.013)
171.4
Dense
0.942 (0.011)
0.942 (0.011)
0.942 (0.011)
64.3
64.3
64.3
20%
0.950 (0.010)
(0.41()
0.945 (0.010)
61.4
57.6
58.4
30%
0.940 (0.01)
(0.011)
0.954 (0.009)
79.0
67.6
66.8
32B
40%
0.918
(0.01))
0,940 (0.011)
100.2
89.7
70.8
50%
0.924 (0.012)
0.924 (0.012)
174.0
100.6
Table 2: Qwen3 MATH500 accuracy under one-shot pruning with SparseGPT. Accuracy with standard error (SE) on the left, total evaluation time in minutes on the right. Best accuracy or fastest runtime in green.
Accuracy 1. SE
Runtime (min)
Model
Sparsity
C4
Prompi-Only
RAC
C4
Prompt
RAC
0.906
0.906
0,906
18.5
18.5
18.5
30%
0.822 (0.017)
0.874 (0.015)
0.880 (0.015)
58.6
33.3
24.1
L.7B
(0.021)
(0.019)
0,858 (0.016)
262.6
87.2
50%
(0.20()
0.648 (0.021)
274.5
101.0
Dense
0.962
0.962
0.962
41.3
41.3
41.3
30%
0.948 (0.010)
0.958 (0.009
0.972 (0.007)
29.6
35.6
29.6
8B
(0.013)
0.944
0.968 (0.008)
57.5
38.0
50%
0.564 (0.022)
(0.02))
0.862 (0.015)
258.8
274.5
17.1
Denve
0.972
0.972
0.972
41.2
41.2
41.2
30%
0.950 (0.010)
0.964 (0.008)
0.970 (0.008)
26.7
22.6
23.6
14B
40%
0.958 (0.009)
0.962 (0. (0.009)
0.970 (0.008)
26.1
33.8
31.3
50%
(0.01))
0.932 (0.011)
0.962 (0.009)
64.4
43.0
31.6

---

Published as a conference paper at ICLR 2026
Table 3: DeepSeck-R1 Llama MATH500 ace@ 1:1 under one-shot pruning. Accuracy with standard error (SF)
on the left, total evaluation time in minutes on the right, Best accuracy or fastest runtime in green.
Model Sparsity
Accuracy 1 SE
Runtime (min)
C4
Prompt-Only
RAC
C4
Prompt-Only
RAC
Donse
0.866 (0.015)
0.866 (0.015)
0.866 (0.015)
8,3
8,1
8,1
30%
0.860
(0.50
0.884 (0.014)
8B
40%
(0.018)
(0.018)
0.834 (0.017)
14.8
14.6
10.3
50%
0.508 (0.022)
0.654 (0.021)
0.714 (0.020)
45.5
30.8
19.3
Densee
0.954 (0.0099
0.954 0.9540)
0.954 (0.009)
13.0
13.0
13.0
30%
(0.06(0
(0.010)
(0.910)
70B
40%
0.910(0.013)
0.936 (0.011)
0.932 (0.01
20.7
20.1
15.1
50%
(..01)
0.892000013)
0.904 (0.013)
40.9
39.8
26.6
Table 4: Qwen3 AIME-25 accuracy under one-shot pruning with SparseGPT. Accuracy with standard error (SE) on the left, total evaluation time in minutes the right. Best accuracy or fastest runtime in green.
Accuracy
Runtime (min)
Model Sparsity
Prompt--ni
C4
Prompt
RAC
Dense
0.333
0.333
0.333
9.9
9,9
9.9
30%
0,267 (0.082)
0,300 (0.085)
0.267 (0.082)
14.5
12.4
11.4
1.7B
40%
0.000 (0.000)
0.33 (0.063)
0.267 (0.082)
27.2
17.5
10.2
(0.000)
0.000 (0.000)
0.200 (0.074)
13.3
Dense
0.633
0.633
0.633
12.4
12.4
12.4
30%%
0.600 (0.091)
0.633 (0,090)
0.667 (0,088)
15.3
15.0
15,9
40%
0.533 (0.093)
0.633 (0.090)
0.633 (0.090)
21.8
14.9
15.2
50%
0.033 (0.033)
0.300 (0.085)
0.500 (0.093)
37.4
25.7
16.5
0.667
13,9
30%
0,700 (0.085)
0.667 (0.088)
0.667 (0.088)
14.2
14.4
14.3
14B
40%
0.500
0.633 (0.090)
0.667 (0.058)
16.6
14.4
17.2
50%
(0.00()
0.433 (0.092)
0.600 (0.091)
23.1
20.1
16.5
Table 5: DeepSeck-RI Qwen LiveCodeBench codegen Sereggenppaseee @ 1:16 under one-shot pruning. Accuracy with standard error (SE) on the left, total evaluation time in minutes on the right. Best accuracy or fastest runtime green.
Accuracy SE
Runtime (min)
Model
Sparsity
Prompt-Ohi
RAC
C4
Prompt
RAC
Dense
0.161
0.161
0,161
20%
0.148 (0.018)
0.156 (0.019)
(0.5188
305.7
511.9
299.4
1.5B
30%
0.127 (0.017)
0.138 (0.018)
(0.008)
724.4
685.3
274.5
40%
(0.011)
(0.013)
0.129 (0.017)
466.8
394.5
277.3
50%
(0.002)
0.024 (0.006)
0.093 (0.014)
464.2
440.3
330.5
0.374
20%
(0.02()
0.367 (0.025)
0.364 (0.025)
591.4 .4
356.6
318.7
7B
30%
0.335 (0.025)
0.341 (0.025)
0.361 (0.025)
348 6
343.4
325.2
40%
0.273((0.023)
0.300 (0.02))
0.333 (0.024)
485.7
847.8
344.7
50%
(0.044))
0.228 (0.02
0,283 (0.023)
692.9
1174,8
381.2
Donse
0,513
0.513
0,513
0.508 (0.027)
0.508 (0:027)
0,508 (0.027)
1202.2
1246.6
1254.6
14B
30%
0.491 (0.027)
0.496 (0.027)
0.496 (0.027)
1292.1
1272.8
759.0
40%
0.447 (0.026)
0.471 (0.027)
0.480 (0.027)
1665.4
1441.2
1416.1
50%
(0.024)
0.385 (0.026)
0.424 (0.026)
1918.8
2472.9
1814.8
Discussion of pruning results. Table 1, Table and Table report MATH500 acc@ 1:1 and run- time across the DeepSeek-RI (Qwen and Llama families) and Qwen3 reasoning families under one- shot pruning with SparseGPT. Across all architectures, RAC consistently outperforms generic C4 calibration and typically improves over prompt-only calibration, especially higher sparsity levels. On DeepSeek- R at 50% sparsity, RAC attains 0.664 accuracy versus 0.496 for prompt- and 0.356 for C4. A similar pattern holds for the 7B model, where at 50% sparsity RAC maintains 0.900 accuracy compared with 0.812 (prompt-only) and 0.744 (C4), and again substantially reduces the runtime (from 135.0 and 11 11..6 minutes 35.3 minutes). For the larger 14B and 32B DeepSeek-

---

Published as a conference paper at ICLR 2026
RI variants, RAC preserves dense-level accuracy much better high sparsity, while avoiding the extreme runtime ulo observed under other forms of calibration.
The Qwen3 results in Table 2 show that this behavior generalizes to an independent reasoning chitecture. For Qwen3-1.7B and 8B, RAC consistently achieves the highest accuracy at 30-50% sparsity, often by wide margin. For instance, 50% sparsity Qwen3-8B, RAC achieves 0.862 accuracy compared to 0.564 (C4) and 0.470 (prompt-only), while reducing evaluation time from over 250 minutes 17.1 minutes. Even on the strongest Qwen3-14B model, RAC closely tracks the dense baseline at 40-50% sparsity (e.g., 0.962 vs. 0.972 dense at 50% sparsity), whereas C4 calibration degrades much more sharply. At more moderate sparsity levels (20% on DeepSeek- 1 30% on Qwen3), all methods remain close to the dense baseline, indicating that the benefits of reasoning-awee calibration become most pronounced as compression becomes more aggressive.
Table 4 extends this comparison to the A AIME-25 benchmark, which consists of harder competition- style math problems. Under C4 calibration, accuracy often collapses at 40-50% sparsity (e.g., Qwen3-1.7B at 40% sparsity drops to 0.000), while prompt-only traces partially mitigate this degra- dation but still underperform. RAC, by contrast, maintains large fraction dense accuracy in the high-sparsity regime. These results suggest that aligning calibration with on-policy reasoning traces is particularly important on challenging benchmarks.
Finally, Table 5 shows the LIVECODEBENCH codegen.passee :16 results for DeepSeek-PP R1. The qualitative pattern mirrors the math experiments: code generation is highly sensitive to pruning, C4 calibration suffers severe accuracy and runtime degradation at high sparsity, prompt-only cali- bration helps but is insufficient, and RAC reduces the loss from pruning the most. Across Math500, AIME, and LiveCodeBench, domain-relevant calibration (prompt-only) consistently outperforms generic C4, but RAC provides the most robust pruned models, especially at high sparsity. Larger models (e.g., 14B and above) are intrinsically more robust pruning, but still benefit meaningfully from RAC in both accuracy and runtime.
RAC mitigates errors during decoding. The purpose of this experiment to test whether RAC reduces reconstruction error on unseen reasoning problems, not just those encountered during cal- ibration. To that end, we evaluate pruned DeepSeek Prepeee--R--isttiiee R1-Distill--BB models on the MATH500 test set, which contains problems the model did not encounter during calibration. For each problem we construct a fixed evaluation sequence: the concatenation of (1) the original prompt - and (2) the dense model's own greedy rollout CT At each step / we feed this shared prefix zo to the dense model and to a pruned model, take the last-layer, last-token hidden state, and compute (meth)
(L) x meth,t 11 1,.... T where "meth" dense,
the tokenwise reconstruction error et refers to pruning with either prompt-only calibration or RAC. The heatmap then plots the ratio (Prompt) rz e, /e, (R(CC) row-wise over problems (columns are token indices). The dashed black line marks the prompt/decoding boundary / = Tin. Grey cells (r, Rs 1) indicate both methods incur simi- lar error. Blue (r, > 1) indicates RAČ achieves smaller error. Red (r, <1) indicates prompt-only has lower error. Two consistent patterns appear: on the left of the black line (prompt tokens), rows skew slightly red, reflecting that prompt-only better preserves prompt activations. On the right (decode tokens), the plot turns predominantly blue, showing that RAC substantially reduces reconstruction error exactly where long reasoning chains occur during autoregressive decoding. This demonstrates that by reconstructing CoT activations during calibration, RAC simulates the inference-time distri- bution and lowers decoding error on held-out test problems.
8

---

Published as a conference paper at ICLR 2026
Error ratio Redic Greya1 Blues1
prompt
Figure 2: Tokenwise reconstruction error ratio on MATH500 test problems for pruned DeepSeek- RI-Distill-7B. Each row is held-out problem, columns are token indices, and the black vertical line marks the prompt/dccoding boundary. Colors show the ratio r; = e,
(Prompt) /e; (R,AC)) grey = indicates equal error, red < indicates lower error from prompt-only calibration, and blue > indicates lower error from RAC. prompt-only slightly outperforms on the input tokens, but RAC has smaller error throughout the much longer decode phase.
5.2 ADDITIONAL EXPERIMENTS
Throughput Analysis with Structured Pruning and Quantization (Table 6). Moving beyond unstructured pruning, we evaluate semi-structured 2:4 sparsity patterns that remove exactly 2 out of every 4 contiguous weights, which can achieve actual speedups on modern hardware platforms such as NVIDIA Ampere (Mishra et al., 2021). The results demonstrate that RAC maintains its effectiveness even with structured pruning constraints. Notably, when applied to different portions of the model (first third, first and last third, entire model), as is done in SparseGPT (Frantar & Alistarh, 2023), RAC consistently preserves accuracy while achieving meaningful throughput improvements. The combination of RAC with FP8 quantization (RAC+FP8) shows particularly promising results, achieving both high accuracy (0.940 for first and last third pruning) and substantial throughput gains (1675 tok/s vs 1426 tok/s baseline).
On-policy vs. Off-policy Calibration (Table 7). We investigate whether the CoT traces used during calibration must come from the same model being compressed (on-policy) or can be gen- erated by a different model (off-policy). Specifically, itself versus using traces generated by the larger 14B model when compressing the 7B model. The results show that on on-policy calibration generally outperforms off- off-policy calibration, particularly at higher sparsity levels (50% sparsity: 0.900 vs 0,876 accuracy). This suggests that the activation patterns during CoT generation are model-specific, and using traces from a different model creates a distribution mismatch. However, the performance gap is relatively modest at lower sparsity levels, indicating some robustness to off-policy data when compression is less aggressive.
compare using traces from the 7B model
Run-time at smaller maximum test-time decoding length (Table 8). To investigate whether the runtime penalties observed with pruned reasoning models are inherent to compression or primar- ily driven by excessive token generation, we evaluated both dense and RAC-prunnd 7B models on MATH-500 under varying maximum decoding constraints of 32,768, 8,192, and 4,096 tokens. The runtime analysis reveals that the problematic behavior of pruned reasoning models is primar- ily confined to scenarios with excessive token budgets. At 32,768 tokens, the RAC-prumed model (40% sparsity) exhibits the pattern documented throughout this work, maintaining reasonable accu- racy (91.2% versus 93.6% for the dense model) but requiring longer inference time (29.1 versus 23.3 minutes) due to longer, more rambling CoT traces. However, this runtime penalty largely disappears

---

Published as a conference paper at ICLR 2026
more practical token limits. At 192 tokens, the RAC-pruned model achieves nearly identical per- formance to the dense baseline on both accuracy (89.8% versus 90.4%) and runtime (10.4 versus 10.5 minutes). At 4,096 tokens, the pruned model actually outperforms the dense model on accu- racy (83.6% versus 82.4%) while maintaining comparable runtime (8.6 versus 8.4 minutes). These results suggest that the runtime overhead from pruning manifests primarily when models per- mitted to generate excessively long outputs, and that RAC-compressed models behave much more similarly to their dense counterparts when constrained reasonable decoding lengths.
Varying the pruning algorithm (Table 9). In preceding sections pruning baseline when contrasting prompt-only versus RAC calibration. As shown in Table 9, RAC also be used in conjunction with other L.M pruning algorithms, such as ALPS and WANDA, and provides gains that are agnostic to the pruning algorithm. Comparing against the SparseGPT also find that ALPS a competitive (and sometimes stronger) pruning baseline
SparseGPT as the default
results in Table in its own right: for example, on Qwen3-1.77 at 50% sparsity, ALPS+RAC attains 0.785 МATI150 accuracy versus 0.648 for SparseGPT+RAC and on Qwen3-8B 3BB at 50% sparsity ALPS+RAC reaches 0.940 compared to 0.862 for SparseGPT+RAC, For completeness, we report AIME-2 25 benchmark results for ALPS in Table 10. We observe similar trends discussed above.
6 CONCLUSION AND LIMITATIONS
Conclusion. We studied the challenge of compressing reasoning language models, where infer- ence is dominated by long chains of self-generated tokens. Standard pruning and quantization pipelines, which calibrate only on prompt activations, strongly degrade accuracy and runtime in this setting, often producing longer and less reliable reasoning traces. We proposed Reasoning-Awar Compression (RAC), a simple modification that augments calibration with on-policy CoT activa- tions. RAC integrates seamlessly with existing pruning algorithms such SparseGPT, requiring no retraining or distillation. Our experiments on mathematics and coding benchmarks demonstrate that RAC substantially mitigates the accuracy loss of pruning and stabilizes CoT generation.
Limitations. While RAC provides simple and effective drop-in improvement, several limitations remain. First, this work does not directly address inference runtime increases. RAC improves pruning quality, which indirectly reduces unnecessary decoding steps, but designing compression methods that explicitly optimize to avoid sequence length increases remains an open direction. We show that this can be mitigated by simply reducing the maximum decoding budget, which is typically excessively high. Second, collecting on-policy CoT activations increases calibration cost relative to prompt-only pipelines, especially for large decode budgets, though this overhead is very modest compared to training or distillation.
7 ACKNOWLEDGEMENTS
Ryan Lucas contributed to this work while he was an intern at LinkedIn during summer 2025. This work is a part of his MIT research. Rahul Mazumder contributed this work while he consultant for LinkedIn (in compliance with MIT's outside professional activities policies). This work is not part of his MIT research.
REFERENCES
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanezyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes, 2024. URL https: /ttp///axivviii wrt. alos////////// 223/ 13649.
Abhinav Bandari, Lu Yin, Cheng-Yu Hsieh, Ajay Kumar Jaiswal, Tianlong Chen, Li Shen, Ranjay Krishna, and Shiwei Liu. Is C4 dataset optimal for pruning? an investigation of calibration data for LLM pruning.
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024. URL https: / /ttp///aiiiv//.a///v///e///..atl org/ abs 2410. 1...0.00 0.4.667763 07461.
DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao
10

---

Published as a conference paper at ICLR 2026
Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang. Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng. Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang. Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang. Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforce- ment learning, 2025. URL https: //tp:::///rriiie aaxiv. org/ abs 2501. 12948.
Tao Fan, Guoqiang Ma, Yuanfeng Song, Lixin Fan, Kai Chen, and Qiang Yang. PPC-GPT: Fed- erated task task-specific compression of large language models via pruning and chain tfutought dis- tillation, arXiv preprint arXiv:2502. 15857, 2025. URL https: / org / abs/ 2502. 15857.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475-4488, 2022.
Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning (ICML), pp- 10323-10337. PMLR, 2023.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
HuggingFace. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: com/hugg
//github.cc. wwtpthh..ccooooetcoooooooooootot wwt.coo////oooe huggiigggg i ng face/ /opeeope..... r1.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Xiang Meng, Kayhan Behdin, Haoyue Wang, and Rahul Mazumder. Alps: Improved optimization for highly sparse one-shot pruning for large language models, 2024. URL https://arxiv. org//bba 2406 07831.
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks, 2021. URL https: /ttss////xvvi.../ rxiv, http://///rvvvvo.aa///////// wttp://gg//////ooooooo// 08800/0080//1
11

---

Published as a conference paper at ICLR 2026
OpenAL. Learning to reason with 1lms, September 2024. URL learning-to- weeeesonnwitten 11ms/ Research release.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(5):548--551, 2020.
Jiaxi Yang Bowen Yu Bo Zheng Dayiheng Liu Shanghaoran Quan. Codeforces: Benchmarking competition-level code generation of lims on codeforces. 2025. traditional code benchmark.
Disclaimer: This non-
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models, 2024. URL https: http///axvvvvv.//ee wwwpa//wwww...////////0000
11695.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of-thouu -ftthughtt prompting elicits reasoning large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 24824-24837, 2022.
Miles Williams, George Chrysostomou, and Nikolaos Aletras. Self-calibration for language model quantization and pruning. In Proceedings of the 2025 Conference of the Nations of the Amer- icas Chapter of the Association for Computational Linguistics: Human Language Technolo- gies (Volume 1: Long Papers), pp. 10149-10167. Association for Computational Linguistics, 2025. doi: 1/225553112225.maaaaaaaa 2025. nwacl 1on1..oog..000 509.
509. URL http: http//xxxxxiixoooo org.10r//100 10t/3330....3/11/
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023.
URL https::///// /ttp//axivvv.///////e//e//aa .rgg abs 2305 10601.
Nan Zhang, Yusen Zhang, Prasenjit Mitra, and Rui Zhang. When reasoning meets compression: Benchmarking compressed large reasoning models on complex reasoning tasks, 2025a. URL
https://arxii: org/ abs///b///// 2504. 02010.
Nan Zhang, Yusen Zhang. Prasenjit Mitra, and Rui Zhang. When reasoning meets compression: Benchmarking compressed large reasoning models on complex reasoning tasks. arXiv preprint arXiv:2504.02010, 2025b. URL https:://aaaa org/ als////b////54// 02010.
8 APPENDIX
Table 6: Actual throughput gains and accuracy with semi-structured 2:4 sparsity of MLP layers. Pruned model Deepseek-Ri-Distil-- 14B on MATH-500. Each pruning scope has two columns: Accuracy (acc@1 1:1 (SE) and throughput (tok/s). Dense baseline throughput 1426 tok/s.
First third
First & last third
Entire Model
Ace
Thr
Ace
Thr
Acc
Thr
RAC
0.950 (0,001)
1532
0.940 (0.012)
1561
0.896 (0.014)
1740
Prompt+FP8
0.952 (0.010)
1532
0.904 (0.0132)
1675
RAC+FP8
0.950 (0.010)
1532
0.920 (0.0106)
1675
0.8920 (0.014)
1823
Dense baseline
Throughput 1426 tok/s
Table 7: On-policy (RAC) vs. off off-policy calibration when pruning DeepSeek- RI-Distill-7BB using PeepSeek-Ri-Distll--ee traces on MATH500. We report ace @ 1:1 with standard error.
Sparsity
On-policy
Off-policy
20%
0.934 (0.011)
0,930 (0.0114)
30%
0.934 (0.011)
0.930 (0.0114
40%
0.912 (0.013)
0.916 (0.0124)
50%
0.900 (0.013)
0.876 (0.0148)
12

---

Published as a conference paper at ICLR 2026
Table 8: MATH500 accuracy and runtime (minutes) across different test-time maximum decode
budgets (number of tokens) for 7B models.
Max tokens 32,768
Max tokens 8,192
Max tokens 4,096
Model
Acc.
Time
Acc.
Time
Acc.
Time
Dense 7B
0.936
23.3
0.904
10.5
0.824
8.4
Pruned 7B (RAC, 40%)
0.912
29.1
0.898
10.4
0.836
8.6
Table 9: MATH500 accuracy SE under one-shot pruning for ALPS family. Best accuracy each row green.
WANDA for the Qwen-3 model
ALPS
WANDA
Model
Sparsity
C4
Prompt-Oni
RAC
C4
Prompt-Only
RAC
Dense
0.906 (0.013)
0.906 (0.013)
0.906 (0.013)
0.906 (0.013)
0.906 (0.013)
0.906 (0.013)
30%
0.796 (0.018)
0.866 (0.015)
0.860 (0.016)
0..74
0.820 (0.017)
0.828 (0.017)
Qwen3-1.7B
40%
0.812 (0.018)
0.854 (0.016)
(..00
0.706 (0.020)
0.746 (0.020)
50%
0.566 (0.022)
0.788 (0.018)
0.436 (0.022)
Dense
0.962 (0.009)
0.962 (0.009)
0.962 (0.009)
0.962 (0.009)
0.962 (0.009)
0.962 (0.009)
30%
0.956 (0.009)
0.956 (0.009)
0.966 (0.008)
0.958 (0.009)
0.966 (0.008)
0.960 (0.009)
Qwen3-8B
40%
0.902)))013)
0.954 (0.009)
0.962 (0.009)
0..361))111
0.928 (0.012)
0.950
50%
(0.014)
0.940 (0.011)
(0.00))
0.878 (0.015)
Dense
0.972 (0.007)
0.972 (0.007)
0.972 (0.007)
0.972 (0.007)
0.972 (0.007)
0.972 (0.007)
30%
0,952 (0.010)
0.962 (0.009)
0.960 (0.009)
0.966 (0.008)
0.962 (0.009)
0.968 (0.008)
Qwen3-14B
40%
(0.000)
0.958 (0.007)
0.962 (0.009)
0.948 (0.010)
0.958 (0.009)
0.962 (0.009)
50%
0.822 (0.017)
(0.01))
0.960 (0.009)
0.926)00012) (0.00))
0.930 (0.011)
Table 10: Qwen3 AIME-25 accuracy under one-shot pruning with ALPS. Accuracy with standard error (SE) parentheses. Best accuracy each row green.
Model
Sparsity
C4
Prompt-Only
RAC
Dense
0.333 (0.088)
0.333 (0.088)
0.333 (0.088)
30%
0.233 (0.079)
0.267 (0.082)
0.400 (0.091)
Qwen3-1.7B
40%
0.033 (0.033)
0.267 (0.082)
0.433 (0.092)
50%
(0.000)
0.033 (0.033)
0.267 (0.082)
Dense
0.633 (0.090)
0.633 (0.090)
0.633 (0.090)
30%
0.567 (0.092)
0.567 (0.092)
0.700 (0.085)
Qwen3-8B
40%
0.433 (0.092)
00.00
0.667 (0.087)
50%
0.033 (0.033)
0.367 (0.089)
0.467 (0.093)
Dense
0.667 (0.088)
0.667 (0.088)
0.667 (0.088)
30%
0.733 (0.082)
0.600 (0.091)
0.633 (0.089)
Qwen3-14B
40%
0.533 (0.093)
0.667 (0.087)
0.700 (0.085)
50%
0.267 (0.082)
0.533 (0.093)
0.667 (0.087)
13


Reviews:
[{"review_id": "J3hOk0t7bN", "reviewer": "Reviewer_bqFF", "summary": "This paper introduces Reasoning-Aware Compression, a pruning method for reasoning language models that maintains high accuracy while improving efficiency. Unlike standard pruning, which calibrates using only prompt signals, RAC aligns pruning with decoding behavior by incorporating internal signals from both the prompt and the model’s own chain of thought. This alignment ensures pruning reflects how the model actually reasons during generation. Experiments show that RAC-pruned models produce shorter, cleaner reasoning chains and retain accuracy close to the dense baseline- a 7B model at 50% sparsity maintains about 90% accuracy, compared to 74.4% with standard calibration.", "questions": "1. Is there any study on how the pruning varies with the CoT quality. With long reasoning traces, many times we observe the model has a long approach then backtracks and then corrects itself etc to get the output. Did you observe any specific behaviors there?\n2. How would this method generalize to other tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Aligns pruning with chain-of-thought activations, effectively preserving reasoning quality. The input column to be considered for pruning considers not just the provided input prompts tokens but aslo the LLM policy's self generated reasoning CoT tokens, which aligns the model with what it would do during inference time.\n\n2. Achieves up to 95% dense-model accuracy at 50% sparsity with no retraining.\n\n3. Simple, plug-and-play method compatible with existing SparseGPT or WanDA approaches.", "weaknesses": "1. How do you pre-calculate $\\mathcal{D}_M$ which tells howmany times you would self-generate tokens from the model?\n2. While the paper has good rigor the presentation and notations can be well-defined, especially the core part in page 4 and 5. Eg. what is $N_P$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TC9aeb10dl", "reviewer": "Reviewer_hbbN", "summary": "This is a method to compress models for reasoning. The key idea is that reasoning involves long generated sequences relatively to the prompts, compared to standard tasks / benchmark where the generated part is short or nonexistent for multi-choice. Hence, using the prompts themselves as \"calibrating data\" to prune is misguided and performance can be improved by using generated sequences as calibrating data. Experiments show that it works very well.", "questions": "I do not have question, but I am curious to see if other reviewers who know the field better confirm that this method is not already well known and used in practice.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The method is simple, straightforward and comparisons to other methods from that domain demonstrate superior performance.", "weaknesses": "The proposed method is too simple and the contribution too limited. It boils down to saying \"use the data of your task to calibrate pruning for your task\".\n\nNow, model compression is not a domain I know well and it is easy to have an illusion of triviality post-hoc.\n\nAbout the form of the paper, some sections are just loading pages with equations which are in my opinion needless and give an impression of gratuitous filling to look impressive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R5NvpqvsAt", "reviewer": "Reviewer_LocM", "summary": "This paper identifies that pruning reasoning models (e.g., DeepSeek-R1) with standard methods fails, causing accuracy drops and slower inference. The authors blame a calibration mismatch: methods use prompt activations, but reasoning is decode-dominated. The fix, Reasoning-Aware Compression (RAC), adds on-policy Chain-of-Thought (CoT) activations to the calibration set. Experiments show RAC significantly boosts accuracy (e.g., 90.0% vs. 74.4% at 50% sparsity for a 7B model) and reduces the \"rambling\" CoTs that slow down inference.", "questions": "1. Will RAC generalize to standard models (Llama, Mistral) using CoT prompting or other reasoning models like Qwen3, QwQ, or is the benefit specific to GRPO-trained models like DeepSeek-R1? And I think I may improve my score if some of the experiments can be done.\n\n2. Please quantify the calibration overhead (wall-clock time) of RAC versus the 1M C4 baseline.\n\n3. Table 4 shows on-policy traces are best. Is matching a model's idiosyncratic activation patterns more important than using 'cleaner' traces from a larger model?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Clear Problem & Diagnosis: Clearly identifies and diagnoses a non-obvious problem: pruning reasoning models can slow them down due to a prompt/decode activation mismatch (Fig. 1).\n\n2. Simple & Practical Solution: The proposed solution (RAC) is simple and practical: a \"drop-in\" fix for the calibration data, not a complex new algorithm.\n\n3. Strong Validation (on DeepSeek): Provides strong empirical validation on the DeepSeek-R1 family, showing consistent gains on math/code and a clear reason for success (lower decode error in Fig. 2).", "weaknesses": "1. Generalizability Concerns: I think this is the primary weakness. The claims are broad, but results are confined only to the DeepSeek-R1 (GRPO-trained) family. There is no evidence this works on standard models (Llama, Mistral) using CoT prompting or other reasoning models like Qwen3, QwQ. This severely limits the contribution's impact.\n\n2. Incomplete Method Comparison: The comparison of pruning methods is incomplete (Table 6). To validate RAC as algorithm-agnostic, the baseline (C4) results for ALPS and Wanda are needed to show relative lift, not just absolute performance.\n\n3. Unquantified Overhead: The paper notes RAC adds overhead but never quantifies the wall-clock time for calibration versus the baseline, making its practicality hard to assess.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W6UIywsUfA", "reviewer": "Reviewer_1Hgg", "summary": "This paper proposes a pruning strategy for reasoning models. Specifically, it uses the fact that there are more decoding tokens than input tokens for reasoning models, so it collects activations for those and applies existing pruning methods.", "questions": "- Are the runtime minutes reversed, or am I misunderstanding something?\n- As in the weaknesses, comparing with efficient fine-tuning for short CoT would make the paper more valuable.\n- Why do you think naive pruning leads to more tokens in the output?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to read and follow.\n- The solution is simple and straightforward.\n- The results show that it preserves accuracy and improves efficiency.", "weaknesses": "- Isn’t the self-calibration a somewhat similar mechanism [1]? Please correct me if I’m wrong.\n- Missing comparison with efficient reasoning on the token side.\n    - What is the pruning cost compared to the fine-tuning cost of training the model to perform concise reasoning?\n    - How is model pruning more effective compared to token reduction? Reducing decoding tokens might reduce the cost more. Also, since both can be applied, I wonder what would happen if we applied both.\n- Runtime minutes appear reversed. It seems like the dense model is the most efficient, and pruning makes inference slower according to the tables.\n- Evaluation sets for math are limited to MATH500, and the models already perform very well on it (90%+ for 7B+ models). It is not surprising that pruning preserves high accuracy. I would like to see results on AIME as well.\n\n[1] Williams et al. “Self-calibration for Language Model Quantization and Pruning”, NACCL 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)