When Reasoning Meets Compression: Understanding the Effects of LLMs Compression on Large Reasoning Models
Authors: Nan Zhang
Eugene Kwek
Yusen Zhang
Ngoc-Hieu Nguyen
Prasenjit Mitra
Rui Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: LLMs Compression
LRMs Compression
Quantization
Pruning
Distillation
Primary area: foundation or frontier models, including LLMs
TL;DR: We run performance benchmarking and mechanistic interpretation to understand the effects of compression on reasoning models.

Abstract: Compression methods, including quantization, distillation, and pruning, improve the computational efficiency of large reasoning models (LRMs). However, existing studies either fail to sufficiently compare all three compression methods on LRMs or lack in-depth interpretation analysis. In this paper, we investigate how the reasoning capabilities of LRMs are compromised during compression, through performance benchmarking and mechanistic interpretation. To uncover the effects of compression on reasoning performance, we benchmark quantized, distilled, and pruned DeepSeek-R1 models on four reasoning datasets (AIME 2024, FOLIO, Temporal Sequences, and MuSiQue). To precisely locate compression effects on model weights, we adapt difference of means and attribution patching techniques, focusing on the activation of every linear component in compressed LRMs, to interpret fine-grained causal relationships between weights and various reasoning capabilities. This fine-grained interpretation addresses a fundamental question of compression: which weights are the most important for reasoning? Overall, we find dynamically quantized 2.51-bit R1 reaches close-to-R1 performance. With empirical verification, we present three main findings that generalize across both R1 and non-R1 LRMs: (1) Weight count has a greater impact on LRMs' knowledge memorization than reasoning, highlighting the risks of pruning and distillation; (2) The MLP up projection in the final layer of distilled LRMs is one of the most important components, offering a new perspective on locating critical weights - a fundamental problem in model compression; and (3) Current quantization methods overly compress the final-layer modules and MLP gate projections, so protecting just 2% of all weights that are excessively compressed can raise average accuracy by 6.57%, greatly surpassing the state-of-the-art.

Published as a conference paper at ICLR 2026
WHEN REASONING MEETS COMPRESSION: UNDER- STANDING THE EFFECTS OF LLMs COMPRESSION ON LARGE REASONING MODELS
Nan Zhang*
Eugene Kwek
Yusen Zhang
Ngoc-Hieu Nguyen *
Prasenjit Mitra o Rui Zhang The Pennsylvania State University fnj25124, emz5227 aes.s.u...... edu
O Carnegie Mellon University Africa
ABSTRACT
Compression methods, including quantization, distillation, and pruning, improve the computational efficiency of large reasoning models (LRMs). However, exist- ing studies either fail to sufficiently compare all three compression methods on LRMs lack in-depth interpretation analysis. In this paper, investigate how the reasoning capabilities LRMs are compromised during compression, through performance benchmarking and mechanistic interpretation. To uncover the ef- fects of compression on reasoning performance, we benchmark quantized, dis- tilled, and pruned DeepSeek-R. R1 models on four reasoning datasets (AIME 2024, FOLIO, Temporal Sequences, and MuSiQue). To precisely locate compression effects on model weights, adapt difference of means and attribution patching techniques, focusing on the activation of every linear component in compressed LRMs, to interpret fine-grained causal relationships between weights and various reasoning capabilities. This fine-grained interpretation addresses a fundamental question of compression: which weights are the most important for reasoning? Overall, we find dynamically quantized 2.51-bit RI reaches close-to-R1 perfor- mance. With empirical verification, we present three main findings that general- ize across both RI and non-R1 LRMs: (1) Weight count has greater impact on LRMs' knowledge memorization than reasoning, highlighting the risks prun- ing and distillation; (2) The MLP up projection in the final layer of distilled LRMs is one of the most important components, offering a new perspective on locating critical weights- fundamental problem in model compression; and (3) Current quantization methods overly compress the final-layer modules and MLP gate pro- jections, so protecting just 2% of all weights that are excessively compressed can raise average accuracy by 6.57%, greatly surpassing the state-of-the-art.
INTRODUCTION
Large reasoning models (LRMs) such as DeepSeek-R1 (Guo et al. 2025) excel at complex reason- ing tasks. However, due their large sizes, deploying them can be costly and even infeasible for individuals, which hinders Al democratization. Compression methods including quantization, distil- lation, and pruning reduce computational resources (e.g., GPU memory and disk space). Represen- tative quantization techniques include dynamic quantization by Unsloth (Daniel Han & team 2023) activation aware quantization AWQ (Lin 2024), and post-training quantization GPTQ Frantar
et al. 2022). Current distillation involves black-box (Li et al. (0224) white-box (Gu et al. 2024)
settings. Representative pruning techniques include unstructured (Zhang et al. 2024a
Frantar &
Alistarh 2023) and structured pruning (Xia et al. 2024 Ma al. 2023).
However, existing works do not sufficiently study the performance of compression method on LRMs (Liu et al. 2025a; Srivastava et al. 2025 Feng et al. 2025). Although current quantization and pruning methods claim to preserve the performance of general-purpose LLMs, benchmarking both of them LRMs with more reasoning intensive datasets helps compare their collapse point. Regarding distillation, recent works either fail to comprehensively evaluate their student models on diverse reasoning benchmarks of varying difficulty neglect to consider distillation effect

---

Published as a conference paper at ICLR 2026
Compression Effect on Reasoning Performance
Compression Effect on Weights
2.51-bit
reesoning
Dynamic
Stcortng vector
Quantization
1.58-bit
Reascning chains of
R1
specifil behaivior
Distilation with SFF
Gradients model's
Weight moprronne
Steering vestor
activations
Distiled
Tasks
Varification of Findings
SparseGPTi
Alpharruning
2024
Top muth chillenges
Selective Osansiizatiin
Disiiled
Various sparsitles
FOLIO
Logical reasoning
AWQ / GPTQ
4-bil
/
Tergoral a1
Event temporsi resecnnnng
ANY413
3-bit
Se lecti Protection
Distiled R1
Quee
Mutthop reasoning
Figure 1: An overview of our pipeline. On the left, we benchmark compressed R1 variants on various reasoning tasks. On the right, by computing weight importance towards a specific reasoning behavior (a dot product of the steering vector and gradients with respect an LRM's activations), we study the compression effects on individual weight matrices. We empirically verify our findings on weight importance by selectively quantizing protecting module test its importance.
knowledge and reasoning (Huang et al. 2024, Agarwal et al. 2024). Another research gap is the lack of interpretability of compression effects LRMs. It is necessary to interpret how compres- sion methods affect LRMs, as such analysis can reveal existing bottlenecks and provide guidance for future compression research.
Therefore, due to the lack of compression works on LRMs, we study this fundamental research question: How the reasoning capabilities of LRMs compressed during compression? We answer it from two perspectives: performance benchmarking and mechanistic interpretation. In the main text, first benchmark compressed DeepSeek-RI on various reasoning tasks investigate how model compression affects performance. We test dynamic quantization (Daniel Han & team 2023). distillation with supervised lne-tuning (SFT) (Guo et al. (025). SparseGiT (Frantar & Al- istarh 2023). AlphaPruning (Lu et al. 2024), AWQ (Lin et al. 2004). GPTQ (Frantar et al.) 2022), GPTAQ (Li et al. 2025), and ANY4/3 (Elhoushi & Johnson 2025) on RI (or distilled R RI). Then, apply mechanistic interpretability to quantify weight contribution towards four core reasoning capabilities of LRMs: backtracking, uncertainty estimation, example testing, and adding knowledge. By focusing the activation of every linear component in compressed LRMs, we adapt difference of means (Arditi et al. 2024) to extract steering vectors and attribution patching (Syed et al.) 2023) compute weight importance. Unlike previous analysis (Venhoff al., 2025 that only measures layer-wise weight contribution, our weight importance scores offer more fine-grained interpreta- tion of weight contribution, addressing the fundamental compression question of locating important weights. By comparing weight importance between compressed and original LRMs, we quantify the effects of distillation, quantization, and pruning on model weights Our analysis framework is in [i
With empirical verification, our key findings are summarized below for better understanding and improving LRMs compression (these findings also generalize to non-R1 families):
Weight count has greater impact on LRMs' knowledge memorization than their reasoning ca- pabilities, highlighting the compression effects of pruning and distillation. Thus, both distillation and pruning are discouraged when tasks require LRMs' parametric knowledge.
The mlp. . up up-proj in the final layer of RI distilled models emerges as one of the most important model components, addressing a core concern in pruning and quantization literature: identifying critical weights. Quantizing only this matrix to 3-bit reduces the average accuracy by 16.3%. which validates its high importance.
Our interpretation code htapa//ttte///tttttecccc///o///p
2

---

Published as a conference paper at ICLR 2026
Final-layer modules, along with the mlp. gate-proj of R1 distilled Llama and Qwen, overly compressed by popular quantization methods, highlighting the need for greater attention to preserving their weight precision. A successful protection of only final-layer MLP modules could raise average accuracy by 6.57%, with gains of up to 23.17% 17% over the state-of-the-art quantization. This key finding also applies to current pruning methods.
PROBLEM FORMULATION
2.1 BACKGROUND
As discussed in Section compression on LRMs (not LLMs) is relatively underexplored. We conduct a thorough literature review in Appendix B
Bottlenecks on evaluation. Few quantization or pruning methodologies have sufficiently demon- strated effectiveness on LRMs. Current works evaluate quantization and pruning performance pri- marily using perplexity and simple end tasks, such as the EleutherAl evaluation harness (Gao et al. 2024) and commonsense reasoning. However, quantized or pruned LRMs should be assessed on more complex reasoning tasks with varying difficulty levels. For distillation, although recent works tend to test on more challenging reasoning tasks (compared to other compression literature) such as GSM8K (Cobbe et al., 1) it is unclear how the compression of LRMs affects models' parametric knowledge and reasoning capability. Some them do not comprehensively select diverse reasoning benchmarks (Agarwal et al. 2024). Our benchmarking aims to address these bottlenecks.
Bottlenecks on in-depth analysis. The lack of interpretability of compression effects on LRMs key bottleneck of in-depth analysis for compressed LRMs. Being able interpret the difference between original and compressed LRMs offers a new way to analyze the effects of compression. As result, better compression approaches can be developed. A recent work (Venhoff et al. 2025) interprets several RI distilled LRMs, but their focus is not on understanding compression effects.
Recent efforts. Recent benchmarking (Liu et al. 2025a and survey (Feng et al. 2025 Srivastaval
et al. 2025) papers have begun to evaluate compressed LRMs on more complex reasoning datasets, but they all lack in-depth interpretation of compression effects and do not comprehensively com- pare different compression strategies. As for compressed LRMs, Unsloth (Daniel Han & team 2023] introduces dynamic quantization by dynamically opting not to quantize certain LLM weights. DeepSeek-RI (Guo et al. 2025) also comes with several distilled models via black-box distillation. Our interpretation analysis aims demystify the effects of LLMs compression on LRMs, providing systematic understanding of existing compressed LRMs.
2.2
MECHANISTIC INTERPRETATION
For our interpretation analysis, we target four core reasoning behaviors following an existing work (Venhoff et al., 2025): backtracking, uncertainty estimation, example testing, and adding knowledge. We prompt GPT-4o to locate token sequences of each behavior from the output to- kens of our LRMss. Our annotation dataset consists of 120 instances drawn from the four benchmark datasets (30 instances from each), so our interpretability analysis spans four different task types and difficulties. Annotation robustness of GPT-4o is demonstrated in Appendix G
To interpret different compression strategies, we adapt difference means and attribution patching by computing the activation of every linear module each layer. This allows us to compute the causal relationship between each weight matrix and our target reasoning behaviors.
Difference of Means. To compute the numerical representation in activation space of each reasoning behavior, we adapt difference of means method (Venhoff et al. 2025) Arditi et al. 2024) to extract the steering vector S for each linear module m at layer & for behavior c:
Une
(s))
1
Σ SJED_
{m)(s;), with (
ame(t)
D
D
SEED
SE
where of denotes the token sequence corresponding its five preceding tokens as output by an LRM, s, is the token sequence of the entire LRM output (prompt and output tokens), D, is the set of output instances containing at least one token sequence
specific reasoning behavior along with
3

---

Published as a conference paper at ICLR 2026
labeled with D_ is the set of all output instances, (t) is the activation of module at layer at token t, (sf) is the average of ame(() across all tokens in sf, and similarly, ame(s) (s;) is the
average of (t) across all tokens in 8j- We then normalize u me if umm T
2
where and denotes the mean activation across all tokens in D_.
Attribution Patching. To find the causally relevant LRMs components with respect to each reason- ing behavior, adapt attribution patching (Syed et al. 2023) method to compute the importance score mn of each linear module.
I ~
Σ ofED+
(ūme)
T
((s))
D+1
damt
where (s{)) is the cross-entropy loss of sf. higher In means stronger causal relationship be-
tween and the linear module m at layer f, helping us locate the most important weights responsible for reasoning capabilities (a fundamental problem for quantization and pruning works).
2.3 DECODING COMPRESSION EFFECTS
To decode compression effects, we compute the relative importance RIT of each weight matrix (I divided by Σm Ee e and track how it changes because of compression (importance shift). Specifically, we measure the change of RI from RI distilled Llama-8B to original meta-1lama/ /aa/mmaaaaasa/ 3. 3-BBB to understand distillation effect (Section 4 Likewise, the im- portance shift from the RI distilled models to their quantized versions indicates quantization effect (Section 5) For the RI distilled models, we also compute the I of their weights to complement findings the distillation effect.
We hypothesize that the importance shift should be minimal in the ideal case, as a compressed LRM should remain close possible its original counterpart (the more reasoning-capable model). When visualizing the importance shift from an LRM to its compressed variant (or from a dis- tilled model to its backbone), we only consider decreases in RI By definition, the relative importance of each weight matrix is normalized to sum to onc, so any increase in relative impor- tance necessarily compensates for decreases elsewhere. Since more informative to track cases reasoning-capable of a weight matrix is diminished), we set all increases in relative importance to zero. Additional justification of only visualizing the decreases is in Appendix
where the RI,,, of a more reasoning- capable model decreases (e.g., when the reasoning capability
2.4 SCOPE
We study three major LLMs compression paradigms, distillation, quantization, and pruning. mak- ing scope comprehensive enough for investigating the effects of diverse compression methods. For distillation, we select four RI distilled models: DeepSeek Bar-Seeek----isseee iRttisttiiiiiiiiieei
Llama-70BB Qwen-
32B, Llama-8B, and Qwen-7B. For quantization, we select 2.51-, 1.73-, and 1.58-bit models by Un- sloth as the choices of quantized I due to their popularity. We also evaluate AWQ et al. 2024) GPTQ (Frantar et al. 2022), GPTAQ (Li et al. 2025), and ANY4/3 (Elhoushi & Johnson 2025) as reproducible state-of-the-art quantization methods designed for relatively smaller L.MS (e.g., the RI distilled models). Specifically, we use all four methods to perform 4-bit quantization, and use GPTQ, GPTAQ, and ANY3 for 3-bit quantization as well, since many AWQ implementations do support 3-bit. For pruning, we run SparseGPT Frantar & Alistarh (2023) and AlphaPruning (Lu et al. 2024) on several distilled models. Inside AlphaPruning, SparseGPT is applied to prune the layers after layerwise compression ratios are computed. We run interpretation analysis on linear modules of all layers within LRMs.
2.5 EVALUATION SETUP
We select four reasoning datasets with varying levels of difficulty: AIME 2024 (Mathematical As- sociation of America) for mathematical reasoning, FOLIO (Han et al.) 2024) for logical reason- ing, Temporal Sequences of BIG-Bench Hard (Suzgun et al. 2022) for temporal reasoning, and
GSUU
4

---

Published as a conference paper at ICLR 2026
MuSiQue (Trivedi et al. 2022) for multihop reasoning. Since MuSiQue requires knowledge orization besides multihop reasoning, we follow closed-book setting (directly prompting LRMs to get final answers) to evaluate both reasoning and knowledge retention capabilities. Additional details of benchmarks, along with Table 5 that shows their statistics, are specified in Appendix C
Accuracy metric is used for AIME 2024, FOLIO, and Temporal Sequences. We adopt exact match (EM) and F1 for MuSiQue. For each model (except RI and those dynamically quantized LRMs), we three times and report its average scores mitigate performance variability, Implementation details are in Appendix D
3 COMPRESSION EFFECTS ON REASONING PERFORMANCE
In this section, we examine the effects of compression on reasoning performance. Interestingly, our key findings from the following sections also generalize to non-RI model families, as elaborated in Appendix
3.1 OVERALL PERFORMANCE
The overall performance of RI and its compressed variants are in Table We show performance of pruned R1-Di ble as it is the default sparsity level current works (Zhang et al.) 2024a Sun et al. 2023 Additional analysis of test-time compute and model collapse behavior in Appendices O and
70B and R1 Sil-Distill-Oeee
32B under 50% sparsity in Ta-
Comparing Compression Strategies. In Table [ the --t RI achieves the highest average ac- curacy overall, since it has the smallest compression ratio. Both R1 distilled Llama-70B and Qwen- 32B reach dlosetto-R/ R1 accuracy scores. On MuSiQue, the 2.51- R1 also achieves performance close to original R1. Therefore, 2-5t-bit R1 has the best overall performance than other compression strategies. Although RI may be aver-parameterized, a compression method with smaller ratio can still offer advantages over methods with higher compression ratios. In contrast, pruning only 50% of the weights causes sigmificant degradation, rendering the pruned LRMs unusable. Thus, we choose interpret the effect of pruning with greater caution and specify the details Appendix As for all distillation-only models, Qwen delivers stronger reasoning performance than Llama (Appendix E)
Comparing Benchmark Difficulties. Comparing the scores using RI distilled Llama-70B on AIME 2024, FOLIO, and Temporal, we see the largest score decrease on AIME 2024. This indicates that AIME 2024 is more challenging than the other two accuracy-based benchmarks. MuSiQue also difficult in terms of knowledge requirement, because its scores in Table Jare much lower than RAG eetrieval-augmeettee generation) setup (Zhang al. 2025). This suggests that existing LRMs lack sufficient knowledge for wewwwldge-intensiin tasks, making RAG more suitable approach.
Takeaway 3.1 for Overall Performance
Considering over- pereparraettrizzation, methods with smaller compression ratios can still offer advantages over those with higher compression ratios. Regardless of whether compression applied, LRMs lack sufficient knowledge for knowledge-intensive tasks.
3.2 COLLAPSE POINT
We investigate whether LRMs degrade as they undergo increasing levels of compression. In Table I the performance of dynamically quantized LRMs steadily declines as we move from 2.51 to 1.58- bit, we not observe clear collapse point. All 4-bit AWQ, GPTQ, GPTAQ, and ANY4 reach performance similar to their unquantized counterparts. which shows the effectiveness of existing 4-bit quantization on LRMs. However, 3-bit GPTQ, GPTAQ, and ANY3 display signs of collapse, indicating bottlenecks of current 3-bit quantization. GPTAQ and ANY4 are newer than AWQ and GPTQ, and they achieve similar performance on 4-bit LRMs. Based on average accuracy, it is noteworthy that GPTAQ surpasses GPTQ 3-bit LRMs for three out of four distilled models. Regarding distillation, RI distilled Llama-SB and Qwen-7B achieve the lowest accuracy among all distillation-only models. Only RI distilled Llama-70B yields decent MuSiQue scores.
5

---

Published as a conference paper at ICLR 2026
Table 1: Benchmark performance of RI and its compressed variants. All four benchmark scores averaged over three passes, except the rows marked with 1. Avg denotes the average scores shown in AIME 2024, FOLIO, and Temporal columns. We segment this table based model families and mark the highest scores within each model family in bold.
Models
Accuracy
Model
Compression
2024
Temporal
Avg
MuSiQue (EM, FT)
DeepSeck-R1t
76.4
99.6
83.1
(17.0, 27.51)
DeepSeek-RR R1T
2.51-bit
100.0
(17.0, 24.43)
DeepSeek-R1
671B
1.73-bit
66.7
78.3
99.6
81.5
(15.0, 22.11)
DeepSeck-RI R1
671B
1.58-bit
66.7
75.4
94.0
78.7
(14.0, 22.34)
RI-Distill-Llam
70B
Distillation
65.6
79.8
99.9
81.8
(13,3, 21,57)
70B
Distillation & 50% Spars:GPT
23.3
71.6
97.6
64.2
(6.7. 13.49)
RI-Distill-
70B
Distillation 50% AlphaPruning
74.2
97.7
12.39)
RI-Distill- Laaamlll-lmma
70B
Distillation 4-bit AWQ
63.4
78.5
99.3
80.4
(10.7, 19.23)
RI-Distil-Llame
70B
Distillation & 4-bit GPTQ
77.0
99,9
(10.3, 18.17)
RI-Distill-Lamm
70B
Distillation 4-bit GPTAQ
64.4
78.8
99.6
80.9
(12.0, 21.57)
RI- -isatill-LLama Llama
70B
Distillation & 3-bit GPTQ
46.7
718
99.3
72.6
(4.7. 11.92)
RI-Discil-Llama
70B
Distillation 3-bit GPTAQ
54.4
77.3
99.7
77.1
(5.7, 13.21)
RI- R1-Distill Qwen
Distillation
10.95)
RI-Distill-
32B
Distillation & 50% GPT
25.6
75.1
97,9
66.2
9,01,
RI-Distill-Qwer
32B
Distillation 4-bit AWQ
67.8
82.3
99.1
83.1
(3.3, 10.28)
R1-Distill Owen
Distillation 4-hit GPTQ
11.78)
RI-Distill-Oue
32B
Distillution & 4-bit GPTAQ
63.3
81.5
99.7
81.5
(2.7, 11.88)
R1 R1-Distill Qwen
32B
Distillation & 4-hit ANY4
68.9
78.0
99.7
82.2
(5.7. 12.68)
RI-Distill- Qwen
32B
Distillation & 3-bit GPTQ
74.2
R RI-Distill-Qwee
32B
Distillation & 3-bit GPTAQ
45.6
77.5
99.5
74.2
(2.3, 9.18)
R-DDsstll- Qwen
32B
Distillation & 3-hit ANY3
53.3
82.6
99.9
78.6
(3.7, 10:27)
RI-Distil-Llamm
Distillation
42.2
71.9
81.5
4.43)
RI R1-Distill-Llam
Distillation 30% AlphaPruning
41.1
68.9
82.1
64.1
4.51)
RI RI-Distill- Llaam
8B
Distillation & 50% AlphaPruning
6.7
61.7
79.6
49.3
(0.0, 2.95)
RI-Distil-Llant
Distillation 4-bit AWQ
68.0
84.0
66.6
(0.3, 5.05)
RI --istillLLaanss
8B
Distillation & 4-bit GPTQ
42.2
66.2
65.9
58.1
(0.3, 4.68)
RI RI-Distil-LLlna
8B
Distillation 4-bit GPTAQ
40.0
66.4
69.3
58.6
(0.0, 3.73)
Rl-Distil--Llama
Distillation 4-hit ANY4
88.7
66.1
(0.0, 3.54)
RI Distill-Llama
8B
Distillation & 3-bit GPTQ
11.1
65.0
67.3
47.8
(0.0, 2.89)
RI-Distill-Lann
8B
Distillation & 3-bit GPTAQ
7.8
65.5
57.2
43.5
(0.0, 3.45)
RI-Distill-Laams
Distillation & 3-bit ANY3
29.4
(0.7, 2.35)
RI-Distill-Ques Qwen
Distillation Distillation 4-bit AWQ
46.7
78.0
75.6
66.8
(..0, 3,57)
R RI-Distill-Que
7B
46.6
75.5
74.9
65.7
(0.0, 3,14)
RI-Distill- Qwen
Distillation & 4-bit GPTQ
72.9
70.3
60.7
(1.0, 4.27)
RI-Distill-Qwer
Distillation 4-bit GPTAQ
47.8
74.4
67.7
63.3
(0.0, 3.96)
R1 Distill Qwen
7B
Distillation & 4-hit ANY4
47.8
75.6
77.1
66.8
(0.0, 3.05)
RI-Distill- Qwen
Distillation & 3-bit GPTQ
17.8
65.7
31.7
38.4
(0.0, 3.12)
RI-Distil--Qwen
Distillation 3-bit GPTAQ
24.4
64.5
48.7
45.9
3.066
R1-1 R1-Disiill Qwen
7B
Distillation & 3-hit ANY3
32.2
69.3
30.1
43.9
(0.0, 3,89)
Table 2 displays performance of our two distilled models under various sparsity levels. Compar- ing distilled models with their sparsified variants, we find the precise collapse points of pruned LRMs. Interestingly, their collapse points correlate to the benchmark difficulty. For example, on AIME 2024, R1 sitiissti iillllli aillllaa ama collapses between 40% and 50% sparsity, since its perfor- mance drops by more than half. However, its collapse points on FOLIO and Temporal are roughly between 60% and 70% sparsity, which occur much later than AIME 2024. The correlation between collapse point and benchmark difficulty can also be seen the sparsified Qwen.
Takeaway 3.2 for Collapse Point
Collapse point correlates with benchmark difficulty. On hard benchmarks, 3-bit quantization and pruning with 50% sparsity or higher still have substantial room for improvement.
3.3 COMPRESSION IMPACT ON KNOWLEDGE AND REASONING
although Qwen demonstrates stronger reasoning capabilities than Llama, it has signifi- cantly lower EM and Fl scores MuSiQue. Because MuSiQue requires knowledge memorization under the closed-bou book setting, the smaller parameter count Qwen puts itself at a disadvantaged position. In other words, models' parameter count affects knowledge than reasoning. In addi-
In Table 2

---

Published as a conference paper at ICLR 2026
Table 2: Performance of two distilled models under various sparsity levels of SparseGPT. We report the one-pass scores for all models in this table.
Models
Accuracy
Model
#Param
Sparsity
AIME 2024
FOLIO
Temporal
Avg
MuSiQue (EM.
R1-Distill- Jama
70B
0%%
63.3
78.8
100.0
80.7
(13.0, 21.80)
RI-Distill-Lamm
70B
10%
60.0
81.3
99.6
80.3
(12.0, 21.69)
R1-Distill-
70B
63.3
79.3
99.6
(14.0, 21.40)
RI-Distill -lama
70B
40%
56.7
73.9
98.8
76.8
(6.0, 13.79)
H1-Distil--L.. -alama
70B
50%
26.7
70.9
97.2
64.9
(6.0, 12.75)
RI--Disill-LLan
70B
60%
65.0
95.6
(0.0,
R1-Distill-Llame
70B
70%
0.0
49,8
15.6
21.8
(0.0, 2.23)
R.I-Disill--Lam
70B
80%
11.8
12.4
8.1
(0.0, 0.94)
Distill Qwen
32B
82.3
100.0
(1.0, 9.38)
R1 Distill-Qween
32B
10%
70.0
81.3
100.0
83.8
(5.0, 13.19)
RI Distill Qwen
32B
30%
56.7
81.3
100.0
79.3
(1.0, 10.47)
R1-Distill-Oue
32B
40%
53.3
78.3
100.0
R1 Bistill-Qween
32B
50%
30.0
75.4
96.0
67.1
(3.0, 9.299
Distiiiill-ue -wen
32B
60%
0.0
65.0
87.2
(0.0, 4.13)
R1 Distill-QQue Bhsilll-Qwen
32B
70%
0.0
32.5
19.6
17.4
(0.0, 1.72)
RI Distill Qwen
32B
80%
0.0
8.7
2.0
3.6
(0.0, 1.29)
tion, we notice that pruned R1 B1-Distill-Llame 70B collapses between 30% and 40% sparsity on MuSiQue, which is even earlier than AIME 2024. This shows that pruning hurts LRMs' knowledge memorization more than quantization. When compression method aggressively re- moves the weights of an LRM, it is expected that the model's knowledge will be more seriously affected. This phenomenon can also be seen on our dynamically quantized models in Table[] Since quantization preserves parameter count and our analysis above shows that many quantized models still retain competitive reasoning capability, quantization is recommended on knowledge-intensive tasks. Additional experiments on knowledge retrieval are specified in Appendix
Takeaway 3.3 for Compression Impact on Knowledge and Reasoning
Pruning and distillation compress knowledge retention more than reasoning capabilities.
DISTILLATION EFFECT ON WEIGHTS
To study the effect of distillation on weights, we compute I of two distilled R1 models and further measure the change of RI as discussed in Section 2.3
4.1 LOCATING IMPORTANT WEIGHTS
The upper part of Figure 2 presents the weight importance of RI distilled Llama-8B in four heatmaps, each corresponding to a reasoning behavior. We observe that the final layer houses several most important linear modules across all four behaviors, with the highest value located at up up_proj. Therefore, the up_prerr proj in the final layer (32.up) stands out as the most important component.
also observe this up proj outlier in the final layer of Qwen in Figure 4 Notably, our finding complements a recent analy- sis (Shao & Wu 2025), which claims the most important module for reasoning is o_proj. Since identifying important weights is a core research problem of compression methodologies, our finding valuable for future works. Additional diagnosis based on AWQ also reinforces finding on the last-layer up projection (Appendix M).
Interestingly, this finding generalizes RI distilled Qwen-7B,
Takcaway 4.1 for Locating Important Weights
Distillation makes up ap-proj in the final layer the most important module for reasoning behaviors, as observed both RI distilled Llama and Qwen models.

---

Published as a conference paper at ICLR 2026
Becktracking
Uncertainty Estimation
0.02
0012
-0000
0.008
-0000
-0.06
gate
gate-
0.004
0004
down
0002
15
Layer
Example Tasting
Adding Knowledge
0.02
0.000
0.10
-.000
0.000
gate-
gate
0.002
0.002
down
down
Layer
mL
C
Layer
Backtracking
Uncertainty
0.025
0.000
0.015
gate
0.000
0.005
00005
down
0.000
00000
10
15
20
Layar
Layer
Example Tosting
Adding Knowlledge
gate
gate
0.00
-0005
0.05
-0.00
0.000
10
15
20
30
15
Layer
Change of RIC ml
Layer
Figure If De--sseee----Disill------------- to Llama-3. 1. 1-8B (lower). Each heatmap displays scores of importance (or importance shift) of every module at each layer, providing a fine- ane-grained analysis of weight contributions the corresponding reasoning capability. For the lower part, in- creases in RI, me are set to 0, as they only offset decreases elsewhere as discussed 2.3 Every cluster of 4 byde-by-side heatmaps (including those displayed below) follow the same scaling to show the precise magnitude of each weight module.
8B (upper) and change RI from
4.2 VALIDATING IMPORTANCE SCORES
We validate Section 4.1 by applying 3-bit round-to-nearest quantization to either 32 up or a com- ponent sharing its column or row the heatmaps, then measuring the resulting accuracy drop (Ta- ble 33. The more important component is, the greater the accuracy drop when it is quantized. Specifically, we select four additional component candidates: the second- and last-ranked modules among the seven in the final layer, and the second- and last-ranked layers across all 32 layers of the up- up-projection. We see that 32.up yields the lowest average accuracy, which clearly demonstrates the validity of our findings in Section 4.1 is quite salient that quantizing only this matrix (merely 0.7% of all weights) reduces the average accuracy by 16.3%. The component rank generally correlates with the accuracy drop, except for 1 up, which incurs the lowest accuracy AIME 2024.
8

---

Published as a conference paper at ICLR 2026
Facktracking
Uncertainty Estimation
-0.150
-0.0100
0.0000
-0.0075
0.0075
gane
gate
0.0050
0.055
4p
down
down
30
30
Example Testing
Adding Knowledge
0.0150
4-
0.1530
k-
0.0125
k-
0.0125
-000775
0.0075
-0.0300
0.0553
0.0025
domn
0.0000
5
10
15
20
25
30
5
10
15
20
25
30
Layer
Layer
Figure 3: Change of RT mf from Saspseeek-R1-Distill1- ama-88 to its 4-bit AWQ variant. Justification of only showing the decrease RI in the main content specified in Appendix H
Table 3: Accuracy after selectively quantizing single component of RI distilled Llama-8B (e.g., Lupp means to only quantize the jp-proj in the first layer) to 3-bit. Ranking of a component based its Ee Im so "2nd col" refers to second place within its column across all four heatmaps (each column consists of linear modules of a layer). "Ist overall" means the global highest ranking.
We provide additional validation on AIME 2024 scores in Appendix N
Quantized Component
Rank
AIME 2024
FOLIO
Temporal
Avg
32_up 32 gate
1t overall
20.0
63.1
63.6
48.9
2nd col
33.3
62.1
67.2
$4.2
32_v
last col
43.3
68.0
79.6
63.6
31_up uup
2nd row
33.3
70.0
64.4
55.9
1 up
last row
64.5
80.4
50.5
4.3 IMPORTANCE SHIFT VIA DISTILLATION
Since R1-Distiilllli Miiiil--llann Llama-aaa-- 8B fine-tuned based on Llama Llama-3.1-88. we compute the change of RI to visualize distillation effect in the lower part of Figure 2 (for Llama) and 5 (for Qwen). Both parts of Figure 2 exhibit similar patterns (e.g., most outliers are the final layer). indicating that the important weights of the distilled model primarily the result of distillation with SFT, while the original Ilama's weight values play little role in shaping its reasoning capabilities (we elaborate Ie me of amlama-3. 1-8B in Appendix K). Thus, distillation effect is quite powerful in transforming a non-reasoning LLM into an LRM. For Qwen, Figures 4 and 5 also show similar patterns, so the utility of distillation generalizes Qwen as well. In conclusion, distillation ef- fect explains why the final-layer up projection is one of the most important LRMs components for reasoning.
Takeaway 4.3 for Importance Shift via Distillation
Important weights of the R1 distilled models are mainly the result of the distillation effect.
QUANTIZATION EFFECT ON WEIGHTS
For quantization effect weights, analyze the decrease of importance shift during quantization. Pruning effect based on AlphaPruning appears very similar to quantization effect and is specified in Appendix

---

Published as a conference paper at ICLR 2026
Table 4: Performance of 3-bit AWQ and selectively protecting the MLP modules in the final layer.
Model
Compression
Full-Preeision Anywhere?
AIME 2024
FOLIO
Temporal
Avg
MuSiQue
R DDistillLllama-S
3-bit AWQ
10.0
59.6
68.4
46.0
(0.0, 3.50)
RI-Distill-Llama- -Lama-8B
3-bit AWQ
Final-layer MLP
16.7
67.0
74.0
52.57
(1.0, 3.62)
5.1 LOCATING QUANTIZATION EFFECT
We show heatmaps visualize the importance shift from R1 distilled Llama-8B to its 4-bit AWQ quantized variant in Figure Across all four heatmaps, we observe a reduction in the significance of the gate projections in the middle layers (e.g. layer compress these modules. Moreover, most linear modules in the final layer are compressed to the greatest extent, which shows the drawback of AWQ. Since 32.up is the most important module as discussed in Section 4.2 its importance shift is little for uncertainty estimation and example test- ing capabilities, AWQ successfully preserves its significance on these two behaviors. However, for backtracking and adding knowledge capabilities, AWQ is not effective at maintaining its importance.
23), suggesting that AWQ may overly
In Figure we visualize the importance shift from RI distilled Qwen-7B its 4-bit AWQ quantized version. We also see shift in the importance of the gate projections on Qwen, but this shift mainly occurs in the early layers (e.g., layer to 10). On Qwen, AWQ does not preserve the importance of _2_up across all four reasoning capabilities, and it also overly compresses 32_k on two capabilities.
As another popular method, we interpret the effect of 4-bit GPTQ in Figure 7 On R1 distilled Llama-8B, observe similar quantization effect AWQ, since GPTQ also overly compresses final-layer modules and the gate projections in the middle layers. Their commonality demonstrates the generality of the bottlenecks identified in existing quantization methods.
Takeaway 5 for Quantization Effect on Weights
State-of-the-- quantization methods fail to preserve the importance of the MLP gate pro- jections and the final layer, which a key bottleneck of performance improvement.
5.2 VALIDATING QUANTIZATION EFFECT
To validate our findings about the bottleneck of current quantization, we design a simple protection mechanism using mixed precision fashion. We run two versions of 3-bit AWQ in Table E In the first version, we run AWQ with their default calibration data. Since we know AWQ overly compresses the MLP modules in the final layer, then choose to protect them by changing their quantized weights to their original values in 16-bit. If they are truly important yet not well protected by AWQ, our protection mechanism should offer a significant improvement. Based on the discussion we perform 3-bit quantization, since it will be the focus of future works.
in Sections 3.1 and 3.2
We see our selective protection boosts 3-bit AWQ on all benchmarks, with an average accuracy im- provement of 6.57%. This is particularly significant given that only about 2% of all weights remain in 16-bit. This mixed precision model outperforms all 3-bit quantization baselines in Table T by at least 4.77% in average accuracy, with gains of up to 23.17%. Therefore, our findings are demon- strated with an indication of substantial room for further improvement. Note that our protection provides relatively marginal increase on MuSiQue, as the weight count stays the same (Section 3.3
CONCLUSION AND FUTURE DIRECTIONS
We study the effects of LLMs compression on LRMs and present key findings for further improving LRMs compression. These findings generalize across both R1 and non-RI models. Future compres- sion works encouraged consider the protection of MLP up projection in the final layer. The excessive compression of current quantization and pruning methods on MLP gate projections and final-layer modules highlights the need for better preserving these weight modules.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by NSF IIS-2338488. We thank Yanchi Liu for his valuable feedback on the project.
REFERENCES
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. In The Twelfth International Conference on Learning Representations, 2024.
URL [KKlhtttps://penrevveee net wttpteee//ettteeeeeeeeeeeeeeeeet
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel
Nanda. Refusal in language models is mediated by a single direction, 2024. URL https:
http://aab2220...111 11717
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT:: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024. URL https:
//openreview wwwp://///oure http://rrrmmmddddxxxxxddddddddd
Guangji Bai, Yijiang Li, Chen Ling, Kibaek Kim, and Liang Zhao. Sparsel.LM: Towards global pruning of pre-trained language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: http:::///ppeeeiee...o wwt.:eet//ooooo certtfforumiie. wtw.ee/wwwwweefooommnem
oXHyYHp4Zb
Maciej Besta, Julia Barth, Eric Schreiber, Ales Kubicek, Afonso Catarino, Robert Gerstenberger, Piotr Nyczyk, Patrick Iff, Yueling Li, Sam Houliston, Tomasz Sternal, Marcin Copik, Grzegorz Kwašniewski, Jürgen Müller, Lukasz Flis, Hannes Eberhard, Hubert Niewiadomski, and Torsten Hoefler. Reasoning language models: A blueprint, 2025. URL http::tt: htthhttss:///rxiie ptttpp::/aaxxi..rr//// 22511.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL /ttpss///axiv.
atg/abs/2110.111
Michael Han Daniel Han and Unsloth team. Unsloth, 2023. URL http://github. com/
Junslothai/unslor
DaYou Du, Yijia Zhang, Shijie Cao. Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 102-1 16, 2024.
Mostafa Elhoushi and Jeff Johnson. any4: Learned 4-bit numeric representation for llms, 2025.
URL htttps://aaxiii wwwpsiivvorrraeeeoee abs 2507. 04610
Sicheng Feng, Gongfan Fang, Xinyin Ma, and Xinchao Wang. Efficient reasoning models: A survey, 2025.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:22 10..22230.7733.... 2022.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL httt/tttt//eeooeeee zenodo. org/ wttgg//cerr//// 12608602
11

---

Published as a conference paper at ICLR 2026
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large lan- guage models, 2024. URL http:::tt///// /ttp://aaxvvvvv.. wttpv.tt...........a.oooooo abs/2305... 08543
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprini arXiv:2501. 2025.
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng. Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong. Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wo- jeiech Maciej Kryscinski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, and Dragomir Radev. FOLIO: Natural language rea- soning with first-order logic. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 22017-22031, Miami, Florida, USA, November 2024. Association for Computational Lin- guistics. doi: 10. 111118663/112224.com maalll-mmii..2295 1229. URL attps://aclanthool org/ 2024. www.eennnnlmmm. 1229/
Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models are reasoning teachers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 14852- 14882, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/ a12002..ac--oong.88 URL https: http:////caaaatthhhooceoo
wwwp/r/g/////.. acl acl-long...... 830/
Coleman Hooper, Schoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length I1m inference with kv cache quantization. Advances in Neural Information Processing Systems, 17:1270-1303, 2024.
Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown. In-context learning distilla- tion: Transferring few-shot learning ability of pre-trained language models. arXiv preprint 2022.
Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yi- wei Qin, Weizhe Yuan, and Pengfei Liu. OI replication journey-part 2: Surpassing ol op-reviww through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.1642
2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, and Tianyi Zhou. Selective eeffection-tuningg Student-sselettd data recycling for L.LM instruction-tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Lin- guistics ACL 2024, pp. 16189-16211, 1, Bangkok, Thailand and virtual meeting, August 2024a. Association for Computational Linguistics. URL htttp:://alaatttlllats wrgp://oooorg//222.. acldings-acl... 958
Shiyang Li, Jianshu Chen, yelong shen, Zhiyu Chen, Xinlu Zhang. Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xifeng Yan. Explanations from large language models make small reasoners better. In 2nd Workshop on Sustainable AI, 2024b. URL https:
7/open ew,
Yuhang Li, Ruokai Yin, Donghyun Lee, Shiting Xiao, and Priyadarshini Panda. Gptaq: Efficient finetuming- free quantization for asymmetric calibration, 2025. URL /tttpp://arxiv.
abs/2504.00 02692
Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less is more: task-aware lase distillation for language model compression. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
12

---

Published as a conference paper at ICLR 2026
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, and Lu Hou. Quantization hurts reasoning? an empirical study on quantized reasoning models, 2025a.
URL
abs/ 200...//254.000 04823
Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, and Kwang-Ting Cheng. LLM- FP4: 4-bit floating-point quantized transformers. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 592 605, Singapore, December 2023. Association for Computational Linguis- URL http::tttp//..mm //acl /ttpa///aaaaaaahhhhaaaa antho 1 logy. org/2233
tics. doi: 10. 12003/3/11123...mmmmmmmii emn p- ma min.n.3. 397
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Lim-qat: Data-free quantization aware training for large language models. Findings of the Association for Computational Linguistics ACL 2024, pp. 467-484, 2024a.
Zihang Liu, Tianyu Pang, Oleg Balabanov, Chaoqun Yang, Tianjin Huang, Lu Yin, Yaoqing Yang, and Shiwei Liu. Lift the veil for the truth: Principal weights emerge after rank reduction for coouueeg--ocuseed focused supervised fine-tuning, 2025b.
http:///axvvvvvoooo www.aiiwggggaaaaaaaaaaaaao 2506.
00772
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Forty-first International Conference on Machine Learning, 2024b. URL htllhtttss//ppeneeee/ .ne.ieewwwee////ee...o.
Corum?id=I.Ce 2Rq80
Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W. Mahoney, and Yaoqing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models, 2024.
10910
Xinyin Ma. Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems, 2023.
Mathematical Association of America. MAA Invitational Competitions. URL https::/maa
org/maa- ona compet itions/ Accessed: 2025-03-19.
OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum. Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Eliz- abeth Prochl, Enoch Cheung. Eric Mitchell, Eric Wallace, Erik Ritter. Evan Mays. Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas. Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart An- drin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O' O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jemry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl
13

---

Published as a conference paper at ICLR 2026
Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kon- draciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofrr Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang. Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agar- wal, Santiago Hernandez. Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang. Tay- lor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojeiech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang. Zheng Shao, and Zhuohan Li. Openai ol system card, 2024. htt.ttps:://////ttt ptttsps:://rriiii http:///aaw....oora///////1 11211...6766. 16720
Jie Shao and Jianxin Wu. Who reasons the large language models?, 2025. URL https:// arxiv. www. abs 225552209999
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Ommidirectionally calibrated quantization for large language models. In The Twelfth International Conference Learning Representations, 2024.
net/ forum? www.wmmmmw..WwuOLO.
Gaurav Srivastava, Shuxiang Cao, and Xuan Wang. Towards reasoning ability of small language models, 2025. htttp://////aa www.aawwwww..aa..a..rrrrrarraaae wttp:///bbbb//2..11.113.
Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thoughe can solve them, 2022. URL http::///////
abs / 2tt//220..0 09261
Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery, 2023. httt:p:://axxi..org//...se
Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown. Hugging Face, 2024.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multi- hop questions via single-hop question composition, 2022. URL htttttsss//xxii wwtp//org/oooo abs/
2108. 005133
Constantin Venhoff, Iván Arcuschin, Philip Torr, Arthur Conmy, and Neel Nanda. Understanding reasoning in thinking language models via steering vectors. In Workshop on Reasoning and Planning for Large Language Models, 2025. URL htttps:/ 1d=0whVWNOBCZ.
forum?
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang. Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models, 2023. htl.:htts:///arrt www.:///.rgg////o//ooooooooss/ 11453
14

---

Published as a conference paper at ICLR 2026
Mengzhou Xia. Tianyu Gao. Zhiyuan Zeng. and Danqi Chen. Sheared LLaMA: Accelerat- ing language model pre-training via structured pruning. In The Twelfth International Confer- Learning Representations, 2024. https:/:: http///ppppeeeeeeeeeeeeeeeeeee net fertfforum?id 0910dae0zpp
https:://pperevieei
Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. The Thirty- eighth Anmual Conference on Neural Information Processing Systems, 2024. URL https:
//openreview wwt./forfom/mm d=7wi G9K FHV
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang. Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168 -111188--18,, 2022.
Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning Ilms to high sparsity, 2025.
URL /ettpps:/azztae org. abs /2100 05175
Nan Zhang. Yanchi Liu, Xujiang Zhao, Wei Cheng. Runxue Bao, Rui Zhang, Prasenjit Mitra, and Haifeng Chen. Pruning as domain-specific LLM extractor. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1417-1428, Mexico City, Mexico, June 2024a. Association for Computational Lin- guistics. doi: 10. 11221866//1/2224.... findings-ninnnnan naac1.91. URL attps://aclantie
orrg/
2024. findingnnnggiii...........g naw.s.aaaa...111
Nan Zhang, Prafulla Kumar Choubey, Alexander Fabbri, Gabriel Bermadeut-Shapiro, Rui Zhang, Prasenjit Mitra, Caiming Xiong, and Chien-Sheng Wu. SireRAG: Indexing similar and related information for multihop reasoning. In The Thirteenth International Conference Learning
Representations, 2025. BRRhhtts::/// //tps/////.///t////////////eeeer net wtt////rrmmm/o 1 http://p.......eeeee...oo.ootott
iorum?i--yp35000U yp95gouAAT]
Yusen Zhang, Sarkar Snigdha Sarathi Das, and Rui Zhang. Verbosity * veracity: Demystify ver-
bosity compensation behavior of large language models, 2024b. URI. httptttts///xiv..ooot
abs/2411 07859
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12: 1556-1577, 2024. doi: 10.1 162/tacl 00704. Gacl-1.85/
A USE OF LLMS
To improve the overall clarity of writing, used ChatGPT-44 and ChatGPTT5 via OpenAl's web interface to polish a small fraction of sentences. LLMs were not used in any steps of the research ideation process. To ensure correctness and precision, we carefully reviewed and adapted all LLM-generated content before incorporating it into our writing.
B RELATED WORK
Our literature review is conducted over existing compression methodologies (quantization, distilla- tion, and pruning), LRMs, and recent module-wiss compression.
B. QUANTIZATION
Quantization reduces the number of bits used to represent LLM weights, thereby lowering their pre- cision (Srivastava et al. 2025). Recent survey (Zhu et al. 2024) categorizes quantization method- ologies into quantization- aware training (QAT) and post-training quantization (PTQ). QAT requires retraining of model weights to recover performance loss during quantization while PTQ does not require retraining. Recent QAT includes I.MM-QAT (Liu et al. 2024a) that adopts distillation to
15

---

Published as a conference paper at ICLR 2026
train quantized LLM, BitDistiller (Du et al. 2024) that develops self-distillation approach for the full-precision model to act as the teacher of its low-bit counterpart, BitNet (Wang et al. 2023) that
proposes 1-bit Transformer architecture for training LLMs from scratch, and OneBit (Xu et al. 2024) that quantizes L.LM weight matrices to 1-bit from a knowledge transfer perspective.
PTQ is more popular in terms of the number of recent publications, because there is retraining involved. For example, GPTQ (Frantar et al. 2022) and GPTAQ eti al. 2025) are one-shot weight quantization methods that use approximate second-order information, while AWQ (Lin al. 2024) leverages activation distribution for finding the salient weight channels to skip. Other PTQ methods
include weight-activg activation quantization (Shao et 2024 Yao et al. 2022 Liu al. 2023 and KV
cache quantization (Hooper et al. 20024 Liu et al. 2024b).
B.2 DISTILLATION
Distillation involves two settings: black-box and white-box settings. For black-box setting, the teacher model is typically closed-source LLM and only the outputs of teacher are available for the student model. For white-box setting, both weights and output distribution of the teacher model
available. Existing black-box distillation (Huang et al. 2024 Li et al. 2024b Ho et al.) 2023
Huang et al. 2022 Li et al. 2024a) prompts the teacher model to generate a training dataset for the student to learn. Specifically, researchers have started to distill OpenAl's s O1 model (Huang al.) 2024), which marks the beginning of LRMs compression. White-box distillation allows the student model learn from teacher's knowledge representation. Works has been done align the output distribution (Agarwal et al.) [0224] Gu et al. 2024) or the hidden representation (Liang et al. 2023) between teacher and student models.
B.3 PRUNING
There are unstructured and structured pruning. For unstructured pruning, individual weights are tar- geted, which leads to irregular sparsity structure. In contrast, structured pruning involves removing entire network components such as channels layers (Zhang et al. (024a). Unstructured pruning usually has better compression performance than structured pruning, while it is casier to achieve inference speedup via structured methods (Zhu al. (024).. Recent unstructured pruning includes one-shot pruning (Frantar & Alistarh| [203)] [unn et al.] 2003). global pruning that makes pruning decisions based on all layers (Bai et al. [2024), and domain-specific pruning (Zhang al. (0244).
Structured pruning includes gradient-based (Xia et al. 2024
Ma el al. 2023) and non-gradient-
based (Ashkboos al. 2024 methods.
B.4 LRMs
Trained with reinforcement learning, LRMs extends LLMs with advanced reasoning mecha- nisms (Besta et al. 2025). Popular closed-source LRMs are OpenAl's ol-mini, ol (OpenAI et al. 2024). and o3-mini. Open-source LRMs include DeepSeck-RI and -o--eie-rre (Team) 2024.. Since quantization, white-box distillation, and pruning methods require access to model weights, they are not suitable for closed-source LRMs. Only black-box distillation will work closed-source models.
B.5 MODULE- WISE COMPRESSION
Recent module-wise compression methods (Lu al., 2024 Yin al. 2025) are designed to ad- dress imbalanced quality among different model components (e.g., layers and weight matrices). This research direction could potentially address the problem of over-compr the important weights, which is also one of our motivations of locating important weights. As a way of enriching module-specific compression, we speculate that using non-uniform compression ratios across differ- ent modules (e.g., unstructured pruning and mix-precision quantization) might pose challenges for inference speedup. A potential solution of being more sardware-efficice tios with protection mechanisms important weights. For example, AWQ protects salient weights identified via activation signals by scaling them up, while still using a uniform compression ratio across modules.
to enforce uniform ra-
16

---

Published as a conference paper at ICLR 2026
Table 5: Dataset statistics of selected reasoning benchmarks.
Size
Answer Type
Metric
Knowledge Required?
AIME 2024
30
Integer True/False/Unccrttain
Accuracy
False
FOLIO
203
Accuracy
False
Temporal
250
(A)(B)((C(())
Accuracy (EM, F1)
False
MuSiQue
100
A few words
True
As the cornerstone of module-wise compression, it is quite valuable to look for novel importance signals for locating salient weights (Liu et al. (025b). For example, our paper decodes the effects of compression on LRMs and validates the usefulness and generalizability of fine-grained mechanistic interpretation for LRMs compression. successful compression method might need to incorporate multiple signals merge them, as some may benefit from change in the compression goal (e.g., changing the domain or the compression ratio).
C ADDITIONAL DETAILS OF REASONING BENCHMARKS
Table shows the statistics of our selected benchmarks. AIME 2024 (parts I and II) represents top match challenges, and its answers are integers. FOLIC requires logical deductions deter- mine whether the provided conclusion is true, false, uncertain based on premise. In Temporal Sequence models are asked to use a provided timeline to determine what time person might be free perform another activity. Since each of its questions comes with four options, we expect our models to output the index (the letter) the selected option. Since MuSiQue involves question answering and its answers are in few words, we adopt exact match (EM) and FI. We randomly sample 100 questions out of 1000 from MuSiQue for our benchmarking analysis.
D IMPLEMENTATION DETAILS
We run the dynamically quantized models on 11 llama. cpg other distilled, pruned, and quantized models on vLLM (Kwon al. 2023) for its fast inference. In order to comprehensively analyze performance change after compression, we also evaluate RI on our reasoning benchmarks by using DeepSeek API. Aligning with DeepSeek-R report (Guo al.) 2025). we keep the same parameters for all models during inference: maximum generation length is set to 32768, temperature is set to 0.6, and top-p value is set to 0.95.
6 based on their requirement. We run all
Within the 2-51-bit LRM, the embedding matrices, output/head, and normalization/router layers kept at higher bit widths, while almost everything else is quantized aggressively. Specifically, the embedding matrix is in 4-bir bit and the final language model head is in 6-bit, while all MoE (Mixture of Experts) routing matrices and layer-norm norm weights are left in full precision (32-bit). The majority (around 88%) of weights namely the bulk of the MoE weights are quantized down to 2.51-bit.
We do not find precise details of the calibration dataset used for 2.51-bit quantization. However, we more than 1.5 million tokens for future quantized models, we can
that Unsloth plans make a reasonable prediction that the dynamic quantization uses more calibration data than other quantization methods we benchmark in our paper. We adopt the same calibration data for all other quantization. The dynamically quantized models work on MoE architecture such as R1, while other quantization methods (e.g., AWQ and GPTAQ) are designed for non-MoE with less weight count. Therefore, due to different compression targets, we are not able to compare dynamic quantization and other methods under identical calibration. In Table [ we aim to report various methods as comprehensive as possible in order to study the effects of compression on reasoning performance.
httets//wuggnntaecc////tttttst/a https: htt.::/thh/.....ooooooooooooooon http://c..aaaaaaaaooo///aaaaaaaa
sequences.
17

---

Published as a conference paper at ICLR 2026
Backtracking
Uncertainty Estimation
4-
k-
0..015
0.015
gate
0.010
gate
0.010
up
up
10
15
20
25
10
15
20
Layer
Layer
Example Testing
Adding Knowledge
0.025
0.025
0.015
gate
0.011
gate
0,010
0.005
0.005
10
10
Layer
Layer
Figure Ihe of
We use AutoAWo as the AWQ implementation for inference due to its speed advantage (vLLM support), while the original AWQ code used generate the pseudo-quantzeed RI distilled Llama- 8B for our analysis in Section
We focus on analyzing the effect of compression methods on performance and thus do not consider inference speedup. The reason is that these methods run on different inference platforms, so it is hard to control the consistency of inference optimization across various platforms.
E COMPARING DISTILLED MODELS
On accuracy-based benchmarks of Table I we see that RI distilled Qwen-32B delivers an aver- age 0.4% improvement over Llama-70 and RI distilled Qwen-7B delivers an average 1.6% im- provement over Llama-8B. Although these two Qwen models have less weights, Qwen deliv- ers stronger reasoning performance than Llama. This phenomenon aligns with DeepSeek re- port (Guo et al. 2025). However, ai1-Dsstil--eee (all---en-3222 32B scores significantly lower than R1 R1-Distill Llama 70B MuSiQue, highlighting its worse ability of memorization.
F ADDITIONAL VISUALIZATION OF WEIGHT IMPORTANCE AND IMPORTANCE SHIFT
Figure shows the weight importance of Deessel--Rl-Disstll---esss heatmaps, each corresponding to a specific target reasoning behavior. Figure 5 displays to To decode the quantization effect on Qwen, Figure 6 shows the change of RI me from Sonsseek-R1--istil-----0
across four
the change of RI from SeeppSee-RR--ae -
to its 4-bit AWQ variant. Similarly, Figure shows the
change of RI from RI distilled Llama-8B to its 4-bit GPTQ quantized variant.
G ANNOTATION ROBUSTNESS
It is important to find validated approach to automatically perform annotation, as large number of of would help the computation of importance scores. We did not change the behavior definitions and
http:://ggittubb.comm/tttttt
awg
18

---

Published as a conference paper at ICLR 2026
Backtracking
Uncertointy Estimation
4-
V-
V-
0.03
0.03
gete
gate
up
up
down
down
0.00
0.00
15
20
25
20
25
Example
Adding
9-
V-
0.03
0.03
gate
0.02
gate
0.02
up
up
down
down
0.00
0.00
10
15
20
25
15
20
25
Figure 5:
Change of RI ml from
DeepSeek-Rl-Distill ama-88
2 th-78
7B (the backbone model).
Backtracking
Uncertainty Estimation
k-
k-
V-
gate
gate
0.01
0.01
down
down
0.00
0.00
Layer
Layer
Example Testing
Adding Knowledge
k-
k-
0.02
0.02
gate
gate
0.01
0.01
down
down
0.00
0.00
Layer
Layer
Figure 6: Change of RI
mf from
its 4-bit AWQ variant.
Table 6: Statistics of ratings under 2 temperature values. "CI" represents 95% confidence interval.
Behavior
Avg (temp=l)
Avg (1emp=0.1)
Spearman p
CI
Backtracking
0.9
0.94
0.892
[0.474, 1.000]
Uncertainty Estimation
0.78
0.75
0.803
[0.416, 1.000]
Example Testing
0.90
0.87
0.862
[0.645, 1.000]
Adding Knowledge 0.86
0.96
0.486
[-0.218, 1.000]
instruction prompts.
The most salient adjustable parameter is the temperature, and used the de-
19

---

Published as a conference paper at ICLR 2026
Backtracking
Uncertainty Estimation
0.020
0.023
gate
gate
-0.05
0.005
0.000
0.003
30
15
30
Layer
Layer
Example Testing
Knowledge
0.020
0.020
0.015
V-
0.015
0.010
0.010
dbem
10
15
20
25
30
10
15
20
25
30
Figure 7: Change of RI from DeepSee--R1-Distil1---- tized variant.
to 4-bit GPTQ quan-
fault temperature value (I) in annotation pipeline. We study whether the change of temperature would overturn our annotation results.
We choose low temperature value (0.1) and whether more deterministic outputs would affect our annotation results significantly. Setting temperature to 0.1, we first use GPT-4o with the exact same behavior definitions and instruction prompts collect annotation. Then, we perform human validation to judge the output quality and stability when a lower temperature is set. To score a model output instance on specific behavior, we instruct the evaluator to go through all the behavior tokens that are tagged and compute the percentage of correctly tagged behavior. For example, suppose GPT- 4o tags 20 "backtracking" token sequences in an output instance and the evaluator finds 18 of them closely following the definition of "backtracking". the score of this instance is 0.9. There are cases when an instance ends up with no tagged behavior. Since we encourage a conservative annotation style (the value of st is large enough for computation such 1000, it is fine if GPT-4o misses some token sequences of the target behavior), we assign the score of (maximum score) to those instances ending up with 0 tagged behavior. For each behavior, the evaluator samples 10 instances out 120 (with replacement) and scores them. We present the average score 10 sampled instances under temperatures and the Spearman P with 95% confidence interval Table
As shown, the average rating scores of both temperatures are high across behaviors, which in- dicates reliable annotation results done by GPT-4o. Comparing the results of both temperatures side-by-side, we observe that a temperature of tends to produce more nested and unclosed token sequences, which expected due higher randomness. This does not affect our interpretation analysis, since we exclude those nested or unclosed sequences.
As for Spearman p, we see that the two rating scores on all reasoning behaviors exhibit at least a moderate and often strong correlation, which demonstrates stable rank ordering. For "adding knowledge" behavior, the 95% confidence interval is wide, indicating uncertainty due the small sample size of 10. Given high average rating scores, we conclude using the default temperature value does not significantly affect the correctness of annotation. Therefore, the robustness of our findings is validated.
H JUSTIFICATION OF VISUALIZATION CHOICE
As elaborated in Section 2.3 we track changes in relative importance from a more ceassoninn-caaaabl model to less cassonnngcaapable model (rather than the other way around). Then, mention to only consider decreases to show our interest in tracking the cases when the reasoning capability of weight matrix is diminished. In other words, when LRM is compressed via suboptimal
20

---

Published as a conference paper at ICLR 2026
Backtracking
Uncertainty Estimation
-0.10
0.003
0.005
0.006
gite
0.004
gato
0.044
-0002
0.002
0.000
0.000
Layer
Layer
Example Testing
Adding Knowledge
gate
gate
0.002
0.000
-0.000
10
15
20
30
15
20
Layer
Layer
Figure 8: Change of RI (increases only) from DeepSeek Seepseek-RR-Dissee RR--iitiill---- -lmma-a------------- 4-bit AWQ variant.
almma-----mmmmmmmmmmm -BB to its
Backtracking
Uncertalntty Estimation
0.0100
0.0100
-20075
0.0075
gate
g0te
0.0255
down
doen
Layer
Layer
Example Testinng
Adding Knowledge
0.0100
0.0100
-0.0755
0.0075
gafa
gile
0.0050
0.0050
drwn
down
Figure 9: Change of RI (net change) from DeepSeek-R1-Disteeeeee
its 4-bit
AWQ variant.
pruning or quantization, salient weight components more likely show decreased rather than increased relative importance, because suboptimal compression tends to preserve or harm, but not improve, the reasoning capabilities of important components. For compression methods that fully preserve or increase the relative importance of some salient components, tracking these increases adds little value, as it essentially confirms that the methods are performing well. more valuable to visualize the effects of compression with an emphasis on the weakness of current compression methods, which can inspire future improvement such as our third key finding.
On the other hand, in order to show all aspects of importance shift, we draw the heatmaps of only considering increases (Figure { and those that reflect the net change (Figure 9 from RI distilled Llama-8B to its 4-bit AWQ variant. We that the net-change heatmaps appear quite similar to those that only consider decreases 3). Specifically, the net-change heatmaps primarily high- light only few components from the increase-onl. heatmaps based on Figure 3] and our third key finding still holds fresearchers are recommended to prioritize the preservation of -ayerll-ly--- module importance in order to improve current quantization methods). The similarity between Figure 3 and
21

---

Published as a conference paper at ICLR 2026
Backtracking
Uncertainty
0.0125
0-
000075
gote
-0.053
gate
0.0050
0.0025
0.0025
down
down
0.0000
0,0000
Exampic Testing
Adding Knowledge
0.0125
0.0125
K--
0.0100
0.0100
0.0075
0.0075
0.0033
0.0000
0000003
0.0000
10
15
20
25
10
25
Layer
Layer
Figure 10: Change of RI from BeepSeek-R1-Distil1-- variant (at 30% sparsity).
to its AlphaPruning
net-change heatmaps showcases a significant shortcoming of current quantization, as most of the net change comes from diminishing capabilities of certain salient modules in Figure
PRUNING EFFECT ON WEIGHTS
we require pruned LRM that remains usable for interpretability anal- ysis. annotating and interpreting outputs that consist largely of gibberish would be meaningless. Therefore, order to provide more insights on pruning, we turn to AlphaPruning (compared to SparseGPT on the R1 distilled Llama-70B, AlphaPruning achieves higher scores most bench- marks in Table []. On RI distilled Llamm-8B, we iteratively reduce the 50% sparsity level and ultimately settle 30%, where much stronger reasoning performance is achieved. Although 30% sparsity (1.4x reduction in size) offers much smaller compression ratio than 4-bit quantization (4x reduction in size), its "avg" score of 64.1 is significantly higher than 4-bit GPTQ and GPTAQ in Table We then select AlphaPruning at 30% sparsity on the R1 R1-distilled Llama-8B as the pruned LRM for our interpretability analysis. We show the corresponding heatmaps in Figure 10
As mentioned in Section 3.1
We observe that Figure 10 appears very similar to 4-bit AWQ on R1 distilled Llama-8B (Figure 3 4- bit GPTQ R1 I Llama-8B (Figure and 4-bit AWQ R1 distilled Qwen-7B (Figure Therefore, our third key finding can be nicely generalized to AlphaPruning as well: AlphaPruning overly compresses the final-layer modules and MLP gate projections. This additional experiment shows the effects of pruning and the generalizability of our third key finding on pruning methods.
J GENERALIZABILLTY OF KEY FINDINGS ON NON-R1 FAMILIES
We are excited to share that our key findings generalize well to Fixxttact/pisei1177-------aaaaa
distilled model from both GPT-OSS-2OB and GPT-OSS-120B via SFT). In order to systematically investigate weight importance and quantization effect, we interpret this distilled model and 4-bit GPTQ quantized variant. We first show their one-pass benchmarking scores in Table 7 It is clear affected by quantization, further implying that preserving weight count with lower precisions is a reasonable strategy for LRMs' knowledge memorization (first key finding).
that MuSiQue scores not
Then, we look at our second key finding. After rumning our fine- fine-grained interpretation analysis on
present its heatmaps in Figure [I] We that 36.up (up projection in the final layer) is either the most important or the second most important com- ponent across four heatmaps, which demonstrates the generalizability of second key finding.
22

---

Published as a conference paper at ICLR 2026
Backtracking
Uncertainty Estimation
0.020
0.00
K-
0.035
V-
0015
6-
gate-
gaze
0.05
0005
dzam
35
50
Layer
Layer
Example Testing
Adding Knowledge
-1070
0000
-0023
0013
0-
gate
gate
0.005
0.00
25
25
30
Layer
Layer
Figure 11: Ii D
Bocktracking
Uncertainty Estimation
0008
6-
0008
3006
V-
0.006
gade
gate
0.000
0.000
Example testing
Adding Knowledge
0.008
0.008
k-
3.006
0.068
-2004
0.004
0.000
0.000
Figure 12: Change of RI me from 1is.::1GT---SS--eee----- to its 4-bit GPTQ variant.
Table 7: Performance Dist::1GTT--SS-oeenn- and its 4-bit GPTQ variant.
Model
AIME 2024
FOLIO
Temporal
Avg
MuSiQue
DistIGPPT-OSS-qeen---P
36.7
82.3
94.4
71.1
(4.0, 6.73)
4-bit GPTQ quantized DistiGGTT-OSS-gwenn--
26.7
76.4
86.4
63.2
(4.0, 6.92)
Moreover, whenever 36_up is ranked second on heatmap, 36_gate is always ranked first. As both up projecteoio and gate projection in the final layer are important to R1 distilled models in Figure (upper part) and [] we see a high similarity of weight importance between R1 distilled models and this GPT-OSS distilled model, which indicates the possibílity of finding common important compo- nents across different LRMs. On the other hand, it is worth noting that different LRMs might end up with slightly different heatmaps. For example, last-layer k-projection but appears to be less important for the GPT-OSS distilled model. This shows that interpretation can produce customized analysis weight importance. Our second key finding aims to locate the import weight component that is shared by different LRMs (e.g., different model sizes and classes).
highlighted in Figure 4
Finally, for our third key finding, we demonstrate its generalizability through the 4-bit GPTQ quan- tized Di st i 1GGT--SS -os---osse--nn--- -wwe-wwweneee -BB. The heatmaps for visualizing decrease of relative importance
23

---

Published as a conference paper at ICLR 2026
Becktracking
Uncertainty Estimation
0.029
V-
0.020
-0.015
0.015
0.010
0.010
0.005
0.005
10
15
20
25
30
10
15
20
30
Layer
Example Testing
Adding Knowledge
9-
0.025
0.025
-0.15
0.015
gate
0.019
gate
0.010
0.005
0.005
down
dbwn
10
15
25
50
16
15
20
30
Figure 13: I of Llama 1.samaa3.111 -3.1-8B
Table 8: Performance of naive RAG and closed-book settings on MuSiQue.
Setting
LRM
MuSiQue
Closed-book
R1-Distill-
(0.0, 4.43)
RAG
R1-Distill-LLae LLama-8B
(2.0, 14.52)
Closed-book
R 11-Distill-Qwen--7
(0.0, 3.57)
RAG
RI RI-Distill-Qwen-7B
(2.0, 13.54)
are shown in Figure 12 Most salient decreases (outliers) come from the last-layer components across all four heatmaps, along with few salient decreases in the second last layer. Out of small number of outliers in a heatmap, gate.projection also has at least outliers on the backtracking and uncertainty estimation capabilities. Therefore, even when we switch GPT-OSS distilled model, GPTQ still overly compresses the final-layer modules and MLP gate projections.
K IMPORTANCE SCORES OF BASE MODEL
Complementing the lower part (change of relative importance from RI distilled Llama-8B to its base model before distillation) of Figure [ we present the importance scores of the base model (Llama-3, 3. 1-88) BB) in Figure 13, As we can see, the heatmaps of the base model look very different from those of the Ri-isiileed model (upper part of Figure There are many differences. For exam- ple, compared to uncertainty estimation behavior, other reasoning behaviors do not present clear outlier of importance, which indicates more evenly distributed causal relationships across weight matrices on backtracking, example testing, and adding knowledge. As for uncertainty estimation, the last-layer o-projection has the highest importance, whereas the up-projection in the final layer is not even ranked among the top three components of that layer. Therefore, pletely different conclusion from key finding if we were to interpret the base model rather than the LRMs.. The commonality between the two parts of Figure ? together with the difference between the base model and the lower part of Figure 2 demonstrates Takeaway 4.3 ("important weights of the RI distilled models are mainly the result of the distillation effect").
could reach com-
L ADDITIONAL RAG PIPELINE
Section 3.3 concentrates the significantly lower MuSiQue scores move from R1 different compressed variants. We hypothesize that parametric knowledge is more sensitive to parameter count, because pruned R1 distilled Llama-70B in Table 2 collapses MuSiQue even earlier than
24

---

Published as a conference paper at ICLR 2026
Table 9: Performance of selectively quantizing 32_up __up on AIME 2024.
Selective Quantization
32 up
1 uuu
4-bit Pisstil-LLlama-e
20.0
43.3
5-bit 11-Distill-Llama-
26.7
30.0
3-bit (R Me1--isilll-Qee--771
16.7
46.7
AIME 2024. Other notable phenomena include the slightly stronger reasoning performance of RI distilled Qwen-32B than Llama-70B, but with collapsed MuSiQue scores at even 0% sparsity.
In order to further validate our key finding 1, we implement RAG pipeline across two RI dis- tilled models with different sizes. Since all MuSiQue scores in Table I follow the closed-book setting, we provide results from naive RAG to decouple reasoning from knowledge memorization. Specifically, we use BAAI -aaaae/ee--aseeen---ee
as the embedding model and retrieve the top 10 chunks whose embeddings are closest to each question in MuSiQue. We run this RAG pipeline using R1-Distill-Bi Table g
and R1-Dist -wwe-wee--weee--- 7B handle questions as shown in
We observe that scores get improved significantly by retrieving relevant knowledge on both LRMs, which demonstrates the lack of memorized knowledge. We would like to highlight that RAG with RI-Distill 11111--wee-7 even surpasses closed-book R1 provides strong evidence that knowledge memorization is largely compromised when compressing from 32B to 7B (a substantial reduction in parameter count), whereas multihop reasoning capability is not clearly affected by this reduction. Therefore, taken together with the discussion in Section 3.3 our key finding is validated.
32B on FI. This
M ADDITIONAL DIAGNOSIS
We performed another diagnosis by running AWQ, checking if there is alignment between our inter- pretation results and AWQ. Since salient weight channel typically gets scaled up more significantly than others, its scaling factor essentially serves as an importance measure that is based on activations. Removing its normalization scaling factors from 4-bit AWQ on RI distilled Llama-8B. We notice that the last-layer up projection gets assigned to the second largest scaling factor across all up projections by excluding the scaling factor of 1 (scaling factor of means no scaling), so we see strong alignment between AWQ and our key finding 2. The incorporation of the gradient signal in attribution patching increases our importance score of the last-layer up projection from the second largest to the largest.
scaling factors that is done component by component,
extract the
N ADDITIONAL SELECTIVE QUANTIZATION
First of all, in Table both 20.0 and 6.7 are pretty low scores on AIME 2024 given that only a single component quantized, since the original model reaches 42.2 over three runs in Table We suspect a greater variability on low-score output, so we take a closer look at the outputs of quantizing 32_up and 1_up. We find that the output of only quantizing 32 32_up contains significantly more gibberish than 1_up, which indicates the low output quality of only quantizing 32_up. For example, the total output size of only quantizing 32 up is 10.9 MB while the total output size of only quantizing 1 up is 7.5 MB.
Taking these signals into consideration, we additionally demonstrate the lower output quality of only quantizing 32_up on AIME 2024 by performing selective 4-bit and 5-bit quantization on 32_up and 1_up. To show generalizability, we also perform selective 3-bit quantization on both components over R1 RI-Distill- -ewe---enn-- 7B. The resulting AIME 2024 scores are shown in Table 9 We see that only quantizing 32.up yields lower scores than 1 l_pp on all three cases. Considering the lower average scores of only quantization 32_up in Table 3l we think the exception in Table 3 (quantiz-
ing only 1 up causes the largest drop on AIME 2024) is due to random perturbation of declined capabilities AIME 2024.
25

---

Published as a conference paper at ICLR 2026
Table 10: Analysis of test-time compute when selecting the shortest and longest 30% of responses output by each model on each benchmark. "Short" column contains performance scores of the shortest 30% of outputs from a model, while "long" column contains scores of the longest 30% of outputs. We compare the scores between "Short" and "Long" for every model and benchmark, and mark the best scores in bold. "Ratio" column represents the ratio of the average length (in number
of tokens) of the longest 30% to that of the shortest 30%.
Models
AIME 2024
FOLIO
MuSiQue
Model
Compression
Short
Long
Ratio
Short
Long
Ratio
Short
Long
Ratio
DeepSeek-RI
88.9
33.3
5.3
83.3
63.3
8.0
(30.0, 42.8)
(3.3, 10.0)
6,4
DeepSeek-RI
2.51-bit
100.0
33.3
4.9
85.0
71.7
6.5
(33.3, 41.9)
(0.0, 6.9)
7.7
DeepSeck-RI
1.73-bit
88.9
22.2
4.4
86.7
73.3
4.9
(30.0, 41.6)
(10.0. 23.5)
6.9
DeepScek-R1
1.58-bit Distillation Distillation & 10% sparse
77.8
44.4
3.8
80.0
65.0
5.0
(30.0, 41.5)
(10.0. 16.6)
5.9
RI-Distill-Llam
88.9
11.1
5,4
80.0
80.0
4.5
(26.7. 38.0)
(6.7. 18.4)
4.0
RI-Distill-Llam
100.0
0.0
6.6
85.0
78.3
4.9
(20.0, 29.6)
(6.7. 13.7)
7,4
RI 11-Distil-Llama
Distillation & 30% sparse
88.9
11.1
5.3
85.0
76.7
5.0
(26.7. 36.9)
(3.3, 12.5)
8.8
RI 1--istill-Qwen
Distillation Distillation & 10% sparse
100.0
11.1
7.1
86.7
75.0
6.9
(16.7, 24.1)
(16.7. 24.7)
7.1
R1-Distil--Qwen
88.9
33.3
4.8
83.3
75.0
5.5
(23.3, 36.7)
(3.3, 12.8)
RI-Distill-Qwen
Distillation & 30% sparse
88.9
11.1
6.8
86,7
78.3
7.8
(26.7. 40.7)
(3.3.9.1)
8.8
Table 11: Analysis of test-time compute when selecting the shortest and longest 20% of responses output by each model each benchmark. Refer Table for the meaning of each column, except that we select shortest and longest 20% instead.
Models
AIME 2024
FOLJO
MuSiQue
Model
Compression
Short
Long
Ratio
Short
Long
Ratio
Short
Long
Ratio
DeepSeek-RI
100.0
16.7
7.1
82.5
57.5
10.9
(30.0, 42.4)
(5.0. 12.5)
8.5
DeepSeck- R1
2.51-bit bit 1.73-bit 1.58 bit Distillation
100.0
33.3
6.6
87.5
62.5
8.6
(40.0, 50.3)
(0.0, 4.5)
10.7
DeepScek-RI
100.0
33.3
5.9
87.5
67.5
(30.0, 43.7)
(5.0, 15.7)
9.3
DeepSeek-RR R1
100.0
33.3
5.6
80.0
72.5
6.2
(35.0, 46.3)
(15.0, 20.0)
7.6
RI-Distill-Llam
83.3
16.7
7.0
82.5
75.0
5.7
(35.0, 45.2)
(5.0, 9.2)
5.3
Distill Llama
Distillation & 10% sparse
100.0
0.0
9.2
90.0
75.0
(20.0, 28.5)
(5.0, 11.7)
10.7
R1 31-Distill-Lamm
Distillation & 30% sparse
100.0
16.7
7.1
90.0
75.0
6.4
(20.0, 31.0)
(0.0, 7.3)
13.0
R1 E1-Distill-OO R1-Distill-Owen
Distillation
100.0
0.0
9.8
87.5
75.0
9.0
(20.0, 28.7)
(20.0, 24.2)
10.4
RI-Distill-Qwer
Distillation & 10% sparse
83.3
33.3
6.1
85.0
75.0
7.2
(30.0, 36.2)
12.0
R RI-Distill-O DistilllQwen
Distillation & 30% sparse
100.0
0.0
8.9
87.5
75.0
11.0
(30.0, 36.6)
(5.0, 11.2)
13.4
O TEST-TIME COMPUTE
We study the behavior of RI and some of its compressed variants by measuring their test-time compute. Table I0 shows the analysis of test-time compute when we select the shortest and longest 30% of responses output by each model on each benchmark. We observe that shorter model outputs consistently yield better performance across three reasoning benchmarks, with only an exception on MuSiQue. Regardless of whether an LRM is compressed, if it generates significantly more tokens for a question than other questions in the same dataset, the answer is likely to be incorrect. We exclude Temporal here, because many of the compressed R1 models achieve close 100% accuracy. The length ratios between the longest and the shortest 30% are typically greater than 4, which indicates nontrivial length differences among model outputs.
Similarly, Table II shows the selection of the shortest and the longest 20% of model responses. Compared to Table 10 as we move toward the extreme, the performance gap between the shortest and longest responses becomes even larger. Both R1 and its compressed variants achieve higher scores when they spend less compute during test time. After manually checking long responses, we notice that longer outputs tend to be more verbose and involve more backtracking in reasoning, This finding demonstrates the need to reduce verbosity to improve reasoning performance, which aligns with recent research (Zhang al. 2024b).
P CASE STUDY
As discussed in Section 3.2 pruned models collapse on all benchmarks at certain sparsity levels. We identify common phenomenon when model collapses: it repeatedly generates sentence
26

---

Published as a conference paper at ICLR 2026
Wait, Beethoven leads the orchestra, so he conductor. Therefore, Beethoven is conductor.
RI-Distill-Qwen--2B 70% sparsity:
So, the conclusion incorrect because Beethoven leads the orchestra. Wait, Beethoven leads the orchestra, so he conductor.
Therefore,
the conclusion incorrect Beethoven leads orchestra.
The conclusion true. The conclusion true. The conclusion true. conclusion
RI-Distill-Qweet -ailll-Qeen-32BB 80% sparsity:
Figure 14: Two examples of the case when model collapses and keeps repeating itself. These are two outputs (for FOLIO question) from DeepSeek - -Disti -Qwen-32R at either 70% or 80% sparsity levels.
chunk until reaching the maximum generation length. We show two examples of this phenomenon Figure 14 For brevity, omit the beginning and the end of outputs.
In both examples, the pruned models are repeating themselves without pushing their reasoning chains forward, which signal of model collapse. When R1-Distill-Owert to 70% sparsity, we see that it can still organize few sentences (e.g., chunk) to repeat. But when is pruned to 80% sparsity, only repeats simple sentence. This decline of linguistic capability common when models are pruned to high sparsities. Therefore, aggressively pruning LRMs requires careful consideration.
pruned
27


Reviews:
[{"review_id": "lqw0KfgUQW", "reviewer": "Reviewer_ST2S", "summary": "This paper studies how compression (quantization, distillation, pruning) impacts reasoning in LRMs (DeepSeek-R1 and distilled Llama/Qwen) via (i) a broad benchmark on AIME-2024, FOLIO, Temporal Sequences, and MuSiQue, and (ii) mechanistic analyses that compute behavior-specific importance for every linear module using difference-of-means and attribution-patching. Key claims include: 2.51-bit dynamic quantization of R1 attains near-R1 performance; the final-layer MLP up-projection is consistently most reasoning-critical; and protecting ~2% of weights (final-layer MLPs) recovers +6.57% average accuracy for 3-bit AWQ.", "questions": "1) Behavior-label robustness.\nVary the behavior taxonomy/prompt/thresholds and report rank stability (Kendall’s τ / Spearman ρ) with 95% CIs; include a leave-one-behavior-out analysis.\n2) RI visualization choice.\nJustify zeroing positive RI deltas and provide an alternative view that reports both increases and decreases, plus a net-change summary.\n3) Selective protection trade-offs. Provide an accuracy vs. protected-ratio (0.5–5%) curve and the latency/memory overhead; compare protecting final-layer gate vs. up vs. both.\n4) 2.51-bit dynamic quantization details.\nSpecify the skip policy (which layers stay high-precision), calibration set, and per-module bit-allocation histograms; compare to AWQ/GPTQ under identical calibration on quantization error, attribution-preservation, and end-task metrics.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1、 Comprehensive scope across three compression families with clear head-to-head tables and multiple distilled sizes (70B/32B/8B/7B).\n\n2、Fine-grained mechanistic lens (module-level DoM + attribution-patching) tied to four reasoning behaviors (backtracking, uncertainty estimation, example testing, adding knowledge).\n\n3、 Actionable insight: the final-layer up-projection is most critical; selectively quantizing it alone (≈0.7% of weights) causes a large drop, while protecting final-layer MLPs at 16-bit raises a 3-bit model by +6.57% on average.\n\n4、 Useful observations on collapse points (e.g., pruning ≥50% collapses; 3-bit baselines stress on harder tasks) and knowledge vs. reasoning separation (MuSiQue).", "weaknesses": "1、Robustness & Statistics. Behavior labeling robustness is under-specified (prompt/threshold/seed); key results lack rank-stability and uncertainty (variance, 95% CIs, significance).\n\n2、Metric & Visualization Choice. RI plots zero out increases, risking masked compensations; provide justified alternatives (report both ↑/↓ and net change).\n\n3、Coverage & Generalization. Pruning analysis is shallow (no mechanistic view beyond collapse); external validity is limited (mainly R1-distilled 8B/7B, few non-R1 families).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WTsgosGljx", "reviewer": "Reviewer_PAMX", "summary": "This paper presents a two-fold analysis of the effects of compression on Large Reasoning Models (LRMs), using DeepSeek-R1 as its primary case study. First, it provides a comprehensive benchmark of three major compression families—quantization, distillation, and pruning—evaluating their impact on performance across a diverse set of reasoning datasets (AIME 2024, FOLIO, Temporal Sequences, and MuSiQue). Second, and more notably, the paper employs mechanistic interpretability techniques (adapting difference of means and attribution patching) to identify which specific model components are causally important for reasoning. The authors find that current compression methods, particularly quantization, disproportionately harm these critical components. They validate this finding by showing that selectively protecting these components (e.g., the final-layer MLP) during quantization significantly recovers lost performance.", "questions": "1. Can the authors comment on whether the \"final-layer importance\" finding is specific to the R1-distillation process? Have you tried applying your interpretability analysis to a standard, non-distilled model (e.g., base Llama 3) that has been fine-tuned for reasoning? Would you expect to see the same components identified as critical?\n\n2. Could you elaborate on the validation process for the GPT-4o labeling of reasoning behaviors? How sensitive are the final importance scores (and the resulting conclusions) to potential inaccuracies or inconsistencies in this automated labeling process?\n\n3. The paper notes (Takeaway 3.3) that pruning/distillation (reducing parameter count) hurts knowledge memorization (MuSiQue) more severely than reasoning (AIME, FOLIO). Quantization (reducing precision) seems to have a less detrimental effect on knowledge. Could you expand on this distinction? Why do you hypothesize that parametric knowledge is so much more sensitive to parameter count than to parameter precision?\n\nI would like to improve my scores if authors can solve my questions.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Novelty of Interpretation: The paper's primary strength is its application of mechanistic interpretability to the problem of model compression. It moves beyond standard \"accuracy vs. bits/size\" tables to provide a causal analysis of why and where performance degrades, which is a valuable contribution.\n\nActionable Findings: The analysis yields clear, actionable insights. The identification of the final-layer mlp.up_proj as a critical component for reasoning (in R1-distilled models) and the finding that popular quantization methods (AWQ, GPTQ) overly compress these final layers are important discoveries.\n\nStrong Validation: The experiment in Section 5.2 is compelling. Demonstrating that protecting just 2% of weights (the final-layer MLPs) from quantization can recover 6.57% in average accuracy on a 3-bit model provides strong validation for the paper's entire interpretability pipeline.", "weaknesses": "Generalizability: The analysis is heavily centered on DeepSeek-R1 and its specific distilled variants (R1-Distill-Llama, R1-Distill-Qwen). It is unclear if the central findings (e.g., the high importance of the final-layer mlp.up_proj) are a general feature of all LRMs or an artifact of the specific distillation-with-SFT process used to create the R1 models.\n\nSubjectivity in Methodology: The interpretability analysis (Section 2.2) relies on prompting GPT-4o to locate token sequences corresponding to four specific reasoning behaviors. This labeling process seems subjective and could introduce noise or bias. The robustness of the resulting steering vectors is highly dependent on the quality of this heuristic.\n\nUnderdeveloped Pruning Analysis: While pruning is introduced as one of the three main compression methods, it is quickly dismissed after benchmarking shows it performs poorly (e.t., at 50% sparsity). The subsequent mechanistic analysis focuses almost exclusively on distillation and quantization, making the pruning aspect of the paper feel incomplete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZXNQd6Duys", "reviewer": "Reviewer_SXNL", "summary": "This paper systematically studies the effective of different model compression strategies on the reasoning capability of LRM, by applying reasoning-diagnosis metrics such as the steering vectors. From extensive empirical studies the authors propose certain properties and locate certain modules in LRM that are most critical to reasoning, providing insights into future designs on compression methods.\n\nAfter reading the paper, I am under the impression that the paper has meaningful motivation, supported by well-designed empirical studies with strong results, and the claims are well-presented. However, the paper \n\n1. Lacks methodological novelty and diversity in the analytical framework, using only one previously-proposed metrics. Some of the main claims of the paper, such as the localization of critical modules.\n2. Could be further supported by more rigorous controlled study and evaluation metrics. \n\nTherefore I recommend weak reject, but with room for further improvement after obtaining more insights from the authors during discussions.", "questions": "1. Could the authors add more discussions on recent works [1] on the principal weights of model weights that are related to reasoning performances. \n2. Since the choosing of tokens related to reasoning behaviors may be critical in the determination of reasoning-related modules, I wonder whether the authors have tried using other models (other than GPT-4o) to determine the tokens. In those cases, would the overall claims still hold? If so, it will strengthen the robustness of the findings.\n3. The authors proposes that certain layers/modules are more important in reasoning process, which relates to the discussion on the imbalanced quality among layers/modules. Do the authors believe that module-specific compression methods could potentially address the problem of over-compressing the important layers? There are recent works discussing module-wise compression [1, 2]\n4. For empirical evidence of layer importance (Takeaway 4.1 & 4.3, Fig. 2 & 3, etc.) it would be more intuitive if the authors could present the $I_{ml}^c$ metric for the base model (before distillation) to give the reader a better idea on how the importance of each module changes, rather than only presenting the $RI_{ml}^c$.\n\n[1] Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models, https://arxiv.org/abs/2410.10912\n\n[2] Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity, https://arxiv.org/abs/2310.05175\n\n[3] Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning, https://arxiv.org/abs/2506.00772", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of the paper is meaningful, as the authors focus on the effect of compression methods to (hard) reasoning tasks, which is both critical and not well-studied.\n2. From comprehensive experiments the authors provides valuable insights on the design of compression methods on LRMs, that is to pay attention to certain modules important to reasoning.", "weaknesses": "1. The paper makes, in my opinion, a quite strong claim on localizing the reasoning ability to certain model layers. The claim that later layer is more critical to reasoning/performances is intuitive, as they have greater impact on the final output, and could be sensitive to perturbations/compressions. However the authors do not go deeper into why certain deep layers are important to reasoning. \n2. For a systematic study, the authors should consider using a more diverse set of diagnosing tools, rather than only using one framework, as it would put the general applicability of the main claim under question.\n3. In the paper, the locating of tokens related to certain reasoning behaviors seems not rigorous enough, as the authors only use GPT-4o to identify related tokens, which itself may have certain biases.\n4. Some missing discussions on the layer/module-wise compression methods [1, 2], and identifying weights important to reasoning [3]\n\n[1] Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models, https://arxiv.org/abs/2410.10912\n\n[2] Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity, https://arxiv.org/abs/2310.05175\n\n[3] Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning, https://arxiv.org/abs/2506.00772", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LNtDIZ5ADQ", "reviewer": "Reviewer_ja42", "summary": "This paper investigates how compression methods affect large reasoning models (LRMs), specifically focusing on DeepSeek-R1 and its variants. The authors benchmark compressed models on four reasoning datasets and employ mechanistic interpretability techniques (difference of means and attribution patching) to identify which weights are most important for reasoning capabilities. Key findings include: (1) weight count impacts knowledge memorization more than reasoning, (2) the MLP up-projection in the final layer is critically important for distilled LRMs, and (3) current quantization methods overly compress final-layer modules and MLP gate projections.", "questions": "See weaknesses above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "**Comprehensive scope.** The paper systematically evaluates three major compression paradigms (quantization, distillation, and pruning) on LRMs, addressing a timely and important research gap.\n\n**Easy to read and well-structured.** The paper maintains a clear narrative flow with consistent notation, provides context for key design choices, and connects results to takeaways, which significantly improves readability.", "weaknesses": "**Limited model coverage.**\nThe analysis centers almost exclusively on DeepSeek-R1 and its distilled variants, which makes several findings read as DeepSeek-specific behaviors rather than properties of compression that generalize across LRMs. Validating the conclusions on additional open-sourced LRM families (e.g., QwQ variants) would strengthen the generalizability claims and help disentangle model-specific effects from compression-induced phenomena.\n\n**Imbalanced treatment across compression methods.**\nThe paper allocates uneven coverage across the three compression families. Distillation is represented by off-the-shelf distilled checkpoints for black-box open-source models, quantization is explored with four distinct methods, while pruning is evaluated only with SparseGPT. This asymmetry makes the comparisons look method-specific rather than family-level and may hurt perceived fairness.\n\n**Coarse knowledge vs. reasoning disentanglement.**\nThe paper infers that parameter count chiefly affects knowledge based largely on lower MuSiQue EM/F1 compared to other tasks. However, this single contrast is not sufficient to attribute effects to knowledge retention versus reasoning capability. MuSiQue itself blends multi-hop reasoning with retrieval-like knowledge and is sensitive to prompt/context choices. As a result, the “parameter count affects knowledge” conclusion feels somewhat overstated. More fine-grained experiments would strengthen the claim, e.g., RAG vs. closed-book ablations across model sizes or other synthetic tasks that decouple reasoning from memorized facts.\n\n**Minor presentation issues.** There’s a typo (“imrpving”) in Section 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aam2QtVjtN", "reviewer": "Reviewer_G6hk", "summary": "This paper investigates how compression methods including quantization, distillation, and pruning, affect the reasoning abilities of large reasoning models (LRMs).", "questions": "(1) Can the findings in this paper also hold for other reasoning models such as GPT-OSS-20B/120B?\n\n(2) Why quantizing the 1_up component causes the largest performance drop on AIME 2024 but still keeps relatively good performence on other tasks.  \n\n(3) Does the identified important component also depend on the task type/task difficulties?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The paper is clearly written, with a well-structured presentation that is easy to follow.\n\n(2) The motivation is articulated in a clear and convincing manner.\n\n(3) The study provides a comprehensive analysis and offers valuable new insights.", "weaknesses": "(1) The chosen reasoning models are all based on DeepSeek-R1 or its distilled variants, which may constrain the generality of the claims. It is unclear whether the findings would also hold for other reasoning models such as GPT-OSS-20B or GPT-OSS-120B.\n\n(2) The proposed layer-importance locating method is not entirely convincing. For example, Figure 2 suggests that the first-layer weights are relatively unimportant, yet Table 3 shows that quantizing the 1_up component causes the largest performance drop on AIME 2024. This apparent contradiction raises concerns about the reliability of the identified importance scores.\n\n(3) Minor concerns: the tick labels in Figure 2 are not clearly visible, and there is a typo in line 483 (“imrpving” → “improving”).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)