Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compression
Authors: Minjun Kim
Jaehyeon Choi
Hyunwoo Yang
Jongjin Kim
Jinho Song
U Kang
Venue: ICLR 2026 Poster (2026)
Keywords: Model Compression
Joint Model Compression
Compression Order
Network Pruning
Network Quantization
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Neural networks compressed by multiple methods perform better when weaker perturbations are applied first and stronger ones later.

Abstract: What happens when multiple compression methods are combined—does the order in which they are applied matter?
Joint model compression has emerged as a powerful strategy to achieve higher efficiency by combining multiple methods such as pruning and quantization.
A central but underexplored factor in joint model compression is the compression order, or the sequence of different methods within the compression pipeline.
Most prior studies have either sidestepped the issue by assuming orthogonality between techniques, while a few have examined them only in highly constrained cases.
Consequently, the broader role of compression order in shaping model performance remains poorly understood.
In this paper, we address the overlooked problem of compression order and provide both theoretical and empirical analysis.
We formulate the problem of optimizing the compression order and introduce the Progressive Intensity Hypothesis, which states that weaker perturbations should precede stronger ones.
We provide theoretical guarantees showing that the relative benefit of one order increases with the underlying performance gap.
Extensive experiments on both language and vision models validate the hypothesis, and further show its generality to broader setups such as multi-stage compression and mixed-precision quantization.

Published as a conference paper at ICLR 2026
PRUNE-THEN-QUANTIE OR QUANTIZE-THEN-PRUNE UNDERSTANDING THE IMPACT OF COMPRESSION ORDER IN JOINT MODEL COMPRESSION
Minjun Kim, Jachyeon Choi, Hyunwoo Yang, Jongjin Kim, Jinho Song & U Kang" Seoul National University, Seoul, South Korea minjjun. kim, ukang) J @nuu. ac.kr
ABSTRACT
What happens when multiple compression methods are combined doee the or- der in which they are applied matter? Joint model compression has emerged as a powerful strategy to achieve higher efficiency by combining multiple methods such as pruning and quantization. A central but underexplored factor in joint model compression is the compression order, or the sequence different methods within the compression pipeline. Most prior studies have sidestepped the issue by assum- ing orthogonality between techniques, while a few have examined them only in highly constrained cases. Consequently, the broader role of compression order in shaping model performance remains poorly understood. In this paper, we address the overlooked problem of compression order and provide both theoretical and empirical analysis. We formulate the problem of optimizing the compression order and introduce the Progressive Intensity Hypothesis, which states that weaker pertur- bations should precede stronger ones. We provide theoretical guarantees showing that the relative benefit of one order increases with the underlying performance gap. Extensive experiments on both language and vision models validate the hypothesis, and further show its generality to broader setups such as multi-stage compression and mixed- mixed-precision quantization.
INTRODUCTION
When combining pruning and quantization, which order leads to better model performance? Although deep neural networks have achieved remarkable success across diverse domains, deploying them on
edge devices remains challenging due to limited computational resources. To bridge this gap, network
compression techniques (Deng et al.) 2020 Liang et al.) 2021 Zhu et al.) [2024] Kim et al.
2025a)
have been proposed, including pruning (Park
2024 Song al. 2024 Park al.
2025b).
quantization (Paoo et al. [2022] ( Ashkboos et al. 20224b Kim et al. 2025b). knowledge distillation (Kim)
al. 2021b Cho & Kang 2022 Jeon al. 2023), parameter sharing (Desai & Shrivastava 2024
Wang et al., 2025a) and low-rank approximation (Jang et al.) 2023 Li ct al., 2025) Wang et al. 2025b). Recent studies highlight that combining these compression methods known as joint model
Sompression-ahiieves achieves better trade-offs between compression ratio and model performance than
applying them separately (Hawks al. 2021 Wang et al.) 2022] (hinde 2024).
A critical yet underexplored issue in joint model compression is the compression order- the sequence which individual compression methods are applied to the target model. As most of these techniques are not simultaneously applicable and should be executed sequentially (Wang et al., 2020 Kuzmin ctt al. (023). identifying an optimal order can yield a "free lunch" by improving performance without any additional computation. Empirical findings (Huang et al.] 2019; Hu et al.] |021 Qu et al.)
2025) show that the performance of the compressed model is sensitive to the compression order, necessitating deeper understanding of when and why certain orders work better.
However, the role of compression order has been largely overlooked by prior studies (Kurtic et al.) 2022 Xiao et al.) 2023 Liu et al. (023). Most existing studies implicitly assume that compression order has no effect on the grounds of orthogonality, naively arguing that different techniques operate
"Corresoonding Author.

---

Published as a conference paper at ICLR 2026
The Progressive Intensity Hypothesis. Neural networks compressed by multiple methods perform better when weaker perturbations are applied first and stronger ones later.
FP16
Pruned
R INT4
Me(er 2)))) > (0 ()))
P ( Pruning
2(): Quantization
.MC O): Performance (T) &: Model
M((b)
Parformance
Stronger
Φ
P (())
@ o ()
Weaker
JM 2)())
(0 r)))
P
(())
(Φ)
(Φ))
(P o (+)
Compression Ratio
(a) Model compression techniques
(b) Joint model compression scenarios
(c) Model performance
Figure I: The Progressive Intensity Hypothesis: Given two compression techniques, we conjecture that compressed models perform better if the stronger method is applied after the weaker one. That said, the optimal order between pruning and quantization varies with their compression ratios.
independently (Kim et al. 202 la
Chitty- Venkata et al. 2023 Song et al. 2024 Motetti et al. 2024).
Only a few works have examined the problem, and most of them merely offer empirical evidence confined to specific settings (Wang et al., 2020; Wu al. 2023 Yu et al. 2023). A notable at- tempt (Harma et al. 2025) presents a theoretical framework, proving the non-ortinn orooooohhgonnlity of pruning and quantization, concluding that pruning followed by quantization is always preferable. However, the scope of the work remains narrow and less practical, focusing only on magnitude- and max-scaled quantization (see Appendix D.5). To date, no study has systematically investigated the tendencies of compression order in general settings, neither empirically nor theoretically.
pruning
In this paper, we demonstrate that applying more aggressive compression algorithms at later stages yields superior performance. We first formulate the problem of joint compression order optimization (see Section 3.1 and Problem[]]. and introduce the Progressive Intensity Hypothesis, which posits that ordering compression methods from weaker to stronger improves performance (see Hypothesis I Figure offers conceptual depiction of the proposed hypothesis. We validate our claim through both theoretical analysis and extensive experiments. Theoretically, we show that the advantage of the compression order grows monotonically with the performance gap between two methods under disjoint sensitivity (see Theorem T and Definition 5). In other cases, we define interference as additional error from mutual interaction and investigate its influence (see Definition 6). Experimen- tally, validate the hypothesis across both language and vision models, covering diverse model architectures, tasks, and compression scenarios (see Sections 5.2 and (.33..3. Our analysis also considers how factors such weight-update strategies and rotations affect the role of compression order (see Figures 4 and (5). Moreover, our results highlight that the hypothesis generalizes to broader paradigms, including multi-stage approaches and mixeddpprccsson quantization (see Section 5.4).
Our contributions are summarized as follows:
Formulation. We formally define the novel problem of optimizing the compression order in joint model compression (see Problem []. and propose the Progressive Intensity Hypothesis, suggesting that stronger perturbations should be applied later to achieve better performance (see Hypothesis D]. Theory. We provide theoretical analysis that quantifies the relationship between method in- teraction and order sensitivity. Specifically, we prove that the superiority of one ordering grows monotonically with the performance gap between the methods (see Theorem [].
Experiments. Extensive and consistent experimental results across various domains, models, and tasks support our hypothesis (see Figures 4 and 6). We further extend the problem to broader setups such multi-stage compression and mixed preediiiniiion quantization (see Figures and 10).
To the best of our knowledge, we are the first to both theoretically and experimentally analyze the impact of compression order in joint model compression under general and practical settings.
Reproducibility.A All our implementation and datasets are available at hittt:s//// snudata ab/PQ0P
htttpss//githubb htggggttubb... com/
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARIES AND RELATED WORKS
We briefly describe the preliminaries and related works on pruning, quantization, and joint model compression. The notations used throughout this paper are formally defined in Appendix A
Pruning and Quantization. Compression" (techniques aim to transform a pre- pre-trained model ø as a more efficient version o' while minimizing performance degradation (Xu & McAuley 2023 Dantas et al. 2024 Liu et al. 2025a). This process inevitably introduces an error term o(-), representing the deviation between outputs of ø³' and é, which typically increases with the compression ratio C.
We define the compression ratio C' as the memory usage of Φ divided by that of ø'. Among various compression techniques f(;; C), our work centers on two major forms: pruning and quantization.
Pruning P-) directly discards less important components of a model to achieve the desired com-
pression ratio while retaining its most critical parts (Nova al. 2023 Ashkboos al. 2024a Park
et al. 2024). Based on the level of granularity, pruning methods fall into three categories: structured pruning (Song et al. (024)) removes entire structural elements such as layers, filters, or attention heads, semi-structured pruning (Xu et al. (0024) enforces fixed sparsity patterns (e.g., 2:4 sparsity) across tensors, and unstructured pruning (Frantar & Alistarh| 2023) prunes weights in fully flexible manner. In the case of structured pruning at the layer level, the induced error &p (W,, s, X4) is -W,,, X, when pruning is applied to layer 4 with weight W; and activation X,, and otherwise. The model achieves a compression ratio Cp 1/(1 - p) by pruning fraction p of weights.
Quantization 2(-) reduces the bit precision used to represent weights and activations by encoding a high-bit network into a fower-bit format (Gholami et al.] (022). Common quantization techniques include uniform (Li al. 2021), non-uniform (Zhao & Yuan (2055), binary coding (Park et al. 2025a), and vector quantization (VQ) (Tseng al. 2024). Although some techniques such VQ focus only on weight quantization without compressing activations, our main scope is on compressing both for practical acceleration. A main challenge towards robust quantization the activation outliers (Xiao) et al.] 2023 Lee et al. (0044) but recent ootation-based methods (Lin et al. 2024 Liu et al. (0225b) error by quantization Q(-) for a layer 4 with weight W; and activation X, is computed as d (W, X,) = Q(W, ((X,) W, W,X,, with compression ratio Co = =Borig/ Bg depending on the original Borig and target Bg bit-widths.
have largely overcome it. The layer-
Joint Model Compression. Joint compression combines two or more compression methods, achiev- ing higher compression ratios while minimizing performance loss (Wang et al.) [2000; Wu al., 2023 Yu et al.) 2023 Harma et al. 2025). These methods fall into two categories: co-designed and
post-hoc frameworks. Although the former offers the benefit of integration- aware design, they tend to
be method-specific and less adaptable to alternative configurations (Qu et al. 2025).
In contrast, combining independently designed techniques allows for method-agnostic pipelines that adapt easily to diverse architectures. Several pruning works (Kurtic et al. [022]] Xiao et al.) [023) Song et al. 2024) empirically confirm that such combinations with quantization are both feasible and beneficial. As independently designed techniques are applied one after another, the order of compression plays a key role. However, the impact of compression order has not been adequately
examined in the current literature. We denote applying f-) before fa(-) fi + fz or (f2 o fi)()).
JOINT COMPRESSION ORDER OPTIMIZATION
3.1 PROBLEM DEFINITION
We are given a pre-trained model and multiple compression techniques, each associated with a specific compression rate. The goal is to find the optimal order in which to sequentially apply these methods. An order is considered optimal if it minimizes the degradation in model performance. We quantify performance using a metric M(-), where higher values indicate better outcomes (e.g.. classification accuracy or the negative of perplexity). We provide the formal definition as Problem []]
Problem (Joint Compression Order Optimization). We have a pre-trained model Φ. a set of compression methods F = { fa.
and performance metric M(-). For set II = { : FF + F| | π is bijective) of all permutations over F, the goal is to find the optimal permutation o E II that maximizes the performance of the compressed model: T² arg max M()(0)).
In the remainder of the paper, we use 'compression' to refer to 'mddel compression' for simplicity.
3

---

Published as a conference paper at ICLR 2026
3.2 CHARACTERIZING COMPRESSION ATTRIBUTES
Two key attributes arise when characterizing compression in a general setting: granularity and intensity. Granularity refers to the smallest structural unit on which compression is applied, and intensity refers to how aggressively the method alters the model, measured by its impact on performance.
Granularity of Compression. Compression methods are not applied to the model as a whole, but rather operate locally on its individual components. We define compression granularity as the atomic level which compression performed. To formalize this notion, we begin by abstracting the model into a set of component types, such as layers, sublayers, or attention heads. We refer to these as abstract rypes, which define the structural units over which compression may act. For two abstract types (n and l2, we say I is larger than l2 if L strictly contains l2 as a structural unit. Among all types that are larger than both L and l2, we define the least upper type hum (i,1,,/2 as the smallest one. For given model 6, let Ty denote the set of abstract types; this set depends on the model architecture.
Each compression method f(-) may be applicable only to a subset of abstract types. We denote this subset by Ty S Tg. representing the structural levels at which f()) can operate. For instance, layer-wise pruning in large language models is applicable only to units coarser than layers. Then, the granularity of f(:) is the smallest unit t, E Ty on which f()) is applicable, defined in Definition Definition (Compression Granularity). For a model 6 with a set To of abstract types and compres- sion method f(), the compression granularity ty arg min |t), where Ty C To denotes the set of abstract types on which f()) operates, and [t] denotes the structural size of type t.
Intensity of Compression. Compression methods affect the model differently even at identical com- pression ratios, so comparing their intensities directly is challenging. To assess compression strength, we introduce three concepts grounded in performance degradation: performance gap ((.. compression equivalent ratio Cj., and compression order advantage A(fi + f2).
Performance differences between two methods fi Ci) and /2(:: C2), each applied at its respective compression ratios C1 and C2. provide a direct measure of their relative intensity. We call this the performance gap G(ø, M; fi(:; C1), f2(:; C2)), or simply G(fi, f2), as defined in Definition 2 If ((f,, f2) > 0, we refer to f2(:: C2) as the stronger compression and fi C1) as the weaker one. Definition 2 (Performance Gap). Given a model o, performance metric M(-), and two compression methods (:: and f2(;;C2). the performance gap between two methods (o, fi (1;; C2)) = M(f Ct)) - -M(f((0;a))
Although g()) offers a clear pairwise comparison, its values in metric units are difficult to interpret and may grow rapidly as the compression ratio increases. Alternatively, mapping methods onto a common scale allows for direct comparison at the level of compression ratios. While multiple choices exist for the baseline method, we select quantization as it exhibits the best performance across diverse models, thereby offering the widest range. Accordingly we define the Compression Equivalent Ratio (CER) C* (f)(),, Q, C), simply C7,,- which expresses the effect of method (-: C') ratio C' as an equivalent ratio of quantization 2(-), as Definition 3 In other words, starting from a 16- bit model, a compression method f(Ø;; C') with C7 = 2 achieves the same performance as 8-bit quantization. Note that CER of quantization Q(-) is naturally equal to its own compression ratio (i.e., We adopt straightforward approach by computing CER through linear interpolation. For instance, f() achieving M(f C) = 65% accuracy maps to Cj 3 when quantization Q(-) yields M(Q; Co 2) 70% and MIQ Co = 4) 60% accuracy, respectively.
Co == =Q).
Definition 3 (Compression Equivalent Ratio). Given a model Φ, performance metric M()), compression method f()), quantization method Q--), and a compression ratio C, the compression equivalent ratio C' (f(), 2, == C" such that M(Q(; (():C")) = =M{((0: C)).
Until now discussion is limited to single methods; but when multiple methods are applied, how should intensity be defined? Our scope centers on measuring how intensity changes by compres- sion order. Accordingly, we capture the gain from applying fi(-) before fz(.) over the reverse as compression order advantage A(o, M: f2(.)), or simply A(Ji + f2), as Definition 4 Definition& 4 (Compression Order Advantage). Given a model é. a performance metric M(-), and two compression methods fi(:;C,) and fe(:;2), the compression order advantage (((C)) f2(: = G(1 f2 + fi) = (i)()) o f2)()))
4

---

Published as a conference paper at ICLR 2026
3.3 THE PROGRESSIVE INTENSITY HYPOTHESIS
Our goal is to uncover general patterns in how compression order affects the model performance in joint compression scenarios. While prior works have focused primarily on isolated settings, we seek to establish a broadly applicable principle. To this end, we propose the Progressive Intensity Hypothesis, which posits that applying stronger compression methods at later stages generally yields better performance. We formalize this hypothesis for pair of methods in Hypothesis ! which serves as the main focus of our analysis; its extension to multiple methods is presented in Appendix [B.3
Hypothesis 1 (The Progressive Intensity Hypothesis). Let fi(:; and fz(:; C2) be two compres- sion methods applied to a model &. Then, the compression order advantage A(fi + f2) grows monotonically with the performance gap ((f1, or equivalently with the CER difference C¹22 -5-h--
As an example, if methods f.()) and fz(.) yields M(fi; C1) = 75% and M[f2; C2) = 70% accuracy, respectively (i.e.,, G(Si, f2) = 5%p),, the compression order advantage A(fi + f2) mild; replacing C2 into C22 at M(f2; C%2 = 60% accuracy (i.e.. G(f,, fa) = 15%p) results in a larger advantage.
THEORETICAL ANALYSIS
We theoretically analyze how compression order affects the model performance. We introduce disjoint selectivity to isolate order-dependent units, and prove in Theorem [ that only these units determine the performance gap. We then show in Theorem 2 that Hypothesis 1 holds due the reduction of order-dependent units. We later extend to non-disjoint cases in which interference occurs. Consistent with earlier works (Harma al. 2025), we investigate each unit, relying on Assumption []
Assumption 1. Given a model & with a set L of layers. performance metric M(.), a compression method f(), and the layer-wise reconstruction loss (14),, assume the followings:
Layer-wise independence. The reconstruction error at one layer does not affect the reconstruction error another: Vl,,y E L, i i j: ( (1,) 0.
Frror-performance trade-off. Model performance is inversely related to total reconstruction error: 33 0, M(Ø) - M(f(o) = 3 Li,e,, (|,, (1.)l..
Disjoint Selectivity. Sequential application of two compression methods leads to two distinct scenar- ios: either there exist units altered by both methods, all units exclusively assigned to one. We define the latter scenario as the case where disjoint selectivity holds, as in Definition 55, This means that while the assignment may vary with order, each unit is ultimately handled by only one method. Definition (Disjoint Selectivity). Given a model o, two compression methods fi(-) and fz(-) with respective granularities t. and f.g. disjoint selectivity holds if Vu, E U(Φ; (t f)))), Vx (x)) (π) = 1, where U(o: t) is the set of all units of model & at granularity
{ff o f2 (f t, and (D((())) (T) denotes whether f-) modifies unit u under the order n (i.e., 1 if modified, 0 otherwise).
Under disjoint selectivity, the compression order advantage A(fi + f2) is proportional to the cumulative sum of error difference g(-) across units assigned differently depending on the order as formulated in Theorem T The underlying intuition is that the performance gap rises solely from units whose assignment varies with the order; for others, the error remains invariant and thus cancels out. To illustrate, consider units u1 u2 and U3 and compression methods (-) and fz()). If un is always handled by (-) regardless of the order, while u2 and U3 are assigned differently depending on the order, then the advantage A(f1 + f2) is proportional to error difference of units U2 and ug.
Theorem 1 (Compression Order Advantage under Disjoint Selectivity). Suppose we compress model é with two compression methods f2(-)) and fz(-) with respective granularities tt, and t/2 where disjoint selectivity holds. Then, under Assumption the compression order advantage
A(f1 + fa) =
(M(6)))((((i)) o fo)( equals to B. (Eu,e,, g(u,) - -E,,c, s(u))..
where 3 is the coefficient between the model performance and total error induced from Assumption 7 I|
g(u:) = | (u.) IF - ||6 (us)]l² is the error gap according to the method applied, and G {u DD (f2 fi) = 1, DD (fi f2) 0} and G2 = {u] DJ (f2 fi) 0. Di (fi f2) = 1} are
groups of onder-dependent units.
Proof. Refer Appendix B.1
5

---

Published as a conference paper at ICLR 2026
2
Interference
A(P 2)
P&Q
(Φ) (Q (b)
P (())
(Q - P))
(a) Pruning gramularity 2 Quantization gramularity
(b) Pruning granularity < Quantization granularity
Figure 2: A case study of pruning P(.) and quantization 2(-) on model @. (a) if pruning granularity (green) is coarser or equal to quantization granularity (orange), disjoint selectivity holds. (b) Other- wise, partial removal of quantization units by pruning introduces extra error, termed interference A.
Monotonicity, Under disjoint selectivity, we show that Hypothesis] I hholds when the two compression methods are well-designed-thee that is, minimally disruptive to the model. We examine this through case study on pruning and quantization. We assume a favorable scenario where pruning is configured to induce minimal degradation, and quantization introduces symmetric, zero-mean errors centered at the original values. These assumptions are formalized in Assumption 2
Assumption 2. Given a model & with a set L. of layers and performance metric M(.), assume that:
Well-designed pruning P(.). The pruming method is chosen from the set of pruning strategies that aim to preserve the model performance: P(-) E PP(Pp) where P(Cp)) = (P:()) Cp, - (M(:(()) S 8) denotes the set of pruning strategies that satisfy the target ratio Cp while keeping performance degradation within small budget 5.
Well-designed quantization 2()). For all layers, quantized outputs follow a symmetric distribution around the original values: v4, L, ((W:))(()))))))))))) ~ X,, where N() is the Gaus- sian distribution. The quantization error is negligible (i.e., ((W.)
W.X; < W.X.).
Theorem 2 states that when disjoint selectivity holds and the compression methods well-designed, A + increases monotonically with CER difference Cip Co for fixed Cp. Note that as Co decreases (i.e.. M(Q(e) increases), both CER difference Cp Co and performance gap ( P) increase monotonically, We show that ((Q + P) increases in this setting because the gap depends solely order-dependent units under Theorem |
We discuss the impact of Cp in Appendix (.6 Theorem (Monotonicity). Suppose we compress a model d with pruning P(.)) and quantization 2(), where disjoint selectivity holds. Then, under Assumptions 7 Mand? 2 given performance metric M(-) and two pairs of compression ratios (C,,, Co.) and (Cp. Co.). if CER difference increases C22- -Ce.1 Cip-- Co. >
then, the compression order advantage increases monotonically:
A(o, M; Q(= Co.) Pp):Cr.)) 2 A(o, M; M: 2(:Co.) + P(:CP.))
Proof. Refer to Appendix B.2
Granularity and Interference. Disjoint selectivity does not always hold in practical joint compres- sion settings for pruning and quantization. As pruning operates by fully discarding or keeping each unit, it always satisfies disjoint selectivity. In contrast, quantization satisfies this condition only when its granularity is finer than or equal to that of pruning. Figure 2 illustrates this: (a) if (p 2 lg. disjoint selectivity is preserved as pruning removes entire quantization units. However, (b) if (p < lg. pruning may partially eliminate quantization unit, introducing regions where both methods interfere.
In general joint compression of two methods fi(.) and fz(-). this violation of disjoint selectivity introduces additional error, which we define interference A(o: fi + f2), or simply A(fi + fa) in Definition 6 Intuitively, interference quantifies how one method disturbs the behavior of the other.
Definition 6 (Interference). Given model & and two methods f1(.) and fa(-), the interference
A / + f2) : (6 (u) - 852 (u)), where X = U(Ú (2) n [ fi) = 1};
uEX
set U(d;t) contains all units of model & at type t, D& indicates whether unit is modified by f-) under order (1 if modified, 0 otherwise), and (66)(1) denotes the error on u after applying f (-).
BB definition, quantization serves as the baseline scale, its CER equals its own compression ratio (i.c.,
C² = Ca). Therefore, Cp Ce represents the CER difference between pruning and quantization.

---

Published as a conference paper at ICLR 2026
m 0.05
0.1
p
0.15
0.2
0.25
0.3
0.0
0.06
0.1
A12 -
AlQ- 100-7)) P)
Ale -1)
0.4
-0.05
-0.1
0.6
-0.10
-0.2
-0.8
0.15
-0.3
2
Cp Co
-55-6p C2
C3 Co
(a) LLaMA 2 7B
(b) LLaMA 13B
(c) LLaMA 8B
Figure 3: Across diverse language models, the compression order advantage A(Q + P) increases monotonically with the CER difference Cg Co. See Section 5.2 for details.
Interference may or may not occur, depending on its applied techniques. A notable example is mixed-precision quantization, where treating each bit-width quantization as distinct method satisfies disjoint selectivity, thereby avoiding interference. In our primary focus of pruning and quantization, the magnitude of interference is determined by the pruning ratio p. When interference occurs, quantization operates on weights altered by pruning, thus the deviation from the original distribution scales with the pruning ratio. As pruning ratio p increases, larger portion of weights is removed before quantization, leading to stronger interference. This additional error depends only on pruning and enters additively into the order advantage, and thus remains independent of quantization strength while preserving the monotonic trend. Consequently, while exact outcomes may vary, A(fi f2) remains a monotonic function of C. Co even under interference. In conclusion, Hypothesis holds under both disjoint and interfering scenarios, highlighting its general validity.
EXPERIMENTAL FINDINGS
We empirically validate our hypothesis in joint compression scenarios, starting with pruning and quantization on language and vision models. We then extend to general pipelines beyond them.
5.1 EXPERIMENTAL SETUP
We briefly introduce the experimental setup. Further setups are detailed in Appendix C
Setup. For language models, focus decoder-only LLMs, mainly LLaMA Touvron al. 2023 models. The main metric is the negative of perplexity on WikiText-2 (Merity et al. 2017) dataset; results on commonsense reasoning tasks appear in Appendix D.3 For vision models, evaluate the classification accuracy of ResNet-18 8 (He et al. 2016) (CNNs) and DeiT-Base (Touvron et al. 2021
(ViTs) models on ImageNet (Deng et al.) 2009) dataset.
Baselines. We evaluate three pruning (SparseGPT (Frantar & Alistarh 2023 Wanda (Sun et al. 2024), and SLEB (Song et al., (0244) and four quantization methods (RTN (Gupta et al. [0155] OPTQ (Frantar et al.,) 2023), QuaRot (Ashkboos et al.) 2024b), and QuaRot + OPTQ) for language models, For vision models, we apply PRACTISE (Wang Wul 2023) and N2UQ (Liu al. 2022)
for CNNs, and adopt SAVIT (Chuanyang et al. 2022) and RepQ- DiT (Li et al. 2023) for ViTs.
5.2 ANALYSIS ON LANGUAGE MODELS
We verify whether Hypothesis) holds for language models. Then, we analyze the effect of weight updates and rotations in quantization, and investigate the impact of pruning granularity on interference.
Compression Order Advantage by CER Differences. We analyze how the compression order ad- vantage A( - P) varies with CERs for SparseGPT (P(-) and QuaRot (Q(+) across three models: LLaMA 7B, 13B, and LLaMA 3 8B. Figure 3 confirms that A(Q - P) increases monotonically with the CER difference for all three models. Each point in the figure corresponds to compression
7

---

Published as a conference paper at ICLR 2026
p =
P
p =
p=
RTN
QuaRot
OPTQ
QuaRot OPTQ
0.1
SparseCPP
4(2- P)
00
-81
6.1
-92
-02
-43
-03
-1.0
0.5
-10
000
0111
Wanda
A12 - P)
-9.5
0.3
0.3
-100
130
16
033
SLBB
AIQ - 17)
-32
-3
-12
110
0.5
12
Cp-Ca
Co
Figure 4: Compression order advantage A(Q - P) against CER difference Cp Co for three prun-
ing (-)) and four quantization Q(.) methods LLaMA 3 8B model. Our hypothesis consistently holds for language models regardless of pruning granularity, rotation, and weight updates.
ratio pair (Cp, Co), defined by pruning ratio p E [0.05, 0.15, 0.2, 0.25, 0.3) and quantization bit- width Bo E [4. 5, 6, 7, 8). We fit an exponential curve per pruning ratio p. reflecting the underlying trend. In this setting, the x-axis C. Co captures the difference between the intrinsic intensities of pruning and quantization, with both Cp and Co calibrated on the original model Φb Consistent trends across diverse language model architectures and scales supports the validity of Hypothesis see Appendices D.2 and D.I for results on encoder-base and other decoder-only models, respectively.
Finding 1. The Progressive Intensity Hypothesis holds across diverse language models of varying scales and architectures, showing that stronger compression should be applied later.
Weight Updates and Rotation-based Transformations. We investigate the hypothesis under prac- tical techniques such as weight-updates and rotations. Figure shows that Hypothesis [ consistenty holds across diverse combinations of methods. Our framework is agnostic to the type of compression methods; weight updates and rotations reduce quantization error, thereby increasing Cp-
The hypothesis generalizes beyond specific design choices of pruning and quantization, remaining robust under weight-update and rotation methods.
Finding 2.
An intriguing phenomenon arises in pruning rotation- based methods: apply- ing pruning after rotation results in drastic performance drop compared to pruning without rotation. Figure 5 illustrates the perplexity changes of LLaMA 3 model pruned by SparseGPT, depending on the Hadamard rotation from QuaRot. Increasing the pruning ratio amplifies the discrep- ancy between rotated and non-rotated settings, as pruning is applied without accounting for rotation. This effect emerges because rotation may intro- duce two types of errors: matrix-wise errors from residual components and element-wist wise errors from altered pruning decisions. We further discuss the details in Appendix [D44 and Table As rotation may intensify pruning, it essential to design pruning approaches compatible with rotation-based quantization, the emerging de facto standard.
10²
No rotation Rotation
Marpeeiiy
10'
0.0
0 8
1.6 Compression Ratio
1.2 1.4
Figure Rotation im- pact on pruning. See Section 5.2 for details.
Finding 3. Rotation amplifies pruning effects, underscoring the necessity of designing rotation- aware pruning methods.
8

---

Published as a conference paper at ICLR 2026
0.5
0.5
0.3
O
8g =1
0g =
A(Q P))
A p2)
9 =9
8 =6
0.2
B
p=9.55
0.0.0
p=0.7
-1.0
0.23
0-0.75
0.1
p-0.0
0-0.0
-
32
33
35
36
C7
Co
Cp
Cq
0.10
0.15
0.20 0.25
0.30
0.35
Ci - Ca.
(a) ResNet-18 (CNN)
(h) DeiT-Base (ViTs)
Figure 6: The Progressive Intensity Hypothesis holds across diverse vision models. See Section 5.3 for details.
Figure 7: Mtlti-stage compression
results on LLaMA 8B model.
Pruning Granularity and Interference. We verify the pres- ence of interference by comparing results across different prun- ing granularities. Table reports A(Q + P) across differ- quantization bit-width Be when applying two 5% pruning methods with QuaRot. For SLEB, which applies structured pruning at sublayer level, there exists regime where no lay- ers differ in their pruning status across orders, leading to an exact A(Q P) of zero (i.e., no interference). By contrast,
Table I: A(Q + P) by quantiza- tion ratio Co.
Co (Ba)
SparseGPT
SLEB
1.78 (9)
0.002
0
2.00
0.001
0
2.28 (7)
-0.003
SparseGPT, as an unstructured pruning method, exhibits inter-
2.68 (6)
-0,013
0
ference in low Co ranges. Notably, empirical results suggest that interference also exhibits a monotonic trend regarding Co-
3.20
-0.017
-0.057
44)
-49.899
-9.379
Finding 4. Pruning granularity determines interference: structured pruning shows no interference in early regimes, while unstructured pruning exhibits monotonic interference.
5.3 ANALYSIS ON VISION MODELS
CNN and VIT Models. We verify whether Hypothesis hollds for vision models, focusing on CNNs and ViTs. In Figure 6 we analyze the behavior of ResNet- 88 and DeiT-Base models under PRACTISE (P(+) and N2UQ (Q()), and SAVIT (P(-)) and RepQ- ViT (Q(-) methods, respectively. The results confirm that both A(2 P) and Cjj- Co increase monotonically for both models, regardless of the pruning or quantization configurations, Notably, the compression order advantage is substantially larger than that observed in language models, where it is often marginally positive.
Finding 5. Vision models consistently satisfy the Progressive Intensity Hypothesis regardless of architecture and applied compression techniques.
5.4 BEYOND PRUNING AND QUANTIZATION: TOWARD GENERAL. PIPELINES
We extend the Progressive Intensity Hypothesis to general compression pipelines. These results align with the n--ethod ordering formulation in Appendix B.3
Multi-stage Compression. Pruning is generally performed in multiple stages to mitigate performance degradation. In Figure 7 we investigate the impact of compression order by alternately applying SparseGPT (P(-) and QuaRot (Q(-)) to the LLaMA 3 8B model where the sum of pruning ratios Pi + P2 0.3 gc. P(:Cp, + 2(-) P(: Cp,)). Our results consistently demonstrate positive advantages, indicating that stronger pruning placed later improves performance under fixed quantization, confirming that our hypothesis holds only for two stages but also for multiple ones.
Finding 6. Beyond pairwise orders, the hypothesis holds practical multi-stage compression, indicating that scheduling stronger compression later in the sequence yields higher accuracy.

---

Published as a conference paper at ICLR 2026
p =
P
p =
p=
0.00
Alprog. regr.
0.2
0.0
-0.02
A(S +))
0.2
A(Q
0.04
-0.4
-10
0.6
0.06
0.8
15
-0.08
2
1.8
-1.4
5
7
C
Co
CΦ
C5
Compression Ratio
Figure 8: Joint compression re- sults with LoRA adapters.
Figure 9: Performance under pruning and parameter sharing.
Figure 10: The impact of com- pression order in MPQ.
Parameter Efficient Fine-tuning, Parameter Efficient Fine-Tuning (PEFT), which introduces lightweight low-rank adapters to mitigate compression-induced performance degradation, has recently become a widely adopted practical approach. We investigate whether the proposed principle remains valid in scenarios where PEFT applied alongside pruning and quantization. Figure g confirms that the same pattern holds for LLaMA 3 8B when combined with SparseGPT (P(-),, RTN (Q(-), and LoRA (Hu al. 2022) (PEFT). Applying LoRA after quantization produced similar corrective effect to rotation, effectively compensating quantization-induee performance loss. Overall, the progressive intensity hypothesis remains robust under practical training pipelines that include PEFT.
Finding 7. PEFT preserves the hypothesis that stronger compression should be applied later, as post-quantization LoRA effectively restores accuracy and maintains the expected ordering.
Parameter Sharing, Beyond pruning and quantization, parameter sharing S(.) is an independent compression technique that ties multiple layers into unified set of weight parameters. We conduct joint compression experiments with pruning and parameter sharing to assess whether the principle generalizes beyond the quanttiztianniiiiinnnnn setting. In Figure results on LLaMA 2 7B model with
Basis Sharing (Wang et al. 2025a (S()) and magnitude- based pruning (Han et al. confirm that the same ordering effect emerges well.
2015) (P(-))
Finding 8. Joint compression with parameter sharing also follows the hypothesis: placing the stronger operation later yields better performance.
Mixed-precision Quantization. As discussed Section 4 Mixed Precision Quantization (MPQ) be formulated joint compression problem where each bit-widhh allocation acts separate compression method, satisfying disjoint selectivity. Figure T0 illlustrrs the effect of compression order in MPQ, where sequentially allocate quantization bit-widths following HAWQ-V2 (Dong) ct al. 2020) on ResNet-18 model under fixed average bit-width (i.e., identical overall compression ratio), comparing progressive (prog.: : 8-2 bits) and regressive (regr: 2-+8 bits) sequential allocations. As the total compression ratio increases, progressive allocation increasingly outperforms regressive allocation in terms of A Aprog. -> regr.). supporting our hypothesis under MPQ settings.
Finding The Progressive Intensity Hypothesis also holds in MPQ. with progressive bit allocation outperforming regressive allocation since lower-bit quantization acts stronger compression.
CONCLUSION
We address the under-explored problem of joint compression order optimization and provide both theoretical and experimental evidences. The Progressive Intensity Hypothesis (Hypothesis[] offers simple yet powerful rule: weaker perturbations first, stronger ones later. Future works include investi- gating interference in more complex pipelines, providing explicit predictive rules, and automating compression order selection (see Appendix D.7..
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No.RS-2020-11200894, Flexible and Efficient Model Compression Method for Various Applications and Environments], INo.RS-2021: 11211343, Artificial Intelligence Graduate School Program (Seoul National University)], [No.RS- 2024-00509257. Global AI Frontier Lab], and NN..RS-0025-24442
AI star Fellowship Support Program(Seoul National University)]. This work was supported by Youlchon Foundation. The Institute of Engineering Research at Scoul National University provided research facilities for this work. The ICT at Seoul National University provided research facilities for this study. U Kang the corresponding author.
REFERENCES
Salch Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. ICLR, 2024a.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. In NeurlPS, 2024b.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. AAAI, 2020.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez- Gazpio, and Lucia Specia. Semeval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Eleventh International Workshop on Semantic Evaluation, 2017.
Krishna Teja Chitty-Venkata, Venkata, Sparsh Mittal, Murali Emani, Venkatram Vishwanath, and Arun Somani. A survey of techniques for optimizing transformer inference. Journal of Systems Architecture, 144:102990, 2023.
Ikhyun Cho and Kang. Pea-kd: Parameter-efficien and accurate knowledge distillation on bert. PLOS ONE, 17(2):1-12, 02 2022.
Zheng Chuanyang, Zheyang Li, Kai Zhang, Zhi Yang. Wenming Tan, Jun Xiao, Ye Ren, and Shiliang Pu. SAVit: Structure-awan vision transformer pruning via collaborative optimization. In NeurIPS, 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 18X3:v8800005557 2018.
Pierre Vilar Dantas, Waldir Sabino da Silva Jr, Lucas Carvalho Cordeiro, and Celso Barbosa Car- valho. A comprehensive review of model compression techniques in machine learning. Applied Intelligence, 54(22): 1804-1804111111 100---18660 11844, 2024.
J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Aditya Desai and Anshumali Shrivastava. In defense parameter sharing for model-compression. In ICLR, 2024.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace- weighte quantization of neural networks. In NeurIPS, 2020.
11

---

Published as a conference paper at ICLR 2026
Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling? In ICLR, 2025.
Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In ICML, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoetler, and Dan Alistarh. OPTQ: accurate quantization for generative pre-traine rre-trained transformers. In ICLR, 2023.
Amir Gholami, Schoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer, A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pp. 291-326. Chapman and Hall/CRC, 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In ICML, 2015.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurIPS, 2015.
Simla Burcu Harma, Ayan Chakraborty, Elizaveta Kostenok, Danila Mishin, Dongho Ha, Babak Falsafi, Martin Jaggi, Ming Liu, Yunho Oh, Suvinay Subramanian, and Amir Yazdanbakhsh. Effective interplay between sparsity and quantization: From theory to practice. In ICLR, 2025.
Benjamin Hawks, Javier Duarte, Nicholas J Fraser, Alessandro Pappalardo, Nhan Tran, and Yaman Umuroglu. Ps and qs: Quantization aware pruning for efficient low latency neural network inference. Frontiers in Artificial Intelligence, 4:676564, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuazzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. ICLR, 2022.
Peng Hu, Xi Peng, Hongyuan Zhu, Mohamed M Sabry Aly, and Jie Lin. Opq: Compressing deep neural networks with one-shot pruning-
pruning-quantizaann quantization. In AAAI, 2021.
Sitao Huang, Carl Pearson, Rakesh Nagi. Jinjun Xiong, Deming Chen, and Wen-mei Hwu. Acceler- ating sparse deep neural networks fpgas. In HPEC, 2019.
Jun-Gi Jang, Chun Quan, Hyun Dong Lee, and U Kang. Falcon: lightweight and accurate convolution based on depthwise separable convolution. Knowl. Inf. Syst., 65(5):2225-2249, 2023.
Hyojin Jeon, Seungcheol Park, Jin-Gee Kim, and U Kang. Pet: Parameter-efficient knowledge distillation on transformer. PLOS ONE, 18(7):1-21, 07 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, al. Mistral 7b. arXiv preprint 1xXi:23110006255. 2023.
Jangho Kim, Simyung Chang, and Nojun Kwak. Pqk: Model compression via pruning, quantization, and knowledge distillation. In Interspeech, 2021a.
Junghun Kim, Jinhong Jung, and U. Kang. Compressing deep graph convolution network with
multi-staged knowledge distillation. PLOS ONE, 16(8):1-18, 08 2021b.
Minjun Kim, Jaehyeon Choi, Jongkeun Lee, Wonjin Cho, and U Kang. Zero-shot quantization: A comprehensive survey. IJCAI, 2025a.
Minjun Kim, Jongjin Kim, and U Kang. SynQ: Accurate zero-shot quantization by synthesis-awan aware fine-tuning. In ICLR, 2025b.
12

---

Published as a conference paper at ICLR 2026
Minjun Kim, Jaeri Lee, Jongjin Kim, Jeongin Yun, Yongmo Kwon, and U Kang. Lampq: Towards accurate layer-wise mixed precision quantization for vision transformers. In AAAI, 2026.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal BERT surgeon: Scalable and accurate second-order pruning 2022.
for large language models. In EMNLP, 2022.
Andrey Kuzmin, Markus Nagel, Mart Van Baalen, Arash Behboodi, and Tijmen Blankevoort. Pruning vs quantization: Which is better? In NeurlPS, 2023.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models. In AAAI, 2024.
Muyang Li. Yujun Lin, Zhekai Zhang, Tianle Cai, Junxian Guo, Xiuyu Li, Enze Xie, Chenlin Meng, Jun- Yan Zhu, and Song Han. SVDQuant: Absorbing outliers by low-rank component for 4-bit diffusion models. In ICLR, 2025.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit post-training quantization by block reconstruction. In ICLR, 2021.
Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. Repq-vit: Scale reparameterization for post-training quantization of vision transformers. In ICCV, 2023.
Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370-403, 2021.
Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized llms. In NeurlPS, 2024.
Defu Liu, Yixiao Zhu, Zhe Liu, Yi Liu, Changlin Han, Jinkai Tian, Ruihao Li, and Wei Yi. A survey of model compression techniques: Past, present, and future. Frontiers in Robotics and AI, 12: 1518965, 2025a.
Zechun Liu, Kwang-Ting Cheng, Dong Huang, Eric P Xing, and Zhiqiang Shen. Nonuniform-to uniform quantization: Towards accurate quantization via generalized straight-through estimation. In CVPR, 2022.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantization with learned rotations. In ICIR, 2025b.
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient Ilms inference time. In /CML, 2023.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: State-of-the-art parameter-efficient fine-tuning methods. htttps://ittht Com/ haam//haggingffaccceees 2022.
Clara Meister and Ryan Cotterell. Language model evaluation beyond perplexity. In ACI,, 2021.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017.
Mistral AI Team. Mistral NeMo: our best small model. htttp::/mistral...// mistrral-nemo July 2024. Accessed: 2025.
Beatrice Alessandra Motetti, Matteo Risso, Alessio Burrello, Enrico Macii, Massimo Poncino, and Daniele Jahier Pagliari. Joint Pruning and Channel-Wise Deep Neural Networks. IEEE Transactions on Computers, 73(11):28) (1111)::6192666 2024.
Quantization for Efficient
Azade Nova, Hanjun Dai, and Dale Schuurmans, Gradient-free structured pruning with unlabeled data. In ICMI., 2023.
13

---

Published as a conference paper at ICLR 2026
D Paperno, G Kruszewski, A Lazaridou, QN Pham, Raffaella Bernardi, S Pezzelle, M Baroni, G Boleda, and Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In ACL, 2016.
Seungcheol Park, Hojun Choi, and U Kang. Accurate retraining-free pruning for pretrained encoder- based language models. In ICLR, 2024.
Seungcheol Park, Jeongin Bae, Beomseok Kwon, Minjun Kim, Byeongwook Kim, Se Jung Kwon, U Kang, and Dongsoo Lee. Unifying uniform and binary-coding quantization for accurate compression of large language models. In ACL, 2025a.
Seungcheol Park, Sojin Lee, Jongjin Kim, Jinsik Lee, Hyunjik Jo, and U Kang. Accurate sublayer pruning for large language models by exploiting latency and tunability information. In IJCAI, 2025b.
Tairen Piao, Ikhyun Cho, and U Kang. Sensimix: Sensitivity-aware 8-bit index & 1-bit value mixed precision quantization for bert compression. PLOS ONE, 17(4):1-22, 2022.
Xiaoyi Qu, David Aponte, Colby Banbury, Daniel P Robinson, Tianyu Ding, Kazuhito Koishida, Ilya Zharkov, and Tianyi Chen. Automatic joint structured pruning and quantization for efficient neural network training and compression. In CVPR, 2025.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications the ACM, 64(9):99-106, 2021.
Tushar Shinde. Adaptive quantization and pruning of deep neural networks via layer importance estimation. In Workshop on Machine Learning and Compression, NeurIPS 2024, 2024.
Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. In ICML, 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In ICLR, 2024.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better IIm quantization with hadamard incoherence and lattice codebooks. In ICMI., 2024.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
Guo-Hua Wang and Jianxin Wu. Practical network acceleration with tiny sets. In CVPR, 2023.
Jingcun Wang, Yu-Guang Chen, Ing-Chao Lin, Bing Li, and Grace Li Zhang. Basis sharing: Cross- layer parameter sharing for large language model compression. In ICLR, 2025a.
Naigang Wang, Chi-Chun Charlie Liu, Swagath Venkataramani, Sanchari Sen, Chia-Yi Chen, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Leland Chang. Deep compression of pre-trained transformer models. In NeurlPS, 2022.
Tianzhe Wang. Kuan Wang. Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han. Apq: Joint search for network architecture, pruning and quantization policy. In CVPR, 2020.
14

---

Published as a conference paper at ICLR 2026
Xin Wang. Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation-awwe aware singular value decomposition for large language model compression. In JCLR, 2025b.
Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, and Yuxiong He. Understanding int4 quantization for language models: latency speedup, composability, and failure cases. In ICML, 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. ICML, 2023.
Canwen Xu and Julian McAuley. A survey on model compression and acceleration for pretrained language models. In AAAI, 2023.
Kaixin Xu, Zhe Wang, Chunyun Chen, Xue Geng, Jie Lin, Xulei Yang, Min Wu, Xiaoli Li, and Weisi Lin. Lpvit: Low-power semi-structured pruning for vision transformers. In ECCV, 2024.
Chong Yu, Tao Chen, Zhongxue Gan, and Jiayuan Fan. Boost vision transformer with gpu-friendly sparsity and quantization. In CVPR, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In ACL, 2019.
Pengxiang Zhao and Xiaoming Yuan. GANQ: GPU-adaptive non-uniform quantization for large language models. In ICML, 2025.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12:1556-1577, 2024.
15

---

Published as a conference paper at ICLR 2026
A
NOTATION
We summarize the frequently used notations in the paper Table 2
Table 2: Frequently used notations.
Symbol
Description
A pre-trained model
d'
A compressed model
f()E C
Compression method from set IF
Compression ratio
M()
Performance metric
W.,X;
Weight and activation matrices of layer l, respectively
Compression order from a set II of all possible permutations
2()
Pruning and quantization methods, respectively
P
Pruning ratio (Cp 1/(1 p))
Borig Be 8,(:)
Original and quantized bit-widths, respectively Error induced by applying f(-)
TE
Abstract data type from the set of all valid types in model e Granularity of f-)
t, u ();
A unit of type within the model Φ
D((T)
Binary indicator of whether /(() modifies unit u under order
C(Ti,z)
Performance gap between (i( and f2(.)
A(fi f2)
Compression order advantage fi - f2 over f2 - f1
C7
Compression Equivalent Ratio (CER) of f(-)
Ad:: fi + f2)
Interference from fi(-) to fz(-)
f2
f1
B DETAILS ON THEORETICAL ANALYSIS
fi
f
We provide the detailed proofs for Theorems and 2 then formulate a generalized version of Hypothesis applicaable to a broader setting with multiple compression methods.
h
G2
G
h
f2
Gz
Gy
B. PROOF OF THEOREM I
Proof. Given two compression methods (-) and f2(.) with respec- tive granularities in and 1/22- disjoint selectivity ensures that ev- ery unit is assigned exclusively to one method. Hence, every unit U(o + is classified into one of four disjoint groups, G1, G2, G3. or G4, according to its assigned method. Then,
Figure 11: We partition all units u into four disjoint groups. Only groups G and G2 influence A(fi fz).
G₁ {u (f (f 0}.
G2 {u D2((2 = 0, o = 1}.
G3 {u (1( 0. D f2) 0},
G1 {u (f D (f1 fz) 1}.
G1 U G2 UG UG, = U(o; (t +2))),
where U(d; t) represents the unit set of model 6 at granularity t, and (T(()) records whether f(-) modifies u under the ordering (1 if yes, 0 no). Note that these four groups mutually exclusive and collectively exhaustive. Also, |G|| = |G2 since compression ratios C₁ and C,2 are identical regardless of the compression order. Figure 11 illustrates the four groups
16

---

Published as a conference paper at ICLR 2026
Under Assumption 1 and the defined partitioning of groups, the compression order advantage A( is expressed in terms of unit-wise reconstruction errors 8y (u;):
Af1 + f2)
=Mi(f
- M((f) f2)())
=-8(0(2051(0) - ((f1s))(o))
=-B( Σ 1 (w))l + Σ (w:)ll +
Σ
(u))( +
Σ (1% (u:)
4
M.
M.EGa
Σ
Σ
(u:))1
Σ
- Σ
|18 (us))l
a.EG,
We
EGG2
a
N.EG,
- B
Σ
(u:
+
Σ
(1x))
Σ
(u:)l:
Σ
118 (us)))
B2E01
01E01
M,EG2
3
Σ g(u,)
Σ
g(ui)),
N:C82
M
where error difference g(u.) | (u:) (u=) ||0 (u)) Note that G3 and G4 are discarded since their effect remains unchanged irrespective of the compression order.
Case study on pruning and quantization. To support intuition, we provide case study on pruning and quantization, which constitute the core scenario of our work. As described the main text, disjoint selectivity holds only when the granularity tp of pruning P-)) is greater than or equal to the granularity W, and X; denote the weight and
to of quantization 2(-). We analyze this at the layer level: activation of layer 4; E L in the model ø. Note that the error 5 (W,, X,) f(W.)f(X, - W,X; for a compression method f()), as described Section 2
We partition the layers L into four disjoint groups G1, G2, G3. and G4 based on their pruning status:
G = {u (P. = (Q: = 0},
G2 {u D Q) 0, (Q. 1}]
G3 {u D. 0, ( =0}
G, = (u (Poo) =1 (2. =]]
G U UG,UG LL
where (T)) records whether f()) modifies under the ordering (1 if yes, O if no). In the pruning quantization setting, the partition is directly determined by whether each layer is pruned in the final model. Pruning behaves an absorbing operator: pruming overrides any modification introduced by quantization. Therefore, the partition above reduces to grouping layers according to whether they pruned under Q o P or P Q, yielding the pruning-status-bon formulation below:
G = Poop \ Proo.
G2 Prog| Poop,
G3 = Poop n nPPoor Proo;
G. L @P U UP
G U G2 U G3 G4 = L,,
where T denote the sets of pruned layers when applying f(-).
17

---

Published as a conference paper at ICLR 2026
Then, the quantization- first advantage A(Q - P) is estimated as follows:
A(2 P) M((P 2) M((Q P) -ō(6poe(0) Sovp(o))
- 3 Σ (
|lóeop ( 10 x.)1
1ELL.
Σ
(16oe(W, - w.x. El1- 11,,,,,113 (eo((w..)))l)
-
4EG
1,GGa
Σ g(W,, X;)
Σ (W)) LEG:
where g( (W, X,) | W, x.)
W This expression holds as for any layer I E II the pruning operator and its associated error are defined as follows:
X
0
if pruned otherwise"
8p 6p(W,,X,) = [ W, if pruned
P(W.)((X)
W,X,
otherwise
B.2 PROOF OF THEOREM 2
Proof. As A(Q + P) and G(-) (or -c) are functions of the compression ratio Co we analyze their behavior separately. Without loss of generality, we consider only the case where Co changes in the direction of increasing Cp Cg. i.e., decreasing Co.
Under Assumption [ which assumes well-designed quantization, decreasing Co preserves the expected value of the quantized outputs while decreasing their standard deviation. As pruning intensity is held constant, the variation across compression orders is attributed solely to the severity of quantization. Lower quantization ratio (i.e., smaller standard deviation) decreases the chance that units behave differently across orders, which can only decrease or preserve the value of |G1] = [G2) but never increase it. We analyze the two possible cases as follows.
Case 1: Number of layers affected by order decreases. Although more than one layer may change simultaneously, any such change can be decomposed into a sequence in which layers are added one by one; thus it suffices analyze the case where exactly one layer added. Let I and lj denote the layers moving from G1 and G22 respectively. Such transition occurs in mudget-preservvvv
manner: one layer from G, and one from G2 are jointly reallocated, with one moving to G3 and the other to Ga, so that the number of pruned layers remains fixed. As only G1 and G2 contribute to A(Q ~ P), removing I G1 and I, E G2 from these groups changes the value by- (b,((, - g(4)) (see Theorem Therefore, to show that A(Q - P) does not decrease, suffices to prove that g(()) g(4) Expanding the definition of g(-), we obtain
((1)) - (l,) (W, I | W, x.) (I W x,x, 11
Under Assumptions and 2 the second term in parentheses is positive, i.e., || w,X,II -1
W,X, > 0. This because under pruning alone, 4 G₁ is pruned while L G2 is not. Under the well-designed pruning assumption, which minimizes performance drop, this is equivalent to minimizing the increased error. Therefore, the pruning error I for pruned less than or equal to W,X, F for unpruned Ij. making the term positive.
Given the assumption of well-designed quantization in Assumption 2] the remaining first term, which denotes the difference in quantization errors, is negligible compared to the pruning-related component. This because the quantization error at each layer modeled as zero-mean noise with small variance, whereas the pruning error term || | X,II 1 (or || W,X, Il corresponds directly to the magnitude of the pruned responses. Thus, the difference | ( (W,, j, - X.) remains uniformly small compared to || w,,,11? dominates g(4g) - g(.). Consequently, we get
|| W, X, 1? so the pruning-induce gap
s(t,) ((.) (11 w w,X - w w. <.
Overall, the decrease in the number of order-dependent layers leads to the increase of the order advantage A(Q P).
18

---

Published as a conference paper at ICLR 2026
Table Bascline methods covered in our experiments across different settings.
Compression
Modality
Target Models
Baseline Methods
SparseGPT (Frantar & Alistarh
2023
Decoder-only models
Wanda (Sun et al. 2024)
Language models
SLEB (ong al.
2024
Pruning P())
Encoder-based models
K-prune (Park et al.
2024
Vision models
CNNSs
PRACTISE (Wang & Wu 2023)
ViTs
SAViT (Chuanyang
2022
RTN Gupta et al.
2015
Decoder-only models
OPTQ Frantar et a
2023
Language models
QuaRoT Ashkboos et al.
2024b)
Quantization 2())
Encoder-based models
UniQuanF Park al.
2025a
Vision models
CNNs
N2UQ (Liu et
2022
ViTs
RepQ-ViT Li et al.
2023
Parameter Efficient
Language models
Decoder-only models
LoRA (Hu al.
2022)
Fine-tuning
Parameter Sharing
Language models
Decoder-only models
Basis Sharing (Wang et al.
2025a)
Mixed-precision
Vision
CNNs
HAWQ-V2 Dong et al.
2020
quantization
models
Case 2: Number of layers affected by order remains unchanged. As the loss-contributing groups G and G2 do change, the compression order advantage A(Q P) remains unaffected.
Therefore, in all cases where Co decreases, the value of A(Q P) does not decrease. In conclusion, monotonicity between A(2 + P) and g(-) holds under fixed Cp.
B.3 GENERALIZATION TO MULTIPLE METHODS
We formulate Hypothesis with compression methods fi(+) and fz(-). This is because if the hypothesis holds for any pair of methods, can be generalized to more than two methods.
Following the setup in Problem 1 suppose sequentially apply set F f1 f2(), f.)() of compression methods to a pre-trained model ø. Then, any pair (m1 T2) of permutations from the set II {I F F is bijective) of all permutations a sequence of adjacent transpositions. This is because the adjacent transpositions generate the full symmetric group. allowing any permutation be constructed from another. Thus, under Hypothesis] [ our original claim shown in Figure [ hholds, that applying stronger permutations later leads to better performance of the compressed model.
be converted into another via
C EXPERIMENTAL SETUP
We describe the details on the experimental setup, including models, datasets, baselines, evaluation protocol, and implementation.
Models. We evaluate representative models across modalities, including LLaMA (7B, 13B) (Tou
vron al. 2023), LLaMA 3 8B (Grattafiori et al., (024).. Mistral 7B (Jiang al. 2023), ) Mistral Nemo- 12B Mistral AT Team 2024), and BERT (Devlin lin et al. (019)) for language, and ResNet-18 (He
al. 2016 and DeiT-Base TTourron al. 2021) for vision.
19

---

Published as a conference paper at ICLR 2026
Datasets. We evaluate decoder-only language models on WikiText-2 Merity al. 2017) and C4 (Raf" fel et al. 2020) datasets for perplexity, and on five commonsense reasoning tasks ARC (Clark et al. 2018), HellaSwag (Zellers et al.] 2019), LAMBADA (Papern al. 2016), PIQA (Bisk et al. 2020), and Winogrande (Sakaguchi et al. 2021). ). For encoder-based models, we evaluate the performance using Spearman's 's rank correlation coefficient the STS-B dataset. For vision models, we report
classification accuracy on ImageNet (ILSVRC 2012) Deng et al.) 2009) dataset.
Baseline Methods. We validate our hypothesis across a total of sixteen pairs of compression methods by incorporating six pruning methods, six quantization methods, and one mixed-precision quantiza- tion method. Table 3 provides an overview of baseline methods categorized by target models and modalities. Refer to the original papers for further details.
Evaluation Protocol. The calibration dataset consists of single batch with 128 samples drawn from the same dataset used for perplexity evaluation. We set the batch size to 16 for perplexity evaluation and to 128 for commonsense reasoning tasks. All quantization methods apply the same bit-width to weights, activations, and KV-cache, with clipping applied during weight quantization. Both models are evaluated without fine-tuning using a batch size of 128. Metrics are reported as the average of five repeated runs, each computed with four-digit precision. We plot the relative values for visualization.
Implementation and Machine. Our implementations are written in Python and rely on PyTorch, Transformers, Accelerate, and Torch Vision libraries. For all bascline methods, reproduce the results based on their open-source code and hyperparameter configurations. All of our experiments done workstation with Intel Xeon Gold 6338 and NVIDIA A100 80GB.
Parameter Efficient Fine-tuning Experiment. We adopt LoRA (Hu et al. 2022 on top of SparseGPT (Frantar& & Alistarh, (2023) and RTN (Gupta al. 2013) to fine-tune the compressed model. The target model is LLaMA 3 8B (Grattafiori et al.) 2024), where fine-tuning is processed after quantization. We select WikiTeext22 the calibration dataset and train for a total of epochs. We follow Basis Sharing (Wang et al.
2025a) for training details of the low-rank adapter, while 2022) library.
exploiting the PEFT (Mangrulkar al.
Parameter Sharing Experiment. We evaluate the performance of LLaMA 2 7B model when ap- plying Basis Sharing (Wang et al.) 2025a) and magnitude-basee pruning (Han et al. 2015). We follow Basis Sharing for hyperparameters regarding parameter sharing, where the group size is 128. QuaRot (Ashkboos et al., (2024) is selected as the quantization baseline for calculating compression equivalent bits. No additional fine-tuning is applied under this setting.
Mixed-precision Quantization Experiment. We base our method on HAWQ-V2 (Dong et al. 2020), but allocate bit-widths iteratively rather than in a single shot following LampQ (Kim et al. 2026). At each iteration, we search per-layer bit-widths from a range of [2. 3, 4, 5, 6, 7, 8] and train for 5 epochs. All other experimental settings, including hyperparameters and quantization techniques, are aligned with the original paper. We run all MPQ experiments on a workstation with Intel Xeon Silver 40 and NVIDIA RTX 4090.
D FURTHER DISCUSSION AND EXPERIMENTS
We present results from extended experiments, and offer further discussion and remarks our work.
D.1 EXPERIMENTS ON DIVERSE LLMS
Our experiments on decoder-only models are limited to LLaMA herd models (LLaMA 2 (Tourron ct al. 2023) and LLaMA 3 (Grattafiori et al. 2024) models), which may not fully reflect broader generality. To address this, we conduct additional experiments on models from the Mistral herd. Figure 12 presents the results of applying SparseGPT (P). and QuaRot (2(-) to Mistral 7B (Jiang et al. 2023) and Mistral Nemo 12B (Mistral AI Team 2024). We have two observations from the result. First, the compression-ordes trend aligns well with the hypothesis across Mistral-based models. The result serves as additional evidence confirming the hypothesis in decoder-only language models. Second, comparing models within the same herd (also refer to Figure 3) we find that smaller models exhibit greater variation in compression-order advantage for identical CER differences. This may be because low-bit quantization (or stronger quantization) causes greater degradation in smaller models, thereby intensifying the observed differences.
20

---

Published as a conference paper at ICLR 2026
p =
p =
p
p =
0.0
-0.2
Al2
2
-0.4
-0.6
-0.8
-1.0
-1.
-1.0
-0.5
0.0
0.5
1.0
1.5
-1.5
-1.0
-0.5
0.0
0.5
Co
Co
Mistral 7B
(b) Mistral Nemo 12B
Figure 12: The compression order advantage A(Q + P) increases monotonically with the CER difference Cp Co also for Mistral herd models. See Appendix D.I for details.
D.2 ANALYSIS ON ENCODER ENCODER-BASED MODELS
p= 0.60
Beyond decoder-only LLMs, we extend our analysis to encoder- based language models validate the generality of our hypothe- sis. Figure 13 presents the performance of BERT (Devlin et al. 2019) model under K-prune (Park al. 2024) (P(-)) and Uni- QuanF PPar et al.) 2025a) (2()). We adopt Spearman correla- tion as the performance metric, measured STS-B dataset (Cer et al. 2017) from GI.UU Wann et al.) 2019) benchmark. We observe a monotonic increase along both axes of A(Q P) and Cp Co, confirming that our hypothesis holds.
0.70
4l0
-0.0044
-0.006
0.5
1.0
1.5
2.0
C₁
- Co
Figure 13: The hypothesis holds
for encoder-based language mod-
D.3 COMMONSENSE REASONING PERFORMANCE
els. See Appendix D.2 for details.
Although the negative of perplexity serves as an intuitive and efficient metric MM-) for evaluating language models, prior studies (Meister & Cotterell, 2021: Fang et al. 2025) suggest it does not always correlate with real-world performance. We thus investigate the performance of LLaMA 3 8B model across five commonsense reasoning tasks in Figure 14 when applying SparseGPT (P(-)) and QuaRot (Q())). Results affirm the generality and metric-agnostic nature of our framework, as hypothesis holds across these tasks.
D.4 IMPACT OF ROTATION ON PRUNING METHODS
In Figure 5 and Finding 3, we observe that applying rotation without quantization may lead to notable degradation on pruning performance. To further analyze this, Table compares the performance of LLaMA 8B model pruned with and without QuaRot- based rotation, across two pruning meth- ods with different granularities. We have two observations from the result. First, rotation-induced degradation scales with pruning ratio. This because higher pruning ratios result more units being pruned that are altered by rotation, thereby increasing the error. Second, unstructured pruning (SparseGPT) exhibits significantly higher error compared to structured pruning (SLEB). This trend is especially evident under high pruning ratios.
We therefore investigate the underlying reason behind this phenomenon. We identify two types of errors induced by pruning, depending its granularity: matrix-wise and element-wise errors. Figure 13 conceptually illustrates these two cases.
First, in the of matrix-wee wise pruning, ignoring rotation during pruning leaves the rotation-induced matrix H intact, introducing extra computation and numerical errors compared to the non-rotated case. As suggested in QuaRot (Ashkboos al. 2024b), the rotation inverse is fused into the target layer,
21

---

Published as a conference paper at ICLR 2026
p -
p
p
p
1.0
0.1
0.5
0.0
-0.1
R
0.2
-1.0
AlQ
0.3
1.5
0.4
2.0
0.5
-2.
-0.6
1.0
0.5
0.0
0.5
1.0
1.5
1.0
0.5
0.0
0.5
1.0
1.5
C5 Cg
CØ
C2
(a) ARC
(b) HellaSwag
1.00
1.0
0.75
0.5
0.50
0.25
0.0
Alg
0.00
0.5
0.25
0075
-1.5
-1.00
-1.0
-0.
0.0
0.5
1.0
1.0
-0.5
0.0
0.5
1.0
C5 Cg
C$
Cg
(c) PIQA
(d) Winogrande
0.1
0.0
-0.2
-0.1
-0.4
AlQ
-0.6
-0.3
-0.8
0.5
-0.5
0.0
0.5
1.0
1.5
-1.5 1.0
-0.5
0.0
0.5
1.0
1.5
C
Ca
(e) LAMBADA
(f) Average
Figure 14: Commonsense reasoning task performance of a LLaMA 8B model for SparseGPT and QuaRot. See Appendix D.3 D.3 for details.
22

---

Published as a conference paper at ICLR 2026
Table WikiText-2 perplexity comparison LLaMA 8B model pruned by SLEB (Song et 2024) and SparseGPT (Frantar & Alistarh 2023) under varying pruning ratios, with and without rotation following QuaRot (Ashkboos
2024b). See Section D.4 for details.
SLEB
SparseGPT
Pruning Ratio
No rotation
Rotation
Difference
No rotation
Rotation
Difference
Original
6.137
0.05
6.857
6.871
0.014
6.140
6.154
0.014
0.1
8.792
8.828
0.036
6.159
6,205
0.046
0.15
12.603
12.615
0.012
6.213
6.352
0.139
0,2
25.289
25.295
0.006
6.330
6.629
0.299
0.25
51.212
51,560
0.348
6.546
7.250
0.704
0.3
61.502
61,901
0.399
6,894
8.504
1.610
0.35
65.997
66.234
0.237
7.474
20.842
13.368
0.4
92.848
93.260
0.412
8.477
98.213
89.736
Weight
Rotation
Fused weight
Pruned
W
Diff
W2
Diff
Wz
W2 Diff
W
W
W
H
W2
W2
(a) Matrix-wise
(b) Instance-wise
Figure 15: Two cases of errors when pruning rotated units. See Section D.4 for details.
while the original transform merged into the preceding normalization layer, leaving un-removed components that generate error during naïve pruning. This type of error scales proportionally with the pruning ratio, as each pruned matrix introduces one such error.
Second, in element- wise pruning, additional eтrors arise due to -ration-induued changes in unit selection, on top of the matrix-wiss error. As the goal of rotation to facilitate quantization by flattening activation outliers, multiplying its inverse results in an error compared to the original matrix. Consequently, the discrepancy in layer content leads to different pruning decisions. Furthermore, this selection-baii based error grows with higher pruning ratios due to greater number of pruned units.
In summary, given these errors, it is crucial to develop pruning techniques that align with rotation- based quantization strategies.
D.5 A DIRECT COMPARISON WITH PRIOR STUDIES
We discuss how our approach differs from prior studies, particularly SmoothQuant (Xiao al. 2023 and Harma et al. (2025). Our contribution lies in establishing general analysis for understanding compression order across diverse methods, whereas these prior works either focus on single-method optimization analyze specific method pairs under restrictive assumptions.
QuaothQuant (Xiso al, 2023). SmoothQuant addresses a fundamentally different problem than joint compression order optimization. Specifically, SmoothQuant optimizes single compression technique (quantization) by mitigating activation outliers through per-channel scaling transformations. While the paper states that "SmoothQuant is orthogonal to quantization schemes." this refers to its compatibility as a pre processinn step that can applied before various quantization methods. However, SmoothQuant does examine the order-dependent interaction problem when combining
23

---

Published as a conference paper at ICLR 2026
quantization with other compression families such pruning. In contrast, our work focuses understanding how compression order affects the model performance when sequentially combining methods from different compression families. SmoothQuant may serve a component within our quantization baselines (i.e., as a pre-processing step before quantization), but our analysis operates at higher determinnn the optimal ordering between different model compression techniques regardless of the specific quantization implementation. This distinction is critical: SmoothQuan addresses intra-method optimization (improving quantization itself), while we address inter-method composition (ordering across different compression types).
Harma al_ (2025). As noted in Section 2 only few studies have addressed how the order of
compression methods affects the model performance. Among them, |arma et al. (2025) stands out as the only study that attempts a theoretical approach to the problem. They examine the interaction between pruning and quantization, showing that the two are not orthogonal as assumed by previous
works. Furthermore, they argue that pruning followed by quantization is universally optimal.
However, their framework suffers from three significant limitations. First, their framework relies on oversimplified assumptions that hinder practical applicability. Specifically, they focus solely on magnitude-based pruning (removes weights based on their absolute values) and max-scaled block- wise quantization (uniformly rescaling blocks using their maximum value), both of which are naive approaches that are less practical and often fail to preserve accuracy. Second, their analysis is confined to minimal set of scenarios, failing to address diverse architectures or methods. Beyond the limited set of methods, their experiments also consider only the combination of two techniques- pruning and quantization-on decoder-based LLMs, lacking broader coverage of models and compression approaches. Lastly, the framework cannot be generalized across different settings, as many coun- terexamples have shown that pruning-before -effore-uuanizzttionn is not always optimal. Motivated by these gaps, we aim for a more general formulation that holds across methods, models, and metries, thereby introducing the Progressive Intensity Hypothesis.
D.6 VIOLATION CASES OF THE HYPOTHESIS
Although our hypothesis is highly general and robust, we still observe cases where it does not hold. These cases largely fall into three categories: severe performance collapse, full model re- re-training, and increase of order-affected layers. First, each model exhibits a different tolerance to compression, with performance dropping exponentially beyond a certain ratio. While these settings are impractical due to severe performance loss, observe cases where applying the stronger method first performs better. This may be because the error is already too large, violating our assumption of well-designed compression in Section 4 applying the stronger method first might help reduce the total error. For less compression-robus! models like decoder-based LLMs, we observe earlier breakdowns-suun diminishing advantage when pruning ratio increases at fixed bit-width (Figure Ba). Second, when strong mull-training is applied, the advantage from compression order may invert. Compression order serves merely as initialization, and the retraining process dominates, making difficult attribute outcomes to order alone. We plan to investigate these and potentially other exceptions more rigorously in future work. Lastly, in practical situations, increasing Cp may result increase of order-affected layers, leading to a violation of the hypothesis. We analyze the details in the following paragraph.
as
Impact Cp. Without loss of generality, we consider the case when only Cp increases. Increasing Op implies a stronger pruning effect, since it lowers (())))), resulting in a decrease in ((P, Q) and a corresponding increase in Ct.. Hence, to ensure monotonicity and satisfy Hypothesis] A(Q P) should increase accordingly. Note that Co is fixed while analyzing the effect of Cp.
To analyze the effect of increasing Cp, we first consider a local step in which the total number of pruned units increases by one. For the initial pruning ratio p and the increased ratio p' under the same granularity /r, the following relation holds:
(U(ott)) - -p- (U(o:t))
From the definition of compression ratio Cp = 11/(1- p). larger pruning ratios correspond to larger compression ratios. By repeating this incremental process, we can construct any pruning ratio. Under disjoint selectivity, each unit is exclusively assigned to one compression method, allowing us to partition the units into four disjoint groups discussed in Appendix B.T
24

---

Published as a conference paper at ICLR 2026
+
2
P
P
P
4
G
4
G
4
G
0
-
-1
0
2
+1
&
A
R
Oi
G2
G3
Of
G2
G3
Ot
G2
G3
B
R
-1
+2
0
+1
+1
Case
Case 2
Case
Figure 16: If the number of pruned units increases by one, the change in unit allocation across groups fall into three distinct cases. See Appendix D.6 for details.
Increasing the pruning ratio from p to p' result in three possible changes in the group configuration: the number of affected layers by order (i.e., |Grl [Ga|; note that two values equal.) 1) decreases by one, 2) remains unchanged, or 3) increases by one. The three cases are visualized in Figure [6 From
Theorem 1 A(Q P) 3- (Eiee,, 9(4) Ei,e,, (44)) (where g(4) 11o2(W; )
W; x (ll))) should be preserved or increased to satisfy Hypothesis However, this condition fulfilled in only Cases and 2, but not in Case 3.
Case I: Number of layers affected by order decreases by one. layer is no longer affected by compression due to order change, then another layer must also be excluded to preserve the total number of pruned layers which should be increased by one. Thus, |G and [G2] each decrease by one, while |Gy| increases by two. Similar Case 1 of Appendix B.2 the increase in Cp eliminates negative loss term, A(Q + P) increases.
Case 2: Number of layers affected by order remains unchanged. If the additionally pruned layer is always pruned regardless of the compression order, then [Gy] increases by 1 while |Ga] decreases by 1. Similar to Case 2 of Appendix B.2 as the loss-contree loss-contributing groups G1 and G2 do not change, the compression order advantage A(Q
remains unaffected.
Case Number layers affected by order increases by one. The aforementioned two cases could not increase the number of order-dependent units, whereas there should exist case where both groups [G1| and |G2| increases by one. As the number of total pruned layers should be increased by one, the added layers should be originated from G4. i.e., |Gal decreases by two. In contrast to Case 1, A(Q + P) may decrease due to the increase of negative loss terms.
As the conditions under which each case emerges differ across specific configurations, this phe- nomenon is not analyzed in general settings. We leave a precise characterization of the conditions under which increasing the pruning ratio may invalidate the progressive intensity hypothesis as an important direction for future work.
D.7 ADDITIONAL REMARKS
Limitations of Current Work. We introduce a broadly applicable hypothesis that can be extended to diverse compression methods and model types across different domains. Still, we acknowledge three important limitations in our current work. First, due to the general nature of our framework, it does not provide detailed analysis for each specific combination of methods. While our hypothesis captures high-level trends, it does not define the best compression sequence for individual cases. This motivates research into discovering the best compression orderings under practical scenarios. Second, our study is limited to joint model compression in plug-and-play settings where methods are combined post-hoc. As demands for higher compression grow, integrated design strategies should be investigated beyond simple combinations. Lastly, framework does not yet provide explicit predictive rules precise estimation of how much better one compression order than another. Capturing the nonlinear and cross-layer effects required for precise sign value prediction of compression-order advantage A(.)
25

---

Published as a conference paper at ICLR 2026
remains open problem. We therefore consider the development of predictive models for A and meta-learning approaches for automatic order selection as promising direction for future research.
Future Work. In addition addressing the aforementioned limitations, future directions may also include extensions of our current framework. First, a systematic study of interference across different pipeline designs would provide deeper insights beyond our current empirical findings. Another direction is to automate compression order selection based on observed trends. unified approach that generalizes across cases may offer better understanding on the role of compression order. Lastly, evaluating our hypothesis on emerging architectures such as Mixture-of-Experts and multimodal LLMs may broaden its generality.
(GPT-4o) and Perplexity Fl exclusively for language polishing purposes; for improving grammar and clarity the sentence level. We do not use them for any research-related tasks, including code implementation, theoretical derivation, and result analysis.
Usage of AI Assistants. We employ ChatGPT 3
httpss//ccatapt.com
httpsss//ww..prrreeeity...
26


Reviews:
[{"review_id": "HIsLxx44Fq", "reviewer": "Reviewer_UfHH", "summary": "This paper revisits an underexplored question in model compression: whether the order of applying pruning and quantization matter, and why? The authors formulate the joint compression order optimization problem and propose the Progressive Intensity Hypothesis (PIH), that weaker perturbations should precede stronger ones. This hypothesis  is proved under the assumptions of disjoint selectivity and well-designed compression (Theorems 1 & 2), and extend the analysis to more realistic interference cases. Empirical validation spans language models (LLaMA-2/3 7B-13B-8B) and vision models (ResNet-18, DeiT-Base), showing consistent monotonic trends where quantization-after-pruning performs worse than pruning-after-quantization. The hypothesis is further generalized to multi-stage compression and mixed-precision quantization.", "questions": "1. How sensitive are the results to the performance metric M(·)? Would monotonicity still hold under different metrics (e.g., loss, perplexity, BLEU)?\n\n2. I am interested whether the Progressive Intensity Hypothesis extend to compression combinations involving low-rank (e.g., Lora) or distillation techniques?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. A new but meaningful problem explored in this paper: the paper introduces a rigorous formalization of compression order optimization, a previously neglected but practically crucial dimension in joint model compression. The Progressive Intensity Hypothesis provides a simple, actionable rule with theoretical grounding and broad implications.\n\n2. Clear theoretical analysis.  Theoretical results are clearly derived. Theorem 1 establishes performance ordering under disjoint selectivity, relating compression order advantage A to per-unit reconstruction errors. Theorem 2 proves monotonicity with respect to compression equivalent ratio (CER) differences. These analyses connect signal perturbation and model compression behavior.\n\n3. The empirical validation covers language, vision, and general pipelines, including experiments on LLaMA and ResNet/ViT variants. The paper goes beyond two-method cases to multi-stage and mixed-precision scenarios, reinforcing the hypothesis’ generality.\n\n4. There are also some useful secondary findings in this paper. For example, rotation exacerbates pruning effects (Fig. 5). Structured vs. unstructured pruning differ in interference behavior (Table 1). Vision models exhibit stronger order sensitivity than LLMs.", "weaknesses": "1. Theoretical assumptions are strong. The “well-designed” compression assumption is idealized and not always met in real pruning heuristics. The analysis could better discuss how violations (e.g., correlated layer errors, adaptive pruning schedules) affect Theorem 2. The independence assumption between layers (Assumption 1) is strong, worth empirically validating with correlation metrics. \n\n2. While extensions to multi-stage and MPQ are demonstrated, the theoretical discussion remains pairwise. A more general n-method ordering theorem (Appendix B.3) is mentioned but not empirically validated.\n\n3. Connection to prior composability studies. The paper could better position itself relative to recent works like SmoothQuant (Xiao et al., 2023) or Harma et al. (2025) by emphasizing differences in analytical generality rather than empirical scope alone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ewZ4ejZYeq", "reviewer": "Reviewer_k2Nf", "summary": "This paper investigates the importance of compression order in joint model compression (e.g., pruning and quantization) and proposes the \"Progressive Intensity Hypothesis\" (PIH). The hypothesis posits that applying weaker perturbations (compressions) first, followed by stronger ones, leads to superior model performance .", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-organized and easy to understand.\n\nThe paper addresses a practical yet underexplored problem in model compression: the order of joint compression.", "weaknesses": "1. The paper introduces the \"Compression Equivalent Ratio\" (CER) to unify the \"intensity\" metric. However, calculating the CER for a pruning method requires running the pruning experiment independently to measure its performance (e.g., 65% accuracy), and then finding (or interpolating) the quantization ratio $\\mathcal{Q}$ that yields the same performance. This implies that to apply the hypothesis, one must first run all compression methods individually to determine their \"intensity\" ranking. It is questionable whether the computational cost of this ranking process is substantially lower than the cost of directly testing both compression orders (e.g., $\\mathcal{P} \\rightarrow \\mathcal{Q}$ and $\\mathcal{Q} \\rightarrow \\mathcal{P}$).\n\n2. The assumptions for the theoretical analysis are an oversimplification of neural networks. The paper assumes \"Layer-wise independence\" , an \"Error-performance trade-off\"  (both in Assumption 1), and \"Disjoint Selectivity\" (Definition 5). These assumptions largely ignore the correlations between layers and the different sensitivities of various layer types.\n\n3. The scope of the hypothesis is mainly focused on \"post-hoc\" compression settings. Additionally, its core analysis revolves almost exclusively around pruning and quantization scenarios. It is unclear how well the hypothesis generalizes to combinations involving other compression techniques (like knowledge distillation or parameter sharing).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lk4tODQDjp", "reviewer": "Reviewer_1tMH", "summary": "This paper investigates how the order of compression operations (e.g., pruning and quantization) affects model performance. The authors formalize the ordering problem, propose quantitative metrics, and validate the intuitive Progressive Intensity Hypothesis—that weaker perturbations should be applied before stronger ones. The paper presents both theoretical insights and extensive experiments on language and vision models, confirming the hypothesis and providing clear practical guidance for compression practitioners.", "questions": "Please see the weakness.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The topic is practical and relevant to real-world model deployment.\n2. Experiments are extensive, covering multiple models and compression settings.\n3. The paper provides a clear and actionable guideline that practitioners can easily adopt.\n4. The writing is clear and the code is open-source.", "weaknesses": "The paper mainly evaluates compression pipelines that consist of two or three stages (e.g., prune–quantize or prune–quantize–prune). Based on the proposed theory, could this framework be extended to longer or more complex multi-stage compression pipelines, and would the same hypothesis still lead to performance gains compared with existing setups/shorter pipelines?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hEs1pC2yEa", "reviewer": "Reviewer_BEC6", "summary": "This paper investigates the interaction between two common model compression techniques — pruning and quantization — and specifically asks whether the order of application (Prune→Quantize vs. Quantize→Prune) affects the final model performance.\nThe authors formalize this question by defining the compression-order advantage\n$A(Q → P)=M((P ∘Q)(φ))−M((Q ∘ P)(φ))$,\nwhere M(⋅) denotes model performance. They also introduce the Compression-Equivalent Ratio $C_P^*$ , the quantization ratio that produces equivalent compression to a given pruning ratio so that both methods can be easily compared/combined under a unified metric.\nThrough a series of controlled experiments across several architectures and datasets, the paper finds that the order of operations does matter if one wishes to preserve the maximum model accuracy", "questions": "$Q_1$. Figures 3, 4 and 6. It is unclear at first glance why multiple points appear for a single pruning ratio or why some curves are nearly flat around zero. I figure that the different points in each curve correspond to different compression bitwidth. Please clarify it in text for a more fluent reading (e.g., the different points in each curve correspond to $C_Q=$a,b,c,d, and e) IMHO it will make it more fluent for readers\n\n$Q_2$. $C_p^* $ is defined for a vanilla model. That is, apply the pruning, measure the model’s accuracy and find the compression ratio $C_p^* $ of the quantization that leads to the same accuracy. I understand the reasoning for this methodology. You want to have a common metric so that you can have a proper $x$ axis. \nBut what happens when you reverse the order of compressions? \nQuantization, if it is applied as the second compression, still compresses by $C_Q$ (Does it?). However, I’m not sure if pruning still achieves the same analogous compression rate $C_p^*$…\n\n$Q_3$. How does this methodology can be used for other compression methods such as joint-compression (e.g., merging similar tokens in KV-Cache, merging heads in Attention, etc.) or Low rank approximations (LoRA). \n\n$Q_4$. Can you give reference to the assumptions in Assumption 1. \n\n$Q_5$. Figure 5 and related text. Can you clarify? The rotation without quantization did not introduce any accuracy degradation. Am I right? Only the pruning on top of the rotation introduced an increased degradation. Is it because you prune additional weights that did not exist prior to rotation?\n\n$Q_6$. Table 1. I guess the $C_Q$ stems from quantization of 16bit to 4,5,…9bit. Please add it to make the table clearer\n\n$Q_7$. Make the result simpler to read. For example, in Finding 5 which you put (wisely) within a frame so that any reader can take-away this message. I think you mean “Quatization and then pruning achieves higher accuracy (than pruning and then quantization)”. Add such simpler phrasing, so that anyone, even one that did not read your hypothesis can take-away your finding.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tClear and focused research question.\nThe “prune–then–quantize or quantize–then–prune?” question is simple but practically relevant for model deployment pipelines.\n2.\tWell-defined theoretical framework.\nThe introduction of the compression-order advantage $A(Q → P)$ and compression-equivalent ratio $C_p^*$  gives a clear formal basis for comparing compression orders.\n3.\tSystematic experimentation and consistency in results.\nThe experiments are comprehensive, covering multiple pruning ratios and quantization levels, with consistent definitions across models. And $A(Q→ P)$ behavior with $\\Delta C$", "weaknesses": "1.\tThe compression order advantage $A$ was empirically measured for a variety of scenarios and it’s trends with $C_p^* - C_Q$ was demonstrated. But the actual value of $A$, and more specifically whether it is positive or negative, cannot be determined by the mere $C_p^*$ and $C_Q$ values. So I don’t understand how can a user decide which method to apply first,\n2.\tI don’t fully understand the practical meaning of \"well-designed pruning\" in Assumption 2. “The pruning method is chosen to minimize performance degradation”.\nIsn’t that conceptually the goal of any pruning methods? \nFrom your phrasing it seems that there is only \"one\" that minimize the performance degradation. Yet, THERE ARE many pruning methods (also in your experiments). So which one of these DID minimize the performance degradation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)