Effective Interplay between Sparsity and Quantization: From Theory to Practice
Authors: Simla Burcu Harma
Ayan Chakraborty
Elizaveta Kostenok
Danila Mishin
Dongho Ha
Babak Falsafi
Martin Jaggi
Ming Liu
Yunho Oh
Suvinay Subramanian
Amir Yazdanbakhsh
Venue: ICLR 2025 Spotlight (2025)
Keywords: theory of compression
model compression
quantization
max-scaled numerical encoding
sparsity
unstructured sparsity
structured sparsity
N:M sparsity
large language models
magnitude pruning
post-training quantization
efficient inference
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We mathematically analyze the interplay of sparsity and quantization, proving they are not orthogonal operations. This means their combined error is greater than the sum of their parts, especially due to quantization.

Abstract: The increasing size of deep neural networks (DNNs) necessitates effective model compression to reduce their computational and memory footprints. Sparsity and quantization are two prominent compression methods that have been shown to reduce DNNs' computational and memory footprints significantly while preserving model accuracy. However, how these two methods interact when combined together remains a key question for developers, as many tacitly assume that they are orthogonal, meaning that their combined use does not introduce additional errors beyond those introduced by each method independently. In this paper, we provide the first mathematical proof that sparsity and quantization are non-orthogonal. We corroborate these results with experiments spanning a range of large language models, including the OPT and LLaMA model families (with 125M to 8B parameters), and vision models like ViT and ResNet. We show that the order in which we apply these methods matters because applying quantization before sparsity may disrupt the relative importance of tensor elements, which may inadvertently remove significant elements from a tensor. More importantly, we show that even if applied in the correct order, the compounded errors from sparsity and quantization can significantly harm accuracy. Our findings extend to the efficient deployment of large models in resource-constrained compute platforms to reduce serving cost, offering insights into best practices for applying these compression methods to maximize hardware resource efficiency without compromising accuracy.

Published as a conference paper at ICLR 2025
EFFECTIVE INTERPLAY BETWEEN SPARSITY AND
QUANTIZATION: FROM THEORY TO PRACTICE
Simla Burcu Harma
Ayan Chakraborty EcoCloud, EPFL
Elizaveta Kostenok EcoCloud, EPFL
EcoCloud, EPFL
simla. hawaaaaamaaappp....mm ch
ayan. chakrabcrty@opfl.... ch cl wwizzvctaa.osttoo a
Danila Mishin
Dongho Ha
Babak Falsafi
EcoCloud, EPFL
MangoBoost Inc.
EcoCloud, EPFL
danila mi shi in@ep11.c. ch
dangho. ha ha@manggcciai mam.mmannnsss.....m.mmmmmmm....m io
babak. fal sa fi @ep ch
Martin Jaggi
Ming Liu Google
Yunho Oh
EcoCloud, EPFL
Korea University
martin. jaa.iin.jjjgieepl...coo
mi miliu@gocgle.com
yunio__ch@koree.ccooo
Suvinay Subramanian Google
Amir Yazdanbakhsh
Google DeepMind
suvinayygooogee.com con
ayazcan@google.. con
ABSTRACT
The increasing size of deep neural networks (DNNs) necessitates effective model compression to reduce their computational and memory footprints. Sparsity and quantization are two prominent compression methods that have been shown reduce DNNs' computational and memory footprints significantly while preserving model accuracy. However, how these two methods interact when combined to- gether remains key question for developers, as many tacitly assume that they are orthogonal, meaning that their combined use does not introduce additional errors beyond those introduced by each method independently. In this paper, we provide the first mathematical proof that sparsity and quantization are oon-orthogonal. We corroborate these results with experiments spanning range of large language models, including the OPT and LLaMA model families (with 125M to 8B parame- ters), and vision models like VIT and ResNet. We show that the order in which we apply these methods matters because applying quantization before sparsity may disrupt the relative importance of tensor elements, which may inadvertently remove significant elements from a tensor. More importantly, we show that even if applied in the correct order, the compounded errors from sparsity and quantization significantly harm accuracy. Our findings extend to the efficient deployment of large models in resource-constrained compute platforms reduce serving cost, offering insights into best practices for applying these compression methods to maximize hardware resource efficiency without compromising accuracy.
INTRODUCTION
Recent breakthroughs in deep neural networks (DNNs) have surpassed human-level capabilities across various tasks such as text generation, machine translation, and computer vision. Unfortunately, this achievement is accompanied by significant challenges due to the exponential growth the size and complexity of DNN models and datasets (Brown et al.) 2020 Zhang et al., 2022b Scao et al.
2022
Touvron et al. 2023 Almazrouei et al. 2023 Anil et al. 2023 Jiang et al., 2023 OpenAI 2023 Mesnard et al. 2024), which complicates their practical deployment and efficient serving.
Delivering efficient and real-time inference for these large models is constrained by arithmetic density (throughput/slliconn area (Drumond et al. [2018a] Darvish Rouhani al.) 2020a| (Hrma et al.) 2022
memory footprint, and the pressure on memory bandwidth across various hardware platforms (c.g.,
GPU (Nvidia 2022). TPU (Google 2023
"oode and data available httph:ttp////. sq-
10/

---

Published as a conference paper at ICLR 2025
Among various efficiency efforts, model compression has emerged as a crucial solution to effectively
address the challenges associated with large models (Micikevicius et al., 2018;: Drumond et al.
2018b a Wang & Kanwar 2019 Darvish Rouhani et al. 2020b Dai al.] 2022 Zhang et al.
2022a Yeh et al.] 2022 Harma et al.
[022] Rouhani et al. 2023a b Hassibi et al. 1993 LeCun et al.
1989 Frantar
2023 Kao et al. 2022 Lasby et et. 2023 Kuzmin al. 2023) with quantization
standing out as a prominent method in terms of overall compression ratio achieved. Quantization
effectively reduces the precision of model tensors from native floating-poing representation to for- mats such as FP16 (Micikevicius et al. 2018). 6 (Wang & Kanwar 2019), and INT8 (van Baalen al., 2023 Zafrir et al. 2019) In recent years, block- wise numerical formats have gained prominence Drumond et al. 201 8b a Darvish Rouhani et al.] 2020b Dai et al. 2021] Zhang et al., 2022a Yeh et al. |022] Harma et al. 2022; Rouhani et al.] 2023a b). particularly due to their ability to reduce memory footprint and increase arithmetic density in sub-8-bit regimes, while preserving accuracy with minimal hardware overhead.
Beyond quantization, researchers are also exploiting sparsity to further compress models, pruning elements of tensors that are least significant to preserve model accuracy. This reduction in the number of parameters reduces the models' memory footprint and eliminates potentially unnecessary computation (Sze et al.] (020). The most popular sparsity method is magnitude- -ased sparsity, which prunes elements of a tensor based on their magnitudes (i.e., a proxy for the importance of an element to model accuracy) (Han et al.] 2015b] Mishra et al. 2021bb bj; Lu et al.) 2023 Ding et al.)
2023; Bambhaniya et al., 2024). Magnitude- based sparsity achieves noticeable compression rates with negligible impact on accuracy when combined with fine-tuning (Kurtic et al. 2022 Sanh et al.
2020
While combining sparsity and quantization provides significant gains in arithmetic density and memory footprint, may inadvertently have high impact on model accuracy. Prior work tacitly assumes that these two methods are orthogonal, meaning that their combined use does not introduce additional errors beyond those of each method individually. These include studies focusing on CNNs (Han et al. 2015a Wang et al. 2020). which are more resilient to quantization errors due to the absence of dot-product outliers in activation tensors (Xiao et al., 2023). Because the quantization error in CNNs is relatively low, additional errors introduced by combining the two are minimal. Other studies focus only on compressing weights without quantizing activations (Li et al. 2020 Liu et al., 2023), which also leads to low overall error when combined with sparsity. These studies short of properly investigating the combined impact of these two compression methods to maximize hardware resource efficiency without compromising accuracy.
In this paper, we study the interplay between sparsity and quantization systematically. Sparsity and quantization leverage fundamentally separate computational properties of DNNs, but their combined impact on model accuracy involves complex interactions due to the introduction of errors in tensors. We hypothesize that sparsity and quantization are non-orthogonal based on the following two insights. First, applying quantization before sparsity (Q+S) may adversely disrupt the relative importance of tensor elements, leading to the removal of significant elements of tensor with a significant impact on model accuracy. Second, applying sparsity before quantization (S-Q) can introduce additional errors in dot product calculations, as these are influenced by the magnitudes and precision of the elements involved, requiring careful investigation. To the best of our knowledge, we are the first to study the interplay between sparsity and quantization in depth to identify the conditions under which accuracy can be preserved compromised. Our contributions are summarized below:
Oot-Oothogynality of Sparsity and Quantization: We prove mathematically that sparsity and quantization are non-orthogonal operations. Our per-layer error analysis shows that the combination of the two introduces compounded errors and a degradation of model accuracy. Our findings challenge the conventional wisdom that these methods can be combined without a significant impact on accuracy.
Corroborating the Optimal Compression Order: Although applying sparsity before quantiza- tion (S-Q) is the most commonly adopted approach, the optimal order has not been formally demonstrated in the literature. We provide the first mathematical proof that applying sparsity before quantization (S+Q) is optimal. Moreover, we derive the upper bound for the error caused by the sub-optimal order of the transformations at the tensor level. We show that it depends linearly on the number of elements in the tensor and the size of the quantization bin.
2

---

Published as a conference paper at ICLR 2025
Validating Non-orthogonality Empirically: We validate our mathematical findings with experi- ments covering a diverse range of models, including prominent LLMs (OPT, LLaMA), ViT, and ResNet. These experiments support hypotheses and mathematical findings, underscoring the non-orthogonality of sparsity and quantization, and the optimal order of compression methods. Our experiments demonstrate that combining sparsity and quantization, even in the optimal order, can cause up to 13% additional error in perplexity.
RELATED WORK
Quantization. The ever-growing size of DNN models has spurred extensive research into using
narrow numerical formats for inference to reduce memory footprint and improve computational effi-
ciency (Micikevicius et al. 2018 Drumond et al. 2018b a Wang Kanwar 2019 Darvish Rouhani
et al. 2020b Dai et [0.21 Zhang et 2022aa Yeh et al. [0022] Harma et al. 2022; Rouhani ct al. 2023a b). Numerical formats employ scaling factors to adjust their dynamic range and can be categonized based on the granularity and levels of the scaling factors.
Element-wise scaling formats, such as FP32, BFloat16 (Wang & Kanwar 2019), and FP16 MiP eikevicius et al., 2018), consist of sign, mantissa, and exponent components, differing in the bit allocation for each component. Conversely, block-wise scaling formats assign scaling factors to blocks of elements, with block sizes varying by format. For instance, INT8 employs per-tensor scaling, where a single scaling factor is shared by around IK elements. Recent research highlights the effectiveness of max-scaled fine-grained block-wise scaling formats with block sizes smaller than 100 elements. especially in the sub-8 8-bitt regime for both training and inference (Drumond et al.
2018a Rouhani et al., 2023a 6 Zhang et al., 2022a; Darvish Rouhani et al.) (0200a). Additionally, modern hardware platforms have adopted these techniques. For instance, the upcoming NVIDIA Blackwell GPUs (Nvidia 2024) will support MXFP.
Sparsity, Sparsity methods (Hassibi et al.) 1993 LeCun et al., 1989; Frantar et al., 2023; Kao et al.
2022 Lasby et al. 2022) aim to reduce computational and memory footprints in DNNs by selectively pruning tensor elements according to various sparsity mask selection criteria. Broadly, these methods
fall into two main categories based on sparsity patterns: unstructured (Han et al., 2015a Guo et al.
2016
Frankle & Carbin 2019 Evci et al.] (020a) and structured (Wen et al.] 2016)] 6 Yao et al.| 2019
Kang 2020 |
Mishra et al. 202 b Pool & Yu 2021 Zhou al. 2021 et al.) 2021 Hubara et al.
2021
Lu el al. (023). Unstructured sparsity (Han et al.) 201 5a Guo et al.) 2016: Frankle & Carbin
2019
Evci al. [dl.,(20))) involves removing individual tensor elements without any specific pattern.
Structured sparsity (Wen et al., 2016), on the other hand, employs specific patterns when pruning tensor elements. Recent work (ao et al. 2019 Kang 2020) has highlighted the effectiveness of fine-grained N:M structured sparsity in mitigating model accuracy loss.
The fundamental operation in any sparsification scheme is selecting candidate elements for pruning. among which magnitude-based sparsity (Han et (015bb) is one of the most widely used methods (Lu ctt al. 2023 Ding et al.) 20233 Bambhaniya et al. (2024). )) In addition, recent work has introduced
one-shot pruning methods, such as SparseGPT (Frantar & Alistarh (023) and Wanda (Sun et al.
2023). aiming to climinate the need for an additional fine-tuning phase. However, evidence suggests
that incorporating fine-tuning still improves accuracies significantly (Sun et al. 2023 Syed et al.)
2023
Lu et al. 2024).
Combining sparsity and quantization. Prior work has studied the combination of sparsity and quantization and its impact on model accuracy in both orders: sparsity followed by quantization Yuu
et al. 2023 Frantar& & Alistarh 22023] [222] Park et al. [2022] Mishra et al.) [202aa] Li et al. 2020
Han et al. 2015a) and quantization followed by sparsity (Hu et al. [222]] Hawks al. 2021 Wu
et al. 2002 Mishra et al. 2021b). There are two missing pieces of information from prior work. First,
consensus on the optimal order of compression operations not established. few studies raise the
question and experiment with both orders to determine the best approach (Wu et al. 2023 Wang
let al. 2022 Zandonati et al. 2023] Park et al., 2019: Yu et al.] 2020 Mishra et al. 2022 la Kozlov et al. 2021 Zhang et al. 202 1), while others treat the methods orthogonal compression schemes.
Second, there is a lack of mathematical grasp on how sparsity and quantization errors interact and influence final model performance.
3

---

Published as a conference paper at ICLR 2025
3 NON-ORTHOGONALITY OF SPARSITY AND QUANTIZATION
This section provides a mathematical analysis of the interplay between sparsity and quantization, formalizing these compression methods and examining their combination, henceforth referred to as (mathematical) composition, at both the tensor and dot-product levels. For the remainder of the paper, we use the following terms: (1) Tensor level refers to structures that encompass both weight and activation tensors; and (2) Dot-product level pertains to the computation of inner products within these tensors, such the matrix multiplication operation between weights and activations during the forward pass. Our analysis centers on quantization methods that reduce the bit-width of model weights and activations using block-wer biike numerical formats, which are prevalent in practical implementations (Darvish Rouhani et al. 2020aa Drumond et al. 201 8a Zhang et al. 2022a Rouhani ctt al. 2023alb by Micikevicius et al. 2022). These formats determine the scaling factor based on the element with the maximum magnitude within the block. We refer to any quantization method employing these numerical formats as max-scaled block-wise quantization. We use magnitude-based sparsity for both unstructured and N.M structured sparsity.
Definition 3.1 (Max-scaled block-wise quantization). Let E IR" be block of n numbers and m N denote the quantization bit-width. Max-scaled block-wise quantization q R² R" is transformation of the block x such that
32 scaaee
where scale = max 11.......,,. |x,l]) is the scaling factor. Qm scale) quantizes the given element with the scaling factor scale and the number of mantissa bits The exact form of Qm depends the numerical format and can be found in Appendix K For instance, INTm quantization transformation defined as follows:
Qm (2i, scale) = 8
Iz where
2m-1 and [.] the rounding to the nearest integer.
scale
(2)
Definition 3.2 (Magnitude-baseer sparsity). Let R" be a block of n numbers. We assume is divisible by M and we consider each group of M elements in the block. The magnitude-based N:M sparsity transformation can be formulated as:
if Laal < .oor i 1,2, M F4 otherwise
where & is the N-th largest element in the set [|.i] | ]] The same formula be adjusted to represent p% unstructured sparsity by defining as the N-th largest element in the tensor, where N [M p/100], MM is the number of elements in the tensor, and [.] the operation of rounding to the nearest integer.
In the remainder of this section, we delve into the composition of sparsity and quantization two different levels. First, we examine the effects of applying this composition in different orders the tensor level, observing how individual tensors are altered. Then, we explore how the composition influences the result of the dot product operation.
3.1 TENSOR-LEVEL ANALYSIS
Sparsity and quantization transformations inherently introduce errors by decreasing precision or pruning tensor elements. To study the composition of sparsity and quantization transformations at the tensor level, we introduce formal definitions of transformation error and orthogonality in compression. We prove that orthogonality in compression between sparsity and quantization does not persist within this composition.
The following definition formalizes the error for specific transformation at the block level, which consists of subset of tensor elements.
Definition 3.3 (Transformation error). Let x E R be a block of numbers, which are the input of transformation "R" + _ We define (x) = f(x) the error of the transformation f.
Definition 3.3 can be extended to the tensor level, despite being defined at the block level. The cumulative enror of tensor be viewed as the summation of individual errors across all its
4

---

Published as a conference paper at ICLR 2025
constituent blocks. Hence, the theorems analyzed at the block level are indicative of the behavior when scaled up to the tensor level.
Composing two compression methods (transformations) is expected introduce additional errors. Any error introduced by the first transformation becomes part of the input to the second transformation, potentially amplifying the initial enror and resulting in larger overall error.
Definition 3.4 (Tensor-level orthogonality). We define two transformations f and g to be orthogonal in compression if any order of their composition does not introduce any additional error, and thus, the following incqualities hold:
Vx E _2, و & || + (x)]] and || (x)ll S || (x)] + (x)]l
where ||- || is an Lp norm, E [1, -00).
Theorem 3.5. Let 9 be the max-scaled block-wind wise quantization and be the magnitude- based sparsity transformation. Applying sparsity before quantization does not introduce any additional error:
Vx E ER, (x)]l [x)](x)]] + [x)x(x)]
(5)
Moreover, the equality is attainable.
The proof of Theorem 3.5 can be found in Appendix | The main idea behind the proof is that as the sparsity transformation does not prune the largest element in the block, the scale quantization parameter remains unchanged. Consequently, the quantization error for the non-zero components before and after sparsity remains the same.
Theorem 3.6. Let g be the max-scaled block wise quantization and s be the magnitude- based sparsity transformation. Applying quantization before sparsity may introduce additional error:
Ex E R", (x)]l [x)]] + (x)
(6)
Moreover, global upper bound exists for the additional error arising from this specific order of transformations (Q-S). This upper bound is solely determined by the quantization method and the parameters of the sparsity type, independent of the input data. The following theorem precisely quantifies the magnitude of this additional error.
Theorem 3.7. Let q be the max-scaled block-wise quantization and be the magnitude- based N:M sparsity transformation. Let step be the least upper bound for the magnitude of the quantization error for one element: step = sup (x):l 8 q with respect to L1 norm has the following upper bound:
IR", E (1...n) .n.).. Then the error of the composition
M -
Vx E IR"", [|Exog (x)I1, < (|s, + (x)]1 + slep
M
additional
The general formulation of Theorem 3.7 for all L, norms and the proof of Theorem 3.6 and 3.7 can be found in Appendix y As a corollary of Theorem 3.5 3.6 and (37] it follows that the optimal order of transformations is sparsity followed by quantization, as this sequence does not introduce any additional error. Moreover, according to Definition 3.4 sparsity and quantization are non-orthogonal the tensor level.
3.2 DOT-PRODUCTLLLEEL ANALYSIS
In this section, we delve into the error linked with the dot product operation, which is the primary operation in DNNs. Our analysis focuses on scenarios where weight tensors undergo sparsity and quantization, while activation tensors solely undergo quantization. We first extend the definition of transformation eттоr to the dot-product level.
Definition 3.8 (Transformation error over the dot product). Let w R" denote the inputs of a transformation ʃ I - IR and the dot product operation (..-) R" x Ra + R. We define of (x, w) (x, w) (x), (ww) as the error of the transformation f over dot product. Similarly, we define E.P (x, w) = (x. w) (f(x), g (w)) as the error over the dot product when different transformations applied to and
In the remainder of the paper, use the term "orthogonal" to refer to "orthogonal in compression" for simplicity.
5

---

Published as a conference paper at ICLR 2025
At the dot-produet level, define two compression methods orthogonal if their composition, in any order, does not introduce additional error, akin to Definition 3.4
Definition 3.9 Dootproduct-leve orthogonality), Let x, E IR" denote the inputs of transforma- tions f R > IR" and g R - IR", and the dot product operation (.. R2" x R + R. Let the transformation f be applied to both and and transformation be applied only to w. Let denote a composition
c== and yg in any order, c:= [ fog g or c = go f. We define two transformations f and
g to be orthogonal on the dot- dot-product level if any order of their composition applied to the second term w does not introduce any additional error:
yx,y E R", I (x, w)l < |e (x, w) +) w)
(8)
In the following theorem, we demonstrate that any composition of sparsity and quantization yields additional error, rendering these two methods non-orthogonal.
Theorem 3.10. Let q be the max-scaled block-wiss quantization, s be the magnitude- sparsity transformation, be the composition which is either q or q and be the identity function. Composition of max-scaled quantization g and sparsity s in any order produces additional error in any order, given that only the second operand, i.e., weight, is pruned:
3x, E R", 1 w)] >> lef, (x, w)l + 1 (x, w)l
(9)
Moreover,
16 w)l S (x,, +e w)
R((x), & (w)) + (eg((),
(10)
additional error
where Ex(x) is defined as the correction error vector of the composition:
(x) = (x) (x) +x(x) + Ex(x)
(11)
Proof of Theorem 3.10 can be found Appendix y
Analysis of the additional error. As corollary of Theorem 3.10 the composition of max-scaled
sparsity and quantization is non-orthogonal, resulting in two additional error terms.
The term &, incorporates the correction vector of composition écs which carries the additional error from the tensor level to the dot-product level. Depending on the order of the composition, the value of E, varies. When quantization precedes sparsity, certain elements within a block may become equal due quantization, leading the sparsity step to prune different elements than it would on the original tensor. This introduces additional error, as previously significant elements may be inadvertently pruned. If sparsity precedes quantization, the correction vector & exclusively comprises the quantization errors of the pruned elements. As a result, the magnitude of the additional error is typically smaller compared the reverse order.
The term e; also contributes to the additional error, encoding the interaction between the error vectors (x) and e.(w), (w). However, as the norm of quantization and sparsity errors are generally smaller than the norm of the quantization block, this term is less significant than Et.
We provide more detailed explanation of the additional error analysis in Appendix M
Finally, to experimentally validate our mathematical findings, we define a metric, orthogonality threshold assess whether the transformations orthogonal.
Definition 3.11 (Orthogonality threshold). Let M be a DNN model under consideration, EM(M) be an evaluation metric that measures the performance of the model M (e.g., perplexity cross- entropy loss), EMc(M) be the evaluation metric of the model M with transformation C, which is either sparsity S or quantization Q. Moreover, let Err, (M) (M) (MM(M)) be the evaluation metric error of the transformation C/ for the model M. We define orthogonality threshold as:
Orthogonality Threshold EM + (M)) + Errs(M)
(12)
the compression methods are non-orthogonal, and the evaluation metric improves with lower values (e.g., perplexity), we expect the compressed model's evaluation metric (e.g., perplexity) to worsen and thus exceed the orthogonality threshold due to compounded errors. Similarly, if the compression methods are non-orthogonal, and the evaluation metric improves with higher values (e.g.. accuracy), we expect the compressed model's 's evaluation metric (e.g., accuracy) to decrease and thus fall below the orthogonality threshold.

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTAL METHODOLOGY AND RESULTS
Models, datasets, and evaluation setup. We study the most widely adopted Transformer-based models, including OPT (Zhang et al. 2022b) and LLaMA (Tourron et al. 2023) model families. In line with prior work (Xiao et al. [ Frantar & Alistarh 2023 Sun et al. 2023), we fine-tune pre-trained models and evaluate perplexity on the WikiText2 (Merity et al., 2017) dataset. The pre-trained L.MSs used in our experiments are base ggeneral-purposte models, not instruct-tuned variants. addition, we assess non-orthogonality across different metries of ViT (Dosovitskiy et al.) 2021) and ResNet (He et al. 2016) on ImageNet-IK (Deng et al. 2009). In all experiments, we designate the dense FP32 configuration as the primary baseline.
Our experiments span diverse range of configurations to validate our mathematical findings, including various variants of max-scaled formats, such as INT8 quantization with per channel scaling (Dettmers et al., 2022), HBFP8/6 (Drumond et al. 2018a), and MXFP8/6 (Rouhani et al.) 2023b We primarily study magnitude-basee 50% unstructured and 2:4 structured sparsity with sparsity-aware fine-tuning. We define N:M structured sparsity as following: in every group of M consecutive weights, at most N weights can have non-zero values. We evaluate the impact of higher compression ratio on VIT-B/16 and ResNet-50 by applying 75% unstructured sparsity, 1:4 structured sparsity, and HBFP4. Detailed results can be found in Appendix G and O We also explore
post-training one-shot sparsity methods like SparseGPT (Frantar & Alistarh 2023) and Wanda (Sun) et al.. 2023) in Appendix
In our experiments, we evaluate the impact of each compression method by analyzing the variations in perplexity and cross-entropy loss compared to the baseline, thereby focusing on the cumulative error in the model output. Additionally, we analyze the errors of intermediate layers to support our mathematical analysis, as detailed in Section 4.3 and Appendix [
Note that we focus on cross-entropy loss for ViT-B/16 and ResNet-50 instead of classification accuracy. This is because accuracy remains unaffected as long as the most likely label remains unchanged, regardless of its absolute value. However, our primary metric for orthogonality threshold is the aggregated eиrors introduced in the model output distribution. Table g in Appendix G and Table 12 in Appendix O present orthogonality threshold on additional metrics.
Experimental setup. We exclusively sparsify and/or quantize layers with trainable parameters. Specificallyy, we target all linear layers in LLMs (excluding the Im-head or embedding layers following the literature (Frantar et al.) [2022] Frantar & Alistarb 2023 Lee et al., 2024)), and all linear and convolution layers (including the initial embedding layer) VIT-B/16 and ResNet-50. These layers collectively constitute approximately 99% of the total parameters. In all experiments, we prune the weights while keeping the activations dense. Both weights and activation tensors are quantized before matrix multiplication operations. For OPT, LLaMA, VIT, and ResNet fine-tuning, fine-tuning on a dense FP32 pre-trained model, recomputing sparsity masks at each iteration. In experiments involving sparsity followed by quantization (S+2)), we apply one-shot quantization to sparse fine-tume tne-tuned models. Conversely, for experiments with the reverse order (Q+S), we directly fine- tune the model in quantized and sparsified manner. At each iteration, we quantize activations and weights while applying sparsity to weight tensors. We validate the effectiveness of these compression recipes through an ablation study, the details of which presented in Appendix [D] To ensure fair comparison, we maintain uniform hyperparameters across various number formats for a given model and sparsity type (details in Appendix [B.. We present limited sensitivity study on the initial seed number in Table in Appendix H. A summary of our experimental setup is presented in Table 6
employ sparse
4.1 EMPIRICAL STUDY 1: ORDER OF SPARSITY AND QUANTIZATION
This section presents empirical evidence demonstrating that applying sparsity before quantization leads to better perplexities compared to the reverse order. These results are aligned with the math- ematical analysis in Section 3.1. Table presents perplexities for OPT-125M and LLaMA-2-7B under various numerical formats and sparsity types considering both orders of transformations. "50%" denotes unstructured sparsity, while "2:4" represents variant of N:M structured sparsity. In the FP32 columns, only one perplexity for each compression order is reported because no quantization is applied. The best results for each (sparsity type, numerical format) pair are highlighted in bold.
7

---

Published as a conference paper at ICLR 2025
Table 1: Model perplexities on WikiTex12 for combined sparsity and quantization. The best results for each (sparsity type, number format) pair are highlighted in bold.
OPT-125M
LLaMA-2-7B
Spansity type
HDFFS
INTS
0% (Lhense)
27.65
28.06
28.45
26.01
27.81
29.91
5.12
5.15
5.17
5.16
5.12
5.24
50%
8-+
29.94
34.71
30.22
31.13
36.39
31.20
35.60
37.48
30.46
40.36
32.51
6,31
6.94
8.13
6.40
8.47
6.38
9.32
6,32
9.86
10.20
6.51
2:4
31.89
45.06
32.76
44.16
33.99
42.25
33.41
32.25
46.57
55.64
34.58
9.30
14,05
9.37
14.35
9.35
14,50
9.32
14.98
9.39
10.68
18.64
Table 2: Model perplexities and CE loss for combined sparsity and quantization. The numbers in the parentheses show the difference in perplexity/CE loss between the sparse and dense configuration.
OPT-125M
OPT-6.78
LLaMA-2-7B
LLaMA-3-88
VIT-B/16
Sparsity type
Numaher format
Orth. threshold
Orth. threshold
Orth. threshold
Orth. threshold
CF
Orth.
thresbold
FP32
27.65
10.85
5.12
0.703
INT8
28.06
10.95
5.19
5.63
0.706
MXPP8
28.45
11.25
5.17
5.62
0.722
0%
MXFP6
28.01
11.02
5.16
5.62
0.715
HRFPP
5.56
HBFP6
29.91
FP32
29.94
11.30
6,31
10.09
0.723
INTS
36.22 (1)
30.35
11.37 (4847)
11.39
6.94 (4:35)
6.38
10.85 (530)
10.19
0.728 (wlattt
0.725
MXFPS
31.13 (12000
30.74
11:74
6.40
10.34
0.745 (408220
0.742
MXFP6
$1.20
30.30
11.53
11:44
6,38
6.35
10.15
10.18
0,734
0.735
30.46 (4240)
30.18
11.31 (4AA)
6,32
6.31
10.12
10.12
0.724 (4UD0
0.723
HBFP6
32.51 (62.0))
11.94 (970
(+0.7)
10.55 14400
0,736 (40000)
0.737
FP52
31, 29
15
9:30
13.07
0.759
INTS
32, (+770)
32.30
15.61 (4400)
15.57
9,37 (-418)
9.37
13.23 (+180)
13.17
0.762 (HAA24)
0.761
MXFPS
33.99 (45.5))
15.70 (+4-5)
(-410)
13.35 (-777)
0.781 (40,59))
0.777
MXFP6
33.41
32.25
15.95
15.64
9.32
9.34
13.20
13.16
0.770
0.771
HBFP8
32.25 (4444)
32,05
15.57 (6400)
15.50
9.39 1-427)
9.31
13.11 (4030)
13.1
0.760 (HALD))
0.75y
HBFP6
(4477)
16.98 (457))
(5440
13.64 (+77))
0.774 (H9256))
0,773
We evaluate the order of transformations across different configurations: sparsity followed by quanti- zation (S-Q) and quantization followed by sparsity (Q-S). Configurations with lower perplexities are highlighted in bold. Consistently, we observe that the S-Q order yields better perplexities across all numerical formats for magnitude- based sparsity. As discussed in Section 3.1 in the case of Q-S, quantizing a tensor can alter the order of its elements due to changes in magnitudes. If magnitude- sparsity prunes elements of the quantized tensor that were originally larger before quantization, the error of the combination may exceed the sum of the errors caused by each transfor- mation individually. This compounded error further propagates through subsequent dot-product and vector operations, impacting overall model performance.
based
4.2 EMPIRICAL STUDY 2: NON-ORTHOGONALITT BETWEEN SPARSITY AND QUANTIZATION
This section demonstrates that combining sparsity and quantization results in additional error, sur- passing the sum of their individual errors. Table 2 presents perplexities for OPT-125M, OPT- 6.7B, LLaMA-2-7B and LLaMA-3-8B on WikiText2, and cross-entropy (CE) loss for ViT-B/16 on ImageNet-Ik lk for various combinations of numerical formats and sparsity types. Following our conclusions regarding the order of transformations, we only report results for S-Q. We compute the orthogonality threshold for each combination by summing the individual errors from sparsity and quantization relative to the baseline dense FP32 model, using Equation 12 Each model's 's performance compared against this bound, with superior results highlighted bold. In the majority of configu- rations, perplexity and cross-entropy loss values exceed the orthogonality thresholds, validating the non-orthogonality of sparsity and quantization.
We mathematically show (Section 3.2) that the additional error introduced by combining sparsity and quantization significantly depends the values of quantized activation tensors and the quantization error of sparsified weights. This error tends to amplify through successive dot products and vector operations. Consequently, the quantization error affects the additional error caused by the combination
8

---

Published as a conference paper at ICLR 2025
more than the sparsity error. The gap between the model's performance and the orthogonality threshold is minimal for numerical formats with minimal performance decrease, whereas larger errors observed for formats with larger emror. For instance, HBFP6 results in 2 2.26 increase in perplexity for OPT-125M, while its combination with 50% unstructured sparsity leads to a 4. 86
increase.
Our analysis reveals that both model size and compression ratio significantly influence the additional error introduced by combining sparsity and quantization. Larger models exhibit greater tolerance to these compression methods, resulting in lower additional errors. Moreover, formats with mini- mal quantization errors (e.g. MXFP8). and sparsity types with minimal sparsification errors (e.g. unstructured sparsity) lead to lower additional errors, even for smaller models. The effect of high quantization error is more pronounced for sparsity types known for higher errors. For instance, HBFP6's combination with 2:4 sparsity causes a 6. 9.3 perplexity increase for OPT-125M and 6. 12 perplexity increase for OPT-6.7B. 7B. In contrast, combining HBFP6 with unstructured sparsity results in smaller increases of 4. 86 and 0. 79 for the respective models.
We also observe instances where the orthogonality threshold is slightly higher than the actual perplexity: (a) INT8 with 50% unstructured sparsity for both OPT models, (b) MXFP8 with 2:4 sparsity for OPT-6.7B, (c) MXFP6 with 2:4 sparsity for LLaMA-2-7B, and (d) MXFP6 50% unstructured sparsity for LLaMa-3-8B. Although these occurrences do not consistently correlate with specific formats, sparsity types, or model sizes, they do not contradict our mathematical analysis, which primarily concerns upper bounds of errors and does not entirely rule out orthogonal cases. Furthermore, orthogonal configurations still result in larger errors compared to applying either sparsity or quantization alone. Our mathematical analysis (Theorem 3.10) indicates that there exists at least one occurrence where the orthogonality is not preserved. This underscores the need for careful examination when applying these compression methods together, as they do not guarantee high accuracies. Delving into cases where orthogonality is preserved falls beyond the scope of this paper, and we leave it for future work.
We observe that although the cross-entropy loss for ViT-B/16 is higher than the calculated orthog- onality threshold in most cases in Table the difference is relatively small. We hypothesize the reason behind this behavior is due to the fact that ViT-B/16, being vision model that operates on images, more robust to sparsity and quantization errors than LLMs that operate on text. Hence, the sparsity and quantization levels shown in Table 2 are not sufficient to induce large errors, and hide the non-orthogynality of sparsity and quantization. To test our hypothesis, we increase the compression further by employing 75% unstructured sparsity, 1:4 structured sparsity and HBFP4 on V (Appendix GG. The results show that the difference between the baseline cross-entropy loss and the calculated orthogonality threshold for these cases increases significantly, validating the non-orthogonality of sparsity and quantization. This increase in cross-entropy loss also translates to significant non-orthogonal drop in accuracy. Additionally, we run the same experiments on CNN-based vision model, ResNet-50 (Appendix Oj. and the experimental results corroborate these findings.
4.3 ABLATION: ERROR PROPAGATION ACROSS LAYERS
In this section, we study eтror propagation across the layers of a deep neural network.
While our mathematical study provides insights into how sparsity and quantization introduce errors at the dot- dot-product and tensor levels, mathematically estimating the cumulative effect of these errors on the model's final loss or performance non- non-trivial. Therefore, we perform a layer-wise empirical analysis to verify that the order of applying sparsity and quantization not only affects per-layer errors but also significantly impacts overall model's 's accuracy. By inspecting the intermediate outputs of the pre-trained OPT-125M model, show that the error accumulation increases with the layer index and that the choice of the order directly influences the final model performance.
We apply quantization and magnitude-based sparsity to all linear layers of the pre-trained OPT-125M model in a zero-shot manner. We feed a sample from the test subset through both the compressed model and the corresponding full precision dense model, and we measure the difference in Feed- Forward outputs for each Transformer block. We repeat this experiment for both the S-Q and Q-+S orders, and compare the La errors at each layer. The results shown in Figure

---

Published as a conference paper at ICLR 2025
350
0+5
300
S
250
-50
100
50
42
+2
+3
+4
+5
+6
+7
+5
-9
+10
+.1
+12
Layer ID
Figure 1: Cumulative error for the combination of 2:4 structured sparsity and HBFP6 quantization
As illustrated in the figure, the per-layer error increases consistently with the layer index, regardless of the order of transformations, and reaches its peak in the final layer. However, we observe that the S S-QQ order produces significantly lower errors at each intermediate layer compared to the reverse order. This pattern indicates that the error introduced by Q-S accumulates more rapidly as layers deepen, highlighting the detrimental effects of applying quantization before sparsity.
DISCUSSION
Our mathematical analysis and experimental results offer multiple insights for ML model practitioners. First, our analysis demonstrates risk-free method to improve model performance, measured by lower perplexity and/or higher accuracy, through choosing the optimal ordering of compression operations for any max-scaled number format and magnitude-based pruning scheme. This contribution is particularly important in the current ML landscape, where sparsity and quantization are pivotal methods for reducing the memory footprint and bandwidth requirements of state-of-the-art LLMs. Second, we show that calculating the orthogonality threshold offers a close enough estimate of model performance (e.g. accuracy, perplexity, etc.) under conditions of sparsity and quantization. This bound can streamline the search for optimal sparse-quantized model configurations by effectively narrowing the search space. There an inherent trade-off between the hardware benefits of various sparse-quantized configurations and the achieved model performance. Quantization bit-width and sparsity level are key factors influencing the memory and bandwidth requirements for serving these models. For example, at 50% sparsity level, 8-bit and 6-bit quantization result in total reductions in memory footprint and bandwidth requirements by x and 10. 7 x, respectively.
Ideally, practitioners aim to maximize compression (inerease sparsity ratio and/or reduce the average bit-width per element). Our analysis clucidates the individual and combined impacts of these factors across range of recent large models, providing practical guidelines achieve the highest compression without compromising model performance. Typically, 8-bit quantization with any max-scaled number format can serve as a direct replacement for FP32 when combined with any form of sparsity in the optimal order (S+Q). As discussed in Section 4 certain models exhibit sensitivity sub-8bit number formats and structured sparsity combinations, even when applied in an optimal order. In scenarios where improvements in arithmetic density (TOPS/mm²) and memory footprint justify slight reduction in model performance, such as the deployment of large models on edge devices, these combinations may still be viable.
In this work, we do not consider heterogeneous sparsity and quantization schemes, where the sparsity fraction and quantization bit- hit-width vary across layers and differ between activation and weight tensors.
Such approaches (Rouhani et al. 2023b; Ma et al. 2024, Harma et al., 2022: Evei et al., 2020b) have demonstrated be effective in maintaining model accuracy perplexity while improving compressing ratio. However, these schemes may not be hardware- hardware-friendly. introduce noticeable overhead, and impractical to implement off-the-shelf hardware platforms (e.g. GPU, TPU). We leave the investigation of the interactions between these heterogeneous sparsity and quantization schemes for future work.
10

---

Published as a conference paper at ICLR 2025
CONCLUSION
We provide a comprehensive analysis of the interplay between sparsity and quantization DNNs, showing that applying sparsity before quantization (S-+Q) minimizes additional errors and yields better model accuracy. Moreover, our mathematical analysis and extensive empirical study with large language models (OPT, LLaMA), vision transformers (ViT), and convolutional neural networks (ResNet) demonstrate that sparsity and quantization are non-orthogonal and their combined use can adversely affect model accuracy. Our findings provide valuable insights for optimizing the compression of large models while preserving accuracy.
ACKNOWLEDGMENTS
We extend our gratitude towards Zhifeng Chen, Cliff Young. James Laudon, and Shashwat Shrivastava for reviewing the paper and providing insightful feedback. We also thank the extended team at Google DeepMind, who enabled and supported this research direction. This work was partially supported by the SNSF project "Unified Accelerators for Post-Moore Machine Learning" (200021 [000211212757] and Microsoft Research PhD Fellowship. A part of this paper was the result of a research project supported by SK hynix Inc.

---

Published as a conference paper at ICLR 2025
REFERENCES
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi. Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Dehbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Damiele Maz- zotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models. CoRR, abs/2311.16867, 2023. doi: 10.48550/ARXIV231....
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Joannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap. Angeliki Lazaridou. Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham. Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty. Eli Collins. Clemens Meyer, Eliza Rutherford, Erica Morcira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roclofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski. and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/2312.1/0000 2023. doi: (0.48550/RXXIV23122. 1105. URL 231.. 11805
https: / /cci. org/ 10.
Abhimanyu Rajeshkumar Bambhaniya, Amir Yazdanbakhsh, Suvinay Subramanian, Sheng- Chun Kao, Shivani Agrawal, Utku Evei, and Tushar Krishna. Progressive gradient flow for robust N: M sparsity training in transformers. CoRR, abs/2402.04744, 2024. doi: 10.46550/ARXIV2400..07777 URL https://doi
org/10 481111.485//aaa. (05550//aXiv... 2402. 04744
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Berbert-Vosss, Gretehen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language Models are Shot Learners. In Hugo Larochelle, Marc" Aurelio Ranzato, Raia Hadsell, Maria-Floo Florina Balcan, and Hsuan-Tien Lin (eds.), NeurlPS, 2020.
Steve Dai, Rangha Venkatesan, Mark Ren, Brian Zimmer, William Dally, and Brucek Khailany. Vs-quant: Per-vector scaled quantization for accurate low-precision neural network inference. Proceedings of Machine Learning and Systems, 3:873-884, 2021.
Bita Darvish Rouhani, Daniel Lo, Ritchic Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vino- gradsky. Sarah Massengill, Lita Yang. Ray Bittner, Alessandro Forin. Haishan Zhu, Taesik Na, Prerak Patel, Shuai Che, Lok Chand Koppaka, XIA SONG, Subhojit Som, Kaustav Das, Saurabh T, Steve Rein- hardt, Sitaram Lanka, Eric Chung, and Doug Burger. Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (cds.), Advances in Neural Information Processing Systems, volume 33, pp. 10271-10281. Curran Associates, Inc., 2020a. URL http::::://///coccccceeecc www.innnnnnnnnusssoee.e.o..e www.:nn.uriscoooooooo.. www.eiis..ccpppppp...eeo httpp/sppccpppppe///.////// 2020://12200//1111///// fbd 3 Paper.
Bita Darvish Rouhani, Daniel Lo, Ritchic Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, et al. Pushing the limits of narrow precision infereneing at cloud scale with microsoft floating point. Advances in neural information processing systems, 33:10271-10288 2020b.
Bita Darvish Rouhani, Nitin Garegrat, Tom Savell, Ankit More, Kyung-Nam Han, Mathew Zhao, Ritchic amd Hall, Jasmine Klar, Erie Chung, Yuan Yu, Michael Schulte, Ralph Wittig, lan Bratt, Nigel Stephens, Jelena Milanovic, John Brothers, Pradeep Dubey, Marius Cornea, Alexander Heinecke, Andres Rodriguez, Martin Langhammer, Summer Deng. Maxim Naumov, Paulius Micikevicius, Michael Siu, and Colin Verrilli. OCP Microscaling (MX) Specification. Open Compute Project, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenct: A large-scale hicrarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Iccc, 2009.
Tim Dettmers, Mike I Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int80:: 8-bit matrix multiplication for transformers at scale. CoRR, abs/2208 07339, 2022. doi: 10.48550/ARXII.2200..77 073339 URL. https
//di. 10. 48550 artov./rivvvvv. 2208. 07339
Shaojin Ding, David Qiu, David Rim, Yanzhang He, Oleg Rybakov, Bo Li, Rohit Prabhavalkar, Weiran Wang. Tara N. Sainath, Shivani Agrawal, Zhonglin Han, Jian Li, and Amir Yazdanbakhsh. Usm-lite: Quantization and sparsity aware fine-tuning for speech recognition with universal speech models. CoRR, a085231, 2023. doi: 00.48500/ARXIV.2312 3.555553. (tt.phttt http://oooooggg///// 00. 48550/
2312..20. 08553
12

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Hcigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: /tttps://cpenrevieer net forum? d- Ybc.Ficbboooo cbNTT
Mario Drumond, Tao Lin, Martin Jaggi, and Babak Falsafi. Training DNNs with Hyhrid Block Floating Point. arXiv; / 804.01526 fcs, stat), December 2018a. httphh:://aww..or//aaaaaaaeaeeee 1804.01526.
01526 arXiv:
Mario Drumond, Tao Lin, Martin Jaggi, and Babak Falsafi, Training dons with hybrid block floating point. Advances in Neural Information Processing Systems, 31, 2018b.
Utku Evei, Trevor Gale, Jacoh Menick, Pablo Samuel Castro, and Frich Elsen. Rigging the lottery: Making all tickets winners. In Procedinggs of the 37th International Conference Machine Learning, ICMI, 2020, 13-18 July 2020, Virtual Event, volume 1 119 of Proceedings Machine Learning Research, pp. 2943-2952. PMLR, 2020a. URL httpttpppp/ppocceddrssssss ml wttps/pe1eeeeeeeeeeeeeee
html
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen, Rigging the lottery: Making all tickets winners. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2943-2952. PMLR, 13-18 Jul 2020b.
mir wwwps///vl//eeeeeeeeeeeeeeeeeeee
html
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representattions ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
net/ forum www.mmmdddllll b3RCF7
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475 4488, 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt. Sivan Sabato. and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 16623-1037.. PMLR, 2023, URL https::/pprceeee
ml press wttp///2222/22222
Elias Frantar, Saleh Ashkboos, Torsten Hoetler, and Dan Alistarh. GPTQ: accurate post-training quantization for generative tre-trained transformers CoRR, ahs/2210.11332. 2022. doi: 10.48550/ARXIV.221100 17323. URL
https:://ci..o wtwp1/...rrrr.... 48550//45500000 2210 17323
Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, and Utku Evei. Scaling laws for conneced foundation models. arXiv preprint (xXi:2330900822 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. org rwtp:///ooorrdeeeooo retpg////ooood/////2222222222222
126038602
Google. puter.
Enabling next-generation workloads:
Announcing tpu v5p and hypercom- httplccgg///oouss//eooo hwtpppooccccttttaacccaiieeeeee wel--maacinneeeee learning/
int wrw.ooicciinoooooooooooooo------ -hyrrsp--tttadi----yyyyyroommmtr 2023.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. Advances in neural information processing systems, 29, 2016.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:15/0.00/44 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015b.
Simla Burcu Harma, Ayan Chakraborty, Babak Falsafi, Martin Jaggi, and Yunho Oh. Accuracy boosters: Epoch-driven mixed-mantissa block floating-point for dan training, arXiv preprint arXiv:2211. 2022.
Bahak Hassibi, David G Stork, and Gregory Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pp. 293-299. IEFE, 1993.
13

---

Published as a conference paper at ICLR 2025
Benjamin Hawks. Javier Duarte, Nicholas Fraser, Alessandro Pappalardo, Nhan Tran, and Yaman Umuroglu. Ps and qs: Quantization aware pruning for efficient low lateney neural network inference. Frontiers in Artificial Intelligence, 4:676564, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training neural networks. ArXiv, abs/2102.00554, 2021.
semantt cschola ar. .rgg http://ooosss:::111000000 ID:
Peng Hu, Xi Peng. Hongyuan Zhu, Mohamed M Sabry Aly, and Jie Lin. Opq: Compressing deep neural networks with one-shot volume 35, pp. 7780-7788, 2021.
In Proceedings of the AAAI conference on artificial intelligence,
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner. Joseph Naor, and Daniel Soudry. Accelerated sparse neu- ral training: A provable and efficient method to find N: M transposable masks, In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin. Percy Liang. and Jennifer Wortman Vaughan (eds.), Advances in Neural Infor- mation Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 21099 21111, 2021. URL/ittppe://proceedee http://ceeeeeiinnnnnree.oeee neurips cc/
2021 0ttp:///2220/aaaa//00055555000
dbbbdb wtw.:a/600000aaa2------a-------
html
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard I Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée I Lacroix, and William EI Sayed. Mistral 7b. CoRR, abs/2310.06825. 2023. doi: 104455WARRXV.2330006222 URL
https://dof http///ii.rrrri////ee /00/ 48550/ a 2310. 06825
Hyeong-Ji Ju Kang. Accelerator-awan pruning for convolutional neural networks. IEEE Trans. Circuits Syst. Video Technol. 3007)000703022222 2103. 2020. doi: 10.1109/TCSVT.2019 29116744 URL tttpp:///// org/
10.
Sheng- Chun Kao, Amir Yazdanbakhsh, Suvinay Subramanian, Shivani Agrawal, Utku Evei, and Tushar Krishna. Training recipe for n: M structured sparsity with decaying pruning mask. arXiv preprint arXiv:2209.07617 2022.
theng-Chun
Alexander Kozlov, Ivan Lazarevich, Vasily Shamporov, Nikolay Lyalyushkin, and Yury Gorbachev. Neural network compression framework for fast model inference. In Intelligent Computing: Proceedings of the 2021 Computing Conference, Volume 3, pp. 213-232. Springer, 2021.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal hert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint arXiv:2203.07255, 2022.
Andrey Kuzmin, Markus Nagel, Mart van Baalen, Arash Behboodi, and Tijmen Blankevoort. Pruning vs quantization: Which is better? In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural Information Processing Systems 36: Annual Con- ference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, Decem- her 10 - 16, 2023, 2023. URI. http:h//ppppprsssep/ep ss .s. cc/paper fi Files/ paper pappp//p220222 /2023/hash/ c4 2.48bc80aa. 3at.Bbaaaaacccbbdddd 1 2d1 cc 076831 9- Abst Con html
Mike Lasby, Anna Golubeva, Utku Evei, Mihai Nica, and Yani Ioannou. Dynamic sparse training with structured sparsity. arXiv preprint arXiv:2305.022299 2023.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing xystems, 2, 1989.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Funhyeok Park. OWQ: outlier-aware weight quantization for efficient fine-tuning and inference of large language models. In Michael J. Wooldridge, Jennifer G. Dy. and Sriraam Natarajan (eds.). Thirty-Eightt AAAI Conference Artificial Intelligence. AAAI 2024. Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Sympositm on Educational Advances in Artificial Intelligence, EAAl 2014, February 20-27, 2024. Vancouver. AAAI Press, 2024, doi: 10. (0..1699/AAAL.33381222..
Canada, pp. org 10g//1..1 (009/aaaa... v38112 29237
https: //oo
Zhuoban Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. Train large. then compress: Rethinking model size for efficient training and inference of transformers. In International Conference on machine learning, pp. 5958-5968, PMLR, 2020.
14

---

Published as a conference paper at ICLR 2025
Zichang Liu, Jue Wang, Tri Dao. Tianyi Zhou, Binhang Yuan, Zhao Song. Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Rc, al. Deja vu: Contextual sparsity for efficient llms inference time. In International Conference on Machine Learning, pp. 22137-22176 PMLR, 2023.
Xudong Lu, Aojun Zhou, Yuhui Xu, Renrui Zhang, Peng Gao, and Hongsheng Li. SPP: sparsity preserved parameter-effiient fine-tuning for large language models. In Forty-first International Conference on Ma- chine Learning, JCML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL [RLaatpp latps
sparsity-peesereed
//openreview.co net http://wwwwnet.ttoommeeeeeeeeeee jttee///ooommmindroooooeee I9G00
Yucheng Lu, Shivani Agrawal, Suvinay Subramanian, Oleg Rybakov, Christopher De Sa, and Amir Yaz- danbakhsh. STEP: learning N: M structured sparsity masks from scratch with precondition. In An- dreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sahato, and Jonathan Searlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 22812-22824, PMLR, 2023, URL https::/pprcceeding mir. press wwwpress//////ud/// 2202//222/2222tr
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang. Shaohan Huang. Li Dong, Ruiping Wang. Jilong Xue, and Furu Wei. The era of 1-bit Ilms: All large language models are in 1.58 bits, 2024.
Stephen Merity, Caiming Xiong. James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL 72udxx
nct forum?
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière. Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette- Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan. George Tucker, George- Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney. Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, and et al. Gemma: Open models based on gemini research and technology. CoRR, abx/2403.08225,, 2024, doi: 10.48550AARTT.22430000555 URL. https://dci. http///ii.rgg//// wtw..:..//.....2222222
08295
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Gins- burg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu, Mixed Precision Training. arXiv:1710.03740 [cs, star). February 2018. URL http:://axxii http://axxxxv.o...//////o/ www.aabal...../. 11t./11110.0740 1710.03740.
arXiv:
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi. Stuart F. Oberman, Mohammad Shoeybi, Michael Y. Siu, and Hao Wu. FP8 formats for deep learning. CoRR, abs/2209.05 004333 2022. doi: 10.48550/acxiv.2209. 054333
URL htttttts:://///m htttp:://do..rg//11 wtt.ooogg1....././.....
485544855511100 2209. 05433
Paulius Micikevicius, Stuart Oberman, Pradeep Dubey, Marius Cornea, Andres Rodriguez, Ian Bratt, Richard Grisenthwaite, Norm Jouppi, Chiachen Chou, Amber Huffman, Michael Schulte, Ralph Wittig, Dharmesh Jani, and Summer Deng. OCP 8-bit Floating Point Specification (OFP8), Open Compute Project, 2023.
Microsoft. MicroXcaling: Library for Microservices Autoscaling, httpp://gitthui wwt.:tttoooccooommmoommmooomm cresoft. mi croxcal ig 2024.
Asit K. Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosie, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. ArXiv, abs/2104,08378, 2021a. URL
Asit K. Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosie, Dusan Stosie, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelcrating sparse deep neural networks. CoRR, abs/2104.08378 2021b. URL
https://arxiie http://wwww....///////.....000
Nvidia. Nvidia ampere architecture whitepaper.
Technical report. NVIDIA, 2021.
URL
.ht fttt::://Images..vviiaa
dam/ en Saamm/ennz//Sann ww/Solutions/aaat cata- cate:/tttaaaaceeae//eaeaeea
nvidia- aar.aaaa-mmee--rrrcieettteeeee whitepeeettteee-etpppeeeee... pdf
Nvidia. Nvidia h100 tensor
core
gpu
architecture.
Technical report, NVIDIA,
2022.
URL
.ttpps: www.:/awwwddc.etteeeeiieeoo com wp-cont
Upa sploods//022/00 wwwpp:/eeeeepppppeaeeapaaa.aeeee tepaper
nvidia ampere hhgprreeeacrtttcurreeehttteeeeee
15

---

Published as a conference paper at ICLR 2025
Nvidia. Nvidia tensor cores: Unprecedented acceleration for generative ai. httpp::/ww..wiieeeee
en http:///2ff...ccceeeeeoc//orooor 2024.
OpenAL. (P7-miichhicall report. CoRR, 087777 2023, doi: 10448550/ARX11222000.07777 URL
httpps://dci...
08774
Jun-Hyung Park, Kang-Min Kim, and Sangkeun Lee. Quantized sparse training: a unified trainable framework for joint pruning and quantization in dnns. ACM Transactions on Embedded Computing Systems (TECS), 21 (5):1 2022.
Mi Sun Park, Xiaofan Xu, and Cormac Brick. Squantizer: Simultaneous learning for both sparse and low- precision neural networks, 2019, URL
Jeff Pool and Chong Yu. Channel permutations for N: M sparsity. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021. December 6-14, 2021, virtual, pp. 13316-13327, 2021. URL https:://proceedinnn (ae.aa1146569--8
nwurips.cc.. htmll html
paper/ 202 2twt/hashhhhhee hash/
Bita Darvish Rouhani, Ritchie Zhao, Venmugil Elango, Rasoul Shafipour, Mathew Hall, Maral Mesmakhos- roshahi, Ankit More, Levi Melnick, Maximilian Golub, Girish Varatkar, Lai Shao, Gaurav Kolhe, Dimitry Melts, Jasmine Klar, Renee I. L''Hureeux, Matt Perry, Doug Burger, Fric S. Chung, Zhaoxia (Summer) Deng, Sam Naghshineh, Jongsoo Park, and Maxim Naumov. With shared micrnexponents, A little shifting goes long way. In Yan Solihin and Mark A. Heinrich (eds.), Proceedings of the 50th Annual International Sympo- sium on Computer Architecture, ISCA 2023, Orlando, FL, USA, June 17-21, 2023, pp. 83:1-83:13, ACM, URL htt.::ttpss//cm /cc.. org/ 10. 10e.1114/3355 000444//5577711 3589351
2023a. doi: 10.1 00.144//57937133589999
Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Comea,, Eric Dellinger, Kristof Denolf, Dusan Stosic, Venmugil Elango, Maximilian Golub, Alexander Heinecke, Phil James- Roxby, Dharmesh Jani, Gaurav Kolhe, Martin Langhammer, Ada Li, Levi Melnick, Maral Mesmakhosroshahi, Andres Rodriguez, Michael Schulte, Rasoul Shafipour, Lei Shao, Michael Y. Siu, Pradeep Dubey, Paulius Micikevicius, Maxim Naumov, Colin Verilli, Ralph Wittig. Doug Burger, and Eric S. Chung. Microscaling data formats for deep learning. CoRR, abs/2310..0SS 2023b. doi: 11.48550/RXIV.23100000
URL[attps wttp://o......... 100 48w50/a:Xlv..222//.// 2310. 10537
Victor Sanh. Thomas Wolf. and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning, Advances in neural information processing systems, 33:20378 203799822999..811 2020.
Teven Le Scan, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang. Benoît Sagot, Niklas Muen- nighoff. Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan- Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifcoluwa Adelani, and al. BLOOM:: A 176h- open-access multilingual language model. CoRR, abs/2211 2022. doi: 4ww5.//...i..222222 05100
01.48550/ARXIV.2211.0
Uttpttps:
Mingjee Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for [ttts:
large language models. CoRR, abs/2306. 11695, 2023. doi: 10448550ARRXV2233611122
//doi.org wrg.//..g...... 11885500aaiii0. 2306. 11695
Wei Sun, Aojun Zhou, Sander Stuijk, Rob G. J. Wijnhoven, Andrew Nelson, Hongsheng Li, and Henk Corporaal. Dominoscarch: Find layer-wise fine-grained N: M sparse schemes from dense neural networks. Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Perey Liang, and Jen- nifer Wortman Vaughan (eds.). Advances in Neural Information Processing Systems 34: Annual Con- ference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, vir- tual, pp. 20721-20732, 2021. URL
wew.eiinnnnnnnnnns........ cc/paper/ ptmpper/22220 /022/ 12022/haa 2022/hashh hash/
Aaquib Syed, Phillip Huang Guo, and Vijaykaari Sundarapandiyan. Prune and tune: Improving efficient pruning techniques for massive language models. In Krystal Maughan, Rosanne Liu, and Thomas F. Burns (eds.), The First Tiny Papers Track or ICLR 2023, Tiny Papers @ ICLR 2023, Kigali, Rwanda, May 5, 2023. pdd diccki (c 7n87
OpenReview.nes 2023. URL htt.:ttts://operreeeeee
16

---

Published as a conference paper at ICLR 2025
Vivienne Sze, Yu-Hssin Chen, Tien-Ju Yang. and Joel S. Emer. Efficient Processing of Deep Neural Net- works, Synthesis Lectures on Computer Architecture. Morgan & Claypool Publishers, 2020. ISBN 978-3- 031-00638-8, 8, doi: 10. 10.2200/S01004EDI V01Y20200CAC0500 htL/:bttt:///o..rr//// /200
101004001001120000000000
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton- Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut I Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinct, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu. Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan. Melanie Kambadur, Sharan Narang. Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuneds chat models. CoRR, abs/2307.0928s 2023. doi: 10.48550/ARXIV.33000 092888. URL htt.tpp:/dbl.ooggg/// /ttp:/10.00550022222232.3..00.
Mart Baalen, Andrey Kuzmin, Suparna S. Nair, Yuwei Ren, Eric Mahurin, Chirag Patel, Sundar Subramanian, Sanghyuk Lee, Markus Nagel. Joseph Soriaga, and Tijmen Blankevoort. FP8 versus INT8 efficient deep learning inference. CoRR, abs/2303.1799 2023. doi: 10.48550/ARXTV.22000 17551.
URL https
//doi.org/11 /e////0..8555//x11111 2303. 17951
Naigang Wang. Chi-Chun (Charlie) Liu, Swagath Venkataramani, Sanchari Sen, Chia- Yu Chen, Kaoutar El Maghraoui, Vijayalakshmi Srinivasan, and Leland Chang. Deep compression of pre-trained transformer models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho. and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 2022, 2022.
Shibo Wang and Pankaj Kanwar. B6 The secret to high performance on Cloud TPUs, August 2019.
Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han. Apq: Joint search for network architecture, pruning and quantization policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2078 2087, 2020.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, 29, 2016.
Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, and Yuxiong Hc. Understanding int4 quantization for transformer models: Latency speedup, composability, and failure cases, 2023.
Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings PMLR, 2023. URL httpp::/proocedaah
of Machine Learning Research, pp. press/ wttas//v22/xxx22 1w..00//xx2222..... htmm
mlr
Zhuliang Yao, Shijie Cao, Wencong Xiao, Chen Zhang, and Lanshun Nie. Balanced sparsity for efficient DNN inference on GPU, In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty- First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances Artificial Intelligence, EAAI 2019, Honolulu, Hawaii USA, January 27 February 1, 2019, pp. 5676-566 5683.666336600 AAAI Press, 2019, doi: 10. 10.1609/AAAL.V33111..0. httos: http:://dcc..ccoc crg. 0. 60 aaa.i.ala..... v33101 01..0111100 33015676
URL
Thomas Ych, Max Sterner, Zerlina Lai, Brandon Chuang, and Alexander Thler. Be like water: Adaptive floating point for machine learning. In International Conference on Machine Learning, pp. 25490-25500. PMLR, 2022.
Chong Yu, Tao Chen, Zhongxue Gan, and Jiayuan Fan. Boost vision transformer with gpu ari-friendly sparsity and quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23658-226666 2023.
Po-Hsiang Yu, Sih-Sian Wu, Jan P Klopp, Liang-Gee Chen, and Shao-Yi Yi Chien. Joint pruning & quantization for extremely sparse neural networks. arXiv preprint arXiv:2010.01892, 2020.
17

---

Published as a conference paper at ICLR 2025
Ofir Zafrir, Guy Boudoukh, Peter Izsak. and Moshe Wasserblat. QSBERT: quantized 8bit BERT. In Fifih Work- shop Energy Efficient Machine Learning and Cognitive Computing NeurlPS Edition, EMC2@ EMC2@NeuIIPS NeurIPS 2019, Vancouver, Canada, December 13, 2019, pp. 36 39. IEEE, 2019, doi: 10. 1001000MMC2N 2019.00016. UtLpttts:://ol
org 10. 1109/ EMC2 WWPS5332222 2019. 9.00166
Ben Zandonati, Glenn Bucagu, Adrian Alan Pol, Maurizio Pierini, Olya Sirkin, and Tal Kopetz. Towards optimal compression: Joint pruning and quantization. CoRR, abs/2302.07612, 2023, doi: 10.48550/ARXIV22222..
07612. /ttps:/:://// org/ 10. 48550/ 2302 07612
Sai Qian Zhang, Bradley McDanel, and HT Kung. Fast: Dnn training under variable precision block floating point with stochastic rounding. In 2022 IEFE International Symposium High-Perfommacc Computer Architecture (HPCA), pp. 846-860. IEFE, 2022a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Out, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, Luke Zeulemoyer. OPT: open pre-trained transformer language models. CoRR, 0bs/2205500106 2022b. doi: 10.48550/ARXXII2205..000 URL
https: http:ac.....g..oooooooe 10. 48550/arXL 2205. 01068
Xinyu Zhang, Ian Colbert, Ken Kreutz- Dellaz,D and Srinjoy Das. Training deep neural networks with joint quantization and pruning of weights and activations. arXiv preprint urXiv:21 2021.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianho Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning N: M fine grained structured sparse neural networks from scratch. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.ned 2021.
URL /RLL/tttss//epeeeeiie...r net forum widddddddddwpepee
18

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
A Additional related work
19
B Hyperparameters
20
Compute resources and runtime
21
D Fine-tuning strategies
21
E Experimental Setup Summary
21
Post-training one-shot sparsity methods
21
Orthogonality threshold for VIT
22
Robustness analysis
23
Layer-wise Error Propagation
24
Proofs for the mathematical analysis
25
K Definitions of Q for numerical formats
29
L Analysis of the upper bound of the dot product error
30
Upper bound is reachable
30
Contribution of additional error terms
31
M Analysis of the Additional Error
31
Combining GPTQ with Sparsity
32
0 Convolutional Networks
32
Optimal order without fine-tuning
32
Collision analysis
33
R OpenLLM evaluation
34
Order of sparsity and quantization during fine-tume tine-tuning
34
A ADDITIONAL RELATED WORK
Quantization. Element-wiss scaling formats, such as FP32, BFLOAT32 (Wang & Kanwar 2019), and FP16 (i Cikevicius et al. 2018), consist of sign, mantissa, and exponent components, differing in the bit allocation for each component. Conversely, block-wise scaling formats assign scaling factors to blocks of elements, with block sizes varying by format. For instance, INT8 employs per-tensor scaling, where single scaling factor is shared by around elements.
Recent research highlights the effectiveness of fine- ine-ggrined block- wise scaling formats with block sizes smaller than 100 elements, especially in the sub-8-bit bit regime for both training and inference (Drumond et al. 2018a Rouhani al. 2023a b) Zhang et al. [202a DDrviish Rouhani al. 2020a). These formats are further categorized into single-level and two-level scaling. Single-level block-wise formats, such HBFP (Drumond
19

---

Published as a conference paper at ICLR 2025
Table Types of max-scaled numerical formats
Element-wise
Single-leve wlse
Two-level block-wise
FP32/FP16
BFloatl6
INT
HEFP
MXINT
MXFP
FP8
Scaling level
Block size
lk
64
32
32
IOK
Scale type
HW
SW
Block size
I
Scaling level
Scale type
HW
HW
ct al. 2018a Darvish Rouhani et al. 2020a and MXINT [Rouhani et al. 2023b), enable fixed-point arithmetic
by sharing a single exponent within block of mantissa or integers. Two-level formats, like MXFP (Rouhani
al. 2023a b Micikevicius al. 2022) and FP8, use more granular scaling factors at the second level, offering greater robustness across diverse range of models.
Sparsity. Unstructured sparsity (Han al. 2015a Guo et al. 2016 Frankle & Carbin 2019 Evci ct al.
2020a
involves removing individual tensor elements without any specific pattern. Structured sparsity (Wen
(016))
on the other hand, employs specific patterns when pruning lensor elements. Recent work (Yao et al.] [019 Kang
2020 has highlighted the effectiveness of fine-graines N:M structured sparsity in mitigating model accuracy
loss. The introduction of the 2:4 structured-sparse Tensor Core in the Nvidia Ampere architecture (Nvidia 202
has further driven research in developing N:M sparsity training recipes (Mishra al. (2021b] Pool Yu 2021
Zhou 2021
Sun et al. 202 Hubara
2021
ct al. 2023).
The fundamental operation any sparsification scheme is selecting candidate elements for pruning among which magnitude-based sparsity (Han et al. 2015b) is one of the most widely used methods (Lu et al. 2023
Ding et al. 2023 Bambbaniya al. 2024). b In addition, recent work has introduced one-shot pruning methods, such SparseGP (Frantar & Alistarn 2023) and Wanda (Sun al. 2023) aiming eliminate the need for an additional fine-tuning phase. While these methods achieve state-of the-art performance one-shot pruning. evidence suggests that incorporating a tine-tuning phase can lead to better model quality (Sun et al. 2023 Syed et al. 2023). Although these methods are proposed to climinate fine-tuning and present statc-of-thc-art accuracies for one-shot pruning, it has heen shown that fine-tuning still improves accuracies significantly (Sun)
2023 Syed
2023)
B HYPERPARAMETERS
We perform full parameter fine-tuning while applying magnitude-bant sparsity methods. find the optimal hyperparameters through grid search for each model and sparsity type and apply the same hyperparameters across all number formats, including FP32. We observe that fine-tuning in a Q-+ order, where we quantize and sparsify tensors at each iteration, leads to highly unstable training process, especially with the structured sparsity. For this reason, we impose limitations the number of training iterations and the learning rate. Thus, we prioritize achieving reproducible and comparable results across all number formats over achieving full convergence for each specific configuration.
Table 4: Details of the sparse fine-tuning experiments
Model
Sparsity type
Batch size
Weight decay
Optimizer
FT num. iterations
Learning rafe
50%
Adam
1776
OPT-125M
2:4
8
Adam
1776
le
50%
4
Adam
1000
5e
OPT-6.7B
2:4
4
Adam
1500
5e
50%
le-3
AdamW
150
2
LLaMA-2-7B
2:4
le
AdamW
Se
50%
2
le-3
AdamW
200
2e-5
LLaMA-3-8B
2:4
le-3
AdamW
300
2
50%
32
le
Adam
30000
5e
VIT-B/16
2:4
32
le
Adam
30000
Se
OPT 125M and OPT-6.7B models are fine-tuned with block sizes of 512 and 1024 respectively, while IJ LaM models utilize block size of 2048. All configurations employ linear learning rate schedule without warm-up.
20

---

Published as a conference paper at ICLR 2025
C COMPUTE RESOURCES AND RUNTIME
We conduct experiments four NVIDIA A10D GPUs with 80GB memory, and for small models, four NVIDIA V100 GPUs with 32GB memory. The hyperparameters used our experiments, including number of fine-tuning epochs, are given in Appendix B In summary. the estimated runtime for each fine-tuning experiments on these hardware platforms are as follows: (a) 20 minutes for OPT-125M, (b) 5-6 hours for OPT-6.7B, (c) 2-3 hours for LLaMA-2-7B and LLaMA-3-8BB 3-B,, and (d) 40 hours for VIT-B/16.
D FINE-TUNING STRATEGIES
Magnitude- sparsity applied in one-shot significant perplexity degradation and thus needs additional fine-tur tinettuning to recover the perplexity. combination quantization, several fine- tne-tuning strategies are possible:
1. Sparse fine-tuning of FP32 model followed by the post-training quantization, sparsity masks applied the FP32 weight tensors
2. Fine-tuning in sparsified and quantized manner, where we sparsify and then quantize tensors at each iteration
Sparse fine-tuning of FP32 model followed by the post-training quantization, sparsity masks are applied to the quantized weight tensors
4. Fine-tuning in quantized and sparsified manner, where quantize and then sparsify tensors each itcration
The former two strategies correspond to the S->Q order transformations, while the latter two correspond to the Q-A order. We conduct ablation experiments for the OPT-125M to compare these fine-tuning strategies. The results presented in the Table S
According
results, post- post-training quantization outperforms
Table 5: Validation perplexities of OPT-125M WikiText2, produced by different fine-tuning strategies. The best results for each configuration are highlighted in bold.
Sparsity type
Number format
Quantization
Order
PPL
during fine-tuning
30.46
S-+Q
33.51
50%
HBFP8
39.04
Q-S
37.48
32.51
S-+Q
36.20
50%
HBFP6
41,97
Q-S
40.86
sparse-and and quantized fune- tane-tuning in S-Q order. In contrast, fine-tuning in quantized and sparsified manner recovers perplexity better than post-training quantization in the reverse order.
E
EXPERIMENTAL SETUP SUMMARY
Table presents summary of experimental setup.
F
POST-TRAINING ONE-SHOT SPARSITY METHODS
Other sparsity schemes, such as Wanda and SparseGPT, have different pruning policy, which uses activations assess the significance the weights and prunes only least significant ones. The pruning metrics Wanda and SparseGPT are
S W b |Xا||2
Sy = [ / (X= XI) XI)
(13)
respectively. If quantization is applied before sparsity, the input values will change, which might also change the set of the nullified weights. However, the significance of those weights will not change considerably. Therefore, the correction vector consists of the least significant weights, which multiplied by the elements of g(x)
21

---

Published as a conference paper at ICLR 2025
Compressed operations by model type
L.MM
ViTs
CNNS
Linear Layers
Yes
Yes
Yes
Convolution Layers
Yes
Embedding Layer
No
Yes
Attention MatMuls
Yes
Yes
Tensor-level granularity for compressed operations
Weights
Activations
Gradients
Forward Pass (for inference and finetuning)
Sparsified + quantized
Quantized
Backward Pass (for finetuning only)
Quantized
Quantized
Quantized
(1) Note: Only matrix multiplications are compressed; other operations (e.g.. optimizer updates) are in FP32
(2) Note: Master weights stored FP32, since our primary aim compress inference.
Workflow for the two different compression orders
Sparse linetuning lollowed by zero-shot quantization
+ Q
Sparsity mask recomputed for each iteration
Quantization and sparse finetuning
Q Q+ S
Quantization and sparsity masks recomputed for each iteration
Table 6: Summary of experimental setup
with the lowest values due to the chosen pruning metrics. As a result, the magnitude of e and the effect of changing the order of the operations is much lower for those sparsity schemes than for the magnitude-based sparsity.
Table 7: Model perplexities on WikiText2 for combined sparsity and quantization. The numbers in the parentheses show the difference in perplexity between the sparse and dense configuration.
(1)
Sparity byc
Spariity methood
Order
FP32
INTS
MXFPS
MXF76
103775
108F76
FP32
INTS
MX3PS
MXFF
HBF7S
21.6
18.06
26445
25,01
27.81
23.91
5.12
5.15
5,17
531
5.34
Magshide
31.13
164
km
6,207
Wands
5-0
9-15
3899-113
20
38.72-
45.50
4044
29.21
42.57-1144
6-00-11
646
655H1.0
6,504.30
6.59-
4.52-
SpurorSPT
9-13
45322490
0.
4.30-40
Magsinak
Q-5
0
11.89
32.70
33,99
55.64
0:30
14.35
14.500944
9.324400
1455
Winds
S-Q
Q-13
85.25
14.10
$4.35 19.30
93 100
91.04-9111
50.69 -9:30
11.30
11 10.20
11 13357
-1.15-301
11.34
11.4
11.35101
11.45
11.36934
12.4
15.6 15
SpansCPPT
Q-15
1940
2011)
50%
46.50 14:00
$7.99
1740
10 19.
10.24 (m
1022
10.1
11. 10
We further explore the effectiveness of post-training one-shot sparsity methods, specifically SparseGPT and Wanda, which utilize selection criterion based on the product of the magnitudes of weights and activations. We report our results in Table We observe that magnitude- due to fine-tuning. However, because of their selection criterion, SparseGPT and Wanda are not affected by the order of the operations. Even when quantization alters the relative magnitudes within weight tensor, the corresponding activations compensate by preserving the original ranking of importance when multiplied together. Consequently, the difference in perplexities between S S-Q and Q-+S for these methods minimal, and in few instances, Q-+S yields better perplexities.
sparsity continues to achieve better perplexities
G ORTHOGONALITY THRESHOLD FOR VIT
Table 8 shows the results of cross-cntropy loss across various combinations of sparsity and quantization schemes for VIT-B/16 performing the image classification task on ImageNet-1k. First, we note that the calculated orthogonality threshold serves as correct lower bound for most configurations, supporting our mathematical analysis. Second, VIT-B/16 significantly more robust the combination of sparsity and quantization schemes compared to the other LLMs studied in this paper. When used with moderate sparsity levels (50% and 2:4) and 8- bit number formats, the actual cross- cross-entropy loss is close to the calculated orthogonality threshold, showing the robustness of ViT-B/16. Only at higher compression rates achieved by using 75% or 1:4 sparsity and 4-bit number formats such HBFP4, do we see the impact of the sparsity and quantization errors affecting the final cross-entropy loss, making significantly higher than the calculated orthogonality threshold.
22

---

Published as a conference paper at ICLR 2025
Table 8: Comparison of evaluation cross-entropy loss with estimated orthogonality thresholds.
VIT-B/16
Sparsity type
Number format
Metric
Orthogonality Threshold
Accuracy
Loss
Accuracy
CE Loss
FP32
81.70%
0.703
INTS
81.64%
0.706
0%
MXFP6
81.26%
0.715
HBFPP
81.67%
0,704
HBFP6
81.35%
0.718
HBFP4
72.73%
1.094
FP32
81.0%%
0,723
INTS
81.03%
0.728
80,98
0.725
MXFPS
80.50
0.745
80.46%
0.742
50%
MXFP6
80.80%
0.734
80.60%
HEFPS
81.00%
0.724
81.01 %
0.723
HBFP6
80.64%
0.736
80.69 %
0.737
HBFP4
73.38%
1.058
72.07%
1.113
FP32
80.06%
0,759
79.95%
0,762
$0.00 %
MXFPS
79.48%
0.781
79.48 %
0.777
2:4
MXPP6
79.73%
0.770
79.62%
0.771
HBFP6
79.69%
0.774
79.71 %
0.773
HBFP4
71.06%
1.163
71.09 %
1.149
FP32
77.26%
0.881
INTS
77.03%
0,897
77.20 %
0.884
MXFPS
76.57%
0,913
76.68 %
75%
MXFP6
76.99%
0.895
76.82%
0.894
HBFPS
77.14%
0.882
77.23%
0.882
HBFP6
76.89%
0.899
76.91
HBRP4
66.84%
1.365
68.29 %
1.272
FP32
73.24%
1.055
INT8
72.90%
1,070
73.18 %
1,088
MXFPS
72.36%
1.095
72.66 %
1.074
1:4
MXPP6
72.92%
1.070
72.80 %
HBFPS
73.16%
1.057
73.21 9
1.056
HBFP6
72.77%
1.078
72.89 %
1.070
HBFP4
59.58%
1.725
64.27%
H ROBUSTNESS ANALYSIS
optimal compression operation order, we conduct limited experiments across three distinct random seeds. We report the mean perplexities and error bars in the Table Given the computational cost of fine-tuning, limit the robustness analysis to the OPT-125M model and HBFPS/6 number format. For both sparsity types observe stable results, consistently affirming the higher efficacy of the S-Q order. Note that deviations causes by different seeds not compromise the integrity of our conclusions.
substantiate our conclusions
Table Validation perplexities OPT-125M on WikiText2 for S-*Q and Q- We report mean and standard deviation over three random seeds.
Sparsity Type
Number Format
Order
PPL
S-Q
30.5 0.2)
IIBFP8
Q-S
37.4 0.3)
50%
S-
32.5 0.2)
HBFP6
Q-S
40.8
S-QQ
32.2
HBFP8
Q-S
46.5
S-Q
Q-S
55.5
23

---

Published as a conference paper at ICLR 2025
I LAYER-WISE ERROR PROPAGATION
We also examine the propagation subsequent layers. Figure [ illustrates how compression errors persist and propagate throughout pre-trained OPT-2.7b model. In this analysis, we quantize each layer in isolation and measure the corresponding propagated emror to the other layers, which remain in full precision. We observe that relatively large errors introduced in carlier layers, despite small fluctuations, stay the same level as they go through the network. As result, a single-layer error can significantly degrade model performance, highlighting the potential threats of errors due to the non-orthogonality compression techniques or applying them the sub-optimal order.
that errors introduced in carlier layers do vanish in
Error propagation
1.75
Compressed
Layer
1.50
Relative L2 error
1.25
12
18
1.00
24
30
0.75
0.50
0.25
0.00
10
15
20
25
30
Layer ID
Figure 2: Error dynamics for single-layer quantization. Each line represents the relative L-2 error of outputs (D);; Yr1l2/ [X|l)/ for the compressed layer the particular index and all subsequent layers, which remain in full precision.
24

---

Published as a conference paper at ICLR 2025
J PROOFS FOR THE MATHEMATICAL ANALYSIS
Proof of Theorem 3.5 Let ft, represent the number of elements pruned from the block by the sparsity transfor- mation. Without loss of generality, we assume that the last n, elements in the block are pruned, as permuting the elements does not affect the block's ' norm. As the sparsity transformation does not prune the largest element in the block, the scole parameter of quantization remains unchanged. Consequently, the quantization error for the non-zero components before and after sparsity remains the same:
J1
31
Eq (x(1
[x)]]
nn-na
In n
Eq (x) n In In-
(14)
In-
0
Zn
(x)x
0
(x)1
(x(x
:
en (x)
Eg
(x))
e. ex(x)n
+
(15)
En +
En
Ex()...
(x(x
ex(x))
E, (x)
Es (x)
[x))(x)) (x)x(
(16)
Eg (x)
ex (x)
(x)
(x),
For Lp norms, where p E (1, +oo)., the upper bound is attainable under one of two conditions: either the pruned clements in originally zero, or the quantization error for all elements of x is zero.
the first case, the first inequality becomes an equality because Vi {n n, + 1, second case, the first inequality also becomes an equality because E (1.....1 n n.) (x); =0
n) 24 In the
Similarly, the second incquality becomes an equality as quantization maps zero to zero. Thus, the quantization error for elements in {n - n, + 1,... n is zero either because the quantization error for all clements of x is zero because the elements
originally zero: Vi E {n n2 ... xx(x) = 0.
For L, norm there exist a non-trivial case. We consider the block Moating-poin numbers x = (4.0, 4.1)T.
INT4 quantization and 1 : 2 sparsity.
(0.0) 4.1
4.0
(0.0)
sxx =
g(x)
((((x)) =
(17)
4.0
(..0)
The LI .l-norms of the transformation errors are the following:
(..0)
(0.9)
[1ea0( - | (
(4.0)
[|)x())],, = 0.0 = 4.0
[1xx())]
= 0.1
=41 (18)
Therefore, (x))s( = (x))l,, + [x),(())]] is attainable.
Proof of Theorem 3.6 Consider the block of floating-point numbers x = (3.9,4.0)* INT4 quantization & and 1:2 sparsity After applying the quantization transformation to the block, initial relation between its elements x, is longer preserved and both elements have equal probability be zeroed out by the sparsity transformation. If sparsity zeroes out the element that was initially larger, the resulting error can exceed the sum of the errors caused by each transformation individually:
(0.0) (.0
4.0
s(((x)) = (4.0)
(t.0)
s(x)
q(x)=
(.0))
(19)
[(.00]
(-011
[x)x0((x)]| = =1((
(0.1)
[s.(()]]
[x)x(()] =
(20)
0.0
1.0
25

---

Published as a conference paper at ICLR 2025
We consider Lp norms, where p E [1; +00). In these norms, Va R (aaa |a| (I(.0) |a]. Therefore:
(
(-0.1
3.9
[x,(())] [x)]]
= 0.11+ 4.0=
(21)
0.0
0.0
=
(0.0)
0.1
4.0
<
4.0
||5x00 (x)l
(22)
Thus, for this particular input the inequality [x)] > [x)]]
[|=,(x)] holds true.
Theorem J.1 (Upper-bound of the error for sub-optimal order, general case). Let be the max-scaled wisk
quantization and be the magnitude-based N:M sparsity transformation. Let step be the least upper bound for the magnitude the quantization error for one element: step = sup[|e (x)x1 ||] x E R", , E (1...n) n.)-
Let I(n, NN, Md Ie" be vector with M-N ones and composition y with respect to L1 norm has the following upper bound:
zeros in any order. Then the error of the
Vx E ERR (x)]] S (x)]] |I + (x)ll + 2 step || N. MII
(23)
additional error
Proof of Theorem J.7 Without loss of generality for simplicity assume that the sparsity operation nullifies the last elements within the vector.
21
q(x)1
ex
[|5so (x]]l
In-ma no En Ru-nat1
Fn-n, (x))--x
(x)nnn
(24)
In-n
n(x)n=nat)
In-natt
:
Fn
In x(x)n
In
(x(x)
0
(x
+
In-nat1
<
c
Eg
nx
+
In #1
=
(25)
Jn
Zn
[=g(())]1 +
= [[(x)]]++ +
+ [x()]]
(26)
In-n
Jn
When quantization is applied first, two distinct numbers can become the same: 2, + g(x) g(x),-
When we sparsify quantized numbers, the number that was smaller might get nullified, as depicted in Figure Therefore, the last component 8 of the upper bound does not equal s.
3
However, in this case we can get an upper bound for the distance between them:
x < x ((x); 2 q(x);
Fi < x, Ii - ((x): 2 q(x);
8 x, >
(27)
Ij x, I q(x); - q(x):
byy 2.5 S lq(x), {(x)+1 [x(x) + (x(x) 2 step
(28)
For each ri that was nullified after quantization followed by sparsification we define of to be an element that would he nullified, if quantization was not applied. Then the vector a(x) consists of all and only such elements at.
26

---

Published as a conference paper at ICLR 2025
22
quantize
step
q(x)1 gxx)
sparsify
Figure 3: A visual representation of applying quantization first and then sparsification. After quantization distinct elements become equal. Then, when sparsification is applied, the element that was originally bigger gets nullified as the sparsification operation cannot differentiate them by their magnitude.
There exists permutation W of the vector sxx such that maps the element z; ((x) to the element x in sxx). Therefore, an upper bound for [[x()))
[x(]] = [ - Ws(x) + (x(x)] S [Ws(x) + [[((x) - Ws(x)ll
(29)
0
0
= =[|s(x]]]
S [|s(()]] +
(30)
In-n
+1
2 step
step
For the of N: M sparsity the number of nullified elements within the block equals M-N Therefore:
2 - step
2 step [,,
N,
(31)
2 step
2
step
As a result, the upper bound for the error of the composition is the following:
[[Exo((x) [|q(x)]] + [[x())] || (x) + [|e.(x)]] 2 step |i M)]l
(32)
Proof of Theorem 3.7 As corollary of Theorem
with respect to T. norm, the last error be
evaluated as follows:
M N
2. step [|ī N, M)]l 2 step
2. step
(33)
M
27

---

Published as a conference paper at ICLR 2025
Proof of Theorem 3.10 Error of the composition can be written the following:
(x,, w) (x, w) (g(x), c(w)
(34)
(x. w) (x (x),,, w Ec(w))
(35)
(x)( w) (x, (w)) - (..()),
(36)
(x), w)
+ +E.(W) + Ew())) (x) (w) + Ee(wW)
(37)
(x, (w)) + (w)) ))+ + (x), w) (g)))) +(x (x (c(w)
(38)
(x.w)
o (x,w)
x(x)
(eo(w), ew))
(39)
(x,, w) + og (x, w) &x (s)(x),
(40)
After adding norms, we obtain the following:
1=2,0(x, w)l (x, w)l + 1e (x, w)l (q(x), + (ex((),
(41)
where E, and E; are additional error terms.
To prove non non-orthogonee orthogonnality, consider the blocks of floating-point numbers x 1.00, w = (0.6, 1.3)".
HBFP4 quantization and 1:2 sparsity s. We assume that q does not affect x: g(z) = x. On the other hand, the block w is transformed in the following way:
0.625 1.25
s(w)
<(w)
s(g(w)) s(((w)) c(w)
(42)
1.3
1.225
The dot product of the composition equals:
< 1.0
[1.0)
00.6)
((100
(100)
Eg.c w) (xx w) (g(x).<(w))
1.3
1.0
1.25
0.65
(43)
The product error of quantization equals:
(
1.00)
(0.6)
) - ((10)
00.625
E (x, w) (x, w) (x(x), g(w))
1.0)
1.3)
(..0)
1.250
0.025
(44)
The dot product error of sparsity equals:
G.x (x, w) = (x, w) (x, sw)) = (1.0)
1100
(0.6)
(1.0)
(
= 0.6
(45)
1.3
1.3
Therefore, for these particular values and w, the inequality: |e (x, w)l > | (x, w)l (xP. w) holds true.
Theorem J.2. Let q be the max-scaled block-wise quantization and be the magnitude-based N:M sparsity transformation. Then:
Vx E R",
(x)]s=(X]]I, I (x)
(46)
Proof. If sparsity is applied first, then from the proof of Theorem 3.5
Ex),, if is sparsified,
(x),
(47)
(x), otherwise.
If quantization is applied first, there are two cases.
Case 1: there duplicates. Quantization cannot reorder elements, it can only make them duplicates. Therefore, if there new duplicates, the same elements will be sparsified after quantization as the order of the elements not change, and the error vector will be the same:
(x) = [E)oo((x)]
(48)
Case 2: there are new duplicates. leet x; and xy be such elements that |ora| < |oy) and g(x), = q(x), =: and the j-th element gets sparsified instead of the i-th. In this case:
Esog (x); (x), and (x)g = Es (x), J.y
(49)
28

---

Published as a conference paper at ICLR 2025
Therefore,
|Esog (X)a + |Esog (x)r1 = (x,( + (xx()), we consider case 24 y < 2, and y 0,
x. + |
(50)
[xg(2); (2) 22 |0yl= (e)x))) (x),))
(51)
Otherwise, assume that æs, my and have the same sign.
Here have three subcases: |0l | Then
ly mal + |2:ي| = lyl
[ass|
[ang]
+ [ma
|zy|
1y
+ mm
(52)
|al <
|arc]. Then
ly - 24| + |25|| [x
1
[gg
221 + [ess
(53)
Jaral lul < lazgl Then
10-2,1+ 22/++1/1rr = 1l |/ryl >> |asyl |11y1-1u0
1v +lyl
1vr + |ar]] | 255+ ||m|
(54)
Therefore,
(ex)))+ = ly 2n. |ass| 2 ly 15 + |ar]| = [=0()))) + (c)
(55)
As a result,
[1exo(())112
Σ
+ 1exoy (2r)s) [e)g)
(56)
Σ
1os0g (orde)) + +1((())) +150(7))l
(57)
(ight.
2
(ppo(())e)) - +le,(x)sl+ (x))l1
(58)
K DEFINITIONS OF Q FOR NUMERICAL FORMATS
1. INT (Symmetric version) Dettmers et al. 2022
Qm (Z,, scale)
w;
where 8
scale
(59)
2m
2. HBFPm (Drumond al. 2018a
Qm (ar,, scale)
=
where 2 (so2l)
(60)
MXFPm Darvish Rouhani et al. 2023
Microsoft 2024
Qm (Ir,, scale)
scale (-1)s 2E (1 (1-2-m M)
(61)
where 5 = sign(x.)
(62)
E Lloga (1 [x:. [}]
bius
(63)
[g
m
(64)
2
x3 2 (acale))
(65)
(66)
Value of bios depends on the chosen configuration (Micikevicius et al. 2023
4. MXINTm DDrvish Rouhani
2023
Microsoft 2024
Qm (Jr,, scale)
2, where
2 (sccalc))))(mm-----))
(67)
29

---

Published as a conference paper at ICLR 2025
Distribution of Deviation
400
300
Count
200
100
1
2
3
10
Deviation
Figure 4: Distribution of the deviation values for several random blocks.
Contribution of the error terms (Q S)
1.0
0.8
Error (normalized)
0.6
0.4
0.2
0.0
9
12
22
27
28
Random block id
Figure 5: Normalized values of each error term of the upper bound for the case of applying quantiza- tion before sparsity.
L ANALYSIS OF THE UPPER BOUND OF THE DOT PRODUCT ERROR
L.1 UPPER BOUND IS REACHABLE
To test the upper bound derived in Theorem 3.10 reachable practice, randomly sampled 1000 blocks of size 64 from standard normal distribution. (O, 1). and applied 2:4 sparsity and HBFPP quantization. We then compute the aggregate error of the composition and individual emor term of the upper bound. Subsequently, we quantified how much the upper bound deviates from the actual composition error using the following formula:
P. (x, w) + 16 (x, w)l + |ex] lea]
Deviation
(68)
1 w)l
As a corollary of Theorem 3.100 the minimal value of deviation 1.
Figure shows the deviation distribution of samples. Most values fall into the first bin, suggesting the upper bound is frequently reached. It can also be seen that the deviation values can be large, almost reaching the value cases too. Theorem does not rule
of 10, which indicates that the upper bound can be pessimistic
30

---

Published as a conference paper at ICLR 2025
Contribution of the error terms (S + Q)
1.0
0.8
Error (normalized,
0.6
0.4
0.2
0
9
12
15
16
20
22
27
28
Random block id
Figure 6: Normalized values of each error term of the upper bound for the case of applying sparsity before quantization. fix the seed and consider the same random blocks as for the order Q - S.
large values of deviation, as it applies the triangle inequality to obtain the upper bound, which leads to dropping the sign of each error term. If the values of the terms negative, they make the overall error of the composition lower than the upper bound, which leads the deviation values larger than one.
L.2 CONTRIBUTION OF ADDITIONAL ERROR TERMS
Section describes how each additional hypothesize that the error term ε contributes
contribute the overall of the composition. We case applying sparsity followed by quantization than case of applying quantization first. We also hypothesize that the magnitude of the error term e: is much lower than the magnitude of Ex. To test our hypotheses. normalized the values of each term of the upper bound to compare their contribution to the error. We considered both orders of applying the transformations. We also only looked at samples with low deviation values (< 1.05) to increase the explainability power bound.
upper
Figures 5 and depict the results of the experiment. If we consider the order Q + S in Figure 5 we can see that the term advocates for almost half of the error of the composition. However, in the order S - Q the term e, has much lower impact, which proves first hypothesis.
We can also that the values of e, are much lower that the values This proves our second hypothesis.
the cases in both orders.
M ANALYSIS OF THE ADDITIONAL ERROR
As a corollary of Theorem 3.10 the composition of max-scaled sparsity and quantization is non-orthogonal, resulting in two additional error terms.
The term incorporates the correction vector of the composition č., which carries the additional eror from the tensor level the dot-product level. Depending the order of the composition, the value of e, varies.
sparsity precedes quantization, the correction vector Eges (w) exclusively comprises negative quantization errors for the elements pruned by the sparsity transformation:
E. <g((w)
(
s(w): 0 otherwise
(w)x
(69)
0,
However, if quantization is applied first, certain elements in the block may become equal, resulting in the sparsity removing different set elements. Formally, w, is pruned by s but not by s g. there exists a wy, where j } i, such that q(w): = o( e(w),. ,. In this scenario, Exag(W) Otherwise, it only contains the quantization errors of the pruned elements. Therefore. the magnitude of the correction vector for the composition
-w)) and (so,,W), = Es Ea(W), ; (W),
is generally larger than that of the reverse order. Besides the
quantization errors, the correction vector also contains pruned elements, which generally larger by orders of
31

---

Published as a conference paper at ICLR 2025
magnitude except in few improbable edge cases. This results in the overall value of et being larger, implying that the order of applying sporsity first and then quantization is optimal for dot products.
The term E also contributes to the additional ertor, encoding the interaction between the error vectors Eg (x) and e, (w). However, E; is less significant than Es. as it contains the quantization error, the norm of which is generally orders of magnitudes lower than the norm of the original block. In addition, contains the sparsity enror, which involves the smallest weights, diminishing the significance of this additional error.
N COMBINING GPTQ WITH SPARSITY
Magnitude-based Sparsity and GPTQ. Although mathematical analysis does not specifically cover GPTQ. we conducted controlled experiments to evaluate performance in the context of our paper. We applied 50% unstructured sparsity to the OPT-125M model combination with GPTQ.
Table 10: GPTQ Magnitude Based for OPT- 125m
Table 11: GPTQ SparseGPT
OPT-3SIM
OPT-L.3
Layer Id
S&Q
Sparsity
Quantization
Order
PPL
Spursity type
Order
Wikitest2
PIB
C4
Wikllest2
PIB
27.65
S- Q Q-+
56.27
M0.67
51.65
27.99
42.32
29.42
24
67.96
91.00
36.57
29.38
45.15
30.15
1, 10,
GPTQ-4b
28.1
1. 10,
50%
34.62
3/4
S-- Q-+
27.38
26,82
36.45
39,62
26,86
27.21
16.73
17:42
24.25
24.30
18.72
18.62
0.
1,
10, 11
50%
GPTQ-4b GPTQ-4b
8 -+0
30.93
S-QQ
43,06
62.84
39.09
22.96
33.61
23,86
0,
1,
10,
11
50%
Q Q
35.59
35.85
When applying GPTQ-+,, finetuning requires quantizing and sparsifying weight tensors in tandem at cach iteration. However, GPTQ operates by quantizing a column and updating the remaining weights to compensate for the introduced errors. Under this scenario, comparing GPTQ and SPPT with finetuning would be fair due the different amounts error compensation. the fine- tine-tuning step and instead sparsified only a subset of layers to contain the sparsity error to a reasonable degree. We determined the number of layers to compress by setting a perplexity threshold equivalent to that achieved by SparseGPT. Table shows that even in this case, magnitude mnagiitude-based sparsity is most effective when applied before quantization (S +GPTQ 30.93 GPTQ-sS: 35.59).
ensure fair comparison, we decided to eliminate
SparseGPT and GPTQ. GPTQ and SparseGPT apply the compression assume that elements the right of the current column remain uncompressed. This is because dense updates propagate through these uncompressed elements to compensate for the introduced error. subsequent columns are compressed, they would not remain so after the first update.
basis and
Given this context, we used the SparseGPT codebase, which natively supports S-+Q. and followed their instructions apply this compression order. We also reached out the author of SparseGPT and followed their recommendation to apply Q-S. We experimented with 4-bit GPT quantization and different variants of OPT models. Table [] summarizes our results, supporting our hypothesis for the optimal order of compression for second-order methods. The mathematical study of the optimal compression order in second-order methods is beyond the scope of our work, and we leave it as future work.
O CONVOLUTIONAL NETWORKS
Our mathematical framework designed to be applicable any matrix mulriplications, regardless of the specific model architecture. This allows us to study the optimal order compression for various models, including CNNs. Therefore, we extended our experiments to include ResNet50 on the ImageNet dataset using all of the same configurations as VIT. The results are present in Table [2 These additional results further validate our findings regarding the optimal compression order and orthogonality threshold.
P OPTIMAL ORDER WITHOUT FINE-TUNING
Magnitude -lageitude-based sparsity, when applied without further re-training, leads to significant accuracy degradation (Hoefler et al. [222 Frantar & Alistarh 2023). In our case, without sparsity tine-tuning. the sparsity error becomes several orders of magnitude larger than the quantization error, causing both S-Q and Q-S to yield predominantly sparsity error. Table 13 shows the WikiText2 perplexities of FP32 sparse models without sparsity- finetuning.
32

---

Published as a conference paper at ICLR 2025
Table 12: Comparison of evaluation cross-entropy loss with estimated orthogonality thresholds
ReaNet50
Sparsity type
Number format
Metric
Orthogonality hhreshold
Accuracy
CE Loss
Accuracy
CE Loss
FP32
76.97%
1,400
76.83%
1.051
MXFPS
69.21%
1.462
0%
MXFP6
70.86%
1,362
HBFPS
76.88%
1.043
HBFP6
74.61%
1.176
FP32
70.33%
1.067
INTS
76.06%
1.072
76.19%
1.078
50%
MXFP8
62.07%
1.917
68.57%
1.489
MXFP6
69.54%
1,420
70.22%
1.389
HBFPS
76.21%
1.072
76.24%
1,070
HBFP6
73.95%
1.186
73.97%
1,204
FP32
76.90%
1.044
INTS
76.49%
1.060
76.66%
1.055
2:4
MXFP6
70.47%
1.351
70.69%
1.366
HAFPX
76.51%
1.053
76.71%
1.047
HBFP6
74.51%
1.158
74.44%
FP32
67.30%
I.569
INTS
67.33%
1.565
67.16%
1,580
75%
MXFP8
62.14%
1.920
59.54%
1,991
MXFP6
58.77%
2.785
61.19%
1.891
HBFPS
67.13%
1.581
67.21%
HBFP6
64.78%
1.733
64.94%
1.706
FP32
73.91%
1.218
73.77%
MXFPS
58.11%
2.314
66.15%
1,640
1:4
MXFP6
63.80%
1,907
67.80%
1.540
HBFP8
73.79%
1.225
73.82%
1,221
HBFP6
71.47%
1.340
71,55%
1.355
Table 13: Magnitude-based sparsity applied without fine-tuning
Sparsity
OPT-125m
OPT-6.7b
LLaMA-2-7B
LLaMA-3-8b
50%
146.
81.
27.
34.
2:4
619.
238.
76.
113.
The significant degradation accuracy makes the scenario without sparsity-aware finetuning impractical. Therefore, reproduced the configurations from Tahle only portion of the layers being sparse and quantized. By applying compression approximately one-third of the layers, specifically those located at the beginning and end of model, we can achieve acceptable perplexity increases. These results support optimality of the S S-xQ order.
Q COLLISION ANALYSIS
proof Theorem J.I we demonstrated that additional error may arise sub-optimal when, after quantization, a larger value and a smaller value collide, causing the originally larger element to be pruned. In this section, we provide empirical evidence of the fact that these collisions take place.
We extracted the weights of all layers of the OPT-1.3b model and calculated the average reduction in unique elements after applying quantization (HBFP6). The reduction in unique elements for each layer is computed as:
Aunique umiquee (W) umiiqe(WW)
(70)
where unigue(X) X) returns the number of unique elements in X. On average Aunique W) 24230, with the total number of unique elements before quantization averaging 24419. Therefore, quantization introduces significant collisions, which may lead to additional
33

---

Published as a conference paper at ICLR 2025
Table 14: Perplexities of OPT-125m model without sparsity- fine-tuning
OPT-125M
S&Q Layer Id
Sparsity type
Order
INT8
MXFP8
MXFP6
HBFP8
HRFP6
HBFP4
S-Q
35.85
34.98
35,02
34.73
35.41
230.
50%
Q-$
35.92
35.03
35.23
34.97
36.05
305.
0. 1, 10, 11
Q
42,97
40.51
43.43
471.
2:4
Q-+S
43.32
40.61
40.59
40.91
48.64
793.
L.L.MA-2-BB
10.04
9,98
18.
50%
Q-+S
8.85
8.74
11.44
8.68
10.43
26.
0-431
S-4Q
9.31
9.39
11.21
9.25
10.06
31.
Q S
9.61
9.92
11.67
9.70
11.24
44.
To further validate findings, analyzed the reduction of unique elements the block level rather than across entire tensor. Specifically, examined the layer of OPT-1.3B, selecting random blocks with a block size of 64. As shown in Figure 7 the reduction of unique elements in block can reach up to 41, representing approximately 64% of the elements in the block. These results confirm that quantization introduces substantial number of duplicates.
Quantization impact on duplicates
Reduction of unique elements
40
30
20
10
0
133
161
165
186
211
Block id
Figure 7: Impact of quantization the number of unique elements in each block.
R OPENLLM EVALUATION
To further validate our findings and expand our experimental setup, we evaluated the models on zero-shot tasks Gao et al. (2024) under both compression orders (S (S++Q and Q-+S.. The results presented in Table [5] for OPT-125M and Table 16 for Llama-3-8B, The results show that all the optimal compression order +Q) yields better performance across models and tasks, validating our theoretical conclusions.
ORDER OF SPARSITY AND QUANTIZATION DURING FINE-TUNING
Our mathematical analysis of sparsity and quantization assumes the model weights to be fixed. However, in our empirical study, the master weights of compressed layers change during fine-tuning. To bridge the gap between the static mathematical analysis and the dynamic case in experimental setup, demonstrate
34

---

model's Published as a conference paper at ICLR 2025
that the optimal compression order independent of the exact model weights, and that quantization minimally affects the model weights during sparsity- aware fine-tuning.
Zero-shot sparsity and quantization. To show that the optimal compression order is independent from the model weights, we apply zero-shot compression to intermediate dense FP32 checkpoints the OPT-125M model on Wiki'exxt2 Table demonstrates final perplexities after applying HBFP6 quantization and 50% unstructured sparsity in S-Q and Q-- orders at various checkpoints of dense fine-tuning. The perplexity increases dense fine-tuning progresses, which is expected, the weights become less tolerant to spansity during the dense FP32 fine-tuning process. For each checkpoint, S-+Q consistently outperforms Q->,, achieving a relative gap up 7%. This confirms that the optimal compression order is independent specific weights, validating our mathematical analysis.
Weight distributions during fine-tuning, our experiments, we employ sparsity- fine-tuning to mitigate sparsity-induce errors. A detailed summary of our experimental set-up is presented in Table During fine- tuning, we store master weights in full precision for each layer, following prior work (Rouhani et al. [023b). This approach enables compression of matrix multiplications during the forward and backward stages while retaining full precision for weight updates. Consequently, quantization during forward and backward phases minimal impact on the learning dynamics the weights.
To demonstrate the minimal effect of quantization during fine-tuning, we compare the distributions of master weights at various iterations during S -Q and Q->S fine-tuning. Figures & 10 show that the master weights remain nearly identical between the schedules, with negligible differences the small magnitude range |-0.2, 0.2]. Although weight values the tails the distributions exhibit more noticeable discrepancies, reordering sparsity and quantization impacts only the smallest values within each block, which lic within the range where the distributions are nearly identical.
To further quantify these differences, Figure II presents the distribution of discrepancies for a single weight component. The absolute difference between weight values remains below 0.02, which negligibly small at the tensor and model levels. Therefore, mathematical analysis remains valid even during fine-tuning.
Weight distribution Iterations sparse-and-qua quantizedddddd fine-tuning
Q-S
102
Count
101
-04
-0.2
0.0
0.2
0.4
Weight Magnitude
Figure 8: Distribution of master weights of Q_proj layer at the heginning of fine-tuning
35

---

Published as a conference paper at ICLR 2025
Weight distribution 1000 iterations sparse-and--uantzer fine-tuning
0
10
102
Count
10a
10a
10P
-0.4
-0.2
0.0
0.2
0.4
Weight Magnitude
Figure 9: Distribution of master weights of Q_proj layer at the middle of fine-tuning
Weight distribution after 1500 lterations of sparse-and-quanttzee fine-tuning
S
10a
Q-5
10a
Count
102
101
100
-022
Weight Magnitude
Figure 10: Distribution master weights of Q_proj layer the end of fine-tuning
Weight difference after 1500 Iterations of sparse-and-quantized fine-tuning
[Ws-0 Wo-sl
10a
101
Count
10a
109
0.0000
0.0025
0.0050
0.0075
0.0000
0.0125
0.0150
0.0175
0.0200
Weight Magnitude
Figure 11: Distribution of discrepancies between weights of Q_proj layer at the end of fine-tuning
36

---

Published as a conference paper at ICLR 2025
Table 15: OPT-125M Zero-Shot Performance. The best results for each configuration are highlighted in bold.
Sparsity
Num format
Order
ARC-c
ARC-e
HellaSWAG
WinoGrande
НВFP6
19.62
41.79
28.74
51.46
0%
INT8
19.03
36.28
28.45
50.99
S-Q
20.31
39.86
28.01
51.38
HBFP6
Q-5
19.04
38.56
26.41
50.67
50%
S-+Q
20.99
34.18
27.49
52.88
INT8
Q-S
20.12
33.70
26.98
51.14
S-Q
20.31
38.26
27.56
52.17
HBFP6
Q-S
18.92
36.68
27.23
49.96
S-QQ
19,97
34,93
27.81
50.67
INT8
Q-S
18.51
30.17
27.35
48.74
37

---

Published as a conference paper at ICLR 2025
Table 16: LLaMA-3 Zero-Shot Performance. The best results for each configuration are highlighted in bold.
Sparsity
Num format
Order
ARC-c
ARC-e
HellaSWAG
WinoGrande
0%
HBFP6
48.21
76.43
59.17
71.19
S-QQ
37.54
69.19
50.64
64.25
50%
HBFP6
Q-S
37.29
67.59
49.54
63.38
S-QQ
31,91
59.6
45.85
60.77
2:4
НВFP6
Q-S
29.83
59.48
42.9
58.43
38

---

Published as a conference paper at ICLR 2025
Table 17: Perplexities of the intermediate checkpoints during dense fine-tuning of OPT-125m, HBFP6+50% sparsity. The best results for each configuration are highlighted in bold.
Steps
500
700
900
1100
1300
1500
1700
S
86.09
95.61
100.35
97.22
119.66
126.04
130.31
Q-S
S
89.03
97.91
107.28
98.58
125.10
127.35
131.01
39


Reviews:
[{"review_id": "lDdVaHFtOv", "reviewer": "Reviewer_KXV4", "summary": "The paper explores the relationship between two widely used compression techniques: sparsity and quantization. Specifically, it demonstrates that these techniques are not independent of one another; the order in which they are applied can significantly impact the results. Additionally, their combination can lead to error propagation, with accumulated errors affecting consecutive layers. The study draws from both theoretical analysis and experimental results conducted on large, modern neural networks.", "questions": "- In Table 2, what do the bold-out results represent? This should be explained in the caption.   \n- In Table 2: perhaps it would be beneficial to show the delta to the sparsity 0% instead/additionally  (e.g. in the appendix)?   \n\nOverall, I find the topic of this paper both interesting and potentially valuable to researchers focusing on sparsity and quantization. The claims and theorems are clearly articulated (I briefly reviewed the details of Theorems 3.5, 3.6, 3.7, and 3.9 in the appendix), and the empirical evaluation, while primarily centered on magnitude-based sparsity, is compelling and conducted across various models and tasks. I believe the strengths of the paper outweigh weaknesses (in fact, I do not have significant concerns regarding weaknesses). Therefore, I am inclined to recommend acceptance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper covers an interesting and timely topic. Given the increasing size of parameters in pre-trained models, there is growing interest in techniques such as quantization and sparsity. Providing both analytical and empirical insights into the relationship between these techniques is valuable, especially as they are often studied separately.The findings in this paper, such as the optimal order for applying sparsity and quantization and the established upper bounds, can offer practical guidance for researchers in this area.  \n- The paper effectively demonstrates the non-orthogonality of sparsity and quantization, determining the optimal sequence for applying these transformations through theoretical analysis, supported by empirical studies on large, modern networks.  \n- The work is well-written, easy to follow, and enjoyable to read.", "weaknesses": "- In the experiments section, the results appear promising and generally align with the theoretical findings. However, it is unclear whether the reported results represent averages of multiple runs or single-run outcomes. If they are averages, what are the standard deviations?  \n- Additionally, I believe the related work section should remain in the main body of the paper, particularly since there is available space before reaching the 10-page limit. Moving it to the appendix could diminish its visibility and importance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SthsrutlKD", "reviewer": "Reviewer_55F2", "summary": "This paper provides a comprehensive theoretical and empirical investigation into the interplay between sparsity and quantization, two widely used model compression techniques. The authors mathematically prove that sparsity and quantization are non-orthogonal operations, meaning their combined use introduces compounded errors beyond those incurred by each method independently. They further derive the optimal order of applying sparsity before quantization (S→Q) to minimize additional errors. These theoretical findings are validated through extensive experiments on large language models (OPT, LLaMA), vision transformers (ViT), and convolutional neural networks (ResNet). The paper also introduces the novel \"orthogonality bound\" metric to efficiently estimate the performance of sparse-quantized models without expensive retraining.", "questions": "- Evaluating the findings on larger, more diverse dataset would be nice. \n- It would be interesting to see how the optimal $S\\to Q$ order and orthogonality bound extend to other sparsity patterns and quantization schemes. Can the authors comment on the generality of their findings in this regard?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper makes significant theoretical contributions by proving the non-orthogonality of sparsity and quantization and deriving the optimal $S\\to Q$ order. These insights challenge conventional assumptions and provide valuable guidance for model compression. \n- The mathematical analysis is rigorous and comprehensive, covering tensor-level and dot product-level errors. \n- The experimental results are extensive, spanning diverse models (OPT, LLama, ResNet, ViT) and settings.\n- The orthogonality bound metric seems like a useful tool for practitioners. \n- Overall the paper is well-structured, with clear definitions, detailed appendices, and informative tables.", "weaknesses": "- While the experiments cover a range of models and settings, the datasets used (WikiText2, ImageNet1k) are relatively small and few. Evaluating on larger, more challenging datasets would further strengthen the findings. \n- The paper does not explore the impact of different sparsity patterns (e.g., block-wise sparsity) or more advanced quantization schemes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bjNgWZDVmI", "reviewer": "Reviewer_j3KK", "summary": "This paper studies the interaction between weight sparsity, weight quantization, and activation quantization in small-to-moderate sized LLMs, ViTs, and CNNs. The authors prove and demonstrate empirically that these methods cannot be considered as purely orthogonal compression modalities under the orthogonality definitions proposed in the paper. Specifically, the authors show that the composition of these strategies is order-dependent and the combined error incurred generally exceeds the sum of the errors produced by applying each method independently.", "questions": "* In the Q->S case, the authors make the argument that this ordering may lead to additional errors when two otherwise unequal weights in the non-quantized precision are set to the same value once quantized. This is an intuitive conclusion but it would be interesting to ground this discussion in empirical evidence of the proportion of weights that this affects, on average, in a pre-trained model. \n* Are the pretrained LLMs obtained from the base models or instruct-tuned variants? Making this explicit in the paper would be beneficial. \n* L312 states that all linear layers were compressed for LLMs. Can you confirm that this included the lm-head, but not the encoder which is typically implemented as an embedding?\n* Table 10 values for 1:4 are counter to typical intuition that higher sparsities generally perform worse. Could the authors confirm that this is 1:4 and not 3:4 sparsity?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* A timely and important topic as sparsity and quantization are promising compression strategies for the large model scales popular today.\n* The paper includes a comprehensive summary of relevant literature.\n* The proofs are relatively easy to follow and explained in an intuitive manner by the authors in the main text.\n* Empirical results generally appear to support the theoretical findings.\n* While many works have studied the combination of sparsity and quantization, this is the first that I am aware of to rigorously consider the interplay between these methods in detail. \n* Empirical experiments include both LLMs and vision models. \n* Extensive supplementary info includes an analysis of several leading SOTA methods from LLM pruning and quantization literature.", "weaknesses": "Overall I am leaning towards accept; however, some concerns regarding the empirical experimental design causes me to doubt the applicability of the results to more general settings:\n\n* The primary metrics considered in the empirical results are perplexity or cross-entropy loss. While these are certainly reasonable proxies for downstream task performance, they are not perfectly correlated. While some accuracy metric for CV models was included in the appendices, it would be beneficial to extend this to downstream tasks for LLMs such as the OpenLLM v1 leaderboard evaluation tasks. It has been shown previously that PPL and CE can be particularly misleading metrics for quantized and sparse models [1]. \n* The experimental design for Section 4.1 is potentially concerning. If I understand the described process correctly, in the Q->S case the pretrained models are pruned and quantized before each forward pass (i.e., instantaneous masking and quantizing). Are the parameters themselves stored as dense fp32 tensors during this process and quantization is simulated similar to QAT approaches? Are the optimizer states left in fp32? The authors note issues with training dynamics in the Q->S case in Appendix A and my concern is that this could be related to numerical precision issues during fine-tuning rather than providing a reliable comparison on the order of compression. Adding a more detailed summary of the fine-tuning approaches in the appendix would potentially clear up any misunderstandings on this point. \n* In the Q->S case quantized activations are used but in the S->Q case it appears the full precision activations are used. It's unclear to me if the dramatic difference in performance is caused by the quantized activations during fine-tuning rather than the specific order of compression for the weights. \n\n\n[1] A. Jaiswal, Z. Gan, X. Du, B. Zhang, Z. Wang, and Y. Yang, “Compressing LLMs: The Truth is Rarely Pure and Never Simple,” Oct. 02, 2023, arXiv: arXiv:2310.01382. doi: 10.48550/arXiv.2310.01382.\n\n\n### Suggestions / Typos:\n* Defining “tensor and dot-product levels” earlier in the text would improve the reader's understanding. Specifically it may be worthwhile to relate these terms to “weights” and “activations” respectively.  I note that activations / dot-products are also represented as tensors. \n* On L68, the authors refer to the challenge of quantizing LLMs due to outliers in “tensor distributions” and reference the smoothquant paper. This should be corrected to “dot-product outliers” as the challenge typically arises from outliers in the activations, not the weights (which instead follow a more gaussian-like distribution typically). \n* I suggest separating references for fine-grained (N:M and similar) and structured (neuron-level or larger NN components) sparsity in the related work discussion on L115. In particular, it would be beneficial to introduce N:M sparsity before it appears in section 3. \n* L469: state-of-the-arts -> state-of-the-art", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ozGKzq1QaT", "reviewer": "Reviewer_e8N1", "summary": "The authors present a theoretical proof showing that the order in which magnitude-based pruning and scaled block quantization are performed is of importance in the context of preserving model performance. The authors define notions such as orthogonality of two operations — which is when the composition of the two operations does not result in any additional error than applying each individual transformation. The authors show theoretically that magnitude pruning and quantization are not orthogonal operations (when going beyond tensor-level) and further showed, both theoretically and empirically, that applying pruning first and then quantization generally leads to better performance.", "questions": "- In Section 3, the authors mention performing quantization at the level of “blocks.\" Could you clarify what you mean by a “block” in this context? Does it refer to a set of weights associated with a CNN filter, or does it resemble the M:N sparsity blocks? Or is it something entirely different?\n- Consider renaming Definition 3.4 to avoid confusion, as it defines \"orthogonality\" between two functions in a way that diverges from the standard interpretation. Traditionally, orthogonality between functions is defined by the condition \\(\\int f(x) g(x) \\, dx = 0\\), so using \"orthogonality\" here might lead to misinterpretation.\n- Why is it important to consider block-wise quantization in Section 3? Since it’s a theoretical derivation, why don’t you simply assume quantization on the tensor level?\n- Theorem 3.5 assumes “max-scaled block-wise quantization”. Is such quantization prevalent in the literature and in practice?\n- Theorems 3.5 and 3.6 imply that the optimal order is pruning followed by quantization. Theorem 3.7 analyses the error for the suboptimal order. Why is that of interest?\n- Is Equation 12 a lower bound or an upper bound? You might want to rename it accordingly to “Orthogonality Lower Bound” or “Orthogonality Upper Bound” to help the reader.\n- “If the compression methods are non-orthogonal, and the evaluation metric indicates better model performance with lower values, we expect the compressed model’s evaluation metric to exceed the orthogonality bound.” — I read this sentence several times and I still can’t understand it. What do you mean by “lower values”? Which values?\n- “For OPT, LLaMA, ViT, and ResNet fine-tuning, we employ sparse fine-tuning on a dense” — what method exactly are you using? Please cite the paper.\n- “we apply one-shot quantization to sparse fine-tuned models” — again, what method exactly are you using? Please cite the paper. is it the “max-scaled block-wise quantization”?\n- “we directly fine-tune the model in a quantized and sparsified manner” — how does one fine tune a quantized model? Isn’t there an issue with doing that?\n- Could you summarize the “Experimental setup” in the form of a table. Otherwise, there are too many details in the paragraph and it’s very hard to digest.\n- In Figure 1, the error accumulates across layers. This stands in contrast to Figure 1 in [1] which shows attenuation of noise injected in intermediate layers. Could it be that the authors should compute a relative error instead of an absolute error (see caption in Figure 1 of that paper)?\n- “and/or reduce quantization effective bitwidth.” — what do you mean by “effective bit width”?\n- “TOPS/mm2” — what’s TOPS and mm^2?\n\n[1] Stronger Generalization Bounds for Deep Nets via a Compression Approach", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Notation, definitions, and theorems in Section 3 are generally clear and their significance is adequately articulated.\n- The authors have addressed an issue that has gone overlooked in the pruning/quantization literature through both theoretical proofs and derivations as well as empirical studies that further solidify their claims.", "weaknesses": "- The discussion following Theorem 3.9 is very hard to digest for a reader who hasn’t spent as much time as the authors thinking about this problem. I’d encourage the authors to prune the text, retaining only the essential message (which presumably is what’s written in italics) and moving other information to the Appendix.\n- Overall, the theoretical claims and experiments are not astonishing as one would perhaps expect that pruning should precede quantization.\n- The theoretical contribution is quite limited as it only holds for magnitude-based pruning (without fine tuning) and block-wise quantization. Importantly, magnitude pruning has gone out of fashion in the context of LLMs because it requires costly fine-tuning to recover model performance and is outperformed by methods like SparseGPT and WANDA when fine-tuning is not performed. The authors mention in the Appendix that, empirically, the order had less of an impact for WANDA and SparseGPT. \n- The experiments seem to be quite orthogonal to the theoretical results. By employing fine-tuning for all the experiments, the authors are making their original theoretical proofs/derivations inapplicable in the context of the experiments as the derivations are based on errors calculated when no fine-tuning is applied. \n- Proof of Theorem 3.5: Only show equality is attained for L1 norm and not all norms. Is it clear that this implies that equality is also achieved for all other norms? Statement of Theorem or proof should be modified to address this. \n- Proof of Theorem 3.6 is only a counter-example for the L1 norm. Is it immediate that the theorem is true in general for norms beyond the L1 norm? Either the statement of the theorem or the proof should be modified to address this.\n- Throughout the paper, some statements are true for all norms, others are only shown for the L1 norm, and then the empirical experiments utilize the L2 norm for measuring errors. \n- The generalization of orthogonality in Definition 3.8 is not clear to me as functions are now being applied coordinate-wise. Is the composition only permitted to happen in one coordinate (similar to in Theorem 3.9). It might be worth it to explicitly write out the definition as the lack of an explicit definition of orthogonality also makes the statement of Theorem 3.9 confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)