Effective Interplay between Sparsity and Quantization: From Theory to Practice

## Abstract

Abstract The increasing size of deep neural networks necessitates effective model compression to improve computational efficiency and reduce their memory footprint.
Sparsity and quantization are two prominent compression methods that have individually demonstrated significant reduction in computational and memory footprints while preserving model accuracy.
While effective, the interplay between these two methods remains an open question.
In this paper, we investigate the interaction between these two methods and assess whether their combination impacts final model accuracy.
We mathematically prove that applying sparsity before quantization is the optimal sequence for these operations, minimizing error in computation.
Our empirical studies across a wide range of models, including OPT and Llama model families (125M-8B) and ViT corroborate these theoretical findings.
In addition, through rigorous analysis, we demonstrate that sparsity and quantization are not orthogonal; their interaction can significantly harm model accuracy, with quantization error playing a dominant role in this degradation.
Our findings extend to the efficient deployment of large models in resource-limited compute platforms and reduce serving cost, offering insights into best practices for applying these compression methods to maximize efficacy without compromising accuracy. 1 1 1 Code and data are available at: https://bit.ly/quantization-sparsity-interplay

### 1 Introduction

Recent breakthroughs in deep neural networks (DNNs) have surpassed human-level capabilities across various tasks such as text generation, machine translation, and computer vision.
However, this success comes with significant challenges due to the exponential growth in the size and complexity of DNN models and datasets [, , , , , , , , ], which complicates their practical deployment and efficient serving.
Delivering efficient and real-time inference for these large models is constrained by arithmetic density (throughput/silicon area [, , ]), memory footprint, and the pressure on memory bandwidth across various hardware platforms (e.g. GPU [], TPU []).

Among various efficiency efforts, model compression has emerged as a crucial solution to effectively address the challenges associated with large models [, , , , , , , , , , , , , , , ]
with quantization standing out as a prominent method in terms of overall compression ratio.
Quantization effectively reduces the precision of model tensors from native floating-point representation to formats such as FP16 [], BFloat16 [], INT8 [, ], and others [, , , , , , , ].

To further enhance compression, researchers often use sparsity, pruning elements of tensors that are least significant to model accuracy.
This reduction in parameters decreases the memory footprint of models and potentially improves computational efficiency by eliminating unnecessary computations [].
Sparsity methods typically prune elements of a tensor based on their magnitudes (a proxy for the importance of an element to model accuracy) [, , , , ].
Magnitude-based sparsity, especially when combined with fine-tuning, offers straightforward implementation and the potential to achieve noticeable compression rates with negligible impact on accuracy [, ].

While prior work has extensively studied the impact of sparsity and quantization in isolation, their combined impact is less explored. A few studies [, , ] suggest that the combination of these two methods does *not* significantly hurt accuracy. Unfortunately, these studies either primarily focus on CNNs [] or do not fully investigate the potential combined impact because they exclusively quantize weight tensors while keeping activations in high precision [, ].

In this paper, we aim to systematically study the interplay between quantization and sparsity.
Quantization and sparsity leverage two fundamentally separate computational properties of DNNs, presenting an opportunity for their combined use.
However, their combined impact on model accuracy involves complex dynamics due to the introduction of errors in tensors. We hypothesize that the interplay between quantization and sparsity can result in additional errors, based on the following insights.
First, applying quantization before sparsity (Q$\rightarrow$S) may adversely disrupt the relative importance of tensor elements, potentially leading to the removal of significant elements of a tensor.
Second, applying sparsity before quantization (S$\rightarrow$Q) can introduce additional errors in dot product calculations, as these are influenced by the magnitudes and precision of the involved elements.
To the best of our knowledge, we are the first to study the interplay between quantization and sparsity in depth, to identify the conditions under which accuracy can be preserved or compromised.
Our contributions are summarized below:

- •
Theoretical Analysis of Sparsity-Quantization Interplay: We provide the first in-depth theoretical analysis of how sparsity and quantization interact. Our mathematical proofs establish that applying sparsity *before* quantization (S $\rightarrow$ Q) is the optimal sequence for compression.
- •
Non-orthogonality and Error Dynamics of Compression Methods: We mathematically demonstrate that sparsity and quantization are *not* orthogonal operations. Combining them introduces additional errors beyond the sum of their individual errors. We analyze the specific error dynamics arising from this interaction, highlighting the dominant role of quantization error in the overall degradation of model accuracy.
- •
Extensive Empirical Study: We validate our theoretical findings through extensive experiments covering a diverse range of models, including prominent LLMs (OPT, LLaMA) and ViTs. These experiments support our hypotheses about the optimal order of compression operations and underscore the non-orthogonality of sparsity and quantization, affirming the practical relevance of our theoretical analysis.
Our experiments demonstrate that applying quantization and sparsity in a sub-optimal order can cause up to 7.96 point increase in perplexity.

### 2 Related work

Quantization.
The ever-growing size of DNN models has spurred extensive research into using narrow numerical formats for inference to reduce memory footprint and improve computational efficiency [, , , , , , , , , ].
Numerical formats employ scaling factors to adjust their dynamic range and can be categorized based on the granularity and their levels. Table 3 in AppendixA summarizes the most common numerical formats.

Sparsity.
Sparsity methods [, , , , ] aim to reduce computational and storage overhead in DNNs by selectively eliminating tensor elements according to various sparsity mask selection criteria.
Broadly, these methods fall into two main categories based on sparsity patterns: unstructured [, , , ] and structured [, , , , , , , , ].

Combining quantization and sparsity.
Prior work has studied the combination of quantization and sparsity and its impact on model performance: sparsity followed by quantization [, , , , , , ] and quantization followed by sparsity [, , , ].
There are two missing pieces of information from these pieces of prior work.
First, consensus on the optimal order of compression operations is not clear. Some favor one order without clear justification, while others treat the methods as independent compression schemes.
Second, there is lack of theoretical grasp on how sparsity and quantization errors interact and influence final model performance.
Appendix A covers additional related work.

### 3 Mathematical analysis of orthogonality of quantization and sparsity

This section provides a mathematical analysis of the interplay between quantization and sparsity, formalizing these compression methods and examining their composition at both the tensor and dot product levels.
Our analysis centers on quantization methods that reduce the bit-width of model weights and activations using block-wise numerical formats, which are prevalent in practical implementations
[, , , , , ].
These formats determine the scaling factor based on the maximum element within a block of elements.
We denote a quantization method employing these numerical formats as max-scaled block-wise quantization.
We use magnitude-based sparsity for both unstructured and N:M structured sparsity.

###### Definition 3.1 ( Max-scaled block-wise quantization ) .

Let $\mathbf{x}\in\mathbb{R}^{n}$ be a block of $n$ numbers and $m\in\mathbb{N}$ denotes the quantization bit-width.
Max-scaled block-wise quantization $q:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$ is a transformation of the block $\mathbf{x}$ such that

$$ $x_{i}\xrightarrow{q}Q_{m}(x_{i},scale)$ (1) $$

where $scale=\max(|x_{1}|,\dots,|x_{n}|)$ is the scaling factor.
$Q_{m}(\cdot,scale)$ quantizes the given element with the scaling factor $scale$ and the number of mantissa bits $m$.
The exact form of $Q_{m}$ depends on the numerical format and can be found in Appendix J.
For instance, INT$m$ quantization transformation is defined as follows:

$$ $Q_{m}(x_{i},scale)=s\cdot\left\lfloor\frac{x_{i}}{s}\right\rceil,\text{where } s=\frac{scale}{2^{m-1}-1},\text{ and }\left\lfloor\cdot\right\rceil\text{ is the rounding to the nearest integer.}$ (2) $$

###### Definition 3.2 ( Magnitude-based sparsity ) .

Let $\mathbf{x}\in\mathbb{R}^{n}$ be a block of $n$ numbers.
We assume $n$ is divisible by $M$ and we consider each group of $M$ elements in the block.
The magnitude-based N:M sparsity transformation can be formulated as:

$$ $\tilde{x}_{i}\coloneqq\begin{cases}0&\text{if }|x_{i}|<\xi\\ x_{i}&\text{otherwise}\end{cases},\text{for }i=1,2,...,M$ (3) $$

where $\xi$ is the $N$-th largest element in $\mathbf{x}=\{|x_{1}|,\dots,|x_{M}|\}$. The same formula can be adjusted to represent $p\%$ unstructured sparsity by defining $\xi$ as the $N$-th largest element in the tensor, where $N=\left[M\cdot p/100\right]$, $M$ is the number of elements in the tensor, and $[\cdot]$ is the operation of rounding to the nearest integer.

In the remaining of this section, we delve into the composition of quantization and sparsity at two different levels.
First, we examine the effects of applying this composition in different orders at the tensor level, observing how individual tensors are altered.
Then, we explore how the composition influences the result of the dot product operation.

###### Definition 3.1 ( Max-scaled block-wise quantization ) .

###### Definition 3.2 ( Magnitude-based sparsity ) .

#### 3.1 Tensor-level analysis

Quantization and sparsity transformations inherently introduce errors by decreasing precision or pruning tensor elements.
To study the composition of quantization and sparsity transformations at the tensor level, we introduce formal definitions of transformation error and orthogonality.
We prove that orthogonality between sparsity and quantization *does not* persist within this composition.
The following definition formalizes the error for a specific transformation at the block level, which consists of a subset of tensor elements.

###### Definition 3.3 ( Transformation error ) .

Let $\mathbf{x}\in\mathbb{R}^{n}$ be a block of $n$ numbers, which are the input of a transformation $f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$.
We define $\varepsilon_{f}(\mathbf{x})\coloneqq\mathbf{x}-f(\mathbf{x})$ as the error of the transformation $f$.

Definition 3.3 can be extended to the tensor level, despite being defined at the block level.
The cumulative error of a tensor can be viewed as the summation of individual errors across all its constituent blocks.
Hence, the theorems analyzed at the block level are indicative of the behavior when scaled up to the tensor level.

Combining two compression methods (transformations) is expected to introduce additional error.
Any error introduced by the first transformation becomes part of the input to the second transformation, potentially amplifying the initial error and resulting in a larger overall error.

###### Definition 3.4 ( Tensor-level orthogonality ) .

We define two transformations $f$ and $g$ to be orthogonal if any order of their composition does not introduce any additional error, and thus, the following inequalities hold:

$$ $\forall\mathbf{x}\in\mathbb{R}^{n},~{}\left\|\varepsilon_{g\circ f}(\mathbf{x} )\right\|\leq\left\|\varepsilon_{f}(\mathbf{x})\right\|+\left\|\varepsilon_{g} (\mathbf{x})\right\|\wedge\left\|\varepsilon_{f\circ g}(\mathbf{x})\right\| \leq\left\|\varepsilon_{f}(\mathbf{x})\right\|+\left\|\varepsilon_{g}(\mathbf{ x})\right\|$ (4) $$

###### Theorem 3.5 .

Let $q$ be the max-scaled block-wise quantization and $s$ be the magnitude-based sparsity transformation.
Applying sparsity before quantization does not introduce additional error:

$$ $\forall\mathbf{x}\in\mathbb{R}^{n},~{}\left\|\varepsilon_{q\circ s}(\mathbf{x} )\right\|\leq\left\|\varepsilon_{q}(\mathbf{x})\right\|+\left\|\varepsilon_{s} (\mathbf{x})\right\|$ (5) $$

Moreover, the equality is attainable.

###### Proof.

Let $n_{s}$ represent the number of elements pruned from the block by the sparsity transformation.
Without loss of generality, we assume that the last $n_{s}$ elements in the block are pruned, as permuting the elements does not affect the block’s norm.
Since the sparsity transformation does not prune the largest element in the block, the $scale$ parameter of quantization remains unchanged.
Consequently, the quantization error for the non-zero components before and after sparsity remains the same:

$$ $\displaystyle\left\|\varepsilon_{q\circ s}(\mathbf{x})\right\|=\left\|\begin{ pmatrix}x_{1}\\ \vdots\\ x_{n-n_{s}}\\ x_{n-n_{s}+1}\\ \vdots\\ x_{n}\end{pmatrix}-q\begin{pmatrix}x_{1}\\ \vdots\\ x_{n-n_{s}}\\ 0\\ \vdots\\ 0\end{pmatrix}\right\|=\left\|\begin{pmatrix}\varepsilon_{q}(\mathbf{x})_{1}\\ \vdots\\ \varepsilon_{q}(\mathbf{x})_{n-n_{s}}\\ x_{n-n_{s}+1}\\ \vdots\\ x_{n}\\ \end{pmatrix}\right\|\leq\left\|\begin{pmatrix}\varepsilon_{q}(\mathbf{x})_{1} \\ \vdots\\ \varepsilon_{q}(\mathbf{x})_{n-n_{s}}\\ 0\\ \vdots\\ 0\\ \end{pmatrix}\right\|+$ (6) $\displaystyle+\left\|\begin{pmatrix}0\\ \vdots\\ 0\\ x_{n-n_{s}+1}\\ \vdots\\ x_{n}\\ \end{pmatrix}\right\|\leq\left\|\begin{pmatrix}\varepsilon_{q}(\mathbf{x})_{1} \\ \vdots\\ \varepsilon_{q}(\mathbf{x})_{n-n_{s}}\\ \varepsilon_{q}(\mathbf{x})_{n-n_{s}+1}\\ \vdots\\ \varepsilon_{q}(\mathbf{x})_{n}\\ \end{pmatrix}\right\|+\left\|\begin{pmatrix}\varepsilon_{s}(\mathbf{x})_{1}\\ \vdots\\ \varepsilon_{s}(\mathbf{x})_{n-n_{s}}\\ \varepsilon_{s}(\mathbf{x})_{n-n_{s}+1}\\ \vdots\\ \varepsilon_{s}(\mathbf{x})_{n}\\ \end{pmatrix}\right\|=\left\|\varepsilon_{s}(\mathbf{x})\right\|+\left\| \varepsilon_{q}(\mathbf{x})\right\|$ (7) $$

To demonstrate that equality is attainable, we consider the block of floating-point numbers $\mathbf{x}=(4.0,4.1)^{T}$, INT4 quantization and $1:2$ sparsity.

$$ $s(\mathbf{x})=\begin{pmatrix}0.0\\ 4.1\\ \end{pmatrix}\quad q(\mathbf{x})=\begin{pmatrix}4.0\\ 4.0\\ \end{pmatrix}\quad q(s(\mathbf{x}))=\begin{pmatrix}0.0\\ 4.0\\ \end{pmatrix}$ (8) $$

The L1-norms of the transformation errors are the following:

$$ $\left\|\varepsilon_{s}(\mathbf{x})\right\|_{1}=\left\|\begin{pmatrix}4.0\\ 0.0\\ \end{pmatrix}\right\|_{1}=4.0\quad\left\|\varepsilon_{q}(\mathbf{x})\right\|_{ 1}=\left\|\begin{pmatrix}0.0\\ 0.1\\ \end{pmatrix}\right\|_{1}=0.1\quad\left\|\varepsilon_{q\circ s}(\mathbf{x}) \right\|_{1}=\left\|\begin{pmatrix}4.0\\ 0.1\\ \end{pmatrix}\right\|_{1}=4.1$ (9) $$

Therefore, $\left\|\varepsilon_{q\circ s}(\mathbf{x})\right\|_{1}=\left\|\varepsilon_{q}(
\mathbf{x})\right\|_{1}+\left\|\varepsilon_{s}(\mathbf{x})\right\|_{1}$ is attainable.
∎

###### Theorem 3.6 .

Let $q$ be the max-scaled block-wise quantization and $s$ be the magnitude-based sparsity transformation.
Applying quantization before sparsity *may* introduce additional error:

$$ $\exists\mathbf{x}\in\mathbb{R}^{n},~{}\left\|\varepsilon_{s\circ q}(\mathbf{x} )\right\|>\left\|\varepsilon_{q}(\mathbf{x})\right\|+\left\|\varepsilon_{s}( \mathbf{x})\right\|$ (10) $$

###### Proof.

Consider the block of floating-point numbers $\mathbf{x}=(3.9,4.0)^{T}$, INT4 quantization $q$ and $1$:$2$ sparsity $s$.
After applying the quantization transformation to the block, initial relation between its elements $x_{i}<x_{j}$ is no longer preserved and both elements have equal probability to be zeroed out by the sparsity transformation.
If sparsity zeroes out the element that was initially larger, the resulting error can exceed the sum of the errors caused by each transformation individually:

$$ $\displaystyle s(\mathbf{x})=\begin{pmatrix}0.0\\ 4.0\\ \end{pmatrix}\quad q(\mathbf{x})=\begin{pmatrix}4.0\\ 4.0\\ \end{pmatrix}\quad s(q(\mathbf{x}))=\begin{pmatrix}4.0\\ 0.0\\ \end{pmatrix}$ (11) $\displaystyle\left\|\varepsilon_{s}(\mathbf{x})\right\|=\left\|\begin{pmatrix} 3.9\\ 0.0\\ \end{pmatrix}\right\|_{1}=3.9\quad\left\|\varepsilon_{q}(\mathbf{x})\right\|= \left\|\begin{pmatrix}-0.1\\ 0.0\\ \end{pmatrix}\right\|=0.1\quad\left\|\varepsilon_{q\circ s}(\mathbf{x})\right \|_{1}=\left\|\begin{pmatrix}-0.1\\ 4.0\\ \end{pmatrix}\right\|_{1}=4.1$ (12) $$

Thus, for this particular input $\mathbf{x}$, the inequality $\left\|\varepsilon_{q\circ s}(\mathbf{x})\right\|_{1}>\left\|\varepsilon_{q}(
\mathbf{x})\right\|_{1}+\left\|\varepsilon_{s}(\mathbf{x})\right\|_{1}$ holds true.
∎

Moreover, a global upper bound exists for the additional error arising from this specific order of transformations (quantization$\rightarrow$sparsity).
This upper bound is solely determined by the quantization method and the parameters of the sparsity scheme, independent of the input data.
The following theorem precisely quantifies the magnitude of this additional error.

###### Theorem 3.7 .

Let $q$ be the max-scaled block-wise quantization and $s$ be the magnitude-based N:M sparsity transformation. Let $step$ be the least upper bound for the magnitude of the quantization error for one element: $step=\sup\{|\varepsilon_{q}(\mathbf{x})_{i}|\;|\;\mathbf{x}\in\mathbb{R}^{n},i
\in\{1\dots n\}\}$. Then the error of the composition $s\circ q$ with respect to $L_{1}$ norm has the following upper bound:

$$ $\forall\mathbf{x}\in\mathbb{R}^{n},~{}\left\|\varepsilon_{s\circ q}(\mathbf{x} )\right\|_{1}\leq\left\|\varepsilon_{q}(\mathbf{x})\right\|_{1}+\left\| \varepsilon_{s}(\mathbf{x})\right\|_{1}+\underbrace{2\cdot step\cdot\frac{M-N} {M}\cdot n}_{\text{additional error}}$ (13) $$

Proof of Theorem 3.7 can be found in Appendix I.
As a corollary of Theorem 3.5, 3.6, and 3.7, it follows that the optimal order of transformations is sparsity followed by quantization, as this sequence *does not* introduce any additional error.
Moreover, according to Definition 3.4, quantization and sparsity are *not* orthogonal at the tensor level.

###### Definition 3.3 ( Transformation error ) .

###### Definition 3.4 ( Tensor-level orthogonality ) .

###### Theorem 3.5 .

###### Proof.

###### Theorem 3.6 .

###### Proof.

###### Theorem 3.7 .

#### 3.2 Orthogonality of dot product operation

In this section, we delve into the error linked with the dot product operation, which is the primary operation in DNNs.
Our analysis focuses on scenarios where weight tensors undergo sparsification and quantization, while activation tensors solely undergo quantization.
We first extend the definition of transformation error to the dot product level.

###### Definition 3.8 ( Transformation error over the dot product ) .

Let $\mathbf{x},\mathbf{w}\in\mathbb{R}^{n}$ denote the inputs of a transformation $f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$ and the dot product operation $\langle.,.\rangle:\mathbb{R}^{n}\times\mathbb{R}^{n}\rightarrow\mathbb{R}$.
We define $\varepsilon_{f}^{D}(\mathbf{x},\mathbf{w})\coloneqq\langle\mathbf{x},\mathbf{w
}\rangle-\langle f(\mathbf{x}),f(\mathbf{w})\rangle$ as the error of the transformation $f$ over dot product.
Similarly, we define $\varepsilon_{f,g}^{D}(\mathbf{x},\mathbf{w})\coloneqq\langle\mathbf{x},\mathbf
{w}\rangle-\langle f(\mathbf{x}),g(\mathbf{w})\rangle$ as the error over the dot product when different transformations are applied to $x$ and $w$.

At the dot product level, we define two compression methods as orthogonal if any order of their composition does not introduce additional error, akin to Definition 3.4.
In the following theorem, we demonstrate that any combination of sparsity and quantization yields additional error, rendering these two methods non-orthogonal.

###### Theorem 3.9 .

Let $q$ be the max-scaled block-wise quantization, $s$ be the magnitude-based sparsity transformation, $c$ be the composition which is either $s\circ q$ or $q\circ s$ and $I$ be the identity function.
Max-scaled quantization $q$ and sparsity $s$ produce additional error in any order, given that only the second operand, i.e., weight, is sparsified:

$$ $\exists\mathbf{x},\mathbf{w}\in\mathbb{R}^{n},~{}|\varepsilon_{q,c}^{D}( \mathbf{x},\mathbf{w})|>|\varepsilon_{I,s}^{D}(\mathbf{x},\mathbf{w})|+| \varepsilon^{D}_{q}(\mathbf{x},\mathbf{w})|$ (14) $$

Moreover,

$$ $|\varepsilon_{q,c}^{D}(\mathbf{x},\mathbf{w})|\leq|\varepsilon_{I,s}^{D}( \mathbf{x},\mathbf{w})|+|\varepsilon^{D}_{q}(\mathbf{x},\mathbf{w})|+ \underbrace{|\langle q(\mathbf{x}),\tilde{\varepsilon}_{c}(\mathbf{w})\rangle| +|\langle\varepsilon_{q}(\mathbf{x}),\varepsilon_{s}(\mathbf{w})\rangle|}_{ \text{additional error}}$ (15) $$

where $\tilde{\varepsilon}_{c}(\mathbf{x})$ is the correction error vector of the composition:

$$ $\varepsilon_{c}(\mathbf{x})=\varepsilon_{q}(\mathbf{x})+\varepsilon_{s}( \mathbf{x})+\tilde{\varepsilon}_{c}(\mathbf{x})$ (16) $$

###### Proof.

Error of the composition can be written as the following:

$$ $\displaystyle\varepsilon_{q,c}^{D}(\mathbf{x},\mathbf{w})$ $\displaystyle=\langle\mathbf{x},\mathbf{w}\rangle-\langle q(\mathbf{x}),c( \mathbf{w})\rangle$ (17) $\displaystyle=\langle\mathbf{x},\mathbf{w}\rangle-\langle\mathbf{x}- \varepsilon_{q}(\mathbf{x}),\mathbf{w}-\varepsilon_{c}(\mathbf{w})\rangle$ (18) $\displaystyle=\langle\varepsilon_{q}(\mathbf{x}),\mathbf{w}\rangle+\langle \mathbf{x},\varepsilon_{c}(\mathbf{w})\rangle-\langle\varepsilon_{q}(\mathbf{x }),\varepsilon_{c}(\mathbf{w})\rangle$ (19) $\displaystyle=\langle\varepsilon_{q}(\mathbf{x}),\mathbf{w}\rangle+\langle \mathbf{x},\varepsilon_{q}(\mathbf{w})+\varepsilon_{s}(\mathbf{w})+\tilde{ \varepsilon}_{c}(\mathbf{w})\rangle-\langle\varepsilon_{q}(\mathbf{x}), \varepsilon_{q}(\mathbf{w})+\varepsilon_{s}(\mathbf{w})+\tilde{\varepsilon}_{c }(\mathbf{w})\rangle$ (20) $\displaystyle=\underbrace{\langle\mathbf{x},\varepsilon_{s}(\mathbf{w})\rangle }_{\varepsilon_{I,s}^{D}(\mathbf{x},\mathbf{w})}+\underbrace{\langle\mathbf{x} ,\varepsilon_{q}(\mathbf{w})\rangle+\langle\varepsilon_{q}(\mathbf{x}),\mathbf {w}\rangle-\langle\varepsilon_{q}(\mathbf{x}),\varepsilon_{q}(\mathbf{w}) \rangle}_{\varepsilon^{D}_{q}(\mathbf{x},\mathbf{w})}+\langle\underbrace{ \mathbf{x}-\varepsilon_{q}(\mathbf{x})}_{q(\mathbf{x})},\tilde{\varepsilon}_{c }(\mathbf{w})\rangle$ (21) $\displaystyle\phantom{{}={}}-\langle\varepsilon_{q}(\mathbf{w}),\varepsilon_{s }(\mathbf{w})\rangle$ (22) $\displaystyle=\varepsilon_{I,s}^{D}(\mathbf{x},\mathbf{w})+\varepsilon^{D}_{q} (\mathbf{x},\mathbf{w})+\langle q(\mathbf{x}),\tilde{\varepsilon}_{c}(\mathbf{ w})\rangle-\langle\varepsilon_{q}(\mathbf{x}),\varepsilon_{s}(\mathbf{w})\rangle$ (23) $$

After adding the norms, we obtain the following:

$$ $|\varepsilon_{q,c}^{D}(\mathbf{x},\mathbf{w})|\leq|\varepsilon_{I,s}^{D}( \mathbf{x},\mathbf{w})|+|\varepsilon^{D}_{q}(\mathbf{x},\mathbf{w})|+| \underbrace{\langle q(\mathbf{x}),\tilde{\varepsilon}_{c}(\mathbf{w})\rangle}_ {\varepsilon_{t}}|+|\underbrace{\langle\varepsilon_{q}(\mathbf{x}),\varepsilon _{s}(\mathbf{w})\rangle}_{\varepsilon_{i}}|$ (24) $$

where $\varepsilon_{t}$ and $\varepsilon_{i}$ are the additional error terms.

To prove non-orthogonality, consider the blocks of floating-point numbers $x=(1.0,1.0)^{T}$ and $w=(0.6,1.3)^{T}$, HBFP4 quantization $q$ and 1:2 sparsity $s$.
For these particular values of $\mathbf{x}$ and $\mathbf{w}$, the inequality: $|\varepsilon_{q,c}^{D}(\mathbf{x},\mathbf{w})|>|\varepsilon^{D}_{q}(\mathbf{x}
,\mathbf{w})|+|\varepsilon_{I,s}^{D}(\mathbf{x},\mathbf{w})|$ holds true. The exact calculations can be found in Appendix I.
∎

Analysis of the additional error.
As a corollary of Theorem 3.9, the combination of max-scaled quantization and sparsity is *not* orthogonal, resulting in two additional error terms.

The term $\varepsilon_{t}$ encompasses the correction vector of the composition $\tilde{\varepsilon}_{c}$, which propagates the error from the tensor level to the dot product level.
Depending on the order of the composition the value of $\varepsilon_{t}$ varies.

If sparsity precedes quantization, the correction vector $\tilde{\varepsilon}_{q\circ s}(\mathbf{w})$ exclusively comprises negative quantization errors for the elements pruned by the sparsity transformation:

$$ $\tilde{\varepsilon}_{q\circ s}(\mathbf{w})_{i}=\begin{cases}-\varepsilon_{q}( \mathbf{w})_{i},&s(\mathbf{x})_{i}=0\\ 0,&\text{otherwise}\end{cases}$ (25) $$

However, if quantization is applied first, certain elements in the block may become equal, resulting in the sparsity removing a different set of elements.
Formally, if $w_{i}$ is pruned by $s$ but not by $s\circ q$, there exists a $w_{j}$, where $j\neq i$, such that $q(\mathbf{w})_{i}=q(\mathbf{w})_{j}$.
In this scenario, $\tilde{\varepsilon}_{s\circ q}(\mathbf{w})_{i}=-\varepsilon_{s}(\mathbf{w})_{i}$ and $\tilde{\varepsilon}_{s\circ q}(\mathbf{w})_{j}=\varepsilon_{s}(\mathbf{w})_{j}
-\varepsilon_{q}(\mathbf{w})_{j}$.
Otherwise, it only contains the quantization errors of the pruned elements.
Therefore, the magnitude of the correction vector for the composition $s\circ q$ is generally larger than that of the reverse order.
Apart from the quantization errors, the correction vector also contains pruned elements, which are larger by orders of magnitude except in a few improbable edge cases.
This results in the overall value of $\varepsilon_{t}$ being larger, implying that *the order of applying sparsity first and then quantization is optimal for dot product.*

The term $\varepsilon_{i}$ also contributes to the additional error, encoding the interaction between the error vectors $\varepsilon_{q}(\mathbf{x})$ and $\varepsilon_{s}(\mathbf{w})$.
However, $\varepsilon_{i}$ is less significant than $\varepsilon_{t}$, as it contains the quantization error, the norm of which is generally orders of magnitude lower than the norm of the original block.
In addition, $\varepsilon_{i}$ contains the sparsity error, which involves the smallest weights, diminishing the significance of this additional error.

Finally, to experimentally validate our theoretical findings, we define orthogonality bound to assess whether the transformations are orthogonal.

###### Definition 3.10 ( Orthogonality bound ) .

Let $M$ be a DNN model under consideration, $\text{EM}(M)$ be an evaluation metric that measures the performance of the model $M$ (e.g. Perplexity or Cross-Entropy Loss), $\text{EM}_{C}(M)$ be the evaluation metric of the model $M$ with transformation $C$, which is either sparsity $S$ or quantization $Q$. Moreover, let $\text{Err}_{C}(M)=\text{EM}_{C}(M)-\text{EM}(M)$ be the evaluation metric error of the transformation $C$ for the model $M$. We define orthogonality bound as:

$$ $\text{Orthogonality Bound}=\text{EM}(M)+\text{Err}_{Q}(M)+\text{Err}_{S}(M)$ (26) $$

If the compression methods are not orthogonal, and the evaluation metric indicates better model performance with lower values, we expect the compressed model’s evaluation metric to exceed the orthogonality bound.

###### Definition 3.8 ( Transformation error over the dot product ) .

###### Theorem 3.9 .

###### Proof.

###### Definition 3.10 ( Orthogonality bound ) .

### 4 Experimental methodology and results

Models, datasets, and evaluation setup.
We study the most widely adopted Transformer-based models, including OPT [] and LLaMA [] model families.
In line with prior work [, , ], we fine-tune pretrained models and evaluate perplexity on the WikiText2 [] dataset.
In addition, we assess orthogonality across different metrics of ViT [] on ImageNet-1k [].
In all experiments, we designate the dense FP32 configuration as our primary baseline.
Our experiments span a diverse range of configurations to validate our theoretical findings, including various variants of max-scaled formats, such as INT8 quantization with per channel scaling [], HBFP8/6 [], and MXFP8/6 [].
We primarily study magnitude-based 50% unstructured and 2:4 structured sparsity with sparsity-aware fine-tuning.
We evaluate the impact of increased compression on ViT-B/16 by applying 75% unstructured sparsity, 1:4 structured sparsity, and HBFP4. Detailed results can be found in Appendix F.
We also explore post-training one-shot sparsity methods like SparseGPT [] and Wanda [] in Appendix E.
In our experiments, we evaluate the impact of each compression method by analyzing the variations in perplexity and cross-entropy loss compared to the baseline, thereby focusing on the cumulative error in the final model output.
Additionally, we analyze the errors of intermediate layers to support our mathematical analysis, as detailed in Appendix G.
Note that we focus on cross-entropy loss for ViT-B/16 instead of classification accuracy.
This is because accuracy remains unaffected as long as the most likely label remains unchanged, regardless of its absolute value.
However, our primary metric for orthogonality bound is the aggregated errors introduced in the final model output distribution.
Table 7 in Appendix F evaluates orthogonality bound on additional metrics.

Experimental setup.
We exclusively sparsify and/or quantize layers with trainable parameters.
Specifically, we target all linear layers in LLMs, and all linear and convolution layers (including the initial embedding layer) in ViT-B/16.
These layers collectively constitute approximately 99% of the total parameters.
In all experiments, we sparsify weights while keeping activation dense.
Both weights and activation tensors are quantized before matrix multiplication operations.
For OPT, LLaMA, and ViT fine-tuning, we employ sparse fine-tuning on a dense FP32 pre-trained model, recomputing sparsity masks at each iteration.
In experiments involving sparsity followed by quantization (S$\rightarrow$Q), we apply one-shot quantization to sparse fine-tuned models.
Conversely, for experiments with the reverse order (Q$\rightarrow$S), we directly fine-tune the model in a quantized and sparsified manner.
At each iteration, we quantized activations and weights while applying sparsity to weight tensors.
We validate the effectiveness of these compression recipes through an ablation study, the details of which are presented in Appendix D.
To ensure fair comparison, we maintain uniform hyperparameters across various number formats for a given model and sparsity type (details in Appendix B).
We present a limited sensitivity study on the initial seed number in Table 8 in Appendix H.

#### 4.1 Empirical study 1: Order of sparsity and quantization

**Table 1: Perplexities of OPT-125M and LLaMA-2-7B on WikiText2 for combinations of different number formats and sparsity types, considering both orders of compression operations. “*50%*” denotes unstructured sparsity, while “*2:4*” represents a variant of N:M structured sparsity. In the FP32 columns, only one perplexity for each compression order is reported because no quantization is applied. The best results for each (sparsity type, number format) pair are highlighted in bold.**
|  |  | OPT-125M | LLaMA-2-7B |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Sparsity<br>type | Order | FP32 | INT8 | MXFP8 | MXFP6 | HBFP8 | HBFP6 | FP32 | INT8 | MXFP8 | MXFP6 | HBFP8 | HBFP6 |
| 0% (Dense) | - | 27.65 | 28.06 | 28.45 | 28.01 | 27.81 | 29.91 | 5.12 | 5.15 | 5.17 | 5.16 | 5.12 | 5.24 |
| 50% | S$\rightarrow$ Q | 29.94 | 30.22 | 31.13 | 31.20 | 30.46 | 32.51 | 6.31 | 6.94 | 6.40 | 6.38 | 6.32 | 6.51 |
| Q$\rightarrow$ S | - | 34.71 | 36.39 | 35.60 | 37.48 | 40.86 | - | 8.13 | 8.47 | 9.32 | 9.86 | 10.20 |  |
| 2:4 | S$\rightarrow$ Q | 31.89 | 32.76 | 33.99 | 33.41 | 32.25 | 34.58 | 9.30 | 9.37 | 9.35 | 9.32 | 9.39 | 10.68 |
| Q$\rightarrow$ S | - | 45.06 | 44.16 | 42.25 | 46.57 | 55.64 | - | 14.65 | 14.35 | 14.50 | 14.98 | 18.64 |  |

This section presents empirical evidence demonstrating that applying sparsity before quantization leads to better perplexities compared to the reverse order.
These results align with the theoretical arguments in Section 3.1.
Table 1 presents perplexities for OPT-125M and LLaMA-2-7B under various number formats and sparsity types.
We compare the order of operations across different configurations: sparsity followed by quantization (S$\rightarrow$Q) and quantization followed by sparsity (Q$\rightarrow$S). Configurations with lower perplexities are highlighted in bold.
Consistently, we observe that the S$\rightarrow$Q order yields better perplexities across all number formats for magnitude-based sparsity.
As discussed in Section 3.1, in the case of Q$\rightarrow$S, quantizing a tensor can alter the order of its elements due to changes in magnitudes.
If magnitude-based sparsity removes elements of the quantized tensor that were originally larger before quantization, the error of the composition may exceed the sum of the errors caused by each transformation individually.
This compounded error further propagates through subsequent dot-product and vector operations, impacting overall model performance.

#### 4.2 Empirical study 2: Non-Orthogonality between sparsity and quantization

**Table 2: OPT-125M, OPT-6.7B, LLaMA-2-7B, and LLaMA-3-8B perplexities on WikiText2 and ViT-B/16 CE loss on ImageNet-1k for the combination of various number formats and sparsity types.**
|  |  | OPT-125M | OPT-6.7B | LLaMA-2-7B | LLaMA-3-8B | ViT-B/16 |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Sparsity<br>type | Number<br>format | PPL$\downarrow$ | Orth.<br>bound | PPL$\downarrow$ | Orth.<br>bound | PPL$\downarrow$ | Orth.<br>bound | PPL$\downarrow$ | Orth.<br>bound | CE<br>Loss$\downarrow$ | Orth.<br>bound |
| 0% | FP32 | 27.65 | - | 10.86 | - | 5.12 | - | 5.53 | - | 0.703 | - |
| INT8 | 28.06 | - | 10.95 | - | 5.19 | - | 5.63 | - | 0.706 | - |  |
| MXFP8 | 28.45 | - | 11.25 | - | 5.17 | - | 5.62 | - | 0.722 | - |  |
| MXFP6 | 28.01 | - | 11.02 | - | 5.16 | - | 5.62 | - | 0.715 | - |  |
| HBFP8 | 27.81 | - | 10.88 | - | 5.12 | - | 5.56 | - | 0.704 | - |  |
| HBFP6 | 29.91 | - | 11.20 | - | 5.24 | - | 5.87 | - | 0.718 | - |  |
| 50% | FP32 | 29.94 | - | 11.30 | - | 6.31 | - | 10.09 | - | 0.723 | - |
| INT8 | 30.22 | 30.35 | 11.37 | 11.39 | 6.94 | 6.38 | 10.85 | 10.19 | 0.728 | 0.725 |  |
| MXFP8 | 31.13 | 30.74 | 11.74 | 11.69 | 6.40 | 6.36 | 10.34 | 10.18 | 0.745 | 0.742 |  |
| MXFP6 | 31.20 | 30.30 | 11.53 | 11.44 | 6.38 | 6.35 | 10.15 | 10.18 | 0.734 | 0.735 |  |
| HBFP8 | 30.46 | 30.18 | 11.31 | 11.32 | 6.32 | 6.31 | 10.12 | 10.12 | 0.724 | 0.723 |  |
| HBFP6 | 32.51 | 32.2 | 11.94 | 11.65 | 6.51 | 6.43 | 10.55 | 10.43 | 0.736 | 0.737 |  |
| 2:4 | FP32 | 31.89 | - | 15.48 | - | 9.30 | - | 13.07 | - | 0.759 | - |
| INT8 | 32.76 | 32.30 | 15.61 | 15.57 | 9.37 | 9.37 | 13.23 | 13.17 | 0.762 | 0.761 |  |
| MXFP8 | 33.99 | 32.69 | 15.70 | 15.87 | 9.35 | 9.35 | 13.35 | 13.16 | 0.781 | 0.777 |  |
| MXFP6 | 33.41 | 32.25 | 15.95 | 15.64 | 9.32 | 9.34 | 13.2 | 13.16 | 0.770 | 0.771 |  |
| HBFP8 | 32.25 | 32.05 | 15.57 | 15.50 | 9.39 | 9.31 | 13.11 | 13.1 | 0.760 | 0.759 |  |
| HBFP6 | 34.58 | 34.15 | 16.98 | 15.82 | 10.68 | 9.42 | 13.64 | 13.41 | 0.774 | 0.773 |  |

This section demonstrates that combining sparsity and quantization results in additional error, surpassing the sum of their individual errors.
Table 2 presents perplexities for OPT-125M, OPT-6.7B, LLaMA-2-7B and LLaMA-3-8B on WikiText2, and cross-entropy (CE) loss for ViT-B/16 on ImageNet-1k for various combinations of number formats and sparsity schemes.
Following our conclusions regarding the order of operations, we only report results for S$\rightarrow$Q.
We compute the orthogonality bound for each combination by summing the individual errors from quantization and sparsity relative to the baseline dense FP32 model, using Equation 26.
Each model’s performance is compared against this bound, with superior results highlighted in bold.
In the majority of configurations, perplexity and cross-entropy loss values exceed the orthogonality bounds, validating the non-orthogonality of sparsity and quantization.

We mathematically show (Section 3.2) that the additional error introduced by combining sparsity and quantization significantly depends on the values of quantized activation tensors and the quantization error of sparsified weights.
This error tends to amplify through successive dot products and vector operations.
Consequently, the quantization error affects the additional error caused by the composition more than the sparsification error.
The gap between the model’s performance and the orthogonality bound is minimal for number formats with minimal performance decrease, whereas larger errors are observed for formats with larger errors.
For instance, HBFP6 results in a 2.26 increase in perplexity for OPT-125M, while its combination with 50% unstructured sparsity leads to a 4.86 increase.

Our analysis reveals that both model size and compression ratio significantly influence the additional error introduced by combining sparsity and quantization.
Larger models exhibit greater tolerance to these compression methods, resulting in lower additional errors.
Moreover, formats with minimal quantization errors (e.g. MXFP8), and sparsity types with minimal sparsification errors (e.g. unstructured sparsity) lead to lower additional errors, even for smaller models.
The effect of high quantization error is more pronounced for sparsity schemes known for higher errors.
For instance, HBFP6’s combination with 2:4 sparsity causes a 6.93 perplexity increase for OPT-125M and a 6.12 perplexity increase for OPT-6.7B.
In contrast, combining HBFP6 with unstructured sparsity results in smaller increases of 4.86 and 0.79 for the respective models.

We also observe instances where the orthogonality bound is slightly higher than the actual perplexity: (a) INT8 with 50% unstructured sparsity for both OPT models, (b) MXFP8 with 2:4 sparsity for OPT-6.7B, (c) MXFP6 with 2:4 sparsity for LLaMA-2-7B, and (d) MXFP6 50% unstructured sparsity for LLaMa-3-8B.
Although these occurrences do not consistently correlate with specific formats, sparsity types, or model sizes, they do not contradict our theoretical analysis, which primarily concerns upper bounds of errors and does not entirely rule out orthogonal cases.
Furthermore, orthogonal configurations still result in larger errors compared to applying either sparsity or quantization alone.
Our mathematical analysis (Theorem 3.9) indicates that there *exists at least one occurrence* where the orthogonality is *not* preserved.
This underscores the need for careful examination when applying these compression methods together, as they do not guarantee high accuracies.
Delving into cases where orthogonality is preserved falls beyond the scope of this paper and we leave it for future work.

We also observe that although the cross-entropy loss for ViT-B/16 is higher than the calculated orthogonality bound in most cases in Table 2, the difference is relatively small. We hypothesize the reason behind this behavior is due to the fact that ViT-B/16 being a vision model that operates on images is more robust to quantization and sparsity errors than LLMs that operate on text.
Hence, the quantization and sparsity levels shown in Table 2 are not sufficient to induce large errors, and hide the non-orthogonality of quantization and sparsity.
To test our hypothesis, we increase the compression further by employing 75% unstructured sparsity, 1:4 structured sparsity and HBFP4 on ViT-B/16 (Appendix F).
The results show that the difference between the baseline cross-entropy loss and the calculated orthogonality bound for these cases increases significantly validating the non-orthogonality of quantization and sparsity.
This increase in cross-entropy loss translates to a significant non-orthogonal drop in accuracy as well.

### 5 Discussion

Our theoretical analysis and experimental results offer multiple insights for ML model practitioners.
First, our analysis demonstrates a risk-free method to improve model performance, measured by lower perplexity and/or higher accuracy, through choosing the optimal ordering of compression operations for any max-scaled number format and magnitude-based pruning scheme.
This contribution is particularly important in the current ML landscape, where sparsity and quantization are pivotal methods for reducing the memory footprint and bandwidth requirements of state-of-the-art LLMs.
Second, we show that calculating the orthogonality bound offers a close enough estimate of model performance (e.g., accuracy, perplexity, etc.) under conditions of quantization and sparsity.
This bound can streamline the search for optimal sparse-quantized model configurations by effectively narrowing the search space.
There is an inherent tradeoff between the hardware benefits of various sparse-quantized configurations and the achieved model performance.
Quantization bit-width and sparsity level are key factors influencing the memory and bandwidth requirements for serving these models.
For example, at a 50% sparsity level, 8-bit and 6-bit quantization result in total reductions in memory footprint and bandwidth requirements by 8$\times$ and 10.7$\times$, respectively.

Ideally, practitioners aim to maximize compression (increase sparsity ratio and/or reduce quantization effective bitwidth.
Our analysis elucidates the individual and combined impacts of these factors across a range of recent large models, providing practical guidelines to achieve the highest compression without compromising model performance.
Typically, 8-bit quantization with any max-scaled number format can serve as a direct replacement for FP32 when combined with any form of sparsity in the optimal order (S$\rightarrow$Q).
As discussed in Section 4, certain models exhibit sensitivity to sub-8bit number formats and structured sparsity combinations, even when applied in an optimal order.
In scenarios where improvements in arithmetic density (TOPS/mm^2) and memory footprint justify a slight reduction in model performance, such as the deployment of large models on edge devices, these combinations may still be viable.

In this work, we do not consider heterogeneous sparsity and quantization schemes, where the sparsity fraction and quantization bit-width vary across layers and differ between activation and weight tensors. Such approaches [, , , ] have demonstrated to be effective in maintaining model accuracy or perplexity while improving compressing ratio.
However, these schemes may not be hardware-friendly, introduce noticeable overhead, and are impractical to implement on off-the-shelf hardware platforms (e.g., GPU, TPU).
We leave the investigation of the interactions between these heterogeneous quantization and sparsity schemes for future work.

### 6 Conclusion

We provide a comprehensive analysis of the interplay between quantization and sparsity in DNNs, showing that applying sparsity before quantization (S$\rightarrow$Q) minimizes additional errors and yields better model accuracy.
Moreover, our mathematical analysis and extensive empirical study with large language models (OPT, LLaMA) and vision transformers (ViT) demonstrate that quantization and sparsity are not orthogonal and their combined use can adversely affect model accuracy.
Our findings provide valuable insights for optimizing the compression of large models while preserving accuracy.

### Acknowledgments and Disclosure of Funding

We extend our gratitude towards Zhifeng Chen, Cliff Young, James Laudon, and Shashwat Shrivastava for reviewing the paper and providing insightful feedback. We also thank the extended team at Google DeepMind, who enabled and supported this research direction.
This work was partially supported by the SNSF project “Unified Accelerators for Post-Moore Machine Learning” (200021_212757) and a Microsoft Research PhD Fellowship.

## Appendix

### Appendix A Additional related work

Quantization.
Element-wise scaling formats, such as FP32, BFLOAT32 [], and FP16 [], consist of sign, mantissa, and exponent components, differing in the bit allocation for each component.
Conversely, block-wise scaling formats assign scaling factors to blocks of elements, with block sizes varying by format. For instance, INT8 employs per-tensor scaling, where a single scaling factor is shared by around 1K elements.

Recent research highlights the effectiveness of fine-grained block-wise scaling formats with block sizes smaller than 100 elements, especially in the sub-8-bit regime for both training and inference [, , , , ].
These formats are further categorized into single-level and two-level scaling.
Single-level block-wise formats, such as HBFP [, ] and MXINT [], enable fixed-point arithmetic by sharing a single exponent within a block of mantissa or integers.
Two-level formats, like MXFP [, , ] and FP8, use more granular scaling factors at the second level, offering greater robustness across diverse range of models.

Sparsity.
Unstructured sparsity [, , , ] involves removing individual tensor elements without any specific pattern.
Structured sparsity [], on the other hand, employs specific patterns when pruning tensor elements.
Recent work [, ] has highlighted the effectiveness of fine-grained N:M structured sparsity in mitigating model accuracy loss.
The introduction of the 2:4 structured-sparse Tensor Core in the Nvidia Ampere architecture [] has further driven research in developing N:M sparsity training recipes [, , , , , ].
The fundamental operation in any sparsification scheme is selecting candidate elements for pruning among which magnitude-based sparsity [] is one of the most widely used methods [, , ].
In addition, recent work has introduced one-shot pruning methods, such as SparseGPT [] and Wanda [], aiming to eliminate the need for an additional fine-tuning phase.
While these methods achieve state-of-the-art performance for one-shot pruning, evidence suggests that incorporating a fine-tuning phase can lead to better model quality [, ].
Although these methods are proposed to eliminate fine-tuning and present state-of-the-art accuracies for one-shot pruning, it has been shown that fine-tuning still improves accuracies significantly [, ].

**Table 3: Types of max-scaled numerical encodings.**
|  | Element-wise | Single-level block-wise | Two-level block-wise |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | FP32/FP16 | BFloat16 | INT | HBFP | MXINT | MXFP | FP8 |  |
| Scaling level 1 | Block size | 1 | 1 | 1k | 64 | 32 | 32 | 10k |
| Scale type | HW | HW | SW | HW | HW | HW | SW |  |
| Scaling level 2 | Block size | - | - | - | - | - | 1 | 1 |
| Scale type | - | - | - | - | - | HW | HW |  |

### Appendix B Hyperparameters

We perform full parameter fine-tuning while applying magnitude-based sparsity methods. We find the optimal hyperparameters through grid search for each model and sparsity type and apply the same hyperparameters across all number formats, including FP32. We observe that fine-tuning in a Q$\rightarrow$S order, where we quantize and sparsify tensors at each iteration, leads to a highly unstable training process, especially with the structured sparsity. For this reason, we impose limitations on the number of training iterations and the learning rate. Thus, we prioritize achieving reproducible and comparable results across all number formats over achieving full convergence for each specific configuration.

**Table 4: Details of the sparse fine-tuning experiments**
| Model | Sparsity type | Batch size | Weight decay | Optimizer | FT num. iterations | Learning rate |
| --- | --- | --- | --- | --- | --- | --- |
| OPT-125M | 50% | 8 | - | Adam | 1776 | 1e^-4 |
| 2:4 | 8 | - | Adam | 1776 | 1e^-4 |  |
| OPT-6.7B | 50% | 4 | - | Adam | 1000 | 5e^-4 |
| 2:4 | 4 | - | Adam | 1500 | 5e^-4 |  |
| LLaMA-2-7B | 50% | 2 | 1e^-3 | AdamW | 150 | 2e^-4 |
| 2:4 | 2 | 1e^-3 | AdamW | 60 | 5e^-5 |  |
| LLaMA-3-8B | 50% | 2 | 1e^-3 | AdamW | 200 | 2e^-5 |
| 2:4 | 1 | 1e^-3 | AdamW | 300 | 2e^-5 |  |
| ViT-B/16 | 50% | 32 | 1e^-3 | Adam | 30000 | 5e^-5 |
| 2:4 | 32 | 1e^-3 | Adam | 30000 | 5e^-5 |  |

OPT-125M and OPT-6.7B models are fine-tuned with block sizes of 512 and 1024 respectively, while LLaMa models utilize a block size of 2048. All configurations employ a linear learning rate schedule without a warm-up.

### Appendix C Compute resources and runtime

We conduct our experiments on four NVIDIA A100 GPUs with 80GB memory, and for small models, we use four NVIDIA V100 GPUs with 32GB memory.
The hyperparameters used in our experiments, including the number of fine-tuning epochs, are given in Appendix B.
In summary, the estimated runtime for each fine-tuning experiments on these hardware platforms are as follows: (a) 20 minutes for OPT-125M, (b) 5-6 hours for OPT-6.7B,
(c) 2-3 hours for LLaMA-2-7B and LLaMA-3-8B, and (d) 40 hours for ViT-B/16.

### Appendix D Fine-tuning strategies

Magnitude-based sparsity applied in one-shot causes a significant perplexity degradation and thus needs an additional fine-tuning to recover the perplexity. In combination with quantization, several fine-tuning strategies are possible:

- 1.
Sparse fine-tuning of FP32 model followed by the post-training quantization, sparsity masks are applied to the FP32 weight tensors
- 2.
Fine-tuning in sparsified and quantized manner, where we sparsify and then quantize tensors at each iteration
- 3.
Sparse fine-tuning of FP32 model followed by the post-training quantization, sparsity masks are applied to the quantized weight tensors
- 4.
Fine-tuning in quantized and sparsified manner, where we quantize and then sparsify tensors at each iteration

The former two strategies correspond to the S$\rightarrow$Q order of transformations, while the latter two correspond to the Q$\rightarrow$S order. We conduct ablation experiments for the OPT-125M to compare these fine-tuning strategies. The results are presented in the Table  5.

**Table 5: Validation perplexities of OPT-125M on WikiText2, produced by different fine-tuning strategies. The best results for each configuration are highlighted in bold.**
| Sparsity<br>type | Number<br>format | Order | Quantization<br>during fine-tuning | PPL |
| --- | --- | --- | --- | --- |
| 50% | HBFP8 | S$\rightarrow$Q | $\times$ | 30.46 |
| $\checkmark$ | 33.51 |  |  |  |
| Q$\rightarrow$S | $\times$ | 39.04 |  |  |
| $\checkmark$ | 37.48 |  |  |  |
| 50% | HBFP6 | S$\rightarrow$Q | $\times$ | 32.51 |
| $\checkmark$ | 36.20 |  |  |  |
| Q$\rightarrow$S | $\times$ | 41.97 |  |  |
| $\checkmark$ | 40.86 |  |  |  |

According to our results, post-training quantization outperforms sparse-and-quantized fune-tuning in S$\rightarrow$Q order. In contrast, fine-tuning in quantized and sparsified manner recovers perplexity better than post-training quantization in the reverse order.

### Appendix E Post-training one-shot sparsity methods

Other sparsity schemes, such as Wanda and SparseGPT, have a different pruning policy, which uses activations to assess the significance of the weights and prunes only the least significant ones. The pruning metrics in Wanda and SparseGPT are

$$ $S_{ij}=|\mathbf{W}_{ij}|\cdot\|\mathbf{X}_{j}\|_{2}\quad\mathrm{and}\quad S_{ ij}=[|\mathbf{W}|^{2}/\text{diag}((\mathbf{X}^{T}\mathbf{X}+\lambda\mathbf{I}) ^{-1}]_{ij}$ (27) $$

respectively. If quantization is applied before sparsity, the input values will change, which might also change the set of the nullified weights. However, the significance of those weights will not change considerably. Therefore, the correction vector $\tilde{\varepsilon}_{c}$ consists of the least significant weights, which are multiplied by the elements of $q(\mathbf{x})$ with the lowest values due to the chosen pruning metrics. As a result, the magnitude of $\varepsilon_{t}$ and the effect of changing the order of the operations is much lower for those sparsity schemes than for the magnitude-based sparsity.

**Table 6: OPT-125M and LLaMA-2-7B perplexities on WikiText2 for the combination of various number formats and sparsity schemes with both orders of operations.**
|  |  |  | OPT-125M | LLaMA-2-7B |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Sparsity<br>type | Sparsity<br>method | Order | FP32 | INT8 | MXFP8 | MXFP6 | HBFP8 | HBFP6 | FP32 | INT8 | MXFP8 | MXFP6 | HBFP8 | HBFP6 |
| 0% | - | - | 27.65 | 28.06 | 28.45 | 28.01 | 27.81 | 29.91 | 5.12 | 5.15 | 5.17 | 5.16 | 5.12 | 5.24 |
| 50% | Magnitude | S$\rightarrow$ Q | 29.94 | 30.22 | 31.13 | 31.20 | 30.46 | 32.51 | 6.31 | 6.94 | 6.40 | 6.38 | 6.32 | 6.51 |
| Q$\rightarrow$ S | - | 34.71 | 36.39 | 35.60 | 37.48 | 40.86 | - | 8.13 | 8.47 | 9.32 | 9.86 | 10.20 |  |  |
| Wanda | S$\rightarrow$ Q | 38.97 | 39.29 | 39.72 | 40.02 | 39.21 | 42.33 | 6.46 | 6.47 | 6.53 | 6.53 | 6.48 | 6.73 |  |
| Q$\rightarrow$ S | - | 40.01 | 40.58 | 40.43 | 40.52 | 42.57 | - | 6.46 | 6.55 | 6.52 | 6.48 | 6.79 |  |  |
| SparseGPT | S$\rightarrow$ Q | 33.24 | 33.22 | 35.27 | 34.22 | 33.41 | 35.86 | 6.51 | 6.51 | 6.58 | 6.58 | 6.52 | 6.77 |  |
| Q$\rightarrow$ S | - | 33.54 | 35.32 | 34.29 | 33.64 | 36.80 | - | 6.53 | 6.60 | 6.58 | 6.55 | 6.93 |  |  |
| 2:4 | Magnitude | S$\rightarrow$ Q | 31.89 | 32.76 | 33.99 | 33.41 | 32.25 | 34.58 | 9.30 | 9.37 | 9.35 | 9.32 | 9.39 | 10.68 |
| Q$\rightarrow$ S | - | 45.06 | 44.16 | 42.25 | 46.57 | 55.64 | - | 14.65 | 14.35 | 14.50 | 14.98 | 18.64 |  |  |
| Wanda | S$\rightarrow$ Q | 79.91 | 79.81 | 85.25 | 84.10 | 80.62 | 90.66 | 11.36 | 11.37 | 11.15 | 11.35 | 11.45 | 12.74 |  |
| Q$\rightarrow$ S | - | 80.28 | 86.69 | 84.38 | 80.69 | 91.04 | - | 11.28 | 11.24 | 11.46 | 11.36 | 13.61 |  |  |
| SparseGPT | S$\rightarrow$ Q | 45.14 | 45.34 | 48.44 | 46.49 | 45.52 | 50.74 | 10.22 | 10.21 | 10.15 | 10.26 | 10.26 | 10.86 |  |
| Q$\rightarrow$ S | - | 44.96 | 48.67 | 46.50 | 45.82 | 57.39 | - | 10.21 | 10.24 | 10.26 | 10.21 | 11.16 |  |  |

We further explore the effectiveness of post-training one-shot sparsity methods, specifically SparseGPT and Wanda, which utilize a selection criterion based on the product of the magnitudes of weights and activations.
We report our results in Table 6
We observe that magnitude-based sparsity continues to achieve better perplexities due to fine-tuning.
However, because of their selection criterion, SparseGPT and Wanda are not affected by the order of the operations.
Even when quantization alters the relative magnitudes within a weight tensor, the corresponding activations can compensate by preserving the original ranking of importance when multiplied together.
Consequently, the difference in perplexities between S$\rightarrow$Q and Q$\rightarrow$S for these methods is minimal, and in few instances, Q$\rightarrow$S yields better perplexities.

### Appendix F Orthogonality bound for ViT

Table 7 shows the results of cross-entropy loss across various combinations of sparsity and quantization schemes for ViT-B/16 performing the image classification task on ImageNet-1k. First, we note that the calculated orthogonality bound serves as a correct lower bound for most configurations, supporting our theoretical analysis. Second, ViT-B/16 is significantly more robust to the combination of quantization and sparsity schemes compared to the other LLMs studied in this paper. When used with moderate sparsity levels (50% and 2:4) and 8-bit/6-bit number formats, the actual cross-entropy loss is close to the calculated orthogonality bound, showing the robustness of ViT-B/16. Only at higher compression rates achieved by using 75% or 1:4 sparsity and 4-bit number formats such as HBFP4, do we see the impact of the quantization and sparsity errors affecting the final cross-entropy loss, making it significantly higher than the calculated orthogonality bound.

**Table 7: Comparison of evaluation cross-entropy loss with estimated orthogonality bounds**
| Sparsity<br>type | Number<br>format |  | ViT-B/16 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Metric |  | Orthogonality Bound |  |  |  |  |
|  | Accuracy | CE Loss |  | Accuracy | CE Loss |  |  |
| 0% | FP32 |  | 81.70% | 0.703 |  | - | - |
| INT8 |  | 81.64% | 0.706 |  | - | - |  |
| MXFP8 |  | 81.12% | 0.722 |  | - | - |  |
| MXFP6 |  | 81.26% | 0.715 |  | - | - |  |
| HBFP8 |  | 81.67% | 0.704 |  | - | - |  |
| HBFP6 |  | 81.35% | 0.718 |  | - | - |  |
| HBFP4 |  | 72.73% | 1.094 |  | - | - |  |
| 50% | FP32 |  | 81.04% | 0.723 |  | - | - |
| INT8 |  | 81.03% | 0.728 |  | 80.98% | 0.725 |  |
| MXFP8 |  | 80.50% | 0.745 |  | 80.46% | 0.742 |  |
| MXFP6 |  | 80.80% | 0.734 |  | 80.60% | 0.735 |  |
| HBFP8 |  | 81.00% | 0.724 |  | 81.01% | 0.723 |  |
| HBFP6 |  | 80.64% | 0.736 |  | 80.69% | 0.737 |  |
| HBFP4 |  | 73.38% | 1.058 |  | 72.07% | 1.113 |  |
| 2:4 | FP32 |  | 80.06% | 0.759 |  | - | - |
| INT8 |  | 79.95% | 0.762 |  | 80.00% | 0.761 |  |
| MXFP8 |  | 79.48% | 0.781 |  | 79.48% | 0.777 |  |
| MXFP6 |  | 79.73% | 0.770 |  | 79.62% | 0.771 |  |
| HBFP8 |  | 80.06% | 0.760 |  | 80.03% | 0.759 |  |
| HBFP6 |  | 79.69% | 0.774 |  | 79.71% | 0.773 |  |
| HBFP4 |  | 71.06% | 1.163 |  | 71.09% | 1.149 |  |
| 75% | FP32 |  | 77.26% | 0.881 |  | - | - |
| INT8 |  | 77.03% | 0.897 |  | 77.20% | 0.884 |  |
| MXFP8 |  | 76.57% | 0.913 |  | 76.68% | 0.900 |  |
| MXFP6 |  | 76.99% | 0.895 |  | 76.82% | 0.894 |  |
| HBFP8 |  | 77.14% | 0.882 |  | 77.23% | 0.882 |  |
| HBFP6 |  | 76.89% | 0.899 |  | 76.91% | 0.896 |  |
| HBFP4 |  | 66.84% | 1.365 |  | 68.29% | 1.272 |  |
| 1:4 | FP32 |  | 73.24% | 1.055 |  | - | - |
| INT8 |  | 72.90% | 1.070 |  | 73.18% | 1.058 |  |
| MXFP8 |  | 72.36% | 1.095 |  | 72.66% | 1.074 |  |
| MXFP6 |  | 72.92% | 1.070 |  | 72.80% | 1.068 |  |
| HBFP8 |  | 73.16% | 1.057 |  | 73.21% | 1.056 |  |
| HBFP6 |  | 72.77% | 1.078 |  | 72.89% | 1.070 |  |
| HBFP4 |  | 59.58% | 1.725 |  | 64.27% | 1.446 |  |

### Appendix G Layer-wise error analysis

Our mathematical study focuses on the per-layer error caused by the combination of quantization and sparsity. To demonstrate that this error is additive and increases with the layer index, we inspect the outputs of intermediate layers. In our experiment, we apply quantization and magnitude-based sparsity to all linear layers of the pre-trained OPT-125M model in a zero-shot manner. We feed a sample from a test subset to the compressed model and the corresponding full precision dense model and measure the difference between Feed-Forward outputs for each Transformer block. We repeat this experiment for S$\rightarrow$Q and Q$\rightarrow$S orders of transformations and show comparison results in Figure 1.

Figure: Figure 1: Per layer error of the combination of HBFP8 quantization and 2:4 structured sparsity
Refer to caption: extracted/5635050/figures/layerwise.png

We observe that the per-layer error increases monotonically with layer index, regardless of the order of transformations, and reaches its maximum value after the final layer. This error directly affects the output of the model and results in the metric degradation, which we have detected in the previous experiments 3.1, 3.2.

Layer-wise analysis also supports our theoretical findings that the optimal order of transformations is sparsity followed by quantization. The S$\rightarrow$Q order of transformations consistently yields lower errors in the intermediate layer outputs compared to the reverse order. We observe an exponential increase in error with the Q$\rightarrow$S order of transformations for the final layers, resulting in an error almost four times larger than with the optimal order.

### Appendix H Robustness analysis

To substantiate our conclusions on the optimal compression operation order, we conduct limited experiments across three distinct random seeds.
We report the mean perplexities and error bars in the Table 8.
Given the computational cost of fine-tuning, we limit the robustness analysis to the OPT-125M model and HBFP8/6 number format.
For both sparsity types we observe stable results, consistently affirming the higher efficacy of the S$\rightarrow$Q order. Note that deviations causes by different seeds do *not* compromise the integrity of our conclusions.

**Table 8: Validation perplexities of OPT-125M on WikiText2 for S$\rightarrow$Q and Q$\rightarrow$S. We report mean and standard deviation over three random seeds.**
| Sparsity<br>Type | Number<br>Format | Order | PPL |
| --- | --- | --- | --- |
| 50% | HBFP8 | S$\rightarrow$Q | 30.5 ($\pm$ 0.2) |
| Q$\rightarrow$S | 37.4 ($\pm$ 0.3) |  |  |
| HBFP6 | S$\rightarrow$Q | 32.5 ($\pm$ 0.2) |  |
| Q$\rightarrow$S | 40.8 ($\pm$ 0.3) |  |  |
| 2:4 | HBFP8 | S$\rightarrow$Q | 32.2 ($\pm$ 0.1) |
| Q$\rightarrow$S | 46.5 ($\pm$ 0.4) |  |  |
| HBFP6 | S$\rightarrow$Q | 34.6 ($\pm$ 0.2) |  |
| Q$\rightarrow$S | 55.5 ($\pm$ 0.8) |  |  |

### Appendix I Proofs for the mathematical analysis

###### Proof of Theorem 3.7 .

Without loss of generality for simplicity we assume that the sparsity operation nullifies the last elements within the vector.

$$ $\left\|\varepsilon_{s\circ q}(\mathbf{x})\right\|=\left\|\begin{pmatrix}x_{1} \\ \vdots\\ x_{n-n_{s}}\\ x_{n-n_{s}+1}\\ \vdots\\ x_{n}\end{pmatrix}-s\begin{pmatrix}x_{1}-\varepsilon_{q}(\mathbf{x})_{1}\\ \vdots\\ x_{n-n_{s}}-\varepsilon_{q}(\mathbf{x})_{n-n_{s}}\\ x_{n-n_{s}+1}-\varepsilon_{q}(\mathbf{x})_{n-n_{s}+1}\\ \vdots\\ x_{n}-\varepsilon_{q}(\mathbf{x})_{n}\end{pmatrix}\right\|=\left\|\begin{ pmatrix}\varepsilon_{q}(\mathbf{x})_{1}\\ \vdots\\ \varepsilon_{q}(\mathbf{x})_{n-n_{s}}\\ x_{n-n_{s}+1}\\ \vdots\\ x_{n}\\ \end{pmatrix}\right\|\leq$ (28) $$

$$ $\leq\left\|\begin{pmatrix}\varepsilon_{q}(\mathbf{x})_{1}\\ \vdots\\ \varepsilon_{q}(\mathbf{x})_{n-n_{s}}\\ 0\\ \vdots\\ 0\\ \end{pmatrix}\right\|+\left\|\begin{pmatrix}0\\ \vdots\\ 0\\ x_{n-n_{s}+1}\\ \vdots\\ x_{n}\\ \end{pmatrix}\right\|\leq\\ \left\|\begin{pmatrix}\varepsilon_{q}^{1}\\ \vdots\\ \varepsilon_{q}^{n-n_{s}}\\ \varepsilon_{q}^{n-n_{s}+1}\\ \vdots\\ \varepsilon_{q}^{n}\\ \end{pmatrix}\right\|+\left\|\begin{pmatrix}0\\ \vdots\\ 0\\ x_{n-n_{s}+1}\\ \vdots\\ x_{n}\\ \end{pmatrix}\right\|=$ (29) $$

$$ $=\left\|\varepsilon_{q}(\mathbf{x})\right\|+\left\|\begin{pmatrix}0\\ \vdots\\ 0\\ x_{n-n_{s}+1}\\ \vdots\\ x_{n}\\ \end{pmatrix}\right\|:=\left\|\varepsilon_{q}(\mathbf{x})\right\|+\left\| \widetilde{\varepsilon_{s}}(\mathbf{x})\right\|$ (30) $$

When quantization is applied first, two distinct numbers can become the same: $x_{i}<x_{j}\rightarrow q(\mathbf{x})_{i}=q(\mathbf{x})_{j}$. When we sparsify the quantized numbers, the number that was smaller might get nullified, as depicted in Figure 2. Therefore, the last component $\widetilde{\varepsilon}_{s}$ of the upper bound does not equal $\varepsilon_{s}$.

Figure: Figure 2: A visual representation of applying quantization first and then sparsification. After quantization two distinct elements become equal. Then, when sparsification is applied, the element that was originally bigger gets nullified as the sparsification operation cannot differentiate them by their magnitude.
Refer to caption: extracted/5635050/figures/math_appendix.png

However, in this case we can get an upper bound for the distance between them:

$$ $\begin{cases}x_{i}<x_{j}\\ q(\mathbf{x})_{i}\geq q(\mathbf{x})_{j}\\ \end{cases}\Leftrightarrow\begin{cases}x_{i}<x_{j}\\ x_{i}-\varepsilon_{q}(\mathbf{x})_{i}\geq x_{j}-\varepsilon_{q}(\mathbf{x})_{j }\\ \end{cases}\Leftrightarrow\begin{cases}x_{j}-x_{i}>0\\ x_{j}-x_{i}\leq\varepsilon_{q}(\mathbf{x})_{j}-\varepsilon_{q}(\mathbf{x})_{i} \\ \end{cases}\Rightarrow$ (31) $$

$$ $\Rightarrow|x_{j}-x_{i}|\leq|\varepsilon_{q}(\mathbf{x})_{j}-\varepsilon_{q}( \mathbf{x})_{i}|\leq|\varepsilon_{q}(\mathbf{x})_{j}|+|\varepsilon_{q}(\mathbf {x})_{i}|\leq 2\cdot step$ (32) $$

For each $x_{i}$ that was nullified after quantization followed by sparsification we define $x_{i}^{t}$ to be an element that would be nullified, if quantization was not applied. Then the vector $\varepsilon_{s}(\mathbf{x})$ consists of all and only such elements $x_{i}^{t}$.

There exists a permutation $W$ of the vector $\varepsilon_{s}(\mathbf{x})$ such that maps the element $x_{i}$ in $\widetilde{\varepsilon}_{s}(\mathbf{x})$ to the element $x_{i}^{t}$ in $\varepsilon_{s}(\mathbf{x})$. This helps us to get an optimal upper bound for the additional error:

$$ $\left\|\widetilde{\varepsilon_{s}}(\mathbf{x})\right\|=\left\|\widetilde{ \varepsilon_{s}}(\mathbf{x})-W\varepsilon_{s}(\mathbf{x})+W\varepsilon_{s}( \mathbf{x})\right\|\leq\left\|W\varepsilon_{s}(\mathbf{x})\right\|+\left\| \widetilde{\varepsilon_{s}}(\mathbf{x})-W\varepsilon_{s}(\mathbf{x})\right\|=$ (33) $$

$$ $=\left\|\varepsilon_{s}(\mathbf{x})\right\|+\left\|\begin{pmatrix}0-0\\ \vdots\\ 0-0\\ x_{n-n_{s}+1}-x_{n-n_{s}+1}^{t}\\ \vdots\\ x_{n}-x_{n}^{t}\\ \end{pmatrix}\right\|\leq\left\|\varepsilon_{s}(\mathbf{x})\right\|+\left\| \begin{pmatrix}0-0\\ \vdots\\ 0-0\\ 2\cdot step\\ \vdots\\ 2\cdot step\\ \end{pmatrix}\right\|$ (34) $$

For the case of $N:M$ sparsity then number of nullified elements within the block equals $\frac{M-N}{M}\cdot n$. Therefore, with respect to $L_{1}$ norm, the last error term can be evaluated as follows:

$$ $\left\|\begin{pmatrix}0-0\\ \vdots\\ 0-0\\ 2\cdot step\\ \vdots\\ 2\cdot step\\ \end{pmatrix}\right\|_{1}=2\cdot step\cdot\frac{M-N}{M}\cdot n$ (35) $$

∎

###### Proof of Theorem 3.9 .

To prove non-orthogonality, consider the blocks of floating-point numbers $x=(1.0,1.0)^{T},~{}w=(0.6,1.3)^{T}$, HBFP4 quantization $q$ and 1:2 sparsity $s$. We assume that $q$ does not affect $x$: $q(x)=x$. On the other hand, the block $w$ is transformed in the following way:

$$ $\displaystyle s(\mathbf{w})=\begin{pmatrix}0\\ 1.3\\ \end{pmatrix}\quad q(\mathbf{w})=\begin{pmatrix}0.625\\ 1.25\\ \end{pmatrix}\quad s(q(\mathbf{w}))=s(q(\mathbf{w}))=c(\mathbf{w})=\begin{ pmatrix}0\\ 1.25\\ \end{pmatrix}$ (36) $$

The dot product error of the composition equals:

$$ $\varepsilon_{q,c}^{D}(\mathbf{x},\mathbf{w})=\langle\mathbf{x},\mathbf{w} \rangle-\langle q(\mathbf{x}),c(\mathbf{w})\rangle=\left\langle\begin{pmatrix} 1.0\\ 1.0\\ \end{pmatrix},\begin{pmatrix}0.6\\ 1.3\\ \end{pmatrix}\right\rangle-\left\langle\begin{pmatrix}1.0\\ 1.0\\ \end{pmatrix},\begin{pmatrix}0\\ 1.25\\ \end{pmatrix}\right\rangle=0.65$ (37) $$

The dot product error of quantization equals:

$$ $\varepsilon^{D}_{q}(\mathbf{x},\mathbf{w})=\langle\mathbf{x},\mathbf{w}\rangle -\langle q(\mathbf{x}),q(\mathbf{w})\rangle=\left\langle\begin{pmatrix}1.0\\ 1.0\\ \end{pmatrix},\begin{pmatrix}0.6\\ 1.3\\ \end{pmatrix}\right\rangle-\left\langle\begin{pmatrix}1.0\\ 1.0\\ \end{pmatrix},\begin{pmatrix}0.625\\ 1.250\\ \end{pmatrix}\right\rangle=0.025$ (38) $$

The dot product error of sparsity equals:

$$ $\varepsilon_{I,s}^{D}(\mathbf{x},\mathbf{w})=\langle\mathbf{x},\mathbf{w} \rangle-\langle\mathbf{x},s(\mathbf{w})\rangle=\left\langle\begin{pmatrix}1.0 \\ 1.0\\ \end{pmatrix},\begin{pmatrix}0.6\\ 1.3\\ \end{pmatrix}\right\rangle-\left\langle\begin{pmatrix}1.0\\ 1.0\\ \end{pmatrix},\begin{pmatrix}0\\ 1.3\\ \end{pmatrix}\right\rangle=0.6$ (39) $$

Therefore, for these particular values of $\mathbf{x}$ and $\mathbf{w}$, the inequality: $|\varepsilon_{q,c}^{D}(\mathbf{x},\mathbf{w})|>|\varepsilon^{D}_{q}(\mathbf{x}
,\mathbf{w})|+|\varepsilon_{I,s}^{D}(\mathbf{x},\mathbf{w})|$ holds true.

∎

###### Proof of Theorem 3.7 .

###### Proof of Theorem 3.9 .

### Appendix J Definitions of Q 𝑄 Q italic_Q for numerical formats

- 1.
INT$m$ (Symmetric version) []
$Q_{m}(x_{i},scale)=s\cdot\left\lfloor\frac{x_{i}}{s}\right\rceil,\text{where }
s=\frac{scale}{2^{m-1}-1},$
(40)
- 2.
HBFP$m$ []
$Q_{m}(x_{i},scale)=s\cdot\left\lfloor\frac{x_{i}}{s}\right\rceil,\text{where }
s=2^{\left\lceil log_{2}(scale)\right\rceil-(m-1)}$
(41)
- 3.
MXFP$m$ [] []
$Q_{m}(x_{i},scale)=scale\cdot(-1)^{S}\cdot 2^{E}\cdot(1+2^{-m}\cdot M)$
(42)
$\displaystyle\text{where }S$
$\displaystyle=\text{sign}(x_{s})$
(43)
$\displaystyle E$
$\displaystyle=\lfloor log_{2}(|x_{s}|)\rfloor-bias$
(44)
$\displaystyle M$
$\displaystyle=\left\lfloor\left(\frac{|x_{s}|}{2^{E}}-1\right)\cdot 2^{m}\right\rceil$
(45)
$\displaystyle x_{s}$
$\displaystyle=\frac{x}{2^{\lceil\log_{2}(scale)\rceil}}$
(46)
Value of $bias$ depends on the chosen configuration [].
- 4.
MXINT$m$ [] []
$Q_{m}(x_{i},scale)=s\cdot\left\lfloor\frac{x_{i}}{s}\right\rceil,\text{where }
s=2^{\lceil\log_{2}(scale)\rceil-(m-1)}$
(48)

Figure: Figure 3: Distribution of the deviation values for several random blocks.
Refer to caption: x1.png

Figure: Figure 4: Normalized values of each error term of the upper bound for the case of applying quantization before sparsity.
Refer to caption: x2.png

Figure: Figure 5: Normalized values of each error term of the upper bound for the case of applying sparsity before quantization. We fix the seed and consider the same random blocks as for the order $Q\rightarrow S$.
Refer to caption: x3.png

### Appendix K Analysis of the upper bound of the dot product error

#### K.1 Upper bound is reachable

To test if the upper bound derived in Theorem 3.9 is reachable in practice, we randomly sampled $1000$ blocks of size $64$ from a standard normal distribution $\mathcal{N}(0,1)$, and applied $2$:$4$ sparsity and HBFP6 quantization.
We then compute the aggregate error of the composition and individual error term of the upper bound.
Subsequently, we quantified how much the upper bound deviates from the actual composition error using the following formula:

$$ $\text{Deviation}=\frac{|\varepsilon_{I,s}^{D}(\mathbf{x},\mathbf{w})|+| \varepsilon^{D}_{q}(\mathbf{x},\mathbf{w})|+|\varepsilon_{t}|+|\varepsilon_{i} |}{|\varepsilon_{q,c}^{D}(\mathbf{x},\mathbf{w})|}$ (49) $$

As a corollary of Theorem 3.9, the minimal value of deviation is $1$.

Figure 3 shows the deviation distribution of samples.
Most values fall into the first bin, suggesting the upper bound is frequently reached.
It can also be seen that the deviation values can be large, almost reaching the value of $10$, which indicates that the upper bound can be pessimistic in some cases too.
Theorem 3.9 does not rule out large values of deviation, as it applies the triangle inequality to obtain the upper bound, which leads to dropping the sign of each error term.
If the values of the error terms are negative, they can make the overall error of the composition lower than the upper bound, which leads to the deviation values larger than one.

#### K.2 Contribution of additional error terms

Section 3.2 describes how each additional error term can contribute to the overall error of the composition.
We hypothesize that the error term $\varepsilon_{t}$ contributes less in case of applying sparsity followed by quantization than in case of applying quantization first.
We also hypothesize that the magnitude of the error term $\varepsilon_{i}$ is much lower than the magnitude of $\varepsilon_{t}$.
To test our hypotheses, we normalized the values of each term of the upper bound to compare their contribution to the error.
We considered both orders of applying the transformations.
We also only looked at the samples with low deviation values ($<1.05$) to increase the explainability power of the upper bound.

Figures 4 and 5 depict the results of the experiment.
If we consider the order $Q\rightarrow S$ in Figure 4, we can see that the term $\varepsilon_{t}$ advocates for almost half of the error of the composition.
However, in the order $S\rightarrow Q$ the term $\varepsilon_{t}$ has a much lower impact, which proves our first hypothesis.

We can also see that the values of $\varepsilon_{i}$ are much lower that the values of $\varepsilon_{t}$ in most of the cases in both orders. This proves our second hypothesis.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - In Table 2, what do the bold-out results represent? This should be explained in the caption.   
- In Table 2: perhaps it would be beneficial to show the delta to the sparsity 0% instead/additionally  (e.g. in the appendix)?   

Overall, I find the topic of this paper both interesting and potentially valuable to researchers focusing on sparsity and quantization. The claims and theorems are clearly articulated (I briefly reviewed the details of Theorems 3.5, 3.6, 3.7, and 3.9 in the appendix), and the empirical evaluation, while primarily centered on magnitude-based sparsity, is compelling and conducted across various models and tasks. I believe the strengths of the paper outweigh weaknesses (in fact, I do not have significant concerns regarding weaknesses). Therefore, I am inclined to recommend acceptance.
soundness: 3
strengths: - The paper covers an interesting and timely topic. Given the increasing size of parameters in pre-trained models, there is growing interest in techniques such as quantization and sparsity. Providing both analytical and empirical insights into the relationship between these techniques is valuable, especially as they are often studied separately.The findings in this paper, such as the optimal order for applying sparsity and quantization and the established upper bounds, can offer practical guidance for researchers in this area.  
- The paper effectively demonstrates the non-orthogonality of sparsity and quantization, determining the optimal sequence for applying these transformations through theoretical analysis, supported by empirical studies on large, modern networks.  
- The work is well-written, easy to follow, and enjoyable to read.
summary: The paper explores the relationship between two widely used compression techniques: sparsity and quantization. Specifically, it demonstrates that these techniques are not independent of one another; the order in which they are applied can significantly impact the results. Additionally, their combination can lead to error propagation, with accumulated errors affecting consecutive layers. The study draws from both theoretical analysis and experimental results conducted on large, modern neural networks.
weaknesses: - In the experiments section, the results appear promising and generally align with the theoretical findings. However, it is unclear whether the reported results represent averages of multiple runs or single-run outcomes. If they are averages, what are the standard deviations?  
- Additionally, I believe the related work section should remain in the main body of the paper, particularly since there is available space before reaching the 10-page limit. Moving it to the appendix could diminish its visibility and importance.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - Evaluating the findings on larger, more diverse dataset would be nice. 
- It would be interesting to see how the optimal $S\to Q$ order and orthogonality bound extend to other sparsity patterns and quantization schemes. Can the authors comment on the generality of their findings in this regard?
soundness: 4
strengths: - The paper makes significant theoretical contributions by proving the non-orthogonality of sparsity and quantization and deriving the optimal $S\to Q$ order. These insights challenge conventional assumptions and provide valuable guidance for model compression. 
- The mathematical analysis is rigorous and comprehensive, covering tensor-level and dot product-level errors. 
- The experimental results are extensive, spanning diverse models (OPT, LLama, ResNet, ViT) and settings.
- The orthogonality bound metric seems like a useful tool for practitioners. 
- Overall the paper is well-structured, with clear definitions, detailed appendices, and informative tables.
summary: This paper provides a comprehensive theoretical and empirical investigation into the interplay between sparsity and quantization, two widely used model compression techniques. The authors mathematically prove that sparsity and quantization are non-orthogonal operations, meaning their combined use introduces compounded errors beyond those incurred by each method independently. They further derive the optimal order of applying sparsity before quantization (S→Q) to minimize additional errors. These theoretical findings are validated through extensive experiments on large language models (OPT, LLaMA), vision transformers (ViT), and convolutional neural networks (ResNet). The paper also introduces the novel "orthogonality bound" metric to efficiently estimate the performance of sparse-quantized models without expensive retraining.
weaknesses: - While the experiments cover a range of models and settings, the datasets used (WikiText2, ImageNet1k) are relatively small and few. Evaluating on larger, more challenging datasets would further strengthen the findings. 
- The paper does not explore the impact of different sparsity patterns (e.g., block-wise sparsity) or more advanced quantization schemes.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: * In the Q->S case, the authors make the argument that this ordering may lead to additional errors when two otherwise unequal weights in the non-quantized precision are set to the same value once quantized. This is an intuitive conclusion but it would be interesting to ground this discussion in empirical evidence of the proportion of weights that this affects, on average, in a pre-trained model. 
* Are the pretrained LLMs obtained from the base models or instruct-tuned variants? Making this explicit in the paper would be beneficial. 
* L312 states that all linear layers were compressed for LLMs. Can you confirm that this included the lm-head, but not the encoder which is typically implemented as an embedding?
* Table 10 values for 1:4 are counter to typical intuition that higher sparsities generally perform worse. Could the authors confirm that this is 1:4 and not 3:4 sparsity?
soundness: 2
strengths: * A timely and important topic as sparsity and quantization are promising compression strategies for the large model scales popular today.
* The paper includes a comprehensive summary of relevant literature.
* The proofs are relatively easy to follow and explained in an intuitive manner by the authors in the main text.
* Empirical results generally appear to support the theoretical findings.
* While many works have studied the combination of sparsity and quantization, this is the first that I am aware of to rigorously consider the interplay between these methods in detail. 
* Empirical experiments include both LLMs and vision models. 
* Extensive supplementary info includes an analysis of several leading SOTA methods from LLM pruning and quantization literature.
summary: This paper studies the interaction between weight sparsity, weight quantization, and activation quantization in small-to-moderate sized LLMs, ViTs, and CNNs. The authors prove and demonstrate empirically that these methods cannot be considered as purely orthogonal compression modalities under the orthogonality definitions proposed in the paper. Specifically, the authors show that the composition of these strategies is order-dependent and the combined error incurred generally exceeds the sum of the errors produced by applying each method independently.
weaknesses: Overall I am leaning towards accept; however, some concerns regarding the empirical experimental design causes me to doubt the applicability of the results to more general settings:

* The primary metrics considered in the empirical results are perplexity or cross-entropy loss. While these are certainly reasonable proxies for downstream task performance, they are not perfectly correlated. While some accuracy metric for CV models was included in the appendices, it would be beneficial to extend this to downstream tasks for LLMs such as the OpenLLM v1 leaderboard evaluation tasks. It has been shown previously that PPL and CE can be particularly misleading metrics for quantized and sparse models [1]. 
* The experimental design for Section 4.1 is potentially concerning. If I understand the described process correctly, in the Q->S case the pretrained models are pruned and quantized before each forward pass (i.e., instantaneous masking and quantizing). Are the parameters themselves stored as dense fp32 tensors during this process and quantization is simulated similar to QAT approaches? Are the optimizer states left in fp32? The authors note issues with training dynamics in the Q->S case in Appendix A and my concern is that this could be related to numerical precision issues during fine-tuning rather than providing a reliable comparison on the order of compression. Adding a more detailed summary of the fine-tuning approaches in the appendix would potentially clear up any misunderstandings on this point. 
* In the Q->S case quantized activations are used but in the S->Q case it appears the full precision activations are used. It's unclear to me if the dramatic difference in performance is caused by the quantized activations during fine-tuning rather than the specific order of compression for the weights. 


[1] A. Jaiswal, Z. Gan, X. Du, B. Zhang, Z. Wang, and Y. Yang, “Compressing LLMs: The Truth is Rarely Pure and Never Simple,” Oct. 02, 2023, arXiv: arXiv:2310.01382. doi: 10.48550/arXiv.2310.01382.


### Suggestions / Typos:
* Defining “tensor and dot-product levels” earlier in the text would improve the reader's understanding. Specifically it may be worthwhile to relate these terms to “weights” and “activations” respectively.  I note that activations / dot-products are also represented as tensors. 
* On L68, the authors refer to the challenge of quantizing LLMs due to outliers in “tensor distributions” and reference the smoothquant paper. This should be corrected to “dot-product outliers” as the challenge typically arises from outliers in the activations, not the weights (which instead follow a more gaussian-like distribution typically). 
* I suggest separating references for fine-grained (N:M and similar) and structured (neuron-level or larger NN components) sparsity in the related work discussion on L115. In particular, it would be beneficial to introduce N:M sparsity before it appears in section 3. 
* L469: state-of-the-arts -> state-of-the-art

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - In Section 3, the authors mention performing quantization at the level of “blocks." Could you clarify what you mean by a “block” in this context? Does it refer to a set of weights associated with a CNN filter, or does it resemble the M:N sparsity blocks? Or is it something entirely different?
- Consider renaming Definition 3.4 to avoid confusion, as it defines "orthogonality" between two functions in a way that diverges from the standard interpretation. Traditionally, orthogonality between functions is defined by the condition \(\int f(x) g(x) \, dx = 0\), so using "orthogonality" here might lead to misinterpretation.
- Why is it important to consider block-wise quantization in Section 3? Since it’s a theoretical derivation, why don’t you simply assume quantization on the tensor level?
- Theorem 3.5 assumes “max-scaled block-wise quantization”. Is such quantization prevalent in the literature and in practice?
- Theorems 3.5 and 3.6 imply that the optimal order is pruning followed by quantization. Theorem 3.7 analyses the error for the suboptimal order. Why is that of interest?
- Is Equation 12 a lower bound or an upper bound? You might want to rename it accordingly to “Orthogonality Lower Bound” or “Orthogonality Upper Bound” to help the reader.
- “If the compression methods are non-orthogonal, and the evaluation metric indicates better model performance with lower values, we expect the compressed model’s evaluation metric to exceed the orthogonality bound.” — I read this sentence several times and I still can’t understand it. What do you mean by “lower values”? Which values?
- “For OPT, LLaMA, ViT, and ResNet fine-tuning, we employ sparse fine-tuning on a dense” — what method exactly are you using? Please cite the paper.
- “we apply one-shot quantization to sparse fine-tuned models” — again, what method exactly are you using? Please cite the paper. is it the “max-scaled block-wise quantization”?
- “we directly fine-tune the model in a quantized and sparsified manner” — how does one fine tune a quantized model? Isn’t there an issue with doing that?
- Could you summarize the “Experimental setup” in the form of a table. Otherwise, there are too many details in the paragraph and it’s very hard to digest.
- In Figure 1, the error accumulates across layers. This stands in contrast to Figure 1 in [1] which shows attenuation of noise injected in intermediate layers. Could it be that the authors should compute a relative error instead of an absolute error (see caption in Figure 1 of that paper)?
- “and/or reduce quantization effective bitwidth.” — what do you mean by “effective bit width”?
- “TOPS/mm2” — what’s TOPS and mm^2?

[1] Stronger Generalization Bounds for Deep Nets via a Compression Approach
soundness: 2
strengths: - Notation, definitions, and theorems in Section 3 are generally clear and their significance is adequately articulated.
- The authors have addressed an issue that has gone overlooked in the pruning/quantization literature through both theoretical proofs and derivations as well as empirical studies that further solidify their claims.
summary: The authors present a theoretical proof showing that the order in which magnitude-based pruning and scaled block quantization are performed is of importance in the context of preserving model performance. The authors define notions such as orthogonality of two operations — which is when the composition of the two operations does not result in any additional error than applying each individual transformation. The authors show theoretically that magnitude pruning and quantization are not orthogonal operations (when going beyond tensor-level) and further showed, both theoretically and empirically, that applying pruning first and then quantization generally leads to better performance.
weaknesses: - The discussion following Theorem 3.9 is very hard to digest for a reader who hasn’t spent as much time as the authors thinking about this problem. I’d encourage the authors to prune the text, retaining only the essential message (which presumably is what’s written in italics) and moving other information to the Appendix.
- Overall, the theoretical claims and experiments are not astonishing as one would perhaps expect that pruning should precede quantization.
- The theoretical contribution is quite limited as it only holds for magnitude-based pruning (without fine tuning) and block-wise quantization. Importantly, magnitude pruning has gone out of fashion in the context of LLMs because it requires costly fine-tuning to recover model performance and is outperformed by methods like SparseGPT and WANDA when fine-tuning is not performed. The authors mention in the Appendix that, empirically, the order had less of an impact for WANDA and SparseGPT. 
- The experiments seem to be quite orthogonal to the theoretical results. By employing fine-tuning for all the experiments, the authors are making their original theoretical proofs/derivations inapplicable in the context of the experiments as the derivations are based on errors calculated when no fine-tuning is applied. 
- Proof of Theorem 3.5: Only show equality is attained for L1 norm and not all norms. Is it clear that this implies that equality is also achieved for all other norms? Statement of Theorem or proof should be modified to address this. 
- Proof of Theorem 3.6 is only a counter-example for the L1 norm. Is it immediate that the theorem is true in general for norms beyond the L1 norm? Either the statement of the theorem or the proof should be modified to address this.
- Throughout the paper, some statements are true for all norms, others are only shown for the L1 norm, and then the empirical experiments utilize the L2 norm for measuring errors. 
- The generalization of orthogonality in Definition 3.8 is not clear to me as functions are now being applied coordinate-wise. Is the composition only permitted to happen in one coordinate (similar to in Theorem 3.9). It might be worth it to explicitly write out the definition as the lack of an explicit definition of orthogonality also makes the statement of Theorem 3.9 confusing.

Meta-review: In this paper the authors consider the error introduced by various model compression schemes for deep networks, such as sparsifying or quantizing the numerical precision of the weights.  While both approaches are commonly used, here the authors consider the combination of the two methods.  In particular, the authors provide theoretical results showing that the ordering of the sparsification and quantification operators matters, with more error occurring if quantification is done before sparsification, and that even if applied in correct order the errors from applying the two techniques in combination can be greater than the sum of the errors from applying each technique independently.

While the basic message of these results are perhaps intuitive and to be expected, the reviewers are largely in agreement that having a theoretical characterization of these effects is a worthwhile contribution.  Some reviewers still have concerns that the applicability of the work may be limited as the results only apply to magnitude based sparsity pruning without fine-tuning, whereas methods like SparseGPT and WANDA may have stronger performance in settings where fine-tuning is too costly.  While this is potentially a valid concern, this seems to be a reasonable avenue for future work, and I agree with the overall consensus of the reviewers to accept the paper for publication.

Decision: Accept (Spotlight)