Pruning vs Quantization: Which is Better?
Authors: Andrey Kuzmin
Markus Nagel
Mart Van Baalen
Arash Behboodi
Tijmen Blankevoort
Venue: NeurIPS 2023 poster (2023)
Keywords: Neural network quantization
neural network pruning
magnitude pruning
post-training quantization
quantization-aware training
TL;DR: An extensive comparison between quantization and pruning containing analytical error computations for data distributions and error lower bounds for post-training scenarios

Abstract: Neural network pruning and quantization techniques are almost as old as neural networks themselves. However, to date, only ad-hoc comparisons between the two have been published. In this paper, we set out to answer the question of which is better: neural network quantization or pruning? By answering this question, we hope to inform design decisions made on neural network hardware going forward. 
We provide an extensive comparison between the two techniques for compressing deep neural networks. 
First, we give an analytical comparison of expected quantization and pruning error for general data distributions.
Then, we provide lower and upper bounds for the per-layer pruning and quantization error in trained networks and compare these to empirical error after optimization.
Finally, we provide an extensive experimental comparison for training 8 large-scale models trained on 3 tasks and provide insights into the representations learned during fine-tuning with quantization and pruning in the loop.
Our results show that in most cases quantization outperforms pruning. Only in some scenarios with a very high compression ratio, compression might be beneficial from an accuracy standpoint.

Pruning vs Quantization: Which is Better?
Andrey Kuzmin, Markus Nagel, Mart van Baalen, Arash Behboodi, Tijmen Blankevoort Qualcomm Research" Amsterdam, The Netherlands
[akuzmin, markusn, mart, behboodi, ti jmenmeqti..... qual comm. com
Abstract
Neural network pruning and quantization techniques are almost as old neural networks themselves. However, to date only ad-hoc comparisons between the two have been published. In this paper, set out to answer the question which is better: neural network quantization or pruning? By answering this question, we hope inform design decisions made neural network hardware going forward. We provide an extensive comparison between the two techniques for compressing deep neural networks. First, give analytical comparison of expected quantization and pruning error for general data distributions. Then, we provide lower bounds for the per-layer pruning and quantization error trained networks, and compare these to empirical error after optimization. Finally, we provide an extensive experimental comparison for training large-scale models on 4 tasks. Our results show that in most cases quantization outperforms pruning. Only in some scenarios with very high compression ratio, pruning might be beneficial from an accuracy standpoint.
Introduction
Recent advances in deep learning led to exceeding human-level performance in many tasks, including computer vision, machine translation, voice recognition, and language understanding. Real-world applications of DNNs rely heavily on their efficiency. Both mobile and cloud platforms greatly benefit from reduced latency and energy efficiency achieved by some form of model compression. In this work, we consider two mainstream techniques used in practice; pruning and quantization.
Pruning methods remove individual weights 77 25 procedure can reduce the memory footprint. Furthermore, not having to perform the computations with weights that are zeroed out can make network inference more efficient. On the other hand, quantization reduces the bit-width used for both the weights and the computation used in networks, leading both predictable memory savings and reductions in the necessary compute. In both scenarios, the hardware used for making use of these optimization schemes needs to take them into account.
sometimes groups of weights [28 47 This
Depending on the availability training data and computing budget, most methods for pruning and quantization fall into one of two families. The first family includes fine-tuning approaches, namely quantization-awee training (QAT) and fine-tuning with pruning in the loop. The second family includes post-training approaches such post-training quantization (PTQ). Previously, pruning techniques primarily relied on fine-tuning; however, some post-training pruning methods appeared recently as fine-tuning is not desirable for large language models [1].
Despite the importance of model efficiency and the plethora of approaches for pruning and quanti- zation, the two fields are mostly disjoint. The literature presents little insight into which of the two
Qualcomm All Research initiative Qualcomm Technologies,
Coode is available at https: //tgtttttbbb////////////t com cooalccomm-ae htw.ahocccccaaaaaaaaaaaaaeeeeaaa or
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

1914 Quarkoot on varsus 25% pruntin NI5.3
Squered ere nnigiteed by P0o
Comperion of 5ki
Mtrgrral
Quatc
ritudia Fruring
Figure 1: Comparison for standard normal distribution. (left) Distributions after pruning and quantization for INT4 and 75% pruning. (middle) The squared error weighted by probability. (right) SNR for different compression ratios.
techniques is more accurate. In practice, there is only limited time to compress network and limited energy to spend on making deep learning inference hardware. For this reason, we ask the question: Should one focus on quantization or pruning for compression?
We present an extensive study comparing pruning and quantization in equal settings. First, we consider different data distributions and analyze the conditions under which each method is preferable. We match findings with real weight tensors from pre-trained models. Second, consider post- training scenario and evaluate single-layer output errors for both methods. Because the comparison might depend on the specific choice of optimization method, we compare the two with theoretical bounds that apply regardless of the optimization method. Finally, we provide full-model comparison for the most common scenario of fine-tuning networks after either pruning or quantization.
In our comparison, we intentionally avoid considering the hardware aspects of pruning and quantiza- tion. Instead, we focus solely the accuracy of both methods, given similar theoretical compression ratios. A coarse discussion on the hardware necessary for both methods can be found in section 6
2 Assumptions
In our work, we assume FP16 the basic data type and measure any gains in compression with respect to it. Using FP16 for inference generally does not lead to a loss in accuracy. Neural networks also very commonly trained with FP16. making it common baseline. Thus, we compare 50% pruning sparsity to INT8 quantization, 75% sparsity to INT4 quantization and so forth. We also assume overhead storing the sparsity mask for pruning and relegate such hardware-speccfic implementations to section(6]
For the pruning experiments, we consider magnitude pruning. It is common to do fine-tuning after or during pruning (70. Several works have independently shown that despite its simplicity, is tough to improve upon magnitude pruning and fine-tuning [19] 4 To our knowledge, no pruning algorithm exists that consistently outperforms this method.
For the quantization experiments, we use symmetric uniform quantization, which is defined by just the quantization scale factor and the bit-width. The scale is represented floating-point number and is used to map floating-poit point values to the integer grid. Further details on symmetrie uniform quantization can be found 44 Uniform quantization is the standard in the quantization literature, and symmetric quantization is mostly employed for the weights. In all our experiments, we use quantization range estimator minimizing the mean-squared error on weights by grid search [49]
3 Comparison on statistical distributions
Before diving into comparison results, we first describe theoretically what the quantization error and pruning error are. Looking this with a theoretical lens helps with understanding the later experimental difference between the two methods. We start off by describing and analyzing both methods on simple data distributions.
In order to compare the error of pruning and quantization, we will frequently use the signal-to-non -o-se ratio measure defined in the log scale: SNR 10 logio (E [w*] /E [(W F[W))?]), where F(W) is the quantization pruning function. This measure is the same scaled logarithm of

---

Pruning SMR
Quanticatin
Pruning SNR Quant SNR
bits
Quertication
2-
30
10
unitng butlier
Figure 2: Comparing the error of pruning and quantization for a student-t distribution, simulating the presence of significant outliers. We plot the results for different magnitudes of the outliers, as the kurtosis on the x-axis. (left) the pruning error, which does not change under the presence of more severe outliers. (middle) the quantization SNR, which is reduced greatly when outliers increase (right) the trade-off regions where quantization and pruning are better.
an MSE measure. Both are often employed to analyze the sensitivity of neural network layers to quantization, and they are theoretically well-founded correlate with network performance [4 48
3.1 Quantization error
For quantization, we consider symmetric uniform quantization, which also called integer quan- tization. Given bit-width b and the scale 8, the grid nodes are defined {-2h, ......... 0,2 1}.. The quantization operation counding-to-neane Q(w) and the corresponding quantization eroor R(w) are defined as:
8i,
(ww) Qi, i = arg min ww 9il,
R(w) Q(w)(w)
(1)
Following 36 model neural network weights random variable W p(w). The expected
value the quantization MSE can be expressed as follows:
Arin
(Q(WW) w) / (((w)p(W)du +
/ (w Jmin) (amin)ppuue p(w()dw + f
(Amar - p( p(w)dw;
qmaz
(2)
where Jmin min; and Jmar max; are the quantization range limits. The left term corresponds to the rounding error, and the right two terms correspond to the clipping error. We use this analytic formulation for our distribution results below, the details are given in appendix
3.2 Pruning error
We consider magnitude pruning T(x) - This simply the values closest actual zero. Given this, the expected error pruning is expressed as follows:
E [T(W)]] - w (upp(w)ddu.
where t is the threshold value that controls how much is pruned. Given the compression ratio e E (0, 1), we find the threshold value which satisfies P(-t S W < t) = c. In case of a symmetric zero-mean distribution, the threshold can be expressed as = Fw ( + 5). where F(w) = P(W < w) the CDF function and F-1(p) is its inverse. The expected pruning error in equation 3 similar to the clipping error for quantization (see the second and the third term in equation 2) and can also be computed analytically. We also use this formulation for our results below.
3.3 Analytical comparison
Standard normal distribution. us first look a standard normal distribution. As many weights in neural networks are roughly Gaussian-shaped this distribution is useful for our understanding of

---

Openidant disan 2a Dn palrc rood way (9744)
Froting MHA Quer
1i cour exantry
38:0um countly
Wore ye caunity
Bute vs 10.u saanday
Suartbatan 2AB
Figure 3: (left) Comparison on all the weights from Py Torch model zoo (46 models) combined with large language models (Bloom-3b, Llama-3b, OPT-2.7b) 2b).. (left) Pruning SNR versus quantization SNR for every tensor. (right) Pruning is preferable at high compression ratios for tensors with high sample kurtosis values.
the comparison. As we can see from figure (middle), the errors for both methods have very different behavior. The quantization error oscillates between the quantization nodes and has a moderate range. The pruning error effectively corresponds to rounding many weights to zero and thus has higher error. As we can see in figure (right), this results a higher SNR for quantization, e.g. 19.1 dB for INT4 quantization versus only 5.6 dB for 75% pruning. We see similar results for different compression ratios. For this distribution, quantization achieves a much higher signal-to-pee to-noise ratio.
Distributions with heavy tails. The trade-off is expected to change when more significant outliers are introduced. The quantization grid is expected be effected strongly by outliers increases the quantization grid in size, whereas the pruning method is expected to be hardly effected with outliers as it only affects weights around zero. We thus analyze both quantization and pruning errors in the presence of many outliers. To simulate a distribution with outliers, we use truncated Student's s-t distribution with v 2, and symmetric range (-r,r) (the PDF defined appendix B This distribution is nice as it gives non-trivial weight to the tail ends of the distribution close to r. The wider the range r is, the heavier are the tails of the distribution.
In order to introduce quantitative measure of the number of outliers, we will use the distribution's
kurtosis given by Kurt[X]= E (x 1)))] E [x -=)²)²² where μ is the mean. We will see later that this kurtosis measure is predictive of quantiation and pruning performance for real layers. To increase the number outliers, we will increase the range r. The results are given in figure 2 The kurtosis range is chosen so that it includes most of the weights from the model zoo. We see that despite the significant outliers and high kurtosis, quantization still has higher SNR in most of the cases for moderate compression. Pruning is better however in the region of high clipping range and very high compression rate, e.g. 2-3 bits per value (see figure 2 on the right).
3.4 Experiments on real weight tensors
The previous discussion was mostly theoretical. We set out to see happens when we do similar analysis on real neural network weights. In order to investigate this, we compare the pruning and quantization SNR the weight tensors for all the pre-terined models from the PyTorch model zod? (46 models total, the details are give in appendix Eb combined with weight tensors from 3 large language models, namely Bloom-3b (3) Llama-3b [20]. OPT-2.7b [67]. Each tensor is quantized using an integer grid of bit widths from 2 to 8. The results are shown in the figure (left).
We similar trend to our previous discussion that pruning becomes more beneficial for lower bitwidth/higher sparsity ratios.
In order match the analytical results from figure we consider the sample kurtosis of every weight tensor given by 1: = N E - x) Ei=(" (a; - ==) See figure (right). We consider range of kurtosis values for every quantization bit-width. Using kernel density estimator, we compute the probability density of encountering a tensor for which pruning has higher SNR than

---

quantization SNR. We compare the PDF to that for quantization and thus determine the region where each method preferable. The results are given in figure on the right. We see that the results from the previous theoretical section (figure on the right) hold very nicely. We can also see that as predicted, the kurtosis is indeed good metric for predicting if a tensor should be quantized or pruned for optimal accuracy.
Per-layer comparison
Most PTQ methods compress the model layer by layer. Given one layer, we use the mean-squared eтror of the output activations as an objective for optimization. As [48] shows, minimizing per layer MSE on the output activations of each layer is computationally affordable second-oree -rderr approximation of the loss function. The local MSE objective correlates well with the task loss and
often used in practice in DNN compression and quantization literature [32 40 68 Our experiments in appendix D confirm this. For the experiments this section, we will use SNR as it represents a
normalized version of MSE. As opposed to section 3 where we used SNR weights, in this section, we will use SNR on the output activations instead.
The goal of a PTQ method is to minimize the error in the output activations of the compressed layer by optimizing over the quantized weights subject integer range constraints. Similarly, for pruning. the weights are optimized subject to a sparsity constraint. As the underlying combinatorial optimization problem for both methods NP-hard [56 [4], in practice, each method relies on some form of heuristic providing reasonably good solution given realistic compute budget. This means that any practical comparison between pruning and quantization would depend on the choice of the method for both and would be open dehate of the optimality of the algorithm. In order to eliminate this dependence, we provide a tight lower bound on the output errors for quantization. For pruning we provide a way to solve the problem exactly for moderate dimensionalities. This way, we can provide a comparison that holds regardless of the algorithm used for each method.
4.1 Post-training quantization
We set out to formulate a way by which we can get relatively tight bounds for comparison when quantizing single layer with the MSE as the objective. The higher bound is simple to obtain by using a solution with heuristic quantization algorithm, but for the lower bound, we have to reformulate the problem. The mean-squared error of the output activations of quantized layer can be expressed as:
min E(w) = X6w
X Ww Worigllg 1
s.t. E Z"
Wmnnn S wi S Wmaz:
where X is the input data in an unfolded form, and Worig are the floating point weights. The quantized weights are computed as the product of the quantization scale ō, and the integer weights w. Wyin and Wmen are the integer limits. We ignore the averaging operation simplify the notation, not important for optimization. We also note that this problem can be solved independently for each output channel of convolution or every row fully-connected layer weight.
This problem is an instance of a mixed-integer quadratic program:
B(w) = w Pw - q" w,
(5)
s.t. w E Zn Wmin < Wmans
where P 28 -= X 2( ( X6.
order to simplify the objective, we can omit the constant term that irrelevant for the optimization
XXwaiggll 11 i.e. Ê(W) E(W) c. In order to find the lower bound of the objective, we follow (55) and relax the integer constraint to wi (w, 1) 2 0, which allows the weight to take values within the interval from 0 to 1. In order to
5

---

obtain the lower bound, we will consider the dual version of the relaxed problem:
L(x)
max 77
(6)
P (A)
q 2
s.t.
(1+
7 =0,
220,
where À E Y E R. The dual problem is convex, and its solution can be used lower bound on the solution of the original problem, i.c., E(w) L()). The dual has semi-definite constraint which can be solved with a semi-definite programming (SDP) solver with (n)²) complexity. In our work, used CVX solver [21]. As discussed in [55), this bound is computationally efficient alternative to branch-and-boun methods introduced in [5]. We use this approach for estimating the lower bound for MSE on the output activations for PTQ below.
approaches, while tightness is better than that for the alternative
4.2 Post-training pruning
We also need similar lower bound for pruning for comparison. To the best of our knowledge we are not aware of the ways to provide tight lower bound for this problem, therefore we formulate way to solve problem for moderate dimensionalities exactly. Similar quantization, post-training pruning of one layer of the network can mathematically be expressed as solving the following optimization problem:
min X Xx Worigllz
s.t.
1 lo S..
where the number of non-zero elements in the solution is theoretically constrained by using the Lo norm, which non-convex and smooth. In order to solve the problem, we introduce the sparsity mask E IR":
E(w) = min X (m @ w)
X Worig 12
(8)
|| =
m @ 1 S ú < m Qu
1 m, E {0, 1},
where is element- wiseent-wise product operation, and L, IR are constants chosen such that any solution satisfies the constraint -m @ v S m ④ u. We solve this problem using the branch-and- bound method implemented in the Gurobi solver 23 that gives the global solution.
4.3 Experiments
With our algorithms in the bag. we can now compare quantization versus pruning in the post-trainin settings with theoretical bounds. In each case, analyze individual layers of several networks. Given a batch of input data, we optimize the pruned or quantized weights to minimize the error between the output activations and the output of the uncompressed layer. We provide range between two SNR values for each method in each case. The performance of the heuristic method gives the first value, and the second value is given by the error lower bound the global solution, which translates into SNR upper bound.
As heuristic method for pruning, optimized weights given by argmin X (m w) X Woriggl? This is convex problem
magnitude pruning with fixed sparsity mask m and data-
w
and has a unique solution. As heuristic method for quantization, we use the mixed-integer solver introduced in [55]. We clip every sample in order satisfy the integer quantization range constraint.
We chose a representative set of layers, including 9 convolutional layers (one 3x3 convolutional layer and 8 point-wise convolutions) from V22 EfficientNet-lite, and Resnet-18. 18, and one fully-connected layer from VIT. The full details for reproducing the experiments are given in appendix F Due to the high computational complexity of the global solution for pruning, the layers

---

Per layer compartion (PP16 basic
BTT 20.0% spenaty
NTT ap5-mpy
IiT--v ppaapaiity
spanity
Pruning StAR
-0
12
Quantratton SNR
Figure 4: Comparison in the post-training scenario. Each box corresponds to a subset of one of 10 layers from the different models that were used, with 7 different bit-width comparison points. The ranges of the box indicate the lower and higher-bounde found by the algorithms.
had to be split into chunks. The slice of 4 input channels all output channels was used for 3x3 convolutions. In the case of linear layers and point-wise convolutions, slices 36 input features over all the output features were used.
The results are shown figure grouped by bit-width. The rectangles indicate the full range of pruning and quantization methods between the heuristic solution and the error lower bound or the global solution. Whenever a rectangle for each chunk intersects the diagonal line, the ranking of the two methods could depend the optimization method, while in cases below above the diagonal, the ranking is guaranteed regardless of the optimizer. see that quantization mostly outperforms pruning for moderate compression, while methods become more comparable for higher compression ratios.
Full-model comparison
Now that we have seen the comparison between the methods in the PTQ setting, we turn to fine-tuning quantized and pruned models. This the setting where pruning is applied in most, and is possible that fine-tuning can change the models significantly enough that the performance between the two methods changes.
In order to provide a fair comparison of pruning and quantization, we chose the two most commonly used methods with performance competitive state-of-the-art. For quantization-aware training, used the widely adapted LSQ method suggested in [12 2] Following this approach, we jointly learn the weights and quantization scales, keep the batch norm layers unfolded, and re-estimated the batch norm statistics after training to avoid wrong running estimates due to oscillations [5I]. We use the method suggested in 77 for pruning, which gradually increases the sparsity during fine-tuning and re-estimates batch norm statistics after training.
In experiments used set of 4 models trained for tasks including Resnet18, Resnet50 27), MobileNet-V V2 (58) MobileNet- V3-small (30], EfficientNet-lite (60), and ViT [1T] trained on Im- ageNet classification [57]: Deeplab-
V3 L3
7 with MobileNet-V2 backbone trained for semantic
segmentation on Pascal VOC [33 EfficientDet 61 trained for object detection on MS COCO 43 OPT-350 fine-tuned on WikiText-103.
For a fair comparison, we used the same amount of epochs of fine-tuning for each method (full details on hyperparameters are given in appendix G). The results given in suggest that pruning almost never leads to higher accuracy than quantization an equal compression rate is considered. The differences sufficiently large enough that the small purported improvements by some methods 59 will likely not close the gap.
To study the effect of training time, we also performed an ablation with 2 times longer fine-tuning on subset of 3 models (Resnet50, EfficientNet, and ViT). The results are given appendix H We observe that pruned models generally benefit from fine-tuning more, and in particular pruning
7

---

Model
Orig.
Metric
Method
8b
7b
6b
5b
4b
3b
2b
quant.
70.5
70.5
70.6
70.3
70.0
68.9
67.3
Resnet-18
69.7
acc.
pruning
70.3
70.1
69.9
69.5
69.3
68.3
66.8
quant.
76.4
76.4
76.4
76.3
76.2
75.5
72.3
Resnet-50
76.1
acc.
pruning
76.6
76.4
76.2
76.1
75.9
75.4
74.3
quant.
71.9
72.0
71.7
71.6
70.9
68.6
59.1
MobileNet-V2
71.7
acc.
pruning
68.1
65.6
61.9
56.3
48.0
34.0
21.2
quant.
75.2
75.3
75.0
74.6
74.0
71.5
60.9
EfficientNet
75.4
acc.
pruning
72.5
70.9
68.1
63.6
56.4
44.5
27.1
quant.
67.7
67.6
67.1
66.3
64.7
60.8
50.5
MobileNet-V. V3
67.4
pruning
65.6
64.4
62.4
60.2
56.1
31.7
0.0
quant.
81.5
81.4
81.4
81.0
80.4
78.4
72.2
VIT
81.3
acc.
pruning
76.6
76.6
76.2
73.1
72.4
71.5
69.4
quant.
72.3
72.3
72.4
71.9
70.8
63.2
17.6
DeepLab-V V3
72.9
mloU
pruning
65.2
62.8
56.8
47.7
32.9
18.6
10.0
quant. 39.6
39.6
39.6
39.2
37.8
33.5
15.5
EfficientDet
40.2
mAP
pruning
34.5
33.0
30.9
27.9
24.2
17.9
8.0
quant.
14.8
14.8
14.9
15.0
15.3
15.9
19.9
OPT-350m
14.8
perpl.
pruning
18.0
19.7
22.6
27.2
35.4
53.5
101.4
Table I:
Comparison of QAT and magnitude pruning with fine-tuning given equal model size and equal number of epochs for fine-tuning.
becomes more beneficial for most compression ratios on Resnet50, However, for the other models, quantization is still more beneficial due to a larger gap in performance.
Combining pruning and quantization Another interesting questions is whether pruning is ben- eficial in combination with quantization. To answer it, we perfromed an experiment pruming quantized Resnet-18. MobileNet-V2 and ViT with different pruning ratios. The results are given on figure [] On x-axis we plot the expected bit-widths which is a product of the base bit-width and the sparsity in the pruned model including the natural sparsity. The points marked by crosses are quantized models with only natural sparsity and no extra pruning applied. As we can see, mild degrees of pruning are beneficial the combinations. However, we note that no extra overhead was assumed for storing the pruning mask.
Figure 5: Combining pruning and quantization on ImageNet models. The average bit-widths shown on x axis is computed a product of the base bit-width and the density of non-zero weight elements. Different pruning ratios are applied to each base bitwidth model. Quantized models with only natural sparsity and no extra pruning are marked with crosses.
Discussion
Other types of pruning While we solely focused in our comparison on unstructured pruning in which individual weights are removed, our results translate to semi-structured and structured pruning. Unstructured pruning has more degrees of freedom and is a strict superset of what can be

---

represented by bemii-sstuuctures pruning. Therefore, unstructured pruning gives an upper bound of the accuracy for all pruning methods. This means that for the cases in which quantization is better than unstructured pruning, quantization will also be better than (semi-)structured pruning. However, we can not make any claims for (semi-)structured pruning for the few scenarios in which pruning better than quantization.
Natural sparsity in quantized tensors In our comparison, we used a theoretical compression ratio for quantization, which depends on the bitwidth. However, we also observe that quantized tensors naturally contain many zeros; for example, 8-bit tensors from Py PyTorch model zoo have an average sparsity of 13% while 4-bit tensors are 35% sparse, We give more details on this in appendix
Representations learned in the compressed models To provide insights into representations learned during pruning or QAT, we studied the evolution of models during fine-tuning. We found that fine-tuning after pruning tends recover the original representation, while quantization-ane training leads to learning completely new representations. We provide further details on these experiments in appendix
Hardware implications So far, we have deliberately avoided discussing the hardware implementa- tions of pruning and quantization and focused solely on the accuracy of both methods at the same ideal compression rates. However, in practice, the hardware considerations do matter for the usability of the methods.
The analysis above assumed an idealistic case for pruning in terms of memory size and data transfer. Since the pruning is unstructured, in order to achieve memory savings practice, one would need at least 1 bit of information for each weight indicating whether weight is pruned or not. On top of 16-bit weights, this gives 6.25% storage overhead at a minimum. Quantization does not have this overhead, as INT8 is just 8 bits smaller than 16 bits, and the only storage overhead is single scaling factor per tensor (or channel).
Also, in terms of the cost of computations done by the hardware, there is a difference between the two methods. For pruning, any hardware would have to take the densely stored weights and mask and either decompress them to the dense format with all weights and many Os or take the pruning into account in the compute itself. No compute benefits are gained in the former, the dense calculations are done in the uncompressed number format. In the latter, dedicated hardware to take into account the is necessary. The overhead for this is generally non-trivial, leading vendors implement more semi-stte semiistrucctued pruming schemes (47). Similarly, it is rare to see unstructured activation compression for the same reason that this needs to happen algorithnically on-the-fly. In contrast, quantization gives quadratic improvements in the compute. Going from INT8 to INT4 theoretically improves the compute performance by factor 4, although practical gains depend on the memory overhead (which improves by only factor 2x) and the existence of other formats in the same hardware compute unit.
Impact Using pruning or quantization leads to power reduction on many architectures and enables new applications on mobile platforms. We see only a positive impact from this on the whole. In some cases both pruning and quantization might lead to biased predictions, a further discussion can be found in [29].
Limitations First, our work has not extensively considered the hardware implications of pruning or quantization. Second, we do not study combinations of pruning and quantization apart from analyzing the inherent sparsity due pruning. We leave this for future work. Finally, we consider only uniform quantization and ignore the other formats, such as low-precision floating or logarithmic quantization, although these are not likely to change the results presented in this paper.
Related work
Quantization Integer quantization, or fixed- fxed-point quantization, one of the most widely used techniques for inference, allowing to reduce the latency and improved energy efficiency. There are two main families of methods for model quantization. The first family includes post-training quantization
(PTQ) methods 42 52 10
optimization of the quantized weights which improve the model accuracy based on par-layer quantization-awant aware training methods
I
9
6
48
40)
data-optimized fashion. The second family includes
22 34
69
8
44
12,
35|
63
5]]
which usually fine-tune

---

the model with quantization in the loop using thtrighttthruuhh estimator (STE) for computing the gradient of rounding operations. more comprehensive overview of quantization methods can be found in 50
Pruning Neural network pruning one of the oldest methods to compress neural networks 37 26
A central problem in pruning is how to choose which weights to prune. Approaches published in the literature include: binary gating, in which a binary gate is learned on each individual weight
45 46
[4]; sensitivity-basel methods (39, 38 66 [8 in which sensitivity. based on weights'
gradient or hessian diagonal value, used, and magnitude pruning [24 54 70, 47, 59]. While
conceptually simple, magnitude- based methods have been shown to consistently outperform more
intricate methods at scale [19] 4). Weight re-initialization schemes [6] or mask reinitialization
[59] yield additional minor improvements. While most pruning approaches require fine-tuning and yield unsatisfactory results in post-training scenarios, recent adaptations of Hessian- flessianbbased sensitivity approaches B7 26 in which the Hessian of layerwise reconstruction loss is used instead of the task loss Hessian, show good pruning results in post-training pruning of large language models [7 [8].
Combining pruning and quantization A number of works study combinations of pruning and quantization with different levels of granularity [24 64 31
651
Comparing pruning and quantization Despite the large amount of work pruning, quantization, and combining them, there is little literature comparing the two methods. To the best of our knowledge, the closest work that performs a comparison of pruning versus non-uniform quantization [33]. The work considers only small-scale models and provides only an empirical comparison with no further analysis. Another related study is (53].
8 Conclusion
We have seen in this paper that in several settings, unstructured pruning only performs better than quantization in rare cases. In our theoretical analysis of distributions and on the real-layer-data, pruning is only better than quantization, compressing the network to an equivalent of 2 or 3 bits. This amount of compression comes with such a degree practice. The post-training quantization results are also informative. In the setting without fine-tuning, we have shown with theoretical bounds on many layers in neural networks that quantization is almost always provably better than pruning. Our hypothesis is that quantized layers are accurate than pruned ones, as shown in the theoretical and PTQ setting, and fine-tuning a network still highly dependent on that. This is in line with fine-tuming results, in which for many networks trained under the same conditions, quantization always has higher performance than pruning.
drop in performance it is rarely used in
The conclusion is clear: Quantization generally outperforms pruning for neural networks. Taking into account the unfavorable hardware implications for pruning described, it could be argued that the conclusion holds even stronger. Based this research, recommend quantizing neural networks when efficiency required before pruning is explored.
Acknowledgement
We would like to thank Marios Fournarakis and Yelisei Bondarenko for their help with performing QAT experiments.
References
[1] Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for mepidddeployment. In Advances in Neural Information Processing Systems, 2019.
[2] Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.
[3] BigScience Workshop. BLOOM (revision 4ab0472), 2022.
10

---

[4] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag, What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129-1466, 2020.
[5] Christoph Buchheim, Ruth Huebner, and Anita Schoebel. Ellipsoid bounds for convex quadratic integer programming. SIAM Journal on Optimization, 25(2):741-769, 2015......
[6] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. arXiv preprint arXiv:2001.0028/ 2020.
[7] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous
convolution for semantic image segmentation, 2017.
[8] Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: parameterized clipping activation for quantized neural networks. arXiv preprint arxiv:805. 06085, 2018.
[9] Yoni Choukroun, Eli Kravchik, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. International Conference Computer Vision (ICCV), 2019.
[10] Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed -reeedd-rrisioon. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 293-302, 2019.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2020.
[12] Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization. International Conference on Learning Representations (ICLR), 2020.
[13] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
[14] Simon Foucart and Holger Rauhut, Mathematical Introduction to Compressive Sensing. Applied and Numerical Harmonic Analysis. Springer New York, New York, NY, 2013.
[15] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprini arXiv: 03635,, 2018.
[16] Jonathan Frankle, Gintare Karolina Dziugaite. Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv: DrXXiv1193.. 1903.00031000 016111611,10 2019.
[17] Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post- training quantization and pruning. arXiv preprini arXiv:2208. 1580, 2022.
[18] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023.
[19] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: 0e... 09574, 2019.
[20] Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023.
[21] Michael Grant and Stephen Boyd. CVX: Matlab software for disciplined convex programming, version 2.1. http:://vvai wtt.comcccc.o cvx March 2014.
[22] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, (ICML),
2015.
[23) Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.
[24] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint 15 100.00019, 2015.
[25] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[26] Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference neural networks, pages 293-299, IEEE, 1993.
11

---

[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition. 2016.
[28] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017.
[29] Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. Characteris- ing bias compressed models. arXiv preprint urXiv:0010003058, 2020.
[30] Andrew Howard, Ruoming Pang, Hartwig Adam, Quoe Le, Mark Sandler, Bo Chen, Wei- jun Wang, Liang- Chinh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Searching for mobilenetv3. In International Conference on Computer Vision (ICCV). 2019.
[31] Peng Hu, Xi Peng, Hongyuan Zhu, Mohamed M Sabry Aly, and Jie Lin. Opq: Compressing deep neural networks with one-shot pruning- quantization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7780-7788, 2021.
[32] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. International Conference on Machine Learning, pages
PMLR, 2021.
[33] Yerlan Idelbayev and Miguel Á Carreira-Perpiîaán An empirical comparison of quantization, pruning and low-rank neural network compression using the Ic toolkit. In 202/ International Joint Conference Neural Networks (IJCNN), pages 1-8. IEEE, 2021.
[34] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-aritheeticonnly inference. Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[35] Sambhav R. Jain, Albert Gural, Michael Wu, and Chris Dick. Trained uniform quantization for accurate and efficient neural network inference on fixed-point hardware. arxiv preprint arxiv: 08066, 2019.
[36] Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort. Fp8 quantization: The power of the exponent. arXiv preprint 22020020.. 2022.
[37] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
[38] Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr A signal propaga- tion perspective for pruning neural networks at initialization. arXiv preprint 100.v7790..06307, 2019.
[39] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruming based on connection sensitivity. arXiv preprint 1rx0v:1810.02340, 2018
[40] Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang. Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang. and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations (ICLR), 2021.
[41] Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization of deep convolutional networks. In International conference on machine learning, pages 2849-2858. PMLR, 2016.
[42] Darryl D. Lin, Sachin S. Talathi, and V. Sreekanth Annapureddy. Fixed point quantization of deep convolutional networks. In International Conference on Machine Learning. 2016.
[43] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014, pages 740-755, Cham, 2014. Springer International Publishing.
[44] Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Relaxed quantization for discretized neural networks. In International Conference on Learning Representations (ICIR), 2019.
[45] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through | 0 regularization. arXiv preprint 1rXiv:1712.01312, 2017.
12

---

[46] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through in regularization. International Conference on Learning Representations (ICLR), 2018.
[47] Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378 2022.
[48] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up down? Adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML), 2020.
[49] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. ArXiv, abs/2106.08295. 2021.
[50] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. ArXiv, abs/2106.8882995
2021.
[51] Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, and Tijmen Blankevoort, Overcoming oscillations in quantization- training. In International Conference on Machine Learning, pages 16318 16330. PMLR, 2022.
[52] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In International Conference on Computer Vision (ICCV), 2019.
[53] Satya Sai Srinath Namburi, Makesh Sreedhar, Srinath Srinivasan, and Frederic Sala. The cost of compression: Investigating the impact of compression on parametric knowledge in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5255-5273, 2023.
[54] Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring sparsity in recurrent neural networks. arXiv preprint arXiv: 2017.
[55] Jachyun Park and Stephen Boyd. A semidefinite programming method for integer convex quadratic minimization. Optimization Letters, 12:499-518 2018.
[56] Alberto Del Pia, Santanu Dey, and Marco Molinaro. Mixed-integer quadratic programming is in np. Mathematical Programming, 162:225-240, 2017.
[57] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy. Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
[58] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[59] Suraj Srinivas, Andrey Kuzmin, Markus Nagel, Mart Baalen, Andrii Skliar, and Tijmen Blankevoort. Cyclical pruning for sparse neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2762-2771, 2022.
[60] Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning (ICML.) 2019.
[61] Mingxing Tan, Ruoming Pang, and Quoc V. Le. Efficientdet: Scalable and efficient object detection, 2020.
[62] Frederick Tung and Greg Mori. Deep neural network compression by in-parallel pruning- quantization. IEEE transactions on paltern analysis and machine intelligence, 42(3):568-579, 2018.
[63] Stefan Uhlich, Lukas Mauch, Fabien Cardinaux, Kazuki Yoshiyama, Javier Alonso Garcia, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Mixed precision duns: All you need is a good parametrization. In International Conference on Learning Representations (ICLR), 2020.
[64] Mart van Baalen, Christos Louizos, Markus Nagel, Rana Ali Amjad, Ying Wang, Tijmen Blankevoort, and Max Welling. Bayesian bits: Unifying quantization and pruning. arXiv preprint 27X0v:2005..7003, 2020.
13

---

[65] Haichuan Yang, Shupeng Gui, Yuhao Zhu, and Ji Liu. Automatic neural network compression by quantiiziquaiiiiioonn joint learning: constrained optimization- based approach. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 2178-21 1778--188 188, 2020.
[66] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 9194-9203, 2018.
[67] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0006 2022.
[68] Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions pattern analysis and machine intelligence, 38(10):1943-195 2015.
[69] Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-nett Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1606.06666 2016.
[70] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint 17Xiv:1710.01878, 2017.
14


Reviews:
[{"review_id": "05XunHxX0z", "reviewer": "Reviewer_S4q6", "summary": "In this paper, the authors try to answer whether pruning or quantization is better for network compression. The paper start by analyze the qunatization error and pruning error under standard normal distribution and heavy-tail distributions. Full-model comparison are done between quantization and pruning with a set of 8 models. In most cases, the quantization method performs better than pruning. Quantization is much better than pruning under extreme compression targets.", "questions": "1. Quantization and pruning are two different network compression techniques. But this paper tries to compare them under the setting of same model size. Nevertheless, model size is not the most important factor for efficient applications. ", "limitations": "The novelty of this paper is quite limited. It is more like an empirical study. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written. The whole writing is to the point. The content is well organized, which makes the paper easy to understand.\n2. Thorough full-model comparison is done for a set of 8 models.", "weaknesses": "1. The standing point of this paper might be biased. Quantization and pruning are not two competitive methods. Each of them have a specific application scenario. They could be used sololy and jointly. According to the experiments, quantization is almost always better than pruning. Yet, this does not mean that pruning is not useful.\n2. The comparison is done such that the model size after pruning and quantization are the same. Not sure whether this comparison is proper.\n3. Even without the investigation in this paper, quantization is still used more often than pruning. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for comments and suggestions. We answer each point below.\n\n**W1**. We totally agree that pruning is useful, and we did not state the opposite in our paper. Rather, for the setups where both methods are supported, using quantization leads to more accurate models. As we mention in the general comment above, quantization is more accurate even in the joint case. \nFor example, pruning is useful in the cases where it is necessary to adjust the model size precisely. The quantised model size is proportional to integer bit-width which makes decrements in the model size relative large. However this can be easily tackled with pruning or a combination of quantization and pruning. \n\n**W2/Q1**. For this work, one might consider at least three metrics: the model size, the bit operations count (BOPs), or run-time on real hardware. The model size is the most simple of the three metrics which is suitable for the direct comparison. In many cases pruning only targets the model size, and resulting memory transfer overheads, unless there are sparse kernels implemented (see section 6 for the discussion on additional pruning overheads). For this reason BOPs count might not be always relevant.\nFinally, as we discuss in section 6 using run-time on real hardware would make the comparison dependent on the specific hardware architecture [mention the discussion]. With that being said, we are open for suggestions on using a different metric if there are some more suitable options.\n\n**W3/Limitation**. We do agree with the reviewer that in practice quantization is used more often than pruning. However, this is in our view not because there are prior extensive and conclusive studies which conclude that quantization is better in terms of accuracy. Likely this is more due to the fact that quantization is more beneficial from a hardware perspective (cf discussion section) and wider supported on common hardware. \nWe respectfully disagree that the novelty of our paper is limited and refer to the general comment above."}, {"review_id": "19CNSTcRJo", "reviewer": "Reviewer_jvrg", "summary": "This paper sets out to answer the question whether quantization or pruning is better. It first provides an analytical analysis of the two methods in terms of signal-to-noise ratio (SNR) and establishes an early relationship between kurtosis and SNR. It then provides a mathematical breakdown of the compression error in both methods. Then, it empirically compares the two methods on 46 models from the PyTorch model zoo to validate the early analytical results. The authors then compress and fine-tune a set of vision models to compare the two methods after fine-tuning and show that quantization still outperforms in this region. Finally it explores the hardware considerations since dense pruned models can be difficult to accelerate on most hardware.", "questions": "Did you consider clipping to make the distribution less sensitive to outliers? This seems standard with weight quantization too since weights are easy to profile and optimize the quantization settings.\n\nFigure 1 (right) is computed with the analytical equations with the normal distribution substituted? \n\nAre there any regions when the combination of the two techniques outperforms the techniques individually?\n\nI'm confused about Appendix H, where the conclusion is that quantization tends to relearn different representations but pruning recovers the previous ones. The figure shows that the distance between the original and new activations is larger for pruning, which should imply that it doesn't relearn the original distributions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The authors use the reasonable FP16 baseline instead of the more common FP32.\n\nThis paper makes an interesting connection between kurtosis and the relative performance of quantization over pruning.\n\nObservation that pruning tends to recover the original representation but quantization builds new ones is intriguing.\n\nThe empirical evaluation spans across image classification and detection tasks.", "weaknesses": "Especially when considering the hardware, it would be useful to consider block sparsity like 2:4, which is supported in some modern GPUs. However, element-wise sparsity presumably has strictly better accuracy / memory performance, so block sparsity would likely be worse.\n\nGiven the recent success of transformers, it may be helpful to include more (in addition to ViT) in the empirical evaluation. That said, I do not expect this will change the conclusion.\n\nYou correctly point out that weight distributions are typically quantized symmetrically but they also typically use channel-wise quantization, which is especially helpful with low bitwidths. I expect channel-wise quantization could further shrink the region where pruning is preferred.\n\nI am slightly concerned about the novelty of this comparison, but I will do a better literature search later in the review process. One other potential critique may be that this conclusion may not be too surprising given the exponential decay in the importance of bits. This decay likely does not hold for individual weights.\n\nMinor: L87 is a little confusing for the definition of T, which is more the error of the magnitude pruning, as opposed to the magnitude pruning itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the details review and useful suggestions. Please find our answers and comments below.\n\n**W1**. Block sparsity is a subset of structured sparsity and therefore indeed we expect it to have strictly worse accuracy for the same model size. This was confirmed in our experiments, e.g., for 50% sparsity when switched from unstructured to 2:4 block sparsity, the validation accuracy decreases for Resnet18 by 0.5%, for Resnet50 by 0.3%, for MNV2 by 1.4%, and for EfficientNet by 1.2%. Hence, more structured sparsity does not change the conclusions in our paper. Structured-sparsity would be accuracy-wise worse, at the trade-off of getting a more efficient hardware implementation. We will include this discussion upon the next revision of the paper.\n\n**W2** We totally agree that adding more transformers is beneficial for our empirical comparison. Below we add the results on SQNR values for tensors of 3 large language models (Bloom-3b, Opt-2.7b, OpenLlama-3b). The conclusions are similar to our experiment in figure 3. We are also working on including quantized and pruned Bloom-560m results with fine-tuning. We will include these results to the next revision of the paper.\n\n| Avg. SQNR       | 8b       | 7b       | 6b       | 5b       | 4b       | 3b       | 2b       |\n|--------------|----------|----------|----------|----------|----------|----------|----------|\n| Quantization \t| 33.8 $\\pm$ 4.7\t| 29.7 $\\pm$  4.2 \t| 25.7 $\\pm$  3.5 \t| 21.6 $\\pm$  2.8 \t| 17.3 $\\pm$  2.0 \t| 12.7 $\\pm$  1.3  \t| 7.5 $\\pm$  0.8  \t|\n| Pruning \t|  12.7 $\\pm$  2.3\t| 10.8 $\\pm$  2.1 \t| 9.2 $\\pm$  1.7 \t| 7.7 $\\pm$  1.4 \t| 6.3 $\\pm$  1.1\t \t| 4.9 $\\pm$  0.8 \t| 3.5 $\\pm$  0.7 |\n\n**W3**. We absolutely agree. We avoided using channel-wise quantization through all parts of the study on purpose in order to clearly demonstrate the region where pruning is preferred. For example, if channel-wise quantization was used for 8-bit quantization, the validation accuracy of Resnet18 would increase by 0.2% and MNV2 by 0.3%. Indeed, the region where pruning is preferred would shrink.\n\n**W4**. To the best of our knowledge this comparison has not been presented in the literature, we refer to the general comment above for the details on our contribution. We are eager to cite any relevant work if found. \n\n**W5**. Thank you for the comment, we are going to clarify  the notation in L87.\n\n**Q1**. Yes, we use clipping with the range estimated using MSE error which is a common practice in quantization literature. \n\n**Q2**. Yes, we use the analytical equations for computing the expected error for quantization and pruning, the details are given in appendix A.\n\n**Q3**. A combination of quantization with some mild pruning outperforms quantization to a lower bit-widths in some cases. However, the picture changes if natural sparsity in quantised tensors is taken into account, we refer the comment on combining pruning and quantization above for further details.\n\n**Q4**. The range of CKA distance is from zero to one (the higher the closer), so larger distance for pruning means distributions closer to the original distributions. We kindly ask the reviewer to clarify the question further in case our comment needs further explanation."}, {"review_id": "rR9GAoWjJm", "reviewer": "Reviewer_XswT", "summary": "The authors compare the performance of post-training quantization and pruning methods with the same compression ratio using a signal-to-noise metric, a kurtosis metric, and, ultimately, model accuracy. They study the expected performance analytically and in simple toy problems, i.e., Gaussian- and Student's t-distributed weights. They also run experiments on real pre-trained models from the PyTorch Model Zoo. They conclude that post-training quantization alone generally outperforms post-training pruning alone based on their per-layer weight fidelity metrics and full-model accuracy.", "questions": "- Why does the SNR metric use weights and not activations?\n- How does the layer-wise SNR metric you evaluate look for the real models in Table 1? Does it match the conclusions based on accuracy?\n- Can you indicate how computationally costly the pruning/quantization methods you employ are?", "limitations": "The authors discuss the limitations that they do not consider combinations of pruning and quantization, hardware considerations, and other quantization or pruning schemes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper represents a complete analysis of post-training pruning alone vs. post-training quantization alone when targeting the same compression ratio.\n- The authors analyze the problem at multiple levels (more theoretical/analytical and more empirical/experiment driven) and find the two viewpoints complement and support each other.\n- The toy model analysis, though straightforward, illuminates the general issues at play for pruning vs. quantization.\n", "weaknesses": "- The SNR appears to be well-motivated, but it would be ideal to demonstrate explicitly that it maps to model accuracy, e.g. by adding a subfigure with accuracy to Fig. 3 and/or adding the SNR metric to Table 1.\n- The authors note that the quantized models naturally exhibit a larger compression ratio than the pruned models because they have a larger fraction of real 0s, which is a minor caveat.\n- Some additional references could be added to the related work, e.g., \n  - combining pruning and quantization: https://arxiv.org/abs/2102.11289, \n  - quantization-aware training: https://arxiv.org/abs/2006.10159, https://arxiv.org/abs/1905.03696\n- In the Impacts section, there is no discussion of the potential decrease in model robustness and generalization when pruning or quantization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments and additional references, please find our comments below.\n\n**W1**. The relation between SNR and model accuracy for both pruning and quantization is demonstrated in appendix D (see figure 6). \n\n**W2**. We agree that naturally appearing sparsity in the quantized value make the direct comparison between pruning and quantization more nuanced, however, if we would take the extra zeros into account that would only improve the quantization results and would not change the conclusions of our paper. We refer the reviewer to the general comment on joint pruning and quantization above.\n\n**W3**. We thank the reviewer for the references, and we are happy to include them to the related work.\n\n**W4**. We thank the reviewer for the comment. We would like to add a discussion on additional bias introduced by both compression techniques upon the next revision of the paper. \n\n**Q1**. We use SNR on activations whenever possible (figures 4,5,7,8), while we begin our presentation with weight SNR experiments (figures 1,2,3) in order to build up the study step by step from the distribution analysis to the full model results.\n\n**Q2**. The per layer SNR metric correlates well with accuracy in post-training quantization or pruning as mentioned in figure 6 of appendix D. In this case each compressed layer is optimized based on local SNR objective. However, the SNR layer does not necessarily make sense when the models are fine-tuned. First, there is a list of SNR values corresponding to each layer. Second, when we fine-tune the model we possibly train a new model by minimizing the loss function, and the new model is not constrained to be close to the original model (see appendix H for the related discussion). In this case the SNR metric might not be related to accuracy.\n\n**Q3**. The computational complexity of pruning and quantization techniques we consider is linear with respect to the number of weights. This overhead is relatively low compared to the computational complexity of per-layer optimization (solving the quadratic program) or fine-tuning. In the latter case the straight-through estimator is used for the gradient’s computation so that the computational complexity of fine-tuning is similar to that of the original model training.\n\n**L1. Combining quantization and pruning**. We refer the reviewer to the general comment above for a discussion on combining quantization and pruning.\n\n**L1. Other pruning or quantization schemes**. we do not expect using different quantization or pruning schemes to change the conclusions of our study. For example using 2:4 sparsity or structured sparsity generally performs worse than unstructured sparsity considered in our work (we discuss this in section 6 \"other types of pruning\"). In our experiment we confirmed that for 50% sparsity when switched from unstructured to 2:4 block sparsity, the validation accuracy decreases for Resnet18 by 0.5%, for Resnet50 by 0.3%, for MNV2 by 1.4%, and for EfficientNet by 1.2%. Using other quantization schemes such as FP8, in its turn, might only improve the results in some cases (https://arxiv.org/abs/2303.17951). "}, {"review_id": "OnX1lTFJWd", "reviewer": "Reviewer_B8sD", "summary": "This submission conducts a series empiricial experiments and anlysis between neural network pruning and quantization. It first used some statistics method to compare pruning/quantization. Then it measure the per-layer error based on a post-training compression framework. Finally, it conducted some experiments using LSQ for quantization and iterative magnitude-based iterative pruning.", "questions": "N.A.", "limitations": "N.A.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Statistics method for compression comparison as a non-parameteric and non-trainable method is interesting.", "weaknesses": "1. The main drawback of the submission is that it barely proposed new algorithm or insight. The submission is a collection of empiricial experiments and analysis.\n2. Comprehensive empiricial experiments can be regarded as contribution. However, the compression methods used in the submission is limited and the experimental setting is not solid. The submission basically focused on magnitude-based pruning and uniform quantization. Author may refer to [1] for some examples.\n\n[1] https://arxiv.org/pdf/1810.05270.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We answer each of their comments below.\n\n**W1**. We agree that our work does not propose a new quantization or pruning method. However, we respectfully disagree that our paper does not bring new insights. We refer to the general comment on the novelty above for the details on our contribution.  In our view a paper with novel insights is also a valuable contribution. In fact, the reference [1] mentioned by the reviewer is a conference paper which also does not suggest any new pruning method. \n\n**W2**. For our comparison, we used gradual magnitude pruning. First, it is not clear which pruning method exactly is the state-of-the-art. For example, the work Gale et. al 2019 claims that magnitude pruning gives state-of-the art results, or the work [1] suggested by the reviewer also suggests that the choice of the pruning method might not be the key for pruned model accuracy. Regarding magnitude pruning accuracy, in fact we report better results for pruning than the reference [1] which is mentioned by the reviewer. The best Resnet50 ImageNet results are reported in table 6 of [1] from the comment above, for example 76.09% of validation accuracy at 60% sparsity while in our experiments we obtain 76.5% of accuracy at 62.5% of sparsity (see Table 1, Resnet-50 at 6 bits).\n\nWe also used uniform quantization for our comparison as a simple and the most widely used technique which is very competitive. Indeed, using non-uniform quantization formats such as FP8 format could lead to slightly better results (see e.g. https://arxiv.org/abs/2208.09225 and https://arxiv.org/abs/2303.17951), however that would only increase the number of cases where quantization is preferrable and thus would not change the overall conclusions of our paper. \n"}]

Author rebuttal:
We thank the reviewers for their thoughtful comments and useful feedback. We are happy to see that they found the paper well written (S4q6), has a thorough empirical evaluation on various tasks, (S4p6, jvrg), that the comparison is performed on various levels (XswT) and that they found our findings interesting such as attributing quantization/pruning performance to the kurtosis values (jvrg) and the analytical analysis on distributions (B8sD). We address some of the common comments below.
## Combining pruning and quantization
While the original scope of the paper was to compare quantization to pruning, we agree that the paper would greatly benefit from also considering their combinations. Below we would like to share the results on combining pruning and quantization for MobileNet-V2 and Resnet18. Each column corresponds to a compression ratio (measured in average bit-width) which can be achieved using different combinations of pruning and sparsity, e.g. a model quantized to INT6 has size equivalent to INT8 model with 25% sparsity. Each diagonal value corresponds to quantization without pruning. We mark the best combination in each column using bold font. 

| MobileNet-V2        | 8b       | 7b       | 6b       | 5b       | 4b       | 3b       | 2b       |
|--------------|----------|----------|----------|----------|----------|----------|----------|
| INT8+pruning         | **71.9** | 71.8     | 59.1     | 47.5     | 35.6     | 0.0      | 0.0      |
| INT6+pruning         | -        | **72.0** | 71.6     | 70.2     | 66.0     | 51.7     | 19.0     |
| INT5+pruning         | -        | -        | **71.8** | 71.4     | 69.1     | 60.1     | 31.7     |
| INT5+pruning | -        | -        | -        | **71.6** | **70.9** | 67.7     | 53.4     |
| INT4+pruning | -        | -        | -        | -        | **70.9** | **70.1** | 64.8     |
| INT3+pruning | -        | -        | -        | -        | -        | 68.6     | **67.9** |
| INT2+pruning | -        | -        | -        | -        | -        | -        | 59.1     |

| Resnet-18        | 8b       | 7b       | 6b       | 5b       | 4b       | 3b       | 2b       |
|--------------|----------|----------|----------|----------|----------|----------|----------|
| INT8+pruning 	| **70.5** 	| 70.3 	| 70.1 	| 69.3 	| 69.0 	| 64.3  	| 61.3  	|
| INT7+pruning 	| -    	| **70.5** 	| 70.2 	| 70.0 	| 69.5 	| 68.0 	| 63.9 	|
| INT6+pruning 	| -    	| -    	| **70.6** 	| 70.1 	| 69.8 	| 68.9 	| 65.9 	|
| INT5+pruning 	| -    	| -    	| -    	| **70.3** 	| **70.0** 	| **69.6** 	| 68.5 	|
| INT4+pruning 	| -    	| -    	| -    	| -    	| **70.0** 	| **69.6** 	| **68.9** 	|
| INT3+pruning 	| -    	| -    	| -    	| -    	| -    	| 68.9 	| 68.4 	|
| INT2+pruning 	| -    	| -    	| -    	| -    	| -    	| -    	| 67.3 	|


As we see from these experiments, quantization alone is better than combinations with pruning in most of cases. Only for a compression ratio below 4 bits, quantization to 3 or 4 bits with added sparsity can be better than the corresponding 2 or 3 bits quantization. 

However, even for the low bit cases the picture changes when taking natural sparsity into account (cf. discussion section and appendix C). 
Below we give the natural sparsity values for quantization:

| Natural sparsity       | 8b       | 7b       | 6b       | 5b       | 4b       | 3b       | 2b       |
|--------------|----------|----------|----------|----------|----------|----------|----------|
| MobileNet-V2 	| 1.3% 	| 2.4% 	| 4.4% 	| 7.7% 	| 13.4% 	| 23.5%  	| 39.9%  	|
| Resnet18 	|  3.4% 	| 5.3% 	| 8.5% 	| 13.3% 	| 20.3% 	| 31.9% 	| 43.4% |


As we see from the tables in some cases of low bit-width the best accuracy is achieved by combining quantization and mild pruning, for example MN-V2 quantized to INT4 model with 33.33% pruning outperforms INT3 model. However, major part of the reason is natural sparsity in quantized tensors. For example, MN-V2 at INT4 is already 13.4% sparse, while it only needs to be 25% sparse to be compressed down to INT3. This model at INT3 has sparsity of 23.3% out of 33.3% needed to compressed it down to INT2 with pruning.  For Resnet18, the natural sparsity values are higher, i.e. 20.3% at INT4 and 31.9% at INT3, so that the natural sparsity almost achieves what is done with pruning without further accuracy loss. For example, Resnet18 at INT3 pruned down to INT2 has accuracy of 68.4%. If we compare this model to INT3 quantization, the latter has a much higher accuracy of 68.9% which has only 2% fewer zeros. And if we compare INT3 with 33.3% pruning to INT2 model, the latter has sparsity of 43% which makes the model even more compressible. 

We further note that pruning only appears beneficial at lower bit-widths where relative overhead of pruning such as storing sparsity mask, which we neglected in this discussion, are potentially much higher (see section 6 of the paper for further details). 
We would be happy to include this discussion upon the next revision of the paper. 

## Comments on the novelty of the paper, the contribution and its value
Some of the reviewers mention lack of novelty in our paper. We respectfully disagree that the novelty of the paper is limited. To the best of our knowledge the insights from the analytical analysis, the lower bounds for post-training pruning/quantization, and the large scale comparison between pruning and quantization across several tasks were not published before. The closest related work we are aware of is [30] which is limited to small scale empirical comparison. It is important to note that these insights are very valuable for ML practitioners as well as HW engineers making decisions on which methods to support, and where to invest their time when making networks more efficient.


Meta-review: Although the reviewers acknowledged the clarity of the paper, the interest of the study, the extensive evaluation on different models, they expressed concerns about the settings used to compare quantization with pruning, the use of uniform pruning instead of structured pruning, the use of a simple uniform quantization method, the lack of evaluation of pruning and quantization together, and the lack of a new algorithm. The authors feedback addressed some of the concerns, but failed to convince all the reviewers, leaving the paper with two acceptance recommendations and two rejection ones.

As the authors provided some experiments of joint quantization and pruning in the feedback, the main point of contention, is the evaluation of a single pruning strategy and a single quantization one. The argument made by the authors, however, correctly answers this: Their conclusion is that, for a given model size, quantization yields better results than pruning in most cases. This was validated for the simplest possible quantization strategy; using a better one would simply further strengthen this argument. This was evaluated for unstructured pruning, which leaves more freedom than structured pruning, and thus, again, using structured pruning would further strengthen the conclusion. The only remaining question is whether the specific unstructured pruning method used by the authors is sufficient to draw this conclusion. The AC agrees with the reviewers, particularly Reviewer B8sD, that evaluating more unstructured pruning methods would make the message stronger. Nevertheless, the method used by the authors is a popular and effective one, and one can expect only relatively small changes in the numbers if using a different method. As such, the AC believes that this study is of interest to the community in its current form. Nevertheless, the AC urges the authors to clarify the reasons why their conclusions generalize beyond the specific methods that they evaluated.
Decision: Accept (poster)