SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In order to provide a fair comparison of pruning and quantization, we chose the two most commonly used methods with performance competitive to state-of-the-art. For quantization-aware training, we used the widely adapted LSQ method suggested in [12, 2]. Following this approach, we jointly learn the weights and quantization scales, keep the batch norm layers unfolded, and re-estimated the batch norm statistics after training to avoid wrong running estimates due to oscillations [51]. We use the method suggested in [70] for pruning, which gradually increases the sparsity during fine-tuning and re-estimates batch norm statistics after training.
In our experiments we used a set of 4 models trained for 4 tasks including Resnet18, Resnet50 [27], MobileNet-V2 [58], MobileNet-V3-small [30], EfficientNet-lite [60], and ViT [11] trained on ImageNet classification [57]; DeepLab-V3 [7] with MobileNet-V2 backbone trained for semantic segmentation on Pascal VOC [13]; EfficientDet [61] trained for object detection on MS COCO [43]; OPT-350 fine-tuned on WikiText-103.
For a fair comparison, we used the same amount of epochs of fine-tuning for each method (full details on hyperparameters are given in appendix G). The results given in table1 suggest that pruning almost never leads to higher accuracy than quantization if an equal compression rate is considered. The differences are sufficiently large enough that the small purported improvements by some methods [59] will likely not close the gap.
To study the effect of training time, we also performed an ablation with 2 times longer fine-tuning on a subset of 3 models (Resnet50, EfficientNet, and ViT). The results are given in appendix H. We observe that pruned models generally benefit from fine-tuning more, and in particular pruning
Model	Orig.	Metric	Method	8b	7b	6b	5b	4b	3b	2b
Resnet-18	69.7	acc.	quant.	70.5	70.5	70.6	70.3	70.0	68.9	67.3
pruning	70.3	70.1	69.9	69.5	69.3	68.3	66.8
Resnet-50	76.1	acc.	quant.	76.4	76.4	76.4	76.3	76.2	75.5	72.3
pruning	76.6	76.4	76.2	76.1	75.9	75.4	74.3
MobileNet-V2	71.7	acc.	quant.	71.9	72.0	71.7	71.6	70.9	68.6	59.1
pruning	68.1	65.6	61.9	56.3	48.0	34.0	21.2
EfficientNet	75.4	acc.	quant.	75.2	75.3	75.0	74.6	74.0	71.5	60.9
pruning	72.5	70.9	68.1	63.6	56.4	44.5	27.1
MobileNet-V3	67.4	acc.	quant.	67.7	67.6	67.1	66.3	64.7	60.8	50.5
pruning	65.6	64.4	62.4	60.2	56.1	31.7	0.0
ViT	81.3	acc.	quant.	81.5	81.4	81.4	81.0	80.4	78.4	72.2
pruning	76.6	76.6	76.2	73.1	72.4	71.5	69.4
DeepLab-V3	72.9	mIoU	quant.	72.3	72.3	72.4	71.9	70.8	63.2	17.6
pruning	65.2	62.8	56.8	47.7	32.9	18.6	10.0
EfficientDet	40.2	mAP	quant.	39.6	39.6	39.6	39.2	37.8	33.5	15.5
pruning	34.5	33.0	30.9	27.9	24.2	17.9	8.0
OPT-350m	14.8	perpl.	quant.	14.8	14.8	14.9	15.0	15.3	15.9	19.9
pruning	18.0	19.7	22.6	27.2	35.4	53.5	101.4
Table 1: Comparison of QAT and magnitude pruning with fine-tuning given equal model size and equal number of epochs for fine-tuning.
becomes more beneficial for most compression ratios on Resnet50. However, for the other models, quantization is still more beneficial due to a larger gap in performance.
Combining pruning and quantization Another interesting questions is whether pruning is beneficial in combination with quantization. To answer it, we performed an experiment on pruning quantized Resnet-18, MobileNet-V2 and ViT with different pruning ratios. The results are given on figure 5. On x-axis we plot the expected bit-widths which is a product of the base bit-width and the sparsity in the pruned model including the natural sparsity. The points marked by crosses are quantized models with only natural sparsity and no extra pruning applied. As we can see, mild degrees of pruning are beneficial in the combinations. However, we note that no extra overhead was assumed for storing the pruning mask.
Figure 5: Combining pruning and quantization on ImageNet models. The average bit-widths shown on x axis is computed as a product of the base bit-width and the density of non-zero weight elements. Different pruning ratios are applied to each base bitwidth model. Quantized models with only natural sparsity and no extra pruning are marked with crosses.
Other types of pruning While we solely focused in our comparison on unstructured pruning in which individual weights are removed, our results translate to semi-structured and structured pruning. Unstructured pruning has more degrees of freedom and is a strict superset of what can be
represented by (semi-)structured pruning. Therefore, unstructured pruning gives an upper bound of the accuracy for all pruning methods. This means that for the cases in which quantization is better than unstructured pruning, quantization will also be better than (semi-)structured pruning. However, we can not make any claims for (semi-)structured pruning for the few scenarios in which pruning is better than quantization.
Natural sparsity in quantized tensors In our comparison, we used a theoretical compression ratio for quantization, which depends on the bitwidth. However, we also observe that quantized tensors naturally contain many zeros; for example, 8-bit tensors from PyTorch model zoo have an average sparsity of  $13\%$  while 4-bit tensors are  $35\%$  sparse. We give more details on this in appendix C.
Representations learned in the compressed models To provide insights into representations learned during pruning or QAT, we studied the evolution of models during fine-tuning. We found that fine-tuning after pruning tends to recover the original representation, while quantization-aware training leads to learning completely new representations. We provide further details on these experiments in appendix I.
Hardware implications So far, we have deliberately avoided discussing the hardware implementations of pruning and quantization and focused solely on the accuracy of both methods at the same ideal compression rates. However, in practice, the hardware considerations do matter for the usability of the methods.
The analysis above assumed an idealistic case for pruning in terms of memory size and data transfer. Since the pruning is unstructured, in order to achieve memory savings in practice, one would need at least 1 bit of information for each weight indicating whether a weight is pruned or not. On top of 16-bit weights, this gives a  $6.25\%$  storage overhead at a minimum. Quantization does not have this overhead, as INT8 is just 8 bits smaller than 16 bits, and the only storage overhead is a single scaling factor per tensor (or channel).
Also, in terms of the cost of computations done by the hardware, there is a difference between the two methods. For pruning, any hardware would have to take the densely stored weights and mask and either decompress them to the dense format with all weights and many 0s or take the pruning into account in the compute itself. No compute benefits are gained in the former, as the dense calculations are done in the uncompressed number format. In the latter, dedicated hardware to take into account the 0s is necessary. The overhead for this is generally non-trivial, leading vendors to implement more semi-structured pruning schemes [47]. Similarly, it is rare to see unstructured activation compression for the same reason that this needs to happen algorithmically on-the-fly. In contrast, quantization gives quadratic improvements in the compute. Going from INT8 to INT4 theoretically improves the compute performance by a factor 4, although practical gains depend on the memory overhead (which improves by only a factor  $2\mathrm{x}$ ) and the existence of other formats in the same hardware compute unit.
Impact Using pruning or quantization leads to power reduction on many architectures and enables new applications on mobile platforms. We see only a positive impact from this on the whole. In some cases both pruning and quantization might lead to biased predictions, a further discussion can be found in [29].
Limitations First, our work has not extensively considered the hardware implications of pruning or quantization. Second, we do not study combinations of pruning and quantization apart from analyzing the inherent sparsity due to pruning. We leave this for future work. Finally, we consider only uniform quantization and ignore the other formats, such as low-precision floating or logarithmic quantization, although these are not likely to change the results presented in this paper.
Quantization Integer quantization, or fixed-point quantization, is one of the most widely used techniques for inference, allowing to reduce the latency and improved energy efficiency. There are two main families of methods for model quantization. The first family includes post-training quantization (PTQ) methods [42, 52, 10, 1, 9, 6, 48, 40], which improve the model accuracy based on per-layer optimization of the quantized weights in a data-optimized fashion. The second family includes quantization-aware training methods [22, 34, 69, 8, 44, 12, 35, 2, 63, 51] which usually fine-tune
the model with quantization in the loop using straight-through estimator (STE) for computing the gradient of rounding operations. A more comprehensive overview of quantization methods can be found in [50].
Pruning Neural network pruning is one of the oldest methods to compress neural networks [37, 26]. A central problem in pruning is how to choose which weights to prune. Approaches published in the literature include: binary gating, in which a binary gate is learned on each individual weight [45, 46, 64]; sensitivity-based methods [39, 38, 66, 17, 18] in which sensitivity, based on a weights' gradient or hessian diagonal value, is used, and magnitude pruning [24, 54, 70, 47, 59]. While conceptually simple, magnitude-based methods have been shown to consistently outperform more intricate methods at scale [19, 4]. Weight re-initialization schemes [15, 16] or mask-reinitialization [59] yield additional minor improvements. While most pruning approaches require fine-tuning and yield unsatisfactory results in post-training scenarios, recent adaptations of Hessian-based sensitivity approaches [37, 26], in which the Hessian of a layerwise reconstruction loss is used instead of the task loss Hessian, show good pruning results in post-training pruning of large language models [17, 18].
Combining pruning and quantization A number of works study combinations of pruning and quantization with different levels of granularity [24, 64, 31, 65, 62, 65].
Comparing pruning and quantization Despite the large amount of work on pruning, quantization, and combining them, there is little literature comparing the two methods. To the best of our knowledge, the closest work that performs a comparison of pruning versus non-uniform quantization [33]. The work considers only small-scale models and provides only an empirical comparison with no further analysis. Another related study is [53].
We have seen in this paper that in several settings, unstructured pruning only performs better than quantization in rare cases. In our theoretical analysis of distributions and on the real-layer-data, pruning is only better than quantization, compressing the network to an equivalent of 2 or 3 bits. This amount of compression comes with such a degree of a drop in performance it is rarely used in practice. The post-training quantization results are also informative. In the setting without fine-tuning, we have shown with theoretical bounds on many layers in neural networks that quantization is almost always provably better than pruning. Our hypothesis is that quantized layers are more accurate than pruned ones, as shown in the theoretical and PTQ setting, and fine-tuning a network is still highly dependent on that. This is in line with fine-tuning results, in which for many networks trained under the same conditions, quantization always has higher performance than pruning.
The conclusion is clear: Quantization generally outperforms pruning for neural networks. Taking into account the unfavorable hardware implications for pruning described, it could be argued that the conclusion holds even stronger. Based on this research, we recommend quantizing neural networks when efficiency is required before pruning is explored.
We would like to thank Marios Fournarakis and Yelisei Bondarenko for their help with performing QAT experiments.
[1] Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. In Advances in Neural Information Processing Systems, 2019.
[2] Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.
[3] BigScience Workshop. BLOOM (revision 4ab0472), 2022.
[65] Haichuan Yang, Shupeng Gui, Yuhao Zhu, and Ji Liu. Automatic neural network compression by sparsity-quantization joint learning: A constrained optimization-based approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2178–2188, 2020.
[66] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9194–9203, 2018.
[67] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[68] Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence, 38(10):1943-1955, 2015.
[69] Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160, 2016.
[70] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017.