Title: PQP: A one-shot collaborative method for post-quantization pruning of LLMs
Primary area: foundation or frontier models, including LLMs
Keywords: model compression, one-shot pruning, one-shot quantization, large anguage models

Abstract: Achieving large-scale model compression at a lower cost without significantly compromising performance remains an important challenge in the deployment of LLMs. To address this, we propose Post-Quantization Pruning (PQP), a framework that enables effective synergy between quantization and pruning. Specifically, we introduce a pruning metric tailored for post-quantization scenarios. In addition, we integrate the statistical properties of quantization error into the loss function for block-wise pruning, allowing the model to simultaneously eliminate weights that are unimportant or heavily affected by quantization noise. The PQP method integrates post-training quantization and one-shot pruning, reducing the compressed model's storage requirements to just 1/8 of the original size, with minimal impact on performance. The entire compression process is lightweight, easy to implement, and does not require fine-tuning. Experimental results on LLaMA and LLaMA-2 models demonstrate that PQP consistently outperforms existing methods in compressing models of various scales. Our code is available at \url{https://anonymous.4open.science/r/PQP}.

TL;DR: A one-shot, high-ratio compression method for LLMs leveraging collaborative quantization and pruning.