Title: Breaking the Efficiency-Accuracy: Fusion of Rotation Quantization and N:M Sparsity for LLMs Inference
Primary area: infrastructure, software libraries, hardware, systems, etc.
Keywords: Large Language Models, Low-bit Quantization, N:M Sparse Pruning, Hardware-aware Optimization, Inference Acceleration

Abstract: Large Language Model (LLM) inference presents substantial computational challenges when executed on commodity hardware, thereby necessitating the development of efficient acceleration techniques. Low-bit quantization has been effectively employed to enhance the speed of LLM inference. Nonetheless, existing methodologies predominantly concentrate on the reduction of weight bit-width, often neglecting the potential for acceleration through the exploitation of weight sparsity at various locations within LLMs. In this paper, we introduce a novel framework that integrates rotation-based low-bit quantization with $N{:}M$ unstructured pruning. This framework encompasses: \textit{i)} $N{:}M$ sparse pattern optimization, which is aimed at identifying optimal sparsity configurations. These configurations are chosen to maximize throughput-per-second (TPS) while minimizing accuracy degradation throughout the low-bit quantization process, and \textit{ii)} in response to the combinatorial explosion associated with identifying optimal $N{:}M$ patterns, we develop hardware-aware performance models that adeptly navigate the expansive $N{:}M$ parameter space to ascertain the optimal sparse pattern. Extensive experiments conducted on the Llama-2 model family reveal that our framework sustains benchmark accuracy degradation within 1\%, concurrently achieving state-of-the-art inference acceleration on NVIDIA RTX 3090 GPUs. On Llama-2-70B, we achieve 3.65$\times$ prefill throughput improvement and 50\% memory reduction. This research bridges the divide between low-bit quantization and unstructured pruning through effective $N{:}M$ exploration strategies, thereby advancing the frontiers of extreme LLM compression for deployment on commodity hardware.