How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study
Authors: name: Wei Huang; name: Xudong Ma; name: Haotong Qin; name: Xingyu Zheng; name: Chengtao Lv; name: Hong Chen; name: Jie Luo; name: Xiaojuan Qi; name: Xianglong Liu; name: Michele Magno
Published: 2024-04-22 10:03:03

Abstract: Meta's LLaMA family has become one of the most powerful open-source Large
Language Model (LLM) series. Notably, LLaMA3 models have recently been released
and achieve impressive performance across various with super-large scale
pre-training on over 15T tokens of data. Given the wide application of low-bit
quantization for LLMs in resource-limited scenarios, we explore LLaMA3's
capabilities when quantized to low bit-width. This exploration holds the
potential to unveil new insights and challenges for low-bit quantization of
LLaMA3 and other forthcoming LLMs, especially in addressing performance
degradation problems that suffer in LLM compression. Specifically, we evaluate
the 10 existing post-training quantization and LoRA-finetuning methods of
LLaMA3 on 1-8 bits and diverse datasets to comprehensively reveal LLaMA3's
low-bit quantization performance. Our experiment results indicate that LLaMA3
still suffers non-negligent degradation in these scenarios, especially in
ultra-low bit-width. This highlights the significant performance gap under low
bit-width that needs to be bridged in future developments. We expect that this
empirical study will prove valuable in advancing future models, pushing the
LLMs to lower bit-width with higher accuracy for being practical. Our project
is released on https://github.com/Macaronlin/LLaMA3-Quantization and quantized
LLaMA3 models are released in https://huggingface.co/LLMQ.