"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM
  Quantization
Authors: name: Eldar Kurtic, name: Alexandre Marques, name: Shubhra Pandit, name: Mark Kurtz, name: Dan Alistarh

Despite the popularity of large language model (LLM) quantization for
inference acceleration, significant uncertainty remains regarding the
accuracy-performance trade-offs associated with various quantization formats.
We present a comprehensive empirical study of quantized accuracy, evaluating
popular quantization formats (FP8, INT8, INT4) across academic benchmarks and
real-world tasks, on the entire Llama-3.1 model family. Additionally, our study
examines the difference in text generated by quantized models versus their
uncompressed counterparts. Beyond benchmarks, we also present a couple of
quantization improvements which allowed us to obtain state-of-the-art accuracy
recovery results. Our investigation, encompassing over 500,000 individual
evaluations, yields several key findings: (1) FP8 weight and activation
quantization (W8A8-FP) is lossless across all model scales, (2) INT8 weight and
activation quantization (W8A8-INT), when properly tuned, incurs surprisingly
low 1-3% accuracy degradation, and (3) INT4 weight-only quantization
(W4A16-INT) is competitive with 8-bit integer weight and activation
quantization. To address the question of the "best" format for a given
deployment environment, we conduct inference performance analysis using the
popular open-source vLLM framework on various GPU architectures. We find that
W4A16 offers the best cost-efficiency for synchronous deployments, and for
asynchronous deployment on mid-tier GPUs. At the same time, W8A8 formats excel
in asynchronous "continuous batching" deployment of mid- and large-size models
on high-end GPUs. Our results provide a set of practical guidelines for
deploying quantized LLMs across scales and performance requirements.