VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large
  Language Models
Authors: name: Yifei Liu; name: Jicheng Wen; name: Yang Wang; name: Shengyu Ye; name: Li Lyna Zhang; name: Ting Cao; name: Cheng Li; name: Mao Yang
Published: 2024-09-25 16:25:45

Abstract: Scaling model size significantly challenges the deployment and inference of
Large Language Models (LLMs). Due to the redundancy in LLM weights, recent
research has focused on pushing weight-only quantization to extremely low-bit
(even down to 2 bits). It reduces memory requirements, optimizes storage costs,
and decreases memory bandwidth needs during inference. However, due to
numerical representation limitations, traditional scalar-based weight
quantization struggles to achieve such extreme low-bit. Recent research on
Vector Quantization (VQ) for LLMs has demonstrated the potential for extremely
low-bit model quantization by compressing vectors into indices using lookup
tables.
  In this paper, we introduce Vector Post-Training Quantization (VPTQ) for
extremely low-bit quantization of LLMs. We use Second-Order Optimization to
formulate the LLM VQ problem and guide our quantization algorithm design by
solving the optimization. We further refine the weights using
Channel-Independent Second-Order Optimization for a granular VQ. In addition,
by decomposing the optimization problem, we propose a brief and effective
codebook initialization algorithm. We also extend VPTQ to support residual and
outlier quantization, which enhances model accuracy and further compresses the
model. Our experimental results show that VPTQ reduces model quantization
perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on Mistral-7B,
4.41-7.34 on LLaMA-3 over SOTA at 2-bit, with an average accuracy
improvement of 0.79-1.5% on LLaMA-2, 1% on Mistral-7B, 11-22% on
LLaMA-3 on QA tasks on average. We only utilize 10.4-18.6% of the
quantization algorithm execution time, resulting in a 1.6-1.8times
increase in inference throughput compared to SOTA.