Scaling Law for Quantization-Aware Training
Authors: name: Mengzhao Chen; name: Chaoyi Zhang; name: Jing Liu; name: Yutao Zeng; name: Zeyue Xue; name: Zhiheng Liu; name: Yunshui Li; name: Jin Ma; name: Jie Huang; name: Xun Zhou; name: Ping Luo
Published: 2025-05-20 12:54:43

Abstract: Large language models (LLMs) demand substantial computational and memory
resources, creating deployment challenges. Quantization-aware training (QAT)
addresses these challenges by reducing model precision while maintaining
performance. However, the scaling behavior of QAT, especially at 4-bit
precision (W4A4), is not well understood. Existing QAT scaling laws often
ignore key factors such as the number of training tokens and quantization
granularity, which limits their applicability. This paper proposes a unified
scaling law for QAT that models quantization error as a function of model size,
training data volume, and quantization group size. Through 268 QAT experiments,
we show that quantization error decreases as model size increases, but rises
with more training tokens and coarser quantization granularity. To identify the
sources of W4A4 quantization error, we decompose it into weight and activation
components. Both components follow the overall trend of W4A4 quantization
error, but with different sensitivities. Specifically, weight quantization
error increases more rapidly with more training tokens. Further analysis shows
that the activation quantization error in the FC2 layer, caused by outliers, is
the primary bottleneck of W4A4 QAT quantization error. By applying
mixed-precision quantization to address this bottleneck, we demonstrate that
weight and activation quantization errors can converge to similar levels.
Additionally, with more training data, weight quantization error eventually
exceeds activation quantization error, suggesting that reducing weight
quantization error is also important in such scenarios. These findings offer
key insights for improving QAT research and development.