Scaling Laws for Floating Point Quantization Training
Authors: name: Xingwu Sun; name: Shuaipeng Li; name: Ruobing Xie; name: Weidong Han; name: Kan Wu; name: Zhen Yang; name: Yixing Li; name: An Wang; name: Shuai Li; name: Jinbao Xue; name: Yu Cheng; name: Yangyu Tao; name: Zhanhui Kang; name: Chengzhong Xu; name: Di Wang; name: Jie Jiang
Published: 2025-01-05 02:30:41

Abstract: Low-precision training is considered an effective strategy for reducing both
training and downstream inference costs. Previous scaling laws for precision
mainly focus on integer quantization, which pay less attention to the
constituents in floating-point quantization and thus cannot well fit the LLM
losses in this scenario. In contrast, while floating-point quantization
training is more commonly implemented in production, the research on it has
been relatively superficial. In this paper, we thoroughly explore the effects
of floating-point quantization targets, exponent bits, mantissa bits, and the
calculation granularity of the scaling factor in floating-point quantization
training performance of LLM models. While presenting an accurate floating-point
quantization unified scaling law, we also provide valuable suggestions for the
community: (1) Exponent bits contribute slightly more to the model performance
than mantissa bits. We provide the optimal exponent-mantissa bit ratio for
different bit numbers, which is available for future reference by hardware
manufacturers; (2) We discover the formation of the critical data size in
low-precision LLM training. Too much training data exceeding the critical data
size will inversely bring in degradation of LLM performance; (3) The optimal
floating-point quantization precision is directly proportional to the
computational power, but within a wide computational power range, we estimate
that the best cost-performance precision lies between 4-8 bits.