Low-Precision Training of Large Language Models: Methods, Challenges,
  and Opportunities
Authors: name: Zhiwei Hao, name: Jianyuan Guo, name: Li Shen, name: Yong Luo, name: Han Hu, name: Guoxia Wang, name: Dianhai Yu, name: Yonggang Wen, name: Dacheng Tao

Large language models (LLMs) have achieved impressive performance across
various domains. However, the substantial hardware resources required for their
training present a significant barrier to efficiency and scalability. To
mitigate this challenge, low-precision training techniques have been widely
adopted, leading to notable advancements in training efficiency. Despite these
gains, low-precision training involves several componentsx2013such
as weights, activations, and gradientsx2013each of which can be
represented in different numerical formats. The resulting diversity has created
a fragmented landscape in low-precision training research, making it difficult
for researchers to gain a unified overview of the field. This survey provides a
comprehensive review of existing low-precision training methods. To
systematically organize these approaches, we categorize them into three primary
groups based on their underlying numerical formats, which is a key factor
influencing hardware compatibility, computational efficiency, and ease of
reference for readers. The categories are: (1) fixed-point and integer-based
methods, (2) floating-point-based methods, and (3) customized format-based
methods. Additionally, we discuss quantization-aware training approaches, which
share key similarities with low-precision training during forward propagation.
Finally, we highlight several promising research directions to advance this
field. A collection of papers discussed in this survey is provided in
https://github.com/Hao840/Awesome-Low-Precision-Training.