###### Abstract  
As Deep Neural Networks (DNNs) continue to increase in complexity, the computational demands of their training have become a significant bottleneck. Low-precision training has emerged as a crucial strategy, wherein full-precision values are quantized to lower precisions, reducing computational overhead while aiming to maintain model accuracy. While prior research has primarily focused on minimizing quantization noise and optimizing performance for specific models and tasks, a comprehensive understanding of the general principles governing low-precision computations across diverse DNN architectures has been lacking. In this paper, we address this gap by systematically analyzing the factors that influence low-precision matrix computations, which are fundamental to DNN training. We investigate three critical factors--accumulation in matrix calculations, the frequency of element usage, and the depth of matrices within the model--and their impact on low-precision training. Through controlled experiments on standard models, as well as customized experiments designed to isolate individual factors, we derive several key insights: layers with higher accumulation and matrices with lower usage frequencies demonstrate greater tolerance to low-precision noise, without significantly compromising the stability of model training. Additionally, while the depth of matrices influences the stability of matrix operations to some extent, it does not have a noticeable effect on the overall training outcomes. Our findings contribute to the development of generalizable principles for low-precision training, offering a systematic framework applicable across various DNN architectures. We provide empirical evidence supporting the strategic allocation of training bit-widths based on the analyzed factors, thereby enhancing the efficiency and effectiveness of DNN training in resource-constrained environments. low-precision training; quantization noise; matrix computations; training efficiency; accumulation; frequency; depth +
Footnote †: journal: Applied  
However, this transformation is not without its challenges. The inherent trade-off between precision and performance remains a significant concern, as the process of quantization inevitably introduces noise, which can degrade model accuracy if not carefully managed. This trade-off highlights the delicate balance that must be struck between reducing computational demands and maintaining the integrity of the model's predictions. The challenge is further compounded by the diverse architectures and varying computational characteristics of different DNN models, which makes it difficult to develop a one-size-fits-all solution. In recent years, considering the diverse computational characteristics across different DNN layers, mixed-precision training--which flexibly allocates different bit-widths to different layers based on low-precision training--has emerged as a promising research direction. Many studies treat the bit-width of each layer as a hyperparameter of the model and determine appropriate bit-widths through Quantization-Aware Training (QAT) [6; 7]. However, this approach primarily focuses on obtaining reasonable mixed-precision inference models without optimizing the DNN training process. Common methods for optimizing DNN training typically require in-depth analysis of the training process and treat bit-width selection as an optimization problem [8; 9; 10], or employ specially tailored data formats and optimization strategies [11; 12; 13]. While these strategies enable more fine-grained trade-offs between computational resources and model accuracy, particularly suitable for energy-efficient scenarios, they often lack universality despite performing well on specific tasks. This makes them challenging to adapt to emerging models and architectures, or they require complex analysis processes with substantial optimization overhead and difficulty.