## 4 Experimental Results on DNNs
### In-Depth Evaluation on Customized Models  
Section 4.2 demonstrates experimental validation using common DNN models, reflecting the impact of three key factors on DNN mixed-precision training. However, due to the structural characteristics of these models, the three factors are not mutually independent (for instance, in CNN models, usage frequency and depth change in tandem), which hinders the separate analysis of each factor's influence. Therefore, we conduct further evaluations and tailor models to achieve precise control over each factor, enabling a more detailed examination. Furthermore, we select different bit-width ranges to more comprehensively validate the training outcomes under various precision levels. **Factor of depth of matrix only. ** We modify the architecture of ResNet-20 on CIFAR: the original ResNet-20 consists of 18 layers divided into three groups, with each group containing three BasicBlocks, excluding the first conv layer and the last linear layer. Except for the first conv layer in each group, the BasicBlocks within the same group are composed of identical conv layers and have feature maps of the same size, meaning that the matrix operations in this part have the same number of accumulations and usage frequency. To investigate the impact of depth on DNN mixed-precision training, we expanded the second group to include 15 BasicBlocks. The model now comprises 44 layers, with 28 layers having the same number of accumulations and usage frequency (as shown in the dashed part of Figure 8), differing only in depth. For this part of the model, we can analyze the effect of depth on mixed-precision training independently. We denote this modified model as ResNet-D.  
Table 3 demonstrates that when depth serves as the only variable (with the number of accumulations in matrix calculations and usage frequency being consistent), there is no discernible correlation between the training bit-widths applied at each layer and the performance of the resulting model. Concurrently, Table 1 illustrates that for models like Transformer and LSTM, where depth and frequency do not change in tandem, the appropriate training bit-width does not exhibit a clear correlation when depth is the sole dominant factor. This indicates that the negative correlation between the appropriate  
Figure 7: Test accuracy and variance of different strategies using ResNet-20 on CIFAR-10/100 and ResNet-50 on ImageNet from 10 replicates. training bit-width and depth observed in the CNN model of Table 1 is likely due more to differences in usage frequency or other factors rather than the depth itself. Additionally, it is observed that although the training bit-width strategy exhibiting a negative correlation with depth does not yield superior final performance, the stability of training is marginally better compared to other strategies. This finding aligns with the analysis presented in Section 3.3. **Factor of usage frequency only. ** From the analysis presented, it is evident that there is no significant correlation between the depth of layers and the suitable training bit-width. This observation raises the question of whether the matrix usage frequency is the primary factor that dictates the resilience against low-precision noise. However, since there is a certain correlation between usage frequency and the accumulation in matrix calculations (as shown in Figures 2 and 3), it cannot be directly analyzed through commonly used models. Therefore, to isolate the impact of usage frequency on the training bit-width at each layer during mixed-precision training, we modify the channel count of each group in the ResNet model to be uniform (with the thickness of the conv layers also being uniform), yet we still reduce the feature map at the first layer of each group.