## 3 Low Precision in Matrix Calculations
### Accumulation in Matrix Calculations  
For DNNs, the computations involved in the model are typically realized through matrix multiplications. Consider the multiplication of two matrices of real numbers,  
Figure 1: Framework of our matrix computation experiments. and \(r_{2}\), with their product represented by \(r_{3}=r_{1}r_{2}\) calculated in full precision (FP32). We denote the entries of each of these matrices \(r_{\alpha}\) (\(\alpha=1\), 2 or 3) and the elements in it as \(r_{\alpha}^{(i,j)}\). In the quantization calculation, quantization errors \(\epsilon_{1}\) and \(\epsilon_{2}\) are added to \(r_{1}\) and \(r_{2}\). At this time, the calculation of elements in \(r_{q,3}\) can be written as  
\[r_{q,3}^{(i,k)}=\sum_{j=1}^{N}(r_{1}^{(i,j)}+\epsilon_{1}^{(i,j)})(r_{2}^{(j,k )}+\epsilon_{2}^{(j,k)}) \tag{1}\]  
which can be rewritten as  
\[r_{q,3}^{(i,k)}=\sum_{j=1}^{N}r_{1}^{(i,j)}r_{2}^{(j,k)}+\sum_{j=1}^{N}r_{1}^{ (i,j)}\epsilon_{2}^{(j,k)}+\sum_{j=1}^{N}\epsilon_{1}^{(i,j)}r_{2}^{(j,k)}+ \sum_{j=1}^{N}\epsilon_{1}^{(i,j)}\epsilon_{2}^{(j,k)} \tag{2}\]  
The first term in the Formula (2) is equal to the value calculated in full precision, and the last three terms are errors caused by low-precision quantization. A direct idea is to reduce the impact of the last three items on the calculation as much as possible. Let us take the term \(\sum_{j=1}^{N}r_{1}^{(i,j)}\epsilon_{2}^{(j,k)}\) as an example. Since \(r_{1}\) is a matrix and \(\epsilon_{2}\) represents the noise of another matrix, they are mutually independent. Therefore, \(E(r_{1}\epsilon_{2})=E(r_{1})\cdot E(\epsilon_{2})\). In the training of DNNs, low-precision quantization errors can be regarded as random errors with a mathematical expectation of zero, i.e., \(E(\epsilon_{2})=0\). Consequently, the mathematical expectation of \(\sum_{j=1}^{N}r_{1}^{(i,j)}\epsilon_{2}^{(j,k)}\), denoted as \(E\Big{[}\sum_{j=1}^{N}r_{1}^{(i,j)}\epsilon_{2}^{(j,k)}\Big{]}\), equals 0. However, when the sample size is relatively small, the sample values may not necessarily converge to their mathematical expectation. Thus, only when \(N\) is sufficiently large can this term approach 0. The same principle applies to \(\sum_{j=1}^{N}e_{1}^{(i,j)}r_{2}^{(j,k)}\) and \(\sum_{j=1}^{N}\epsilon_{1}^{(i,j)}\epsilon_{2}^{(j,k)}\). In DNN training, various matrices of different size are required to perform multiplication operations. Figures 2 and 3 illustrate the accumulation count of elements involved in computations across different layers for models based on ResNet-50 and Transformer, respectively. It is evident that in DNN computations, there is a significant disparity in the number of elements participating in calculations along the model, suggesting that the tolerance for low-precision noise also varies considerably. Figure 3: The accumulation in matrix calculations and the frequency of elements in matrix used for Transformer-based model. Figure 2: The accumulation in matrix calculations and the frequency of elements in matrix used for layers of ResNet-50. We employ 50 random experiments for our observations: As shown in Figure 1, initially, we perform matrix multiplication using full precision, and we consider the outcome as the ground truth. Subsequently, we quantize one of the matrices to low precision (i.e., 4-bit and 2-bit) and carry out the same multiplication operation, recording the average distance of each element in the computed result from its corresponding element in the ground truth. To examine the impact of the accumulation count of elements involved in the computation, we alter the matrix size to progressively increase the number of elements (from 9 to 2304, from the same distribution) participating in the matrix calculation, and sequentially document the distance between the computed results and their ground truth at this time. Figure 4 illustrates that regardless of the bit-width (4-bit or 2-bit), the distance introduced by low precision consistently decreases as the number of elements increases.