# Learning and Compressing: Low-Rank Matrix Factorization for Deep Neural Network Compression
## 4 Experiments
### Comparison Baselines Introduction
After a certain number of iterations, the ranks of each layer are gradually determined, and the low-rank approximation matrix \(\mathbf{\Theta}\) is also determined, so that the learning loss of the compressed DNN model increases until the ranks of each layer are all determined, and then gradually decreases. Considering that similar convergence behavior (e.g., Figure 1) generally does not mean that different values of \(\lambda\) lead to similar accuracy, we analyze the performance sensitivity of our algorithm to \(\lambda\). Ideally, \(\mathbf{W}_{k}\) should exhibit the same low-rank properties as the compressed low-rank matrix \(\mathbf{\Theta}_{k}\), with high accuracy in classification tasks. To examine the desired low-rank behavior of **W**, we record and observe the Frobenius norm, which measures the similarity between **W** and \(\mathbf{\Theta}\), as shown in Figure 2. According to the compression step, \(\mathbf{\Theta}\) is constantly updated using low-rank matrix factorization methods. The curves shown in Figure 2 show that **W** exhibits the low-rank features indeed. Figure 3 shows the classification accuracy of the DNN model with different \(\lambda\) values during compression. It can be seen from this figure that the compression process shows a gradual upward trend whether it is a simple or a complex network. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{1}{c}{**Model**} & \multicolumn{1}{c}{**Algorithm**} & \multicolumn{1}{c}{\(\mathbf{E_{test}\) (\%)}} & \multicolumn{1}{c}{\(\mathbf{\rho_{FLOPS}}\)} & \multicolumn{1}{c}{\(\mathbf{\rho_{storage}}\)} \\ \hline \multirow{6}{*}{**Model**} & Uncompressed & 34.60 & 1.00 & 1.00 \\ \cline{2-5}  & TSVD & 38.96 & 2.05 & 1.96 \\  & CUR & 38.20 & 2.19 & 2.07 \\  & Standard TR [47; 48] & 36.45 & - & 4.00 \\  & Ours (\(\lambda=2\times 10^{-4}\)) & **33.22** & **2.53** & **2.34** \\  & Ours (\(\lambda=8\times 10^{-4}\)) & **36.08** & **4.17** & **4.00** \\ \hline \multirow{6}{*}{**Model**} & Uncompressed & 31.90 & 1.00 & 1.00 \\ \cline{2-5}  & TSVD & 36.47 & 3.05 & 2.74 \\ \cline{1-1}  & CUR & 35.93 & 3.44 & 2.95 \\ \cline{1-1}  & Standard TR [47; 48] & 33.30 & - & 4.80 \\ \cline{1-1}  & Ours (\(\lambda=2\times 10^{-4}\)) & **30.86** & **4.08** & **3.82** \\ \cline{1-1}  & Ours (\(\lambda=8\times 10^{-4}\)) & **33.24** & **5.01** & **4.87** \\ \hline \multirow{6}{*}{**Model**} & Uncompressed & 30.15 & 1.00 & 1.00 \\ \cline{2-5}  & TSVD & 35.28 & 2.20 & 1.94 \\ \cline{1-1}  & CUR & 35.17 & 3.06 & 2.50 \\ \cline{1-1}  & Standard TR [47; 48] & 32.73 & - & 4.60 \\ \cline{1-1}  & Ours (\(\lambda=4\times 10^{-4}\)) & **31.62** & **3.13** & **2.66** \\ \cline{1-1}  & Ours (\(\lambda=16\times 10^{-4}\)) & **32.74** & **5.03** & **4.86** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison of compression rate and error rate for VGG16 ResNets on CIFAR-100. In addition, Figure 3 not only examines the classification accuracy brought by different values of \(\lambda\), but also measures the difference in classification accuracy brought by different \(\lambda\) values under similar convergence behavior. It can be seen from this figure that the smaller the value of \(\lambda\), the higher the final classification accuracy, and vice versa. As shown in Figure 2, the gap in the Frobenius norm is relatively large at the beginning. The above situation occurs because the ranks of the weight matrices in the convolutional layers have not been determined at this time, thus the results in Figure 3 are equivalent to random classification. After a period of iterations, the \(\mathbf{W}\) and \(\mathbf{\Theta}\) in each layer begins to be determined, and the gap in the Frobenius norm gradually narrows, which makes the classification accuracy of the compressed DNN models increase rapidly. Figure 1: The model Learning Loss with different \(\lambda\times 10^{-4}\), for LeNets on MNIST, VGG16 and ResNets on CIFAR-10.