# Learning and Compressing: Low-Rank Matrix Factorization for Deep Neural Network Compression
## 4 Experiments
### Comparison Baselines Introduction
The gradient descent curve of the model in the learning step is shown in Figure 1. It is obvious that the overall trend of the curve is declining, which means that the curves have similar convergence characteristics under different \(\lambda\) (convergence is reached sooner or later in different iterations). Therefore, these curves provide an experimental guarantee for the convergence of our algorithm, and indicate that different values of \(\lambda\) hardly affect the convergence of this algorithm. Additionally, in this figure, the numerical jitter of the enlarged part is caused by the learning step in multiple iterations. In other words, each compression step of iteration generates the new low-rank matrix \(\mathbf{\Theta}\), and the weight matrix \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Algorithm** & \(\mathbf{E_{test}}\) **(\%)** & \(\mathbf{\rho_{FLOPs}}\) & \(\mathbf{\rho_{storage}}\) \\ \hline \multirow{6}{*}{\begin{tabular}{} \end{tabular} } & Uncompressed & 6.75 & 1.00 & 1.00 \\ \cline{2-5}  & TSVD & 7.93 & 1.08 & 1.65 \\  & CUR & 7.92 & 1.30 & 1.82 \\  & BayesOpt [34] & 8.71 & - & 8.97 \\  & SLR-a [27] & 7.41 & - & 9.17 \\  & Ours (\(\lambda=0.2\times 10^{-4}\)) & \(\mathbf{6.15}\) & \(\mathbf{4.52}\) & \(\mathbf{7.37}\) \\  & Ours (\(\lambda=0.4\times 10^{-4}\)) & \(\mathbf{7.00}\) & \(\mathbf{5.87}\) & \(\mathbf{9.71}\) \\ \hline \multirow{6}{*}{\begin{tabular}{} \end{tabular} } & Uncompressed & 8.15 & 1.00 & 1.00 \\ \cline{2-5}  & TSVD & 9.80 & 1.94 & 2.35 \\  & CUR & 9.77 & 2.01 & 2.59 \\  & Standard TR [47; 48] & 12.50 & - & 5.40 \\  & CNN-FCF [18] & 10.47 & 3.226 & 3.17 \\  & Learning LDNN [26] & 10.57 & 3.26 & - \\  & Ours(\(\lambda=4\times 10^{-4}\)) & \(\mathbf{9.38}\) & \(\mathbf{2.66}\) & \(\mathbf{2.65}\) \\  & Ours(\(\lambda=16\times 10^{-4}\)) & \(\mathbf{10.58}\) & \(\mathbf{4.30}\) & \(\mathbf{4.34}\) \\ \hline \multirow{6}{*}{\
\begin{tabular}{} \end{tabular} } & Uncompressed & 7.22 & 1.00 & 1.00 \\ \cline{2-5}  & TSVD & 9.41 & 1.63 & 2.42 \\  & CUR & 9.36 & 1.82 & 2.60 \\  & Standard TR [47; 48] & 9.40 & - & 5.10 \\  & CNN-FCF [18] & 9.26 & 3.35 & 3.27 \\  & Learning LDNN [26] & 9.45 & 3.93 & - \\  & Ours (\(\lambda=4\times 10^{-4}\)) & \(\mathbf{8.54}\) & \(\mathbf{2.96}\) & \(\mathbf{3.05}\) \\  & Ours (\(\lambda=8\times 10^{-4}\)) & \(\mathbf{9.02}\) & \(\mathbf{3.43}\) & \(\mathbf{3.66}\) \\ \hline \multirow{6}{*}{
\begin{tabular}{} \end{tabular} } & Uncompressed & 6.50 & 1.00 & 1.00 \\ \cline{2-5}  & TSVD & 8.61 & 1.51 & 2.05 \\ \cline{1-1}  & CUR & 8.42 & 1.56 & 2.30 \\ \cline{1-1}  & CNN-FCF [18] & 6.62 & 1.77 & 1.76 \\ \cline{1-1}  & CNN-FCF [18] & 8.08 & 3.44 & 3.30 \\ \cline{1-1}  & Learning LDNN [26] & 8.45 & 3.75 & - \\ \cline{1-1}  & Ours (\(\lambda=4\times 10^{-4}\)) & \(\mathbf{7.46}\) & \(\mathbf{3.00}\) & \(\mathbf{3.01}\) \\ \cline{1-1}  & Ours (\(\lambda=8\times 10^{-4}\)) & \(\mathbf{8.11}\) & \(\mathbf{4.21}\) & \(\mathbf{4.29}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparison of compression rate and error rate for VGG16 and ResNets on CIFAR-10. **W** uses SGD methods to approximate the new \(\mathbf{\Theta}\) and calculate many loss values in the learning step. The first loss calculation in each iteration is a high value, and then gradually decreases, which causes this jitter phenomenon. It is worth noting that the learning loss of some models has a tend of increasing first and then decreasing (Figure 1c-f). We will explain this trend below. Our algorithm iteratively finds the optimal solution. For more complex networks, at the beginning, the optimal rank of the weight matrix in the convolutional layer has not been determined (or partially determined). There is no gap between **W** and \(\mathbf{\Theta}\), thus the learning loss is small at this time.