# Learning and Compressing: Low-Rank Matrix Factorization for Deep Neural Network Compression
## 4 Experiments
### Comparison Baselines Introduction  
We use two single low-rank matrix compression methods as baselines and briefly describe the experimental procedure. In each compression experiment, a reference network with full-rank weight matrices is trained first, and then a lower rank is chosen for each layer. We first obtain the optimal ranks \(r\) (Equation (10)) using our compression algorithm (Algorithm 2) so that the architecture can be determined. Then we conduct experiments of compression using single TSVD (Equations (2) and (3)) and CUR decomposition (Equation (1)) under above optimal rank. Furthermore, to demonstrate our low-rank compression algorithm can be generally applied for model compression using other low-rank MFs, we also perform our compression algorithm based on TSVD as the compression step (replacing CUR decomposition with TSVD). It should be noted that all compression algorithms are applied in convolutional layers of DNNs models only. We use the processed MNIST dataset (normalize pixels in \([0,1]\) at first, and then subtract their mean) to train LeNet300 and LeNet5. The experimental results with different \(\lambda\) in Table 1 clearly show that our algorithm has a slightly decreased or even a lossless classification test error.For the experiments on the MNIST dataset, compared to employing single SVD and the CUR decomposition techniques in LeNet300 models, we can easily recognize that our performance outperforms across the board from Table 1; the test error of our algorithm is lossless and about 1.90%. Additionally, on LeNet5, with a high storage compression ratio of 17.85\(\times\), our method has the same error test 0.94% as the original one. Similarly, for the experiments on CIFAR-10 dataset (Table 2), we also obtain impressive storage compression ratio and FLOPs compression ratio from using VGG16, ResNet20, ResNet32, and ResNet56 models. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Algorithm** & \(\mathbf{E_{test}}\) **(\%)** & \(\mathbf{\rho_{\text{FLOPs}}}\) & \(\mathbf{\rho_{\text{storage}}}\) \\ \hline \multirow{6}{*}{**Training**} & Uncompressed & 0.94 & 1.00 & 1.00 \\ \cline{2-5}  & TSVD & 1.61 & 3.50 & 4.12 \\ \cline{1-1}  & CUR & 1.60 & 3.71 & 4.70 \\ \cline{1-1}  & Ours (\(\lambda=1\times 10^{-4}\)) & **0.94** & **7.02** & **17.85** \\ \hline \multirow{6}{*}{**Training**} & Uncompressed & 1.90 & 1.00 & 1.00 \\ \cline{2-5}  & TSVD & 2.62 & 1.67 & 1.67 \\ \cline{1-1}  & CUR & 2.46 & 1.67 & 1.67 \\ \cline{1-1}  & Ours (\(\lambda=1\times 10^{-4}\)) & **1.90** & **1.95** & **1.95** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of compression rate and error rate for LeNet on MNIST. Finally, for the experiments on the CIFAR-100 dataset (Table 3), the compressed ResNet20, ResNet32 and ResNet56 models have achieved moderate storage compression ratios of 2.34\(\times\), 3.82\(\times\), and 2.66\(\times\), respectively. Here, the test error in case of the compressed ResNet20 is lower than the TSVD model, around 5.74%, and most notably, our ResNet20 and ResNet32 networks surpass the uncompressed version by 1.38% and 1.04% in terms of accuracy, respectively. It is not only that all error tests are excellent, but also the weight parameters are reduced significantly. It is worth noting that the part of VGG16 network of this table shows that its error test (6.15%) is higher than the uncompressed one (6.75%), under high storage-compression ratio (7.37\(\times\)). Each iteration contains the compression step and the learning step, and the former performs the low-rank matrix factorization (Algorithm 1) to obtain the low-rank approximation matrix \(\mathbf{\Theta}\), and the latter performs multiple SGD methods to make the weight matrix \(\mathbf{W}\) approximate \(\mathbf{\Theta}\).