# Learning and Compressing: Low-Rank Matrix Factorization for Deep Neural Network Compression
## 5 Conclusions  
Due to recent technological advancements, DNNs have been more widely used as advanced and powerful methods. However, calculation resource and space are high as a result of the huge number of parameters. Moreover, the high-dimensional data in DNNs usually contains useless information, which will also reduce the accuracy to a certain extent. An optimization algorithm that reduces the calculation resource and space cost of the DNN model while maintaining the original accuracy is required to compress (and optimize) the DNN models. The main goal of this paper is to propose a DNN compression algorithm which incorporates the matrix factorization method and the linear compression function into the joint function optimization framework, reducing the number of parameters in the networks and thereby reducing storage space and computational resource consumption. We deal with this joint function using the ALM and split it into learning and compression steps, respectively, so that the networks can be trained and compressed independently without interfering with each other. All experiments are carried on classification tasks over three public datasets. It is worth noting that the experimental results show that our algorithm has the highest accuracy and compression ratios when compared with other baselines and the state-of-the-art research. For example, we can make a VGG network faster than ResNets and with nearly the same classification accuracy. In addition, the compressed models of VGG16 on CIFAR-10 and ResNet20 on CIFAR-100 using our algorithms achieve test errors of 6.15% and 33.22%, which are 0.60% and 1.38% lower than the uncompressed models, respectively. The current work also aims to examine the convergence and sensitivity of our algorithm. The results show that our algorithm has stable convergence and the weight matrices have low-rank properties, as can be seen from the values of learning loss and Frobenius norm \(\left\|\mathbf{W}-\mathbf{\Theta}\right\|_{\text{F}}^{2}\). Moreover, under same ranks, we use the classic TSVD, CUR decomposition, and the joint optimization framework-based TSVD compression methods as baselines to carry out experiments. The comparison results prove the superiority in compression effects and compatibility with MF methods. However, as mentioned above, both TSVD and CUR decomposition are matrix factorization methods with cubic level complexity, and GPU is not good at SVD and CUR decomposition in practice. In other words, the larger the  
Figure 8: Comparison of classification test error (Y-axis), FLOPs (X-axis) and the number of storage parameters (ball size for each net) in Error-compression space, for ResNets and VGG16 trained on CIFAR-10, the algorithm using TSVD and CUR decomposition. network size, the longer compression runtime required. For future works, we attempt to develop more research on how to cut down the runtime of compression for DNNs. Conceptualization, G.C. ; Methodology, G.C. and J.L. ; Software, G.C. and X.L. ; Validation, G.C., J.L. and Z.C. ; Formal analysis, G.C., X.L. and H.Z. ; Investigation, G.C., J.L. and H.Z. ; Resources, Z.C. ; Data curation, G.C. ; Writing--original draft, G.C. ; Writing--review and editing, G.C., J.L., X.L., Z.C. and H.Z. ; Visualization, G.C. ; Supervision, J.L. and X.L. ; Project administration, Z.C. ; Funding acquisition, Z.C. All authors have read and agreed to the published version of the manuscript. This research was funded by the National Natural Science Foundation of China, grant number 32071775. Informed Consent: Not applicable. * [PERSON] et al. (2022) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] Bi-Endmember Semi-NMF Based on Low-Rank and Sparse Matrix Decomposition. _IEEE Trans. Geosci. Remote Sens._**2022**, _60_, 1-16. [CrossRef]
* [PERSON] et al. (2020) [PERSON]; [PERSON]; [PERSON]; [PERSON] VGG deep neural network compression via SVD and CUR decomposition techniques.