# Learning and Compressing: Low-Rank Matrix Factorization for Deep Neural Network Compression
## 2 Related Works  
### Matrix Factorization Techniques  
Non-negative matrix Factorization (NMF) [19] is a fundamental data analysis technique that is applied in many real world systems, such as recommender systems and audio processing systems. The purpose of NMF is to express a non-negative matrix (i.e., a matrix with non-negative entries) using the product of two non=negative matrices. This method is widely used because it is computationally fast and uses nonsubtractive linear combinations to make factorization easily interpretable in many cases. [PERSON]. [20] exploited the random projections of the data structure in NMF to reduce memory demands without any major performance deterioration. [PERSON]. [21] proposed a sparse deep NMF model that satisfied different sparsity requirements, and adopted the [PERSON]'s accelerated algorithm to accelerate the computing process during optimization. This model has competitive classification accuracy and a more intuitive interpretation of features. Ref. [22] presented an iterative updating algorithm called self-paced learning and adaptive neighbors methodology (SPLNMFAN) which incorporated the self-paced learning regularization and the adaptive graph to improve NMF solution performance. However, it is difficult to accurately restore the original matrix with NMF and it can only be used as a means of approximate restoration. The singular value decomposition (SVD) [23; 24; 25] is among the most ubiquitous and powerful methods for data processing in the computational area. Additionally, SVD performs best in Equation (1), which means it can decompose and restore the original matrix almost without losses. In SVD, matrix can be decomposed into the product of three matrices \(\mathbf{U}\), \(\mathbf{\Sigma}\), and \(\mathbf{V}^{\mathbf{T}}\). It can be approximated by truncating the largest \"\(k\)\" singular values of the above three matrices, which is called the TSVD. Ref. [26] combined SVD with DNNs; the authors first decomposed each layer into its full-rank form using SVD, then directly performed training on the decomposed full-rank weights. In order to encourage low ranks to avoid gradient vanishing or exploding, they also added the orthogonality regularization to the singular vectors. Ref. [27] proposed sparse low-rank (SLR) factorization, which computed the importance of neurons, and used the lower rank to disperse the SVD matrix for the unimportant neuron, resulting in a better compression ratio.However, it requires more time for fine-tuning and retraining. In the applications of DNNs, it can reduce the number of model parameters through factorization and low-rank approximation. Although the TSVD is applied to many sizable applications, as far as the domain of origin of the factorization data is concerned, its decomposition components still lack meaning and interpretation. Recently, there has been interest in a low-rank MF called the CUR decomposition [28; 29; 30] (also called the pseudo skeleton approximation). The CUR decomposition first selects some rows and columns in the original matrix, and then constructs three low-dimensional matrices using the above-selected rows and columns, and finally calculates the product of above three matrices to approximate the original one. Although the approximations are suboptimal [31], it preserves the original properties and makes selected columns or rows retain their original meaning. Therefore, the CUR decomposition has the natural interpretability, traceability, and explicit physical meaning. In particular, [PERSON]. [32] proposed an efficient CUR decomposition algorithm to decompose and approximate a given matrix. Moreover, it provided error bounds of choosing \(\mathbf{C}\) and \(\mathbf{R}\) from the given matrix \(\mathbf{W}\) based upon the probabilistic method. Ref. [17] gave another CUR decomposition algorithm using a way of columns selection, which made the error bounds \(\left\|\mathbf{W}-\mathbf{C}\mathbf{R}\right\|_{\mathrm{F}}\) less than or equal to \(\left(2~{}+~{}\varepsilon\right)\left\|\mathbf{W}\right\|_{\mathrm{F}}\).