# Learning and Compressing: Low-Rank Matrix Factorization for Deep Neural Network Compression
## 4 Experiments
### Comparison Baselines Introduction
Figure 2: The F-norm between the origin and compression matrices with different \(\lambda\times 10^{-4}\), for LeNets on MNIST, VGG16 and ResNets on CIFAR-10. In Figure 4, the results of our algorithm over different \(\lambda\) values for a given network span a curve, shown as connected circles which start on the lower right at the reference labeled \"R\" (\(\lambda=0\)) and then move left and up. In this figure, ideal models are small balls (having few parameters) on the left-bottom (where both test error and FLOPs are the smallest). Each color corresponds to a different reference net. The area of a circle or square is proportional to the number of weight parameters (storage) in the corresponding compressed model. Figure 3: The model classification accuracy with different \(\lambda\times 10^{-4}\), for LeNets on MNIST, VGG16 and ResNets on CIFAR-10. We compute FLOPs based on fusing multiplication and addition as the assumption, and treat it as one FLOP. For example, if the forward passes through a fully connected layer with \(n\times m\) weight matrix and \(n\times 1\) bias, which has \(nm\) multiplications and \(n\) additions, then the FLOPs are \(nm\). In this figure, each color corresponds to a different reference grid, and the ideal models are generally the small ball in the lower left corner, that is, with fewer parameters, lower error rate, and smaller FLOPs. It can be seen from the figure that the FLOPs of the VGG16 network are lower than that of the uncompressed ResNet model (labeled \"R\") when the classification accuracy is almost the same. That is, the VGG16 network compressed by our algorithm can be faster than the uncompressed ResNets, but with almost the same test error. From the experimental results it can be seen that the compression ratio grows with the growth of \(\lambda\). In addition to CUR decomposition, we also use TSVD as the low-rank matrix decomposition method of the compression step in our proposed algorithm, and compare the classification accuracy, loss, compression space (storage parameters and FLOPs), and Frobenius norm \(\|\mathbf{W}-\boldsymbol{\Theta}\|_{\text{F}}^{2}\) of the two decomposition methods in the compression step based on the classification task. All hyperparameters involved are controlled, and the experimental results are shown in Figures 5-8. It can be clearly seen from Figures 5-7 that the performance (classification accuracy, Frobenius norm and learning loss) of our algorithms using TSVD and CUR decomposition, respectively, is basically the same when \(\lambda\) is small. However, with the increase of \(\lambda\), the experimental results of the algorithm using CUR decomposition gradually surpass the latter in terms of accuracy and have a faster classification-loss drop rate. In terms of F-norm, similarly, the numerical difference between above two decomposition methods is not obvious when \(\lambda\) is small. As \(\lambda\) gradually increases, compared with the joint optimization of TSVD, the one using CUR decomposition makes the inflection point recording the Frobenius norm appear earlier and take fewer iterations to reach the approximate zero point. The experimental performance in Figure 8 is also the same. Compared with the joint optimization of TSVD, the algorithm using CUR decomposition is closer to the lower left position (ideal position) when \(\lambda\) increases, which means that our algorithm using CUR decomposition performs better at high compression ratios. Figure 4: The error-compression space of test error (Y axis), inference FLOPs (X axis) and number of parameters (ball size), for LeNets on MNIST, VGG16 and ResNets on CIFAR-10. Figure 5: Comparison of accuracy with different \(\lambda 10^{-4}\), for LeNets on MNIST, VGG16 and ResNets on CIFAR-10. Figure 6: Comparison of F-norm with different \(\lambda 10^{-4}\), for LeNets on MNIST, VGG16 and ResNets on CIFAR-10. Figure 7: Comparison of learning loss with different \(\lambda 10^{-4}\), for LeNets on MNIST, VGG16 and ResNets on CIFAR-10.