# Compressed Learning of Deep Neural Networks for OpenCL-Capable Embedded Systems
## 2 Method
### Optimization Algorithm
Our method SpC reaches top accuracy and compression much faster than MM. Note that the last iterations are 60k (SpC) and 120k (MM). \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**GPU** & **NVIDIA GTX 1080 TI** & \multicolumn{3}{c}{**ARM Mali-T860**} \\ \hline Compression & Yes & No & Yes & No \\ Model Size & 148 KB & 5.0 MB & 148 KB & 5.0 MB \\ \hline Inference Time & 8572 ms & 16,977 ms & 506,067 ms & 606,699 ms \\ Speed up & & 2\(\times\) & & 1.2\(\times\) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Inference speedups by model compression (Lenet-5, MNIST). ## Appendix A Layer-Wise Compression Rate

Here, we check the compression rates of each layer of the three networks, Lenet-5, AlexNet, and VGGNet when we apply our sparse coding methods SpC and SpC(Retrain). We chose the regularization parameter value \(\lambda\) for each method so that the compressed models achieved at least 99% of the test accuracy of the reference model while maximizing compression rates among trials. Tables A1-A4 show the results. We can again confirm our discussion in the previous experiments that our method without retraining, SpC, provides superb compression rate being on par with the reference model in prediction accuracy. SpC showed compression rates: Lenet-5 (96.9%), AlexNet (90.65%), VGGNet (94.41%), and ResNet (20.67%). Further compression was also possible with retraining. SpC(Retrain) showed compression rates: Lenet-5 (97.16%), AlexNet (97.88%), VGGNet (98.13%), and ResNet (87.50%), while prediction accuracy values of compressed models were similar or slightly improved from those of SpC. From the tables, we can also observe that the layers that are near the input and the output layers are compressed less than the other layers in the middle. One could use such information to redesign and optimize the architecture of the neural networks for better computational efficiency. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Methods** & \multicolumn{2}{c}{**SpC**} & \multicolumn{2}{c}{**SpC(Retrain)**} \\ \hline
**Layers** & **NNZ/Total Weights** & **Compression Rate** & **NNZ/Total Weights** & **Compression Rate** \\ \hline conv1 & 379/432 & 12.27\% (1\(\times\)) & 139/432 & 67.82\% (3\(\times\)) \\ conv1-1-1 & 1844/2304 & 19.97\% (1\(\times\)) & 327/2



## References

* (1) [PERSON]; [PERSON]; [PERSON] ImageNet Classification with Deep Convolutional Neural Networks. In _Advances in Neural Information Processing Systems 25_; Curran Associates, Inc.: New York, NY, USA, 2012; pp. 1097-1105. * (2) [PERSON]; [PERSON] [PERSON] Framewise phoneme classification with bidirectional LSTM and other neural network architectures. _Neural Netw._**2005**, _18_, 5-6. [CrossRef] [PubMed]
* (3) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] Natural Language Processing (Almost) from Scratch. _[PERSON]. Learn. Res._**2011**, _12_, 2493-2537. * (4) [PERSON]; [PERSON]; [PERSON]; [PERSON] Gradient-based learning applied to document recognition. _Proc. IEEE_**1998**, _86_, 2278-2324. [CrossRef]
* (5) [PERSON]; [PERSON] Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proceedings of the International Conference on Learning Representations, San Diego, CA, USA, 7-9 May 2015. * (6) [PERSON]; [PERSON]; [PERSON] [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] Going Deeper with Convolutions. In Proceedings of the Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7-12 June 2015. * (7) [PERSON]; [PERSON] Visualizing and Understanding Convolutional Networks. In Proceedings of the Computer Vision (ECCV 2014), Zurich, Switzerland, 6-12 September 2014; pp. 818-833. * (8) [PERSON]; [PERSON] Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32 nd International Conference on Machine Learning, Lille, France, 6-11 July 2015; pp. 448-456. * (9) [PERSON]; [PERSON] [PERSON]; [PERSON]; [PERSON]; [PERSON] Rethinking the Inception Architecture for Computer Vision.