# Compressed Learning of Deep Neural Networks for OpenCL-Capable Embedded Systems
## 2 Method
### Optimization Algorithm
Table 2 shows the comparison between our method SpC and the state-of-the-art method MM, regarding two convolutional neural networks tested in the original paper [33]. Note that the comparison would be more favorable to MM because MM is allowed (i) to start from the state-of-the-art pretrained models and (ii) to use different solvers and auxiliary parameter control strategies as in the original paper [33]. Nevertheless, our method SpC shows a competent compression performance even though its compressed learning starts from random weight values. Figure 8 shows the convergence behavior of SpC and MM. The compression is performed every update in SpC and every 4k updates in MM, and therefore the compression rate convergence may look smoother in MM (In fact, it is another tuning parameter in MM how often it performs weight compression, and we found that the algorithm is quite sensitive to different choices). However, our method SpC achieves the top compression rate and test accuracy much faster than MM. Therefore, our method suits better for embedded systems where we can afford only a few thousand training iterations due to time and energy constraints. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Network (Data)** & \multicolumn{2}{c}{**Lenet-5 (MNIST)**} & \multicolumn{2}{c}{**ResNet32 (CIFAR10)**} \\ \hline Method & SpC & MM & SpC & MM \\ \hline Pretrained Model & - & Required (Test Acc = 99.1) & - & Required (Test Acc = 92.28) \\ Solver & Prox-Adam & SGD with Momentum & Prox-Adam & Nesterov \\ Aux. Parameter (\(\mu\)) & - & \(9.76\times 10^{-5}\) (\(\times 1.1\) per 4k iter) & - & \(10^{-3}\) (\(\times 1.1\) per 2k iter) \\ \hline Accuracy & 97.25\% & 97.65\% & 89.22\% & 92.37\% \\ Compression Rate & 0.98 & 0.98 & 0.88 & 0.85 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparison between our method SpC and the state-of-the-art MM. For MM, we used the hyperparameters and the best pretrained models as a starting point following the original paper [33]. Note that our method SpC starts from random weights and does not require complicate control of the auxiliary parameter \(\mu\). ### Performance on Embedded Systems

We also tested our suggested framework in an embedded system, which has a similar spec to the Samsung Galaxy S7 smartphone (Seoul, Korea). Our test system has six-core 64-bit ARM CPU, 4 GB main memory, ARM Mali-T860 GPU, OpenCL 1.2 support, and Ubuntu 16.04 operating system. In this case, we are interested in the inference time comparison between full reference models and compressed models, since in the latter the weights are stored in a compact form and will be computed with sparse matrix operations, and therefore some speedups can be expected. Table 3 shows the results on the test embedded system, also making a comparison to the results on a workstation with an NVIDIA GPU. Despite that we have achieved some speedup, it seems to fall short considering the size of compressed models. One reason is that the compressed convolution filters have irregular nonzero patterns for which full GPU acceleration is difficult. We plan to investigate this issue further in our future research. ## 5 Conclusions

In this paper, we proposed an efficient model compression framework for deep neural networks based on sparse coding with \(\ell_{1}\) regularization, proximal algorithms, debiasing, and compressed computation of sparse weights using OpenCL. We believe that our method will be more versatile for embedded system applications than the previous methods as it does not require a pre-trained model as an input, while it can produce competent compressed models. Conceptualization, S.L. ; formal analysis, S.L. and J.L. ; software, S.L. and J.L. ; writing--original draft preparation, S.L. ; writing--review and editing, S.L. This work was supported by the research fund of Hanyang University (HY-2017-N). The authors declare no conflict of interest. Figure 8: Convergence behavior of the our method SpC (left) and the state-of-the-art MM (right) in terms of compression rate and test accuracy on Lenet-5 (MNIST).