# Compressed Learning of Deep Neural Networks for OpenCL-Capable Embedded Systems
## 2 Method
### Optimization Algorithm
### Compression Rate and Prediction Accuracy

In the training problem with sparse coding (1), we use the \(\ell_{1}\)-regularizer \(\Psi(w)=\lambda\|w\|_{1}\) where \(\lambda>0\) is a hyperparameter that determines the amount of compression (higher \(\lambda\) gives larger numbers of zero entries in \(w\) and thereby higher compression rate). Figure 6 shows test accuracy and compression rate of the four network-dataset pairs with respect to the change of the \(\lambda\) parameter value, for our sparse coding approach (Figure 5(a), SpC) and the existing pruning approach (Figure 5(b), Pru). The test accuracy values of the reference networks (trained without sparse coding) are shown as the horizontal dotted lines. **SpC**: at small \(\lambda\) values, we can see that the test accuracy of compressed models is usually better than that of the reference models. This may happen when the reference model is overfitting the data and \(\ell_{1}\) regularization is mitigating the effect. When the test accuracy values of the compressed models were similar to those of the reference models (crossings of vertical and horizontal dotted lines), our method was able to remove about 90% of the weights (one exception was ResNet-32, for which both Pru and SpC without retraining showed poor compression). The plots also indicate that further compression will be possible if we are ready to sacrifice prediction accuracy by a small margin. **Pru**: test accuracy values drop much more rapidly as we increase the compression rate, compared to our sparse coding approach. Similar test accuracy to the reference model has been achieved with about 40% of compression in case of Lenet-5, but no compression was possible for AlexNet, VGGNet, and ResNet-32 if we wanted to achieve similar test accuracy to the reference model. \[\text{col\_indices} =[\ 0\ \ 1\ \ 1\ \ 2\ \ 0\ \ 2\ \ 3\ \ 1\ \ 3\ ]\]5 _Cont._

### The Effect of Retraining

Compressed models can be debiased by means of an extra training the compressed networks, where the weights at the zero value are fixed and not updated during retraining. Figure 7 shows the prediction accuracy of the neural networks at different compression rates, comparing models produced by SpC and Pru, and their retrained versions, SpC(Retrain) and Pru(Retrain). We can see that retraining is indeed a necessary step for the pruning approach (Pru) as previously known [23], since otherwise compressed models show impractically low prediction performance. The pruning method with retraining, Pru(Retrain), shows similar prediction accuracy to our sparse coding method SpC (without retraining) up to a moderate compression rate. However, when the compression rate is very high, our method SpC clearly outperforms Pru(Retrain). In addition, retraining improves the prediction accuracy of our method SpC even further, especially when the compression rate is very high, which can be seen from the solid and dotted vertical lines where SpC and SpC(Retrain), respectively, achieve at least 99% of the reference accuracy with maximal compression. Therefore, for our sparse coding method SpC, retraining can be used when we aim for very high compression. Table 1 summarizes the compression and prediction results of all cases. \[\text{col\_indices} =[\ 0\ \ 1\ \ 1\ \ 2\ \ 0\ \ 2\ \ 3\ \ 1\ \ 3\ ]\]6 Compression rate and test accuracy with respect to the change of the \(\lambda\) parameter value, for (**a**) our sparse coding approach Spc and (**b**) the existing pruning approach. Higher \(\lambda\) values result in more compression with a decrease in prediction accuracy.