# Compressed Learning of Deep Neural Networks for OpenCL-Capable Embedded Systems
## 2 Method
### Optimization Algorithm
ViennaCL has a collection of kernels for matrix unary elementwise operations, which provides a basis for implementing our own kernel. Our code in Figure 4 assigns thread groups to rows and threads in groups to columns, similar to the two OpenCL kernels we discussed above. \[\text{col\_indices} =[\ 0\ \ 1\ \ 1\ \ 2\ \ 0\ \ 2\ \ 3\ \ 1\ \ 3\ ]\]1 OpenCL kernel for the proximal operator, where the matrix A contains \(w^{k}-\eta G(w^{k})\). \[\text{col\_indices} =[\ 0\ \ 1\ \ 1\ \ 2\ \ 0\ \ 2\ \ 3\ \ 1\ \ 3\ ]\]2 OpenCL kernel for the Dense(Dmat) \(\times\) Compressed(Cmat) operation. ## 4 Experiments

To show the effectiveness of our compression method for deep neural networks, we tested our method on four popular convolutional neural networks on image recognition tasks: (i) Lenet-5 on the MNIST dataset [4] and (ii) AlexNet [1], (iii) VGG16 [5], and (iv) ResNet-32 [11] on the CIFAR-10 dataset [65]. In both datasets, we have used the original split of training and test data (MNIST has \(28\times 28\) grey-scaled images with no. of train/test examples = \(60k/10k\), and CIFAR-10 has \(32\times 32\) color images with train/test = \(50k/10k\)). We fixed the number of training updates to \(60k\) with a minibatch size of 128 (so that the effective number of example iterations will be \(60k\times 128\)) since the number was large enough for the resulting neural networks to reached known top prediction accuracy. We compared our proposed method based on \(\ell_{1}\)-regularized sparse coding (denoted by SpC), to the existing pruning method [23] (denoted by Pru) based on thresholding and retraining and the state-of-the-art method [33] (denoted by MM) based on the method of multipliers optimization algorithm. To initialize weight values, we have used the initialization from [PERSON] et al. [66], which is known to work well with neural networks containing the ReLU activation. Our implementation is available as open-source software ([[https://github.com/sanglee/caffe-mc-opencl](https://github.com/sanglee/caffe-mc-opencl)]([https://github.com/sanglee/caffe-mc-opencl](https://github.com/sanglee/caffe-mc-opencl))). ### Comparison of Training Algorithms

Our algorithms Prox-RMSProp (Algorithm 1) and Prox-ADAM (Algorithm 2) use random initialization and random minibatch examples in weight updates. Furthermore, model compression will impose statistical bias compared to the reference models without compression. Hence, it is natural to expect some variation if we repeat training with different random seeds. Here, we compare our two algorithms in order to investigate how much variation they will exhibit in training, regarding test accuracy and compression rate (the ratio of the number of zero entries to the total number of learning parameters). We have trained Lenet-5 (MNIST), AlexNet (CIFAR-10), VGGNet (CIFAR-10), ResNet-32 (CIFAR-10) multiple times with different random seeds. Amongst the experiments, our algorithms showed the most distinctive characteristics on VGGNet, as shown in \[\text{col\_indices} =[\ 0\ \ 1\ \ 1\ \ 2\ \ 0\ \ 2\ \ 3\ \ 1\ \ 3\ ]\]3 Overall, Prox-ADAM showed smaller variance in both test accuracy and compression rate than Prox-RMSProp. This behavior is expected since the search directions produced by Prox-ADAM is more stable than those of Prox-RMSProp: in the former, the direction is a composition of a momentum direction (which brings stability) and a minibatch gradient direction, whereas in the latter it is solely based on a noisy minibatch gradient. For this reason, we chose Prox-ADAM for the rest of our experiments. \[\text{col\_indices} =[\ 0\ \ 1\ \ 1\ \ 2\ \ 0\ \ 2\ \ 3\ \ 1\ \ 3\ ]\]4 Comparison of our suggested algorithms (**a**) Prox-RMSProp and (**b**) Prox-ADAM, on VGGNet training with the CIFAR-10 dataset. Prox-ADAM produced more stable trained models in terms of test accuracy and compression rate.