# Compressed Learning of Deep Neural Networks for OpenCL-Capable Embedded Systems
## 2 Method
### Optimization Algorithm
timestep); while\(w_{t}\) not convergeddo \(t\gets t+1\);  Compute a minibatch gradient: \(g_{t}\gets g_{B}(w_{t-1})\);  Update 1 st moment: \(m_{t}\leftarrow\beta_{1}m_{t-1}+(1-\beta_{1})g_{t}\);  Update 2 nd moment: \(v_{t}\leftarrow\beta_{2}v_{t-1}+(1-\beta_{2})g_{t}\odot g_{t}\);  Bias correction 1 st: \(\hat{m}_{t}\gets m_{t}/(1-\mathrm{pow}(\beta_{1},t))\);  Bias correction 2 nd: \(\hat{o}_{t}\gets v_{t}/(1-\mathrm{pow}(\beta_{2},t))\);  Update: \(w_{t}\leftarrow\mathrm{prox}_{\eta\|\cdot\|_{1}}(w_{t-1}-\eta\cdot\hat{m}_{t}/( \sqrt{\hat{o}_{t}}+\epsilon))\); return\(w_{t}\);
```  
**Algorithm 2**Prox-ADAM Algorithm  
```
Input: a learning rate \(\eta\), a safeguard parameter \(\epsilon>0\), \(\beta_{1},\beta_{2}\in[0,1)\): decay rates  Initialize the weight vector \(w\in\mathbb{R}^{p}\); \(m_{0}\gets 0\) (Init. 1 st moment vector); \(v_{0}\gets 0\) (Init. 2 nd moment vector); \(t\gets 0\) (Init. timestep); while\(w_{t}\) not convergeddo \(t\gets t+1\);  Compute a minibatch gradient: \(g_{t}\gets g_{B}(w_{t-1})\);  Update 1 st moment: \(m_{t}\leftarrow\beta_{1}m_{t-1}+(1-\beta_{1})g_{t}\);  Update 2 nd moment: \(v_{t}\leftarrow\beta_{2}v_{t-1}+(1-\beta_{2})g_{t}\odot g_{t}\);  Bias correction 1 st: \(\hat{m}_{t}\gets m_{t}/(1-\mathrm{pow}(\beta_{1},t))\);  Bias correction 2 nd: \(\hat{o}_{t}\gets v_{t}/(1-\mathrm{pow}(\beta_{2},t))\);  Update: \(w_{t}\leftarrow\mathrm{prox}_{\eta\|\cdot\|_{1}}(w_{t-1}-\eta\cdot\hat{m}_{t}/( \sqrt{\hat{o}_{t}}+\epsilon))\); return\(w_{t}\);
```  
**Algorithm 3**Prox-ADAM Algorithm  
``` Input: a learning rate \(\eta\), a safeguard parameter \(\epsilon>0\), \(\beta_{1},\beta_{2}\in[0,1)\): decay rates  Initialize the weight vector \(w\in\mathbb{R}^{p}\); \(m_{0}\gets 0\) (Init. 1 st moment vector); \(v_{0}\gets 0\) (Init. 2 nd moment vector); \(t\gets 0\) (Init. timestep); while\(w_{t}\) not convergeddo \(t\gets t+1\);  Compute a minibatch gradient: \(g_{t}\gets g_{B}(w_{t-1})\);  Update 1 st moment: \(m_{t}\leftarrow\beta_{1}m_{t-1}+(1-\beta_{1})g_{t}\);  Update 2 nd moment: \(v_{t}\leftarrow\beta_{2}v_{t-1}+(1-\beta_{2})g_{t}\odot g_{t}\);  Bias correction 1 st: \(\hat{m}_{t}\gets m_{t}/(1-\mathrm{pow}(\beta_{1},t))\);  Bias correction 2 nd: \(\hat{o}_{t}\gets v_{t}/(1-\

### Retraining

Once we have obtained a sparse model via sparse coding, we can consider an optional step to train the weights again without any regularization, starting from the previously trained weight values, while excluding the zero-valued weights from training. This type of retraining is known as _debiasing_[57], which can be used to remove the unwanted reduction of weight values due to regularization. It has been shown that debiasing can improve the quality of estimation [58], although it may undo some desired effect of regularization, e.g., the reduction of distortion caused by noise [59]. Our methods work well without retraining, but our experiments show that retraining can achieve further compression without sacrificing prediction accuracy. ## 3 Accelerated OpenCL Operations

In order to perform forward and backward computations of deep neural networks in an accelerated manner, we must use an efficient representation of sparse weight matrices, which can be also stored in a compact form to reduce memory footprint. In the previous research, sparse data structure and matrix operations have not been discussed fully enough [23; 24; 25], due to the difficulty of using sparse matrix operations efficiently on GPUs. Therefore, implementations and testing were performed mainly on CPUs [23; 24], or some vendor-specific support was used, e.g., the cuSPARSE library, which is not an open-source software and available only for NVIDIA hardware [25]. In this section, we discuss the details how we implement sparse weight matrices and efficient sparse matrix operations for training deep neural networks with sparse coding, making use of our proposed algorithms. We base our discussion on the OpenCL-Caffe [60] software, an OpenCL-capable version of the favorite open source deep learning software Caffe [61].