# Compressed Learning of Deep Neural Networks for OpenCL-Capable Embedded Systems
## 2 Method
### Optimization Algorithm
(**a**) Spc: using sparse coding, about 90% of model parameters can be compressed achieving similar prediction accuracy to the reference models (test accuracy is shown as the horizontal dotted lines, and their crossings with accuracy curves of compressed models are marked as vertical dotted lines); (**b**) Pru: the existing pruning approach does not compress the networks as much: about 40% compression for Lenet-5 but no compression for the other networks at the similar accuracy level to the reference model. ### Comparison to the State-of-the-Art Approach

We also compare our method to the state-of-the-art network pruning approach based on the method of multipliers [33], which we refer to as MM. In this method, one first duplicates the learning parameters \(w\) with an auxiliary variable \(\theta\), to convert the training regularization problem (1) into the following constrained optimization problem:

\[\text{col\_indices} =[\ 0\ \ 1\ \ 1\ \ 2\ \ 0\ \ 2\ \ 3\ \ 1\ \ 3\ ]\]7 \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multicolumn{2}{c}{**Network**} & **Lenet-5** & **AlexNet** & **VGGNet** & **ResNet-32** \\ \hline \multirow{3}{*}{Pru (Retrain)} & Data & MNIST & CIFAR-10 & CIFAR-10 & CIFAR-10 \\  & Ref. Accuracy & 98.61\% & 78.61\% & 84.88\% & 90.05\% \\ \hline \multirow{3}{*}{Pru} & Accuracy & 37.01\% & 11.24\% & 10.00\% & 89.76\% \\  & Compression Rate & 0.97 (29\(\times\)) & 0.91 (10\(\times\)) & 0.94 (15\(\times\)) & 0.21 (1\(\times\)) \\ \cline{1-1}  & Accuracy & 98.19\% & 42.67\% & 10.00\% & 88.16\% \\ \cline{1-1}  & Compression Rate & 0.97 (33\(\times\)) & 0.97 (35\(\times\)) & 0.98 (51\(\times\)) & 0.88 (8\(\times\)) \\ \hline \multirow{3}{*}{SpC} & Accuracy & 97.78\% & 80.93\% & 85.53\% & 90.22\% \\ \cline{1-1}  & Compression Rate & 0.97 (32\(\times\)) & 0.91 (10\(\times\)) & 0.94 (17\(\times\)) & 0.21(1\(\times\)) \\ \cline{1-1} \cline{2-6}  & Accuracy & 98.29\% & 78.84\% & 84.63\% & 89.22\% \\ \cline{1-1}  & Compression Rate & 0.97(35\(\times\)) & 0.98 (47\(\times\)) & 0.98 (53\(\times\)) & 0.88 (8\(\times\)) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of network compression results. (Pru: network pruning, SpC: our method). Figure 7: The effect of retraining. The maximal compression rates where our methods SpC and SpC(Retrain) achieve at least 99% of the reference prediction accuracy are indicated by vertical lines (solid: SpC, dotted: SpC(Retrain)). Retraining is indeed required for Pru since otherwise high compression rate is not possible to achieve good prediction performance. On the contrary, our method SpC without retraining can achieve high accuracy at high compression. Retraining for SpC can be considered when we aim for very high compression. where \(L(w)=\frac{1}{n}\sum_{i=1}^{n}\ell(w;x_{i},y_{i})\) is the total loss and \(\Psi(\theta)\) is a regularizer. Then MM constructs an augmented Lagrangian function

\[\mathcal{L}(w,\theta,\lambda;\mu)=L(w)+\frac{\mu}{2}\|w-\theta\|^{2}-\lambda^{ T}(w-\theta)+\Psi(\theta). \tag{4}\]

Here, \(\lambda\in\mathbb{R}^{p}\) is the Lagrange multipliers and \(\mu>0\) is an auxiliary parameter. Then, MM iterates (i) the minimization of \(\mathcal{L}\) (in terms of \(w\) and \(\theta\) in an alternating fashion) and (ii) a gradient ascent step of \(\mathcal{L}\) in \(\lambda\), while driving \(\mu\rightarrow\infty\). Our approach has several benefits over the MM method. First, MM requires a pre-trained model to start network pruning, whereas our method can start from random weights. Second, due to the duplication of learning weights \(w\in\mathbb{R}^{p}\) by \(\theta\) in Equation (3) and the introduction of the Lagrange multipliers \(\lambda\in\mathbb{R}^{p}\), MM requires about double amount of memory for training than our method (ours: \((w,\
abla_{w}L(w))\) and MM: \((w,\
abla_{w}L(w),\theta,\lambda)\)). Third, the convergence of MM is quite sensitive to how we control learning rates and the auxiliary parameter \(\mu\rightarrow\infty\) in the augmented Lagrangian, where our method has no such sensitivity.