# CitiusSynapse: A Deep Learning Framework for Embedded Systems
## 5 Experiments
### Comparison of Inference Times  
Table 6 presents a comparison of the inference times between OpenCL Caffe and our framework. For the performance evaluation, inference was continuously performed on 100 images in each DNN, and then the average inference time was measured. The images which were used for inference were randomly selected from the dataset used in each DNN. In addition, the first inference time was excluded, because the build time of OpenCL kernels was added during the first inference, when the DNN inference was accelerated based on OpenCL. For our framework, the inference time, to which the model and NDRange optimization were applied, was also measured. The performance of VGG-16, to which the model and NDRange optimization were applied, was not evaluated, because batch normalization was not used for VGG-16. The evaluation results show that the GPU-based inference performance in OpenCL Caffe was considerably reduced when compared with that of the CPU. Because OpenCL Caffe is optimized for a general PC environment, it does not guarantee that the performance is optimized for an embedded system. The GPU-based inference performance using OpenCL had improved further than the CPU in most cases in our framework. In particular, the performance improved even further when model optimization (MO) and NDRange optimization (RO), which were proposed in our framework, were applied. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**System**} & \multirow{2}{*}{**DNN**} & \multicolumn{3}{c}{**OpenCL Caffe**} & \multicolumn{3}{c}{**Our Framework**} & \multirow{2}{*}{**Speed-Up**} \\ \cline{3-4} \cline{6-9}  & & **CPU** & **GPU** & & **GPU** & **(+MO 1)** & **GPU** \\ \hline \multirow{6}{*}{Odroid N2+} & VGG-16 & 1618 & 12,005 & 1606 & 1207 & - & - & 134 3 \\  & ResNet-18 & 236 & 1792 & 227 & 203 & 141 & 129 & 183 4 \\  & ResNet-101 & 877 & 7522 & 814 & 908 & 536 & 515 & 170 4 \\  & MobileNet-v2 & 308 & 3082 & 280 & 232 & 62 & 40 & 770 4 \\  & Yolo v3 & Not supported & 841 & 367 & 118 & 80 & 1051 5 \\  & EAST & Not supported & 795 & 459 & 176 & 153 & 520 5 \\  & CRNN & Not supported & 80 & 58 & 42 & 40 & 200 5 \\ \hline \multirow{6}{*}{RK3399} & VGG-16 & 2562 & 10,164 & 2855 & 2218 & - & - & 116 3 \\  & ResNet-18 & 366 & 1656 & 354 & 363 & 255 & 246 & 149 4 \\ \cline{1-1}  & ResNet-101 & 1930 & 7242 & 1299 & 1467 & 963 & 928 & 208 4 \\ \cline{1-1}  & MobileNet-v2 & 444 & N/A (error) & 930 & 354 & 94 & 64 & 694 4 \\ \cline{1-1}  & Yolo v3 & Not supported & 1603 & 529 & 193 & 146 & 1098 5 \\ \cline{1-1}  & EAST & Not supported & 1457 & 652 & 254 & 224 & 650 5 \\ \cline{1-1}  & CRNN & Not supported & 99 & 81 & 63 & 61 & 162 5 \\ \hline \hline \end{tabular}  
* Model optimization. 2 NDRange optimization. 3 CPU in OpenCL Caffe/GPU in our framework. 4 CPU in OpenCL Caffe/GPU (+MO/RO) in our framework. 5 CPU in our framework./GPU in our framework.  
\end{table} 
Table 5: List of DNNs used in performance evaluation. When the inference time was compared between OpenCL Caffe and our framework, the results show that performance was improved by 134% for heavy DNNs, such as VGG-16, and by 770% for light DNNs, such as MobileNet-v2. VGG-16 has a large number of parameters, and the operation is executed sequentially in each layer; thus, an optimization method applicable to an embedded system is limited, because operational resources are also limited. In contrast, the GPU acceleration performance significantly improved through the methods proposed in our framework for lightened DNNs, such as ResNet or MobileNet. In particular, in-place kernel optimization that simultaneously executes operations for weight and bias parameters can be applied if model optimization, which combines the batch normalization layer, is applied, which minimizes unnecessary operating costs. The evaluation also shows that the OpenCL-based acceleration performance can be maximized by setting the optimal parallel processing space for each OpenCL kernel when the NDRange optimizer is applied.