# CitiusSynapse: A Deep Learning Framework for Embedded Systems
## 6 Conclusions and Discussion  
In this study, we have proposed a deep learning framework that is specialized for embedded systems. The proposed framework provides a deep learning core, including data and layer structures, an OpenCL-based accelerator engine, and an on-device optimizer to improve the learning and inference performance on real devices, and a model converter for compatibility. The deep learning core of our framework was designed for unified memory, thus minimizing the operation costs required for OpenCL data management by preventing unnecessary data transfer between the CPU and GPU. In addition, the accelerator engine provides the BLAS library, which is optimized for the embedded GPU, in-place kernel optimization that minimizes the repeated access cost of data, and asynchronous queue execution for maximizing the GPU utilization. The on-device optimizer for inference provides a model optimizer for improving the inference performance by combining layers, an NDRange optimizer for searching for the optimal parallel processing space, and a quantization optimizer, which reduces the computation amount through INT8 quantization. Finally, our framework provides a model converter for compatibility, enabling it to use models trained in conventional deep learning frameworks. The performance evaluation shows that our framework compares favorably in terms of inference time and memory usage with OpenCL Caffe, which is an existing deep learning \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**DNN**} & **PyTorch** & \multicolumn{2}{c}{**Our Framework**} \\ \cline{3-4}  & **Jetson Nano with CUDA/cuDNN** & \multicolumn{2}{c}{**Odroid N2+ with OpenCL**} \\  & **(472 GFLOPS)** & \multicolumn{2}{c}{**(163 GFLOPS)**} \\ \cline{2-4}  & **GPU** & **GPU** & **GPU (+MO)** & **GPU (+MO\&RO)** \\ \hline ResNet-18 & 43 1 & 203 & 141 & 129 \\ MobileNet-v2 & 51 1 & 58 & 42 & 40 \\ \hline \hline \end{tabular}  
* It uses TorchVision pre-trained model: pytorch/visionv0.10.0.  
\end{table} 
Table 8: Comparison of inference times (ms) of PyTorch and our framework. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**System**} & \multirow{2}{*}{**DNN**} & **Usage** & \multicolumn{2}{c}{**OpenCL Caffe**} & \multicolumn{2}{c}{**Our Framework**} & \multirow{2}{*}{**Reduction**} \\  & & & **Type** & **CPU** & **GPU** & **CPU** & **GPU** & **Ratio (\%)1** \\ \hline \multirow{6}{*}{Odroid N2+} & VGG-16 & Average & 1164 & 1724 & 944 & 1288 & 25 \\  & Max & 2334 & 2620 & 1842 & 2236 & 15 \\  & ResNet-18 & Average & 395 & 688 & 496 & 500 & 27 \\  & Max & 395 & 707 & 602 & 657 & 7 \\  & ResNet-101 & Average & 807 & 1551 & 809 & 744 & 52 \\  & Max & 807 & 1710 & 1116 & 1302 & 24 \\  & MobileNet-v2 & Average & 550 & 1007 & 468 & 831 & 17 \\  & Max & 550 & 1226 & 685 & 939 & 23 \\ \hline \hline \end{tabular}  
* 1 – (GPU in our framework)/(GPU in OpenCL Caffe).  
\end{table} 
Table 9: Comparison of memory usage (MB) in Odroid N2+. framework. Furthermore, our framework was confirmed as being competitive against NNAPI, according to the performance evaluation results of the AI benchmark. Currently, research on on-device learning, based on field data in embedded systems, is challenging. Our framework focuses on optimizing the inference performance of embedded systems, and provides a limited functionality for on-device learning. Therefore, we plan to study an acceleration engine and an on-device optimizer for on-device learning. Also, our framework is optimized for the ARM CortexA CPU and Mali GPU in a Linux environment. Therefore, the framework will be expanded in the future to enable its use in other HW environments and operating systems that are used in various common mobile devices, such as Android or Qualcomm. Finally, we plan to study a tuning algorithm that automatically analyzes the problems and potential of parallelism, based on the characteristics of the embedded system, and explore the optimization mechanism for this. Conceptualization, S.H. ; methodology, S.H. and H.C.; software, S.H.