# Moving Deep Learning to the Edge
## 4 Deep Learning Inference on Edge Devices
### Edge Computing Architectures for Deep Learning  
Hardware solutions exist to run deep learning services at edge servers. However, running large deep learning models on end devices can be cumbersome because they have to cope with the intensive computation of full deep learning models [170]. One possible solution is to use an edge computing platform with the possibility to access cloud computing platforms if needed. Different approaches for deep learning execution on the edge can be identified depending on how the task load is distributed among end devices, the edge, and the cloud. A simple approach is to send the complete task to be executed on the edge or the cloud. The end device decides to offload or not the task and to where it should be offloaded. This approach was followed in [171]. The framework decides if the inference of a deep learning model should be executed in the end device or offloaded to an edge server. The decision depends on the state of the end device (battery) and the network conditions. The edge is also resource limited compared to the cloud, and therefore, different deep learning models with different accuracy and complexity are available. Depending on the application, a smaller model may be enough to achieve the required accuracy. A broader distributed solution permits the partial offloading of the inference to the edge or the cloud. A few solutions already exist to deploy partial offloading of tasks to edge servers. In [172], task partition can be done statically or dynamically, where the partition depends on the network and device conditions. Other partial offloading solutions exist specifically for deep learning. In [173], the model was partitioned and uploaded in pieces. The edge incrementally builds the model and can start the execution of the model before the whole deep learning model is loaded. The objective is to reduce the latency response by overlapping communication and computation. The solution also allows some layers to be executed in the end device whenever the computational complexity associated with a layer does not justify the communication of the layer. The edge server of an edge computing platform is also memory- and resource-limited. When multiple end devices ask for a specific DL service, the edge may not be able to process them on time. In these cases, the edge uses cloud services. The decision to offload the model to the edge or the cloud can be made by the end device. In [23], the end device had a performance model to determine the \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Accelerator** & **Year** & **Model** & **GOPs** & **GOPs/W** \\ \hline \multicolumn{5}{c}{**ASIC Technology**} \\ Gyrfalcon [151] & 2018 & CNN & 16.8 TOPs & 24 TOP/W \\ LinleyGroup [152] & 2018 & CNN & 12.5 TOPs & — \\ EVex [153] & 2017 & DNN & 9.0 TOPs & 4 TOPs/W \\ Tensilica DNA [154] & 2017 & DNN & 24 TOPs & 6.8 TOPs/W \\ Myriad X [155] & 2017 & DNN & 4 TOPs & — \\
[156] & 2017 & CNN & — & 3.9 TOPs/W \\ Snapdragon [157] & 2019 & DNN & 15 TOPs & 3 TOPs/W \\ Kirin 900 [158] & 2019 & DNN & 8 TOPs & — \\ Ascend 910 [159] & 2019 & DBB & 512 TOPs & 1.7 TOPs/W \\ \hline \multicolumn{5}{c}{**Reconfigurable Computing Technology**} \\ Eyeriss [161] & 2018 & CNN & 46.2 GOPs & 166 GOPs/W \\ Thinker chip [162] & 2018 & CNN & 368 GOPs & 1 TOPs/W \\ DRP [163] & 2018 & CNN & 960 GOPs & — \\
[167] & 2018 & CNN & 13 GOPs & — \\
[168] & 2018 & CNN & 84 GOPs & — \\
[169] & 2018 & CNN & 363 GOPs & 33 GOPs/W \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Summary of hardware accelerators for deep learning on the edge. expected execution times of each part of the model in each computing platform (end device, edge, and cloud). Based on the estimated performance, the network state, and the energy level, the device decides how the model should be partitioned among the different computing platforms. Another solution [174] considers an iterative inference process, where initially, a simple DL model is used and the accuracy is assessed.