FPGA-based Deep Learning Inference Acceleration at the Edge

Over the last few years, with the exponential increase in the amount of data available, Deep Neural Networks (DNNs) have become the most widely used computational model in solving complex problems for most of applications, due to the extremely high level of accuracy they are capable of attaining compared to that of traditional Machine Learning (ML) algorithms, even better than human capabilities. However, in order to achieve this level of accuracy on the results produced, the number of layers and neurons of DNNs have become increasingly high over time, resulting in a huge demand of computational and storage resources to handle the execution on the increasing number of operations and parameters required in DL processing in real-time, both during the training and inference phases. While currently the training requires too high computational performance and arithmetic precision to think of an execution different than on the cloud, the current trend is to move the inference execution from the cloud to the edge, close to where the data are collected, saving the latency and energy consumption required to transfer the processing to the server. However, while the execution at the cloud is based on maximizing performance, the requirements for inference processing at the edge can be very different, focusing on high throughput, low latency and low energy consumption. Thus, maintaining a high accuracy for DNN models implemented on devices with a limited amount of resources results to be a challenging task. To address this problem, a large number of specialized Artificial Neural Network (ANN) accelerators based on different processing units have been proposed over time, designed to accelerate the inference processing in hardware by exploiting the high level of inherent parallelism exhibited by DNN models. In general, the standard internal architecture of each of these processing units is composed of an array of Processing Elements (PEs) responsible for the execution of the multiplyand-accumulate (MAC) operations within the fully connected (FC) and convolutional (CONV) layers that compose the DNN model and to which a memory system is connected. The later is usually split into on-chip and off-chip storage elements that hold the network parameters of all the computational units of the implemented DNN model. However, depending on the selected hardware platform, this architectural organization undergoes variations, adapting itself more or less well to the DL algorithm to be executed. Therefore, for the realization of the ANN inference accelerators, of fundamental importance turns out to be the choice of the processing unit architecturally better suited for DNN computation, to minimize the latency and energy consumption, and especially of the memory accesses, which often represent the main DNN processing bottleneck. In this scenario, all the possible processing units capable of meeting the latency and energy consumption constrains imposed by edge computing represent a possible solution for DL inference processing at the edge. These include general-purpose processors (GPPs), such as Central Processing Units (CPUs), Graphical Processing Units (GPUs), re-configurable hardware platforms, such as Field Programmable Gate Arrays (FPGAs), and special-purpose Application Specific Integrated Circuits (ASICs) designed specifically for ML applications, Tensor Processing Units (TPUs). Each solution has specific features in terms of flexibility, speed and energy efficiency,