FPGA Acceleration of a Multimodal Neural Network

Currently, Internet of Things and Deep Learning (DL) are merging into one domain and creating outstanding technologies for various classification tasks. Such technologies require complex DL networks that are mainly targeting powerful platforms with rich computing resources like servers. Therefore, for resource-constrained embedded systems, new challenges of size, performance and power consumption have to be considered, particularly when edge devices handle multimodal data, i.e., different types of real-time sensing data. Our ongoing project is focused on DeepSense, a multimodal DL framework combining Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) to process accelerometer and gyroscope data to detect human activity. We aim at accelerating DeepSense by FPGA in a hardware-software co-design manner. Our demo will show the latest achievements through latency and power consumption evaluations. 1 Background and Motivation With the emergence of multi-sensor edge devices, embedded systems need to process big data collected in real time from different sensors whose data are defined in the different type. This is also the case for neural networks running on the edge devices. Such networks are called “multimodal neural networks” and can be used for efficient detection/classification by leveraging combination of voice and video or text and images or even complex human body vital signals. The main interest of our project is to accelerate the multimodal neural networks, particularly the ones that handle vital signals with time-series data like heart rate, temperature, and motion signals. In this project, we specifically target DeepSense [1] which is a multimodal deep learning framework designed for processing multiple time-series data. We utilize FPGA for the acceleration since FPGAs are the promising competitor to GPUs. With customized FPGA implementations, Convolutional Neural Networks (CNNs) have demonstrated better efficiency in both power consumption and execution time (e.g., up to 9× in power reduction and up to 2.5× speedup in latency) than GPUs. 2 FPGAAcceleration of DeepSense Figure 1 depicts the overview of DeepSense, which has three “individual CNN layers” (the bottom three layers) collecting data from K sensors. After extracting the features from individual layer, three “Merge CNN layers” learn interactions among all K sensors. Then two “Recurrent Neural Network (RNN) layers” extract the feaFigure 1: An overview of DeepSense [1] tures from time sequences, and finally a fully connected “Output Layer” (the top layer) is used for classification. In our implementation, K=2 is set (a gyroscope and an accelerometer) to detect six types of human activity. In our work, a Xilinx Zynq UltraScale board is used to co-design DeepSence – hardware part on the FPGA reconfigurable fabrics and the software parts on an ARM CPU. Figure 2 illustrates the current implementation on the board. While the data collection and pre-processing are done on the CPU, CNNs are implemented as hardware so that the computation parallelism can be fully utilized. The extracted features are then sent back to the CPU to conduct the RNN on the CPU again. The fully-connected CNN is then performed as hardware. Our presentation will compare the latency and power consumption of this architecture with the original work [1]. Figure 2: Zynq Implementation of DeepSense