HASP: Hierarchical Asynchronous Parallelism for Multi-NN Tasks

Authors:
raw_author_name: Guanrui Wang
raw_affiliation_string: Lynxi, Company, Ltd., Beijing, China
raw_author_name: Junfeng Lin
raw_affiliation_string: Center for Brain-Inspired Computing Research, Optical Memory National Engineering Research Center, Tsinghua University-China Electronics Technology HIK Group Co., Joint Research Center for Brian-inspired Computing, IDG/McGovern Institute for Brain Research at Tsinghua University, Department of Precision Instrument, Tsinghua University, Beijing, China
raw_author_name: Cheng Ma
raw_affiliation_string: Center for Brain-Inspired Computing Research, Optical Memory National Engineering Research Center, Tsinghua University-China Electronics Technology HIK Group Co., Joint Research Center for Brian-inspired Computing, IDG/McGovern Institute for Brain Research at Tsinghua University, Department of Precision Instrument, Tsinghua University, Beijing, China
raw_author_name: Jing Pei
raw_affiliation_string: Center for Brain-Inspired Computing Research, Optical Memory National Engineering Research Center, Tsinghua University-China Electronics Technology HIK Group Co., Joint Research Center for Brian-inspired Computing, IDG/McGovern Institute for Brain Research at Tsinghua University, Department of Precision Instrument, Tsinghua University, Beijing, China
raw_author_name: Rong Zhao
raw_affiliation_string: Center for Brain-Inspired Computing Research, Optical Memory National Engineering Research Center, Tsinghua University-China Electronics Technology HIK Group Co., Joint Research Center for Brian-inspired Computing, IDG/McGovern Institute for Brain Research at Tsinghua University, Department of Precision Instrument, Tsinghua University, Beijing, China

Abstract: The rapid development of deep learning has propelled many real-world artificial intelligence applications. Many of these applications integrate multiple neural networks (multi- NN) to cater to various functionalities. There are two challenges of multi-NN acceleration: (1) competition for shared resources becomes a bottleneck, and (2) heterogeneous workloads exhibit remarkably different computing-memory characteristics and various synchronization requirements. Therefore, resource isolation and fine-grained resource allocation for each task are two fundamental requirements for multi-NN computing systems. Although a number of multi-NN acceleration technologies have been explored, few can completely fulfill both of these requirements, especially for mobile scenarios. <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">This paper reports a Hierarchical Asynchronous Parallel Model (HASP) to enhance multi-NN performance to meet both requirements. HASP can be implemented on a multicore processor that adopts Multiple Instruction Multiple Data (MIMD) or Single Instruction Multiple Thread (SIMT) architectures, with minor adaptive modification needed. Further, a prototype chip is developed to validate the hardware effectiveness of this design. A corresponding mapping strategy is also developed, allowing the proposed architecture to simultaneously promote resource utilization and throughput. With the same workload, the prototype chip demonstrates 3.62×, and 3.51× higher throughput over Planaria and 8.68×, 2.61× over Jetson AGX Orin for MobileNet-V1 and ResNet50, respectively.