Scaling for edge inference of deep neural networks

© 2018 Macmillan Publishers Limited, part of Springer Nature. All rights reserved. © 2018 Macmillan Publishers Limited, part of Springer Nature. All rights reserved. 1Department of Computer Science, University of Notre Dame, Notre Dame, IN, USA. 2Department of Computer Science, University of California, Los Angeles, CA, USA. 3School of Optical and Electronic Information, Huazhong University of Science and Technology, Wuhan, China. *e-mail: yshi4@nd.edu Deep neural networks (DNNs) have recently demonstrated performance comparable with, and in some cases superior to, that of human experts in areas such as image recognition1–3 and the game of Go4. As the field develops, DNNs will be instrumental in delivering breakthroughs in various disciplines, including disease diagnosis, real-time language translation and autonomous driving5–9. Such accomplishments can be largely credited to everincreasing computing power and a growing abundance of data. As larger clusters of faster computing nodes become available at lower cost and in smaller form factors, more data (assuming abundant data is readily available for the application of interest) can be used to train DNNs with more layers and neurons. This should translate to higher inference accuracy10. Network sizes have increased drastically over time, reaching beyond the petascale for some applications. Figure 1 shows, for example, how the number of parameters in popular deep neural networks has increased over the last twenty years, and clearly illustrates an exponential growth. For applications where powerful computing resources are easily accessible through network connections, large networks such as these may not present substantial challenges. However, for edge computation on embedded hardware platforms, where security, privacy and/or latency are critical considerations (such as smart sensors, wearable devices, autonomous driving and unmanned aerial vehicle tracking), inference must be performed locally or at the edge of the network, and such computation is subject to stringent area and power constraints due to the limited resources available. To address the computational demands of ever-increasing network sizes, hardware architects have begun exploring techniques to compress DNNs for efficient edge inference. The ultimate judgment of such techniques is that lower power and area overheads can be achieved with minimal loss in inference accuracy. As many data scientists are focusing on increasing inference accuracy through designing more complex DNNs, there is a race between data scientists and hardware architects. Complementary metal–oxide semiconductor (CMOS) technology scaling based on Moore’s law has provided hardware designers with a relatively easy path towards accommodating increasing network sizes. However, with the slowdown of CMOS scaling trends, novel architectures specifically designed for neural networks have emerged. In this Perspective, we demonstrate that gaps exist between Moore’s-law-based CMOS scaling and the scaling of DNNs for edge inference, and discuss various architecture and algorithm innovations that could help to bridge these gaps.