TwinDNN: A Tale of Two Deep Neural Networks

1 INTRODUCTION . Machine learning is one of the most popular fields in the current era . It is used in various ways , such as speech recognition , face recognition , medical diagnosis , etc . However , the problem is that the neural networks for machine learning applications Krizhevsky et al . ( 2012 ) ; He et al . ( 2016a ; b ) are becoming too large and slow to be on a small chip for real-time systems . As a result , there has been a significant amount of research to reduce the size of the neural networks so that their inference latencies are low enough to handle real-time inputs Zhang et al . ( 2020 ) ; Zhou et al . ( 2016 ) ; Zhang et al . ( 2018 ) . There are quite a few approaches to compress existing neural networks , but for field-programmable gate arrays ( FPGAs ) , quantization of network is the most popular and effective method to reduce the size and inference latency at the same time Han et al . ( 2016 ) . In particular , extremely low bit-width networks on FPGAs , such as binary or ternary neural networks have been studied recently Wang et al . ( 2018 ) ; Courbariaux & Bengio ( 2016 ) ; Courbariaux et al . ( 2015 ) ; Zhao et al . ( 2017 ) ; Yao Chen & Chen ( 2019 ) ; Li & Liu ( 2016 ) ; Blott et al . ( 2018 ) . These networks provide an extra benefit that normal quantized networks do not provide in terms of multiplier ( DSP ) utilization . The idea is that extremely low bit-width weights allow multiplications to be done in conditional logic , which can be implemented by logic gates , without using a special hardware for multiplication ( DSP ) . This fact can allow developers to utilize additional DSPs in other ways where DSPs can be useful . However , this benefit is not free , of course . One major downside of these low bit-width networks is that they tend to have even more accuracy drop than regular quantized neural networks , as a result of further reduced precision . Therefore , it is more difficult to use binary or ternary neural networks as they are , especially in the fields such as surveillance or medical diagnosis systems , where the cost of that accuracy drop is much larger than the latency improvement . The goal of this study is to accelerate neural network inferences by using an extremely low bitwidth network implementations on FPGAs , while maintaining the accuracy of the original network by using relatively high precision network concurrently , without having to develop a single DNN accelerator that meets both accuracy and latency requirements . The main contribution is to find a mechanism to choose the right network to infer for specific inputs , and this is done by creating a hierarchical structure of two different compressed networks and utilizing the output of the initial inference to determine the need of extra verification . In summary , we propose a system that consists of two distinct networks : one extremely low bit-width network that is focused on latency , and the other moderately quantized network that is focused on accuracy . In this paper , extremely low bit-width network will be called a compressed network , and moderately quantized network will be called an original network . These two networks work in a way that can exploit advantages in both latency and accuracy at the same time . The overall mechanism is similar to the one presented in Mocerino & Calimera ( 2014 ) . However , there has not been any study of this concept in FPGA accelerators for deep neural networks . This represents a novel direction in neural network research , to pair compressed network and original network in parallel in order to improve latency while maintaining the original accuracy . Our main contributions are as follows : • Accelerators that are designed and optimized to exploit low bit-width networks , with pipelined and parallelized computation engines that use a minimum number of DSPs as possible . • A software solution that allows two accelerators to be run in hierarchical fashion , utilizing confidence of a compressed network prediction . • For ImageNet and ResNet-18 , our TwinDNN solution can deliver up to 1.9× latency improvement with only 3 % of extra DSPs used for compressed network , and up to 95 % of accuracy loss is recovered during hierarchical inference . In Section 2 , some background information related to this work will be introduced . In Section 3 , design flow of our implementation and experiment will be explained . In Section 4 , the results of our experiments will be described . Section 5 will conclude the paper with future explorations . 2 BACKGROUND . 2.1 EXTREMELY LOW BIT-WIDTH NEURAL NETWORKS . Recent researches have succeeded in binarizing or ternarizing parts of layers in neural networks Wang et al . ( 2018 ) ; Courbariaux & Bengio ( 2016 ) ; Courbariaux et al . ( 2015 ) ; Zhao et al . ( 2017 ) ; Yao Chen & Chen ( 2019 ) . Many experiments claim that these compression methods are very effective in terms of latency reduction with some accuracy drops . As one would expect , as the number of bits used to represent either weights or feature maps decreases , accuracy drops more significantly . Because the goal of our study is to compensate for the accuracy loss caused by compression , we can forgive moderate accuracy loss , as long as the benefit of using those networks is significant . The following equations : wb = { −wscale if b = 0 +wscale if b = 1 wt = { −wscale if t = −1 +wscale if t = 1 0 if t = 0 ( 1 ) show how these extremely low bit-width weights are used in computation . b is a 1-bit value that can be either 0 or 1 , and t is a 2-bit value that can take either -1 , 0 , or 1 . The key idea here is that wscale value is the same across the weights . The bits are only used in sign representations . In binary , as an example , a single bit of 0 represents negative and 1 represents positive , and this logic can be implemented in a simple condition , or a multiplexer in FPGAs . wscale value is stored separately , and the same wscale value is multiplied over all binary weights to get the actual weight values . However , we do not need to perform all of these multiplications separately . Consider b1 = 0 and b2 = 1 for the binary case . a stands for activation , or feature map , then we can express a very simple neural network computation as follows : anext = wb1 × a1 + wb2 × a2 = −wscale × a1 + wscale × a2 = wscale × ( −a1 + a2 ) ( 2 ) This shows how binary and ternary weight computations can be handled with a single multiplication . Reducing the number of actual multiplications reduces the need of DSPs , and indeed makes the overall computation faster . For ternary , the only difference is that two bits now represent positive , negative , and zero . Therefore , the main benefit of using extremely low bit-width neural networks is more effective and balanced resource utilization , specifically on FPGAs . For a typical DNN implementation on FPGAs , DSP is the one that directly determines the performance , and so is the limiting factor of the performance . Therefore , typical DNN implementations on FPGAs utilize nearly all the DSPs available , and other resources are left under-utilized . Extremely low bit-width networks , on the other hand , only uses a minimal number of DSPs , and mainly utilizes look-up tables ( LUTs ) , which are generally left unused for computation ( e.g. , only used for control logic ) . In this study , we instantiate both typical DNN ( original DNN ) and extremely low bit-width DNN ( compressed DNN ) at the same time , in a way that original DNN uses most of the DSPs available on the board , and compressed DNN uses extra LUTs that were not used by original DNN . This method allows us to utilize both DSP and LUT resources as much as possible to ultimately reduce the overall latency . 2.2 FINAL LAYER OUTPUTS IN NEURAL NETWORK . In neural network image classifications , output of the final layer is a list of values for each class , and the class with the highest final layer output is typically chosen as a prediction . Here , each value represents how possible is that image in the class , and we will call these values probabilities for convenience . Note that in order to compute the actual probabilities , we need to apply a softmax function to these values . However , we do not apply a softmax function here , because we only need to find the class with the highest value . Anyways , the point is that the output of a neural network provides more information than just a prediction . Specifically , it provides information about all probabilities of predictions that the input can be classified as . It is definitely possible to utilize this additional information to enhance the prediction . In this study , we will make use of the probability of the second most possible label . From compressed network inference output , along with the prediction itself , we also need to find out whether to infer the original network or not . The probability of the second most possible label is used here to determine if the prediction of compressed network is confident enough to be used as an actual output without verification from the original network . Here is an example of how we can utilize this information during inference . For handwritten digit recognition , let ’ s define Out ( x ) as final output value for label x . There is no problem if Out ( 0 ) = 0.9 and Out ( 1 ) , . . . , Out ( 9 ) < 0.1 , because the network is almost sure that the digit is 0 . However , in case where Out ( 1 ) = 0.5 and Out ( 2 ) = 0.4 , although Out ( 1 ) is greater than Out ( 2 ) , we can not guarantee that 1 is actually a correct label , because that 0.1 difference could have been resulted from the noise of using a low precision network . This is what we use as a confidence of the prediction . Instead of just finding a label with maximum probability , the system will now find two labels with first and second maximum probabilities , and compute the difference between those two probabilities . If the difference is large ( i.e. , beyond a threshold determined empirically ) , it means that the label with highest probability dominates other labels , so compressed network prediction can be considered confident . If the difference is small , it means that two labels with highest probabilities both have potentials to become a true label , so the prediction is considered not confident . In this case , the input needs additional verification from the original network that is designed to have maximum accuracy . 3 IMPLEMENTATION . Our implementation flow consists of three parts : creating original network and compressed network models , implementing high-level-synthesis ( HLS ) accelerator intellectual properties ( IPs ) for those networks , and creating a software system for hierarchical inference .

Summary: This paper proposes a framework to accelerate DNN inference on small embedding systems using an extremely low bit network and a moderately quantized network jointly. The mechanism of the proposed work is to first compute the difference using top2 prediction scores from the compressed network to determine if the inference is further needed for the original network. The proposed framework is evaluated on both CIFAR10 and ImageNet with different network structures and the empirical results indicate better accuracy and latency over baselines.