TwinDNN: A Tale of Two Deep Neural Networks
Authors: Hyunmin Jeong
Deming Chen
Venue:  (2021)
Keywords: Hardware Accelerator
High-Level-Synthesis
Machine Learning
Neural Network Quantization

Abstract: Compression technologies for deep neural networks (DNNs), such as weight quantization, have been widely investigated to reduce the DNN model size so that they can be implemented on hardware with strict resource restrictions. However, one major downside of model compression is accuracy degradation. To deal with this problem effectively, we propose a new compressed network inference scheme, with a high accuracy but slower DNN coupled with its highly compressed DNN version that typically delivers much faster inference speed but with a lower accuracy. During inference, we determine the confidence of the prediction of the compressed DNN, and infer the original DNN for the inputs that are considered not confident by the compressed DNN. The proposed design can deliver overall accuracy close to the high accuracy model, but with the latency closer to the compressed DNN. We demonstrate our design on two image classification tasks: CIFAR-10 and ImageNet. Our experiments show that our design can recover up to 94% of accuracy drop caused by extreme network compression, with more than 90% increase in throughput compared to just using the original DNN. This is the first work that considers using a highly compressed DNN along with the original DNN in parallel to improve latency significantly while effectively maintaining the original model accuracy. 

Under review as a conference paper at ICLR 2021
TWINDNN: A TALE OF TWo DEEP NEURAL NET- WORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Compression technologies for deep neural networks (DNNs), such as weight quantization, have been widely investigated to reduce the DNN model size so that they can be implemented on hardware with strict resource restrictions. However, one major downside of model compression is accuracy degradation. To deal with this problem effectively, we propose new compressed network inference scheme, with high accuracy but slower DNN coupled with its highly compressed DNN version that typically delivers much faster inference speed but with a lower ac- curacy. During inference, we determine the confidence of the prediction of the compressed DNN, and infer the original DNN for the inputs that are considered not confident by the compressed DNN. The proposed design all accuracy close to the high accuracy model, but with the latency closer to the compressed DNN. We demonstrate our design on two image classification tasks: CIFAR-10 and ImageNet. Our experiments show that our design can recover up 94% of accuracy drop caused by extreme network compression. with more than 90% increase in throughput compared to just using the original DNN. This is the first work that considers using a highly compressed DNN along with the original DNN in parallel to improve latency significantly while effectively maintaining the original model accuracy.
deliver over-
INTRODUCTION
Machine learning one of the most popular fields in the current era. is used in various ways, such as speech recognition, face recognition, medical diagnosis, etc. However, the problem is that the neural networks for machine learning applications Krizhevsky et al. (012)); He et al.) (2016a b) are becoming too large and slow to be a small chip for real-time systems. As a result, there has been a significant amount of research to reduce the size of the neural networks so that their inference latencies are low enough to handle real-time inputs Zhang et al. (2020); Zhou et al. (206)); Zhang et al. (2018). There are quite a few approaches to compress existing neural networks, but for weld-programmable gate arrays (FPGAs), quantization of network is the most popular and effective method to reduce the size and inference latency at the same time Han et al. (2016). In particular, FPGAs, such binary ternary neural networks have been
extremely low bit-width networks studied recently Wang et (2018): Courbariaux & Bengio (016)); Courbariaux et al. (2015); (Zhao al. (017)): Yao Chen & Chen (2019;; Li & Liu (2016): Blott et al. (2018). These networks provide an extra benefit that normal quantized networks do not provide in terms of multiplier (DSP) utilization. The idea is that extremely low bit-width weights allow multiplications to be done in conditional logic, which can be implemented by logic gates, without using a special hardware for multiplication (DSP). This fact can allow developers to utilize additional DSPs other ways where DSPs can be useful. However, this benefit is not free, of course. One major downside of these low bit-width networks is that they tend have even more accuracy drop than regular quantized neural networks, result of further reduced precision. Therefore, it is more difficult to use binary or ternary neural networks as they are, especially in the fields such as surveillance or medical diagnosis systems, where the cost of that accuracy drop is much larger than the latency improvement.
The goal of this study is to accelerate neural network inferences by using an extremely low bit- width network implementations by using relatively high precision network concurrently, without having to develop a single DNN accelerator that meets both accuracy and latency requirements. The main contribution is to find
FPGAs, while maintaining the accuracy of the original network

---

Under review as a conference paper at ICLR 2021
mechanism choose the right network infer for specific imputs, and this is done by creating hierarchical structure two different compressed networks and utilizing the output the initial inference to determine the need of extra verification.
In summary, we propose a system that consists two distinct networks: one extremely low bit- bit-width network that is focused latency, and the other moderately quantized network that is focused on accuracy. In this paper, extremely low bit-width network will be called a compressed network, and moderately quantized network will be called an original network. These two networks work in way that can exploit advantages in both latency and accuracy at the same time, The overall mechanism is similar the one presented in Mocerino & Calimera (2014). However, there has been any study of this concept in FPGA accelerators for deep neural networks. This represents a novel direction in neural network research, to pair compressed network and original network in parallel in order to improve latency while maintaining the original accuracy. Our main contributions are follows:
Accelerators that are designed and optimized to exploit low bit-width networks, with pipelined and parallelized computation engines that use minimum number of DSPs as possible.
A software solution that allows two accelerators to be run in hierarchical fashion, utilizing confidence of a compressed network prediction.
For ImageNet and ResNet-18, TwinDNN solution deliver up to 1.9 latency im- provement with only 3% of extra DSPs used for compressed network, and up to 95% of accuracy loss is recovered during hierarchical inference.
In Section 2, some background information related to this work will be introduced. In Section 3, design flow of our implementation and experiment will be explained. In Section 4, the results of our experiments will be described. Section 5 will conclude the paper with future explorations.
BACKGROUND
2.1 EXTREMELY Low BIT-WIDTII NEURAL NETWORKS
Recent researches have succeeded in binarizing or ternarizing parts of layers in neural networks Wang et al. (2018); Courbariaux Bengio (2016); Courbariaux et al. (2013): Zhao al.] (2017); Yao Chen & Chen (219).. Many experiments claim that these compression methods are very effec- tive in terms of latency reduction with some accuracy drops. As one would expect, as the number of bits used to represent either weights or feature maps decreases, accuracy drops more significantly. Because the goal of our study is to compensate for the accuracy loss caused by compression, we can forgive moderate accuracy loss, long the benefit of using those networks is significant. The following equations:
W Uscale if = 0 if
W/
W Wocale
(I)
W Wsente
Wi Weenle
if / 1
0
if =
show how these extremely low bit-width weights are used in computation. b is 1-bit value that can be either 0 or and is a 2-bit value that can take either -1, O, or 1. The key idea here that Wseale value is the same across the weights. The bits are only used in sign representations. In binary, as an example, a single bit of represents negative and I represents positive, and this logic can be implemented in simple condition, the same Wscale value is multiplied over all binary weights to get the actual weight values. However, do not need to perform all of these multiplications separately. Consider b1 0 and 62 1 for the binary case. a stands for activation, or feature map, then we can express a very simple neural network computation as follows:
multiplexer in FPGAs. Wscale value is stored separately, and
2

---

Under review as a conference paper at ICLR 2021
Onent Wb1 a1 W62 a2
- Wscale X 01 + Wscale X 02
Wscale X (-a1 02))
This shows how binary and ternary weight computations can be handled with a single multiplication. Reducing the number of actual multiplications reduces the need of DSPs, and indeed makes the overall computation faster. For ternary, the only difference is that bits now represent positive, negative, and zero. Therefore, the main benefit of using extremely low bit-width neural networks is more effective and balanced resource utilization, specifically FPGAs. For typical DNN implementation on FPGAs. DSP is the one that directly determines the performance, and so is the limiting factor of the performance. Therefore, typical DNN implementations on FPGAs utilize nearly all the DSPs available, and other resources are left under-utilized. Extremely low bit-width networks, on the other hand, only uses a minimal number of DSPs, and mainly utilizes look-up tables (LUTs), which are generally left unused for computation (e.g.. only used for control logic). In this study, we instantiate both typical DNN (original DNN) and extremely low bit-width DNN (compressed DNN) at the same time, in way that original DNN uses most of the DSPs available on the board, and compressed DNN uses extra LUTs that were not used by original DNN. This method allows us to utilize both DSP and LUT resources much possible to ultimately reduce the overall latency.
2.2 FINAL LAYER OUTPUTS IN NEURAL NETWORK
In neural network image classifications, output of the final layer is a list of values for each class, and the class with the highest final layer output is typically chosen as prediction. Here, each value represents how possible is that image in the class, and we will call these values probabilities for convenience. Note that in order to compute the actual probabilities, we need to apply a softmax function to these values. However, we do not apply a softmax function here, because we only need to find the class with the highest value. Anyways, the point is that the output of neural network provides more information than just prediction. Specifically, it provides information about all probabilities of predictions that the input can be classified as. It is definitely possible to utilize this additional information to enhance the prediction. In this study, we will make use of the probability of the second most possible label. From compressed network inference output, along with the prediction itself, we also need to find out whether to infer the original network or not. The probability of the second most possible label is used here to determine if the prediction of compressed network is confident enough to be used an actual output without verification from the original network.
Here is an example of how recognition, let's define Out (x) as final output value for label x. There is no problem if (ut = 0.9 and (I)( (9) 0.1, because the network is almost sure that the digit is O. However, in case where Ou((1) 0.5 and (2)(2) = 0.4, although Out(1) is greater than (u) we cannot guarantee that 1 is actually a correct label, because that 0.1 difference could have been resulted from the noise of using low precision network. This is what we use as confidence of the prediction. Instead of just finding label with maximum probability, the system will now find two labels with first and second maximum probabilities, and compute the difference between those two probabilities. If the difference is large (i.c., beyond a threshold determined empirically), it means that the label with highest probability dominates other labels, so compressed network prediction can be considered confident. If the difference is small, it means that two labels with highest probabilities both have potentials to become true label, so the prediction is considered not confident. In this case, the input needs additional verification from the original network that is designed to have maximum accuracy.
utilize this information during inference. For handwritten digit
3 IMPLEMENTATION
Our implementation flow consists of three parts: creating original network and compressed network models, implementing high-level-synthesii (HLS) accelerator intellectual properties (IPs) for those networks, and creating software system for hierarchical inference.
3

---

Under review as a conference paper at ICLR 2021
3.1 MODEL GENERATION
Creating original network starts with a typical floating-point training, which can also be completed by using pre-trained model available. For training. we used Caffe framework Jia et al. (2014), which was also customized to be used by other works (e.g., Yao Chen & Chen (0119). In order to enhance the accuracy, we use a variety of well-known techniques, such as learning rate decay and batch normalization. Specifically, batch normalization of activations allows us to standardize and stabilize the outputs, which can speed up the training process. After these networks are trained
to the point where accuracy does not improve anymore, we merge these batch-normalization layers into convolutional layers using
Y X wold
Wnew
VV Varriane
(3)
bnew Y X (bold MMaan) + B
Variance
where Meen, V Varionce, 7 and 9 are the parameters trained in natth-normalization layers, Wold and dotd are weight and bias values before merge, and Wnew and bnew those value after merge. This is typically done in neural networks to reduce the extra latency of normalizing the activations during inference, and it gives us exactly the same inference results.
The network weights are then quantized to designated bit-widths. Quantization scheme is deter- mined by accuracy drop and distribution of weights. First, we try a uniform quantization scheme. We always use uniform quantization whenever possible because non-uniform quantization requires extra logic and computation required for bit shifting in hardware. If accuracy drop is significant, we then try non-uniform quantization scheme depending on the distribution of weights and activations. There can still exist a slight accuracy drop after non-uniform quantization, and there are few ways presented in Zhao et al. (2019); NVIDIA to recover this accuracy drop, which can be implemented the future.
Compressed network model, on the other hand, cannot be generated without training scheme that is specifically designed for binary and ternary neural networks. For binary neural network, which was used in our CIFAR-10 10 experiment, using the method proposed by [Courbariaux 5% accuracy drop compared to the original network model. As explained in Zhao et al. (201)) there are some advanced training techniques available, but they are not used in our work. Although is still better to have higher accuracy for the compressed network, it is not necessary especially for the purpose of this work, as we will show how our design can recover the accuracy drop caused by extreme network compression.
used the same model in Zhao et al. (0117), which was trained
Bengio (2016). This model showed approximately
For ternary neural network, which was used in our ImageNet experiments, we trained the model by using the framework explained in Yao Chen & Chen (2019). Our trained model, however, could not reach the exact accuracy reported in Yao Chen & Chen (2019), and this is due to additional fine- tuning and data augmentations that they performed. Our trained model also showed approximately 5%-8% accuracy drop compared to the original network model, which seems valid for the purpose of this work.
3.2 ACCELERATOR DEVELOPMENT
Xilinx's Vivado high-level-syee sevel--ynthesis tool was used to generate IPs for both original and compress networks. Their tools allow developers to apply various optimizations, such as loop pipelining and arra partitioning, more easily on their FPGAs. We targeted Ultra96 and ZCU102 FPGAs, which are both Arm-based Xilinx Zynq UltraScale+ MPSoC development boards. ZCU102 has more overall resources than Ultra96, and is used for MobileNetV2 experiment only.
Figure [ shows the overall architecture of accelerators. For convolutional and fully connected layer computations, the main technique we used was to have multiple pipelined computation engines that compute partial multiply-accumulae of weights and features, and then compute the sum of the products using an adder tree. These computation engines pipelined that they can produce MAC of 16 weights and 16 features
(MAC) operations. It will perform element-wise multipiccation
4

---

Under review as a conference paper at ICLR 2021
Weight Buffer
Input Bulfer
C Compute
Compute Engine
DRAM
Engine
Output Buffer
Figure Basic accelerator architecture
every single cycle. For 16-bit network, 16 of these computation engines are instantiated to serve 256 elements parallel, by using 256 DSPs, and these compute engines are shared between layers for maximum resource utilization. As further optimization for 8-bit network, we can double the efficiency of using DSPs using the method proposed by Xilinx
For binary and ternary networks, computation engines do not use any DSPs. Instead, multiplexers (MUX) are used determine the sign of the weights. For binary networks, wh is used 1-bit selector to determine the output between - A and +A. For ternary networks, selector to determine the output between A, 0, and +A. Then, the sum of those outputs will be computed using the adder tree, same before. At the end of all computations, will multiply Wscale values from Equation
is used 2-bit
3.3 SOFTWARE DEVELOPMENT
Original Network (High Accuracy)
Input Source
Final Output
Queue
Not Confident
Compressed Network
(Low Latency)
Confident
Figure 2: Graphical representation of hicrarchical architecture
The accelerators are invoked from the software running inside the processing system of the FPGA. Because accelerators both instantiated in single design, they support concurrent execution. Figure shows a graphical representation of the software system, which is designed to fully utilize both networks. First, image from the source be processed will wait for either network be- come idle. Whatever network that becomes idle first will perform the initial inference for that image. Note that for the original network, idle means that its queue is empty as well. If the original network was used for the initial inference, its prediction will always be used as the final prediction, because has higher accuracy. However, if the compressed network was used for the initial inference, the software will compute the confidence of the compressed network prediction to determine if the image needs additional inference on the original network. Confidence is defined as the difference between two largest output values. If confidence is above the threshold, the prediction of the com- pressed network will be used as a final prediction. If confidence below the threshold, however, the input image will go into the queue for the original network inference. Meanwhile, when the image from the source starts initial inference, the next image becomes ready immediately to wait for either network to become idle. This is ensure that both accelerators running for the entire time until

---

Under review as a conference paper at ICLR 2021
Table 1: Resource utilization ratio of accelerators
Dataset
Network
Precision
DSP
LUT
ResNet-18
16-bit
256
20110
CIFAR-10
ConvNet
Binary
25074
16-bit + Binary
260
63727
ResNet-18
16-bit
274
24114
ResNet-18
8-bit
274
30970
ResNet-18
Ternary
25416
16-bit + Ternary
282
56922
ImageNet
8-bit Ternary
282
63710
MobileNetV2
32-bit
536
60424
MobileNetV2
Ternary
64
27507
32-bit Ternary
600
119851
Table 2: Accuracy and latency comparison for all configurations
Dataset
Network
Precision
T
Accuracy (%)
R* (%)
Latency (ms)
ResNet-18
16-bit
94.1
391
CIFAR-10
ConvNet
Binary
89.6 (V4.5)
64 (6.1x)
16-bit Binary 1.5
92.8 (V1.3)
71.1
77.5 (5.0x)
ResNet-18
16-bit
69.5
306
ResNet-18
8-bit
67.9 (V1.6)
255 (1.20x)
ResNet-18
Ternary
63.6 (V5.9)
244 (1.25x)
16-bit + Ternrr 0.7
69.2 (V0.3)
94.9
160 (1.91x)
ImageNet
8-bit + Ternary
1.0
67.1 (V0.8)
81.4
153 (1.66x)
MobileNetV2 MobileNetV2
32-bit
69.8
231
Ternary
62.2 (v7.6)
288 (0.80x)
32-bit + Ternary 0.3
68.5 (V1.3)
82.9
140 (1.65x)
T*; Threshold, R#: Accuracy Recovery (1
Accuracy
Accuracg
all the images are processed, and is the most important difference between this work and Mocerino & Calimera (2014). In contrast the CPU implementation Mocerino & Calimera (2014), where the worst case latency is combined latency of original and compressed networks, our FPGA parallel implementation allows both networks run in parallel for the entire time until the input source is depleted, so the amortized worst case latency is just the latency of the original network.
Threshold value is determined from experiment. Threshold value of network predictions will be considered confident, and none of the inputs will go into the queue. This results both networks running parallel independently, as original network will also get the input from source. High threshold value means that more images go into the original network, thus it results in high accuracy and high latency. Note that when the threshold value goes above a certain point, the queue will contain some images even after all images from the source are depleted. From that moment, only the original network will be running, and this reduced parallelism impacts the latency significantly. Therefore, it is recommended to choose threshold value that will keep the queue small. Threshold value of infinity, in fact, is the same as just running the original network alone, because all compressed network output will be considered not confident and require original network inference. We test a variety of threshold values to see which one gives a most balanced result between accuracy and latency, and will use obtain the final result.
all the compressed

---

Under review as a conference paper at ICLR 2021
(a) CIFAR-10 16 bit Binary
[6] ImageNet ResNet-18 16-bit Ternary
84-
Average Labency per Image: (ms)
82-
Sverage Lateacy Image
80-
1.0
1.5
170
74-
1.0
72
150
70-
91.5
92.0
92.5
92.0
66.0
66.5
67.0
57.5
68.0
68.5
09.0
09.5
70.0
Accunatty (%)
(c) ImageNet ResNet-18 Ternary
(d) ImageNet Mobile 32 bit Ternary
180
Average Latency per Image (ims)
Average Latency per Image (ms))
170
177
1.0
.1.0
150
1.0
.03
+0.
130
05.5
66.0
66.5
87.0
60.0
Acourady 196)
Accurasy (%)
Figure 3: Accuracy and latency plots for different experiment configurations
EXPERIMENT
4.1 CIFAR-10
We first tested our design on CIFAR-10 dataset. The experiment was performed with 16-bit ResNNtt-1--baee network created by us, and binary ConvNet-based network created by Zhao al. (2017), on Ultra96 development board, with frequency of 100MHz. Table T shows the source utilization ratio for individual accelerators. As expected, 16-bit network mainly uses DSPs 256 DSPs were used for 16 x 16 computation engines. Binary network, on the other hand, only uses 4 DSPs, which are used for Wscale multiplication. It uses more LUTs than 16-bit network LUTSs. Table [ also shows the resource utilization ratio
since it mainly performs computation for the entire design. Note that the entire design uses more LUTs than the sum of two accelerators. This is because extra LUTs are used for interconnects and memory interfaces. Although LUT usage for individual models may seem small, we are actually utilizing more than 90% of LUT resources available for the TWinDNN solution.
Figure (a) shows the accuracy and latency plot for different threshold values. We chose 0.5, 1, 1.5, and our threshold values. Higher threshold value resulted in higher latency and accuracy. A threshold value of 1.5 gives the most balanced result between latency and accuracy, thus we used for the final experimental result. Table 2 shows the comparison between configurations, with 16- bit network baseline. A combination of 16-bit and binary network was able to recover more than 71% of accuracy loss caused by binarizing the network. Average latency is 5x faster than the baseline network, which proves
using the binary network effectively.
Note that the binary network results in much higher latency than the result reported in Zhao et al. 2017). This is because first, used much lower frequency than Zhao al. (2017). and second, we converted their accelerator built with Vivado SDSoC platform into Vivado HLS accelerator with customized software part that supports parallel inference of two accelerators.

---

Under review as a conference paper at ICLR 2021
4.2 IMAGENET
Next, we tested our design on much bigger dataset, ImageNet. This time the experiment was performed with two different networks on different FPGA configurations: ResNet-18 on Ultra96 with 150MHz frequency and MobileNetV2 on ZCU102 with 200MHz frequency.
We use 16-bit, 8-bit, and ternary versions of ResNet-1 18, and 32-bit and temarr versions of Mo- bileNetV2. Ternrry networks were used as compressed networks, and other networks were used as original networks. Table T shows that similar to our CIFAR- 10 experiment, ternary network uses much less number of DSPs compared to other fixed-point networks. Also, for ResNet-18, 8-bit net- work uses more LUTSs than 16-bit network, and this is because 8-bit network uses additional logic for bit shifting and introduces additional parallelism from using 1 DSP for 2 multiplications. Note that ternary networks are parallelized to the extent where they would fit on the FPGAs along with original network accelerators, so they do not have much speedup compared to baseline original net- works. The meaningful point is that
combine these two accelerators in parallel to increase throughput with only a small number of extra DSPs compared to original network accelerators.
Figure (b)-(d) shows the accuracy and latency plots for different threshold values. This time, we tested more variety of threshold values so that as much design space is explored as possible. In these graphs, only the points, or threshold values, with the most balanced and efficient results labeled, for a clear view. For 18 16-bit and ternary network configuration, threshold value 0.7 gives the most balanced result between latency and accuracy. For ResNet- 18 8-bit and ternary network configuration, threshold value of 1.0 gives the most balanced result, and for MobileNetV2 32-bit and ternary configuration, threshold value of 0.3 gives the most balanced result. After a threshold value of about 1.0, accuracy of combined network becomes almost identical to that of original network. This result suggests that our software can identify the majority of inputs that are likely to be incorrect with compressed network by using confidence.
Note that for some high threshold values, final accuracy is higher than the accuracy of original network. We believe that this is just result of noise, where there are few cases ternary network predicts correctly while 16-bit network does not. However, there is a chance that these two networks actually complement each other in terms of accuracy. This means that each network is more accurate on certain types of inputs, and our hierarchical design can exploit this hypothesis by finding the confidence of prediction instead of just prediction. This can be one of the areas where we can study further.
Table shows the final result and comparison with original networks as baselines. Note that 8-bit network does not show the ideal speedup of 2x, expected from DSP usage, and this is because there are other factors, such as memory bandwidth, that also limit the speed of the inference. For ResNet- 18 16-bit and ternary configuration, with threshold value of 0.7, our design was able to recover almost 95% of accuracy loss caused by network compression, with more than 1.91 reduced latency compared to 16-bit network alone. For 8- bit and ternary configuration. with threshold value of 1.0, our design was able to recover more than 81 % of accuracy loss, with 1. 1.66x reduced latency compared to 8-bit network alone. Note that 16-bit and ternary configuration gives better results than 8-bit and ternary configuration overall. This is because our 8-bit model is just low- www-precision version of 16-bit model. If we perform additional retraining for 8-bit model, 8-bit and ternary configuration is expected to have a better result than what is currently reported. Finally, for MobileNet 32-bit bit and ternary configuration, with threshold value of 0.3, our design able recover 82.9% of accuracy loss, with 1.65x reduced latency.
Overall result indicates that our design works well for accuracy recovery given optimized neural network accelerators, with only a small amount of extra inferences. Our definition of confidence is also proved to be a useful metric of verifying neural network output.
5 CONCLUSION
In this paper, we proposed a TwinDNN system with a high-accuracy network and a low-latency net- work using hierarchical inference logic that will infer high-accuracy network when the prediction of low-latency network is not considered confident. This design becomes especially more effective on the FPGAS where DSP resources are limited compared to LUT resources, as compressed network
8

---

Under review as a conference paper at ICLR 2021
latency will solely depend on the number of LUTs. There are several aspects that make this study stand out. The first aspect is its high flexibility. Although in this project we mostly used ResNet- the same dataset. There are already many neural network
can basically put any two networks accelerators that are built for different focuses: accuracy and latency. We only need to find two accelerators that would fit on the target FPGA, and simply apply the same logic for experiment. Use Zhao et al. (2117) shows this idca, although we did not reach their exact performance. Second is better concentration. Accelerator development becomes much more difficult when developers need to care about multiple aspects at the same time. However, this work can potentially allow one group of developers to solely focus on increasing accuracy, and the other group of developers to solely fo- cus on reducing latency. It will ultimately reduce the time and effort it takes to build a high-quality accelerator that achieves both accuracy and latency goals.
There are also several aspects where this work can be enhanced further. The first aspect is specialized training. If we can train specialized network that is trained only to classify between top few predictions of the compressed network, we may be able save resources and improve the confidence of the compressed network. Another specialized training scenario can be to train the compressed network to classify detect easy objects and train the original networks to target difficult objects. This way, the two networks can complement each other better. Second is heterogeneous computing with graphics processing unit (GPU). GPUs are usually much more efficient than FPGAs on floating- point operations, and floating-point precision indeed gives higher accuracy compared to low bit- width fixed-point precision. If we can make GPU run the original network as poaaing--poin, and make FPGA run the compressed network, we may be able to achieve an even more efficient solution for this study,
REFERENCES
Michaela Blott, Thomas B PreuBer, Nicholas J Fraser, Giulio Gambardella, Kenneth O'brien, Yaman Umuroglu, Miriam Leeser, and Kees Vissers. Finn-r: An end-to-end deep-learning framework for fast exploration of quantized neural networks. ACM Transactions on Reconfigurable Technology and Systems (TRETS), 2018.
Matthieu Courbariaux and Yoshua Bengio. Binarynet: Training deep neural networks with weights and activations constrained +1 or -1. Computing Research Repository (CoRR), 2016.
Matthieu Courbariaux, Yoshua Bengio, and Jean lean-Pierre David. Binaryconnect: Training deep neu- ral networks with binary weights during propagations. Neurul Information Processing Systems (NIPS), 2015.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning. International Conference on Learning Representations, ICLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Conference on Computer Vision and Puttern Recognition (CVPR), 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings deep residual networks. Computer Vision ECCV, 2016b.
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embed- ding. arXiv preprint arXiv:1408.5093, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep volutional neural networks. International Conference on Neural Information Processing Systems (NIPS), 2012.
Fengfu Li and Bin Liu. Ternary weight networks. Computing Research Repository (CoRR), 2016.
Luca Mocerino and Andrea Calimera. Coopnet: Cooperative convolutional neural network for low- power meus. Computing Research Repository (CoRR), 2014.
NVIDIA. Tensort. URL wtt.thttt//ccm BtK.:Ittpss//bbeeloper..mm nvi www.iiiii...mmmmeeeeeeee /wt.a///aaaooooot//////////////o

---

Under review as a conference paper at ICLR 2021
Junsong Wang, Qiuwen Lou, Xiaofan Zhang. Chao Zhu, Yonghua Lin, and Deming Chen. Design flow accelerating hybrid extremely low bit-width neural network in embedded FPGA. Interna- tional Conference on Field Programmable Logic and Applications (FPL), 2018.
Xilinx. Deep learning with int8 optimization xilinx devices. URL https: www xilinx.
Com/ https/ppppooococoetttttatatttttt http:////ww.iteeapppreeee//eeeee deep learning int8.
pdf
Cheng Gong Cong Hao Xiaofan Zhang Tao Li Yao Chen, Kai Zhang and Deming Chen. T-dla: An open-source deep learning accelerator for ternarized dnn models on embedded fpga. IEEE Computer Society Annual Symposium on VLSI (ISVLSI), 2019.
Xiaofan Zhang, Junsong Wang, Chao Zhu, Yonghua Lin, Jinjun Xiong, Wen mei Hwu, and Deming dnn hardware accelerators
Chen. Dnnbuilder: an automated tool for building for fpgas. International Conference on Computer-Aided Design (ICCAD), 2018.
Xiaofan Zhang, Haoming Lu, Cong Hao, Jiachen Li, Bowen Cheng, Yuhong Li, Kyle Rupnow, Jin- jun Xiong. Thomas Huang, Honghui Shi, Wen-mei Hwu, and Deming Chen. SkyNet: a hardware- efficient method for object detection and tracking on embedded systems. Conference on Machine Learning and Systems (MLSys). 2020.
Ritchie Zhao, Weinan Song. Wentao Zhang, Tianwei Xing, Jeng- Hau Lin, Mani Srivastava, Rajesh Gupta, and Zhiru Zhang. Accelerating Binarized Convolutional Neural Networks with Software- Programmable FPGAs. International Symposium on Field- Pooldd-rorrmaablee Gate Arrays (FPGA), 2017.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Christopher De Sa, and Zhiru Zhang. Improving neu- ral network quantization without retraining using outlier channel splitting. Computing Research Repository (CoRR), 2019.
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. Computing Research Repository (CoRR), 2016.
10


Reviews:
[{"review_id": "l5oalRCNAlv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY:\nThe paper at hand discusses a compressed network inference scheme, where two networks are trained to solve a give classification task. One network aims at achieving a high accuracy, whereas the other network is a highly compressed network which is able to highly speed-up inference. The compression of the network is performed by quantizing the parameters of the network to two or to three different values. The approach of the authors consists of estimating the uncertainty of the highly-compressed network by thresholding the scores. In the case that one score does not exceed the other scores by the pre-defined threshold, the sample will additionally be feed into the high-accuracy network. Experiments are performed on the CIFAR-10 and the ImageNet dataset using a convolutional neural network, a ResNet and a MobileNetV2 network. The speed-up is evaluated with respect to specialized hardware (FPGAs).\n\nREASONS FOR SCORE:\n1. The novelty of the described approach is very limited. The approach of training two different neural networks, one for high accuracy and one for low latency, was already proposed by Mocerino & Calimera (2014). Also estimating the uncertainty of the prediction by analyzing the score vectors should be considered common-knowledge.\n2. The description of batch normalization in Section 3.1 is wrong. From the description it seems like the weights and biases are normalized, which is not the case for batch-normalization where activations are normalized!\n3. The mathematical writing is flawed at various places, e.g., Equation (1) is not properly integrated in the sentence. The same holds for Equation (2).\n4. Overall, the paper does not explain the specialized hardware used to implement the proposed approach. There are many abbreviations like HLS, IPS, FPGA, DPS and LUT that are not even defined. This makes the paper hard to understand for readers that are not familiar with FPGAs. In my opinion the paper needs a rewrite before publication.\n5. Missing labels for points in Figure 3 b)-d). It is not clear to the reader what the unlabeled grey dots show. Perhaps, the interpretabilty of this plots could be improved by adding color and a colorbar?\n6. Are the numbers in Table 2 test accuracies? If yes, how are the thresholds determined? This should have been done using a validation set.\n\nDECISION:\nOverall, I recommend rejecting this paper because of the issues stated above.\n\nUPDATE AFTER REBUTTAL:\nI would like to thank the authors for their responses. After reading the updated version I still would recommend to reject the paper. The reason is that the paper is written for a very narrow audience and is hard to understand for readers who are not familiar with this area of research. Also I  feel like some of my concerns where not properly addressed in the updated version (e.g., issue 2 and 6). After reading the authors' response, I still think that the novelty of this paper is very limited. I decided to keep my score at 4.\n\nMINOR REMARKS AND TYPOS:\n- The last sentence of introduction: section numbers are roman instead of arabic as in the template and \"section\" should be capitalized.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bHWD2074TUK", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a framework to accelerate DNN inference on small embedding systems using an extremely low bit network and a moderately quantized network jointly. The mechanism of the proposed work is to first compute the difference using top2 prediction scores from the compressed network to determine if the inference is further needed for the original network. The proposed framework is evaluated on both CIFAR10 and ImageNet with different network structures and the empirical results indicate better accuracy and latency over baselines. \n\n\nPros:\n- The idea of how to utilize both compressed and original networks is novel.\n- The framework has high flexibility that the network component can be replaced by any other networks based on user demands. \n- The experiments show some promising results on both accuracy and latency.\n\n\nCons: \n-  The authors state that w_scale value is the same across the weights in equation (1). However, it is not clear to see how it is implemented. \n- In section 2.2, I am confused about the statements “...output of the final layer is …., and this is very similar to the probability…””. Is softmax included? Why not use the outputs with softmax?\n- Many terms are not well-explained in the paper, such as FPGA, IPs (page 4), LUT(table 1). I recommend the authors to give some explanations on these terms for the audience without experience in this domain. \n- Figure 2 seems to be confusing. Why are there two inputs to the original network if the input of the original network is determined by the compressed network? \n- In table 2, “accuracy recovery” is not clearly defined.\n- Caption for Figure 3 is missing, e.g., configurations. In addition, I recommend making the makers bigger.\n- Overall, the presentation of the paper needs to be improved and further clarifications are needed for some parts.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VuuQ6ORu2W7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper claims that this is the first work to consider \"using a high compressed DNN along with the original DNN in parallel\". The idea is to use the compressed DNN when possible and to fall back on the original DNN when the confidence in the prediction is low. \n\nThis is an interesting idea and area, as the authors outline there is considerable scope to improve throughput/latency (or save power) with such an approach. The authors report they can reduce latency by nearly 2x at a low cost. Experiments are performed using an FPGA. The compressed networks are created using binary or ternary neural networks.  \n\nUnfortunately, I believe this is an idea that has been explored by numerous previous works (and in some cases evaluated using FPGAs) e.g.: CascadeCNN ( https://arxiv.org/abs/1807.05053 )\n\nOf course, cascade architectures, e.g. for keyword spotting are also common. The idea of anytime predictors and early exit may also be relevant here and even schemes such as dynamic channel pruning (https://arxiv.org/abs/1810.05331). Interesting work has also been undertaken to provide improved confidence metrics for such approaches.\n\nSome related work:\nBig/little DNN: https://ieeexplore.ieee.org/document/7331375\nhttps://arxiv.org/abs/1710.03368\nBranchyNet: https://arxiv.org/abs/1709.01686\nhttps://arxiv.org/abs/1708.06832\netc.\n\nIt is unclear what specific new contribution the paper makes over previous work?\n\nWould there be an argument to compare to a pruned network? \n\nIt would be useful to indicate the size of each network too (rather than just DSP/LUT counts), again to help comparisons to previous work. \n\nUPDATE AFTER REBUTTAL: Many thanks to the authors for their comments. I still believe novelty is limited and will leave my score unchanged. \n\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GSg6nLJ8WNn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## writing\n-pros\n\t- The language is concise and easy to understand, and can clearly convey their own academic views\n\t- The overall quality of the article is nice, and the structure of the paper is very complete. The engineering ability reflected in this article is also relatively strong. \n- cons \n\t- Some of the points that may be questioned are not explained. e.g. Why on the latencies of ResNet and mobile-net, the model with the highest compression rate is slower than the one proposed in this article, when structure is chose as ResNet-18.\n## originality\n- pros \n\t- Innovation can be guaranteed\n-cons \n\t- The validity of confidence as the output of network has been concerned in the previous research or application.\n## significance\n- pros\n\t\n\t- This paper presents a method to balance the precision and latency of Deep Neural Networks. The core idea of this method is to take the output confidence of the compressed network as the criterion. If the confidence levels of different classes differ greatly (i.e. exceeding the threshold determined by experience), it means that the label with the highest probability dominates other tags, so the prediction of compressed network can be considered as reliable. Otherwise they are likely to be unreliable. In this case, the input requires additional validation from the original network. For the model that can be input into FPGA, this method proposes an idea to accelerate the parallel operation of highly compressed network and original uncompressed network. This method enables developers and researchers to determine the balance point between speed and precision by adjusting the trust degree of the output confidence of compressed network (by setting threshold). It used to be a discrete choice, but now it's continuously adjustable. In addition, such a network also makes the quantization compression technology more simple and of more application value.\n- cons\n - The core idea of compressed network, on the one hand, is to obtain higher real-time performance, on the other hand, because the volume of large-scale network is too large. Therefore, this method is equivalent to sacrificing one party to the other. And we cannot use this method with a network unless the uncompressed version of the network is able to be stored in the FPGA, which greatly limits the application.\n-The theoretical explanation of confidence is not enough, and there are no corresponding more controlled experiments, so the persuasiveness is not strong enough. Because this article should be based on experiments, but the number of experiments is still relatively small, which may need to be further supplemented, especially more about the comparison of low-precision quantization networks such as 2bit and 3bit with the methods mentioned in the article. The exchange of precision and speed is equivalent to solving the transition problem before the quantization network and the actual network. But is it possible to get the same result by directly quantifying it with a slightly higher number of digits, which is better in storage cumsuming?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All four knowledgeable referees have indicated reject mainly because the novelty is limited - they thought (and I also agreed) that  it would be difficult to argue the novelty of the proposed framework simply by considering the more recent compressed network training technique, as the reviewer mentioned through rebuttal. In addition, there were concerns about various terms and basics specialized for hardware that are not kindly explained for more diverse audiences in the machine learning field. It improved a little through revision, but I think it needs a more kind explanation. It seems that more thorough experimental verifications  are needed.
Decision: Reject