Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

This paper proposes a framework to accelerate DNN inference on small embedding systems using an extremely low bit network and a moderately quantized network jointly. The mechanism of the proposed work is to first compute the difference using top2 prediction scores from the compressed network to determine if the inference is further needed for the original network. The proposed framework is evaluated on both CIFAR10 and ImageNet with different network structures and the empirical results indicate better accuracy and latency over baselines. 


Pros:
- The idea of how to utilize both compressed and original networks is novel.
- The framework has high flexibility that the network component can be replaced by any other networks based on user demands. 
- The experiments show some promising results on both accuracy and latency.


Cons: 
-  The authors state that w_scale value is the same across the weights in equation (1). However, it is not clear to see how it is implemented. 
- In section 2.2, I am confused about the statements “...output of the final layer is …., and this is very similar to the probability…””. Is softmax included? Why not use the outputs with softmax?
- Many terms are not well-explained in the paper, such as FPGA, IPs (page 4), LUT(table 1). I recommend the authors to give some explanations on these terms for the audience without experience in this domain. 
- Figure 2 seems to be confusing. Why are there two inputs to the original network if the input of the original network is determined by the compressed network? 
- In table 2, “accuracy recovery” is not clearly defined.
- Caption for Figure 3 is missing, e.g., configurations. In addition, I recommend making the makers bigger.
- Overall, the presentation of the paper needs to be improved and further clarifications are needed for some parts.

review_title: Good work but improvements are needed

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## writing
-pros
	- The language is concise and easy to understand, and can clearly convey their own academic views
	- The overall quality of the article is nice, and the structure of the paper is very complete. The engineering ability reflected in this article is also relatively strong. 
- cons 
	- Some of the points that may be questioned are not explained. e.g. Why on the latencies of ResNet and mobile-net, the model with the highest compression rate is slower than the one proposed in this article, when structure is chose as ResNet-18.
## originality
- pros 
	- Innovation can be guaranteed
-cons 
	- The validity of confidence as the output of network has been concerned in the previous research or application.
## significance
- pros
	
	- This paper presents a method to balance the precision and latency of Deep Neural Networks. The core idea of this method is to take the output confidence of the compressed network as the criterion. If the confidence levels of different classes differ greatly (i.e. exceeding the threshold determined by experience), it means that the label with the highest probability dominates other tags, so the prediction of compressed network can be considered as reliable. Otherwise they are likely to be unreliable. In this case, the input requires additional validation from the original network. For the model that can be input into FPGA, this method proposes an idea to accelerate the parallel operation of highly compressed network and original uncompressed network. This method enables developers and researchers to determine the balance point between speed and precision by adjusting the trust degree of the output confidence of compressed network (by setting threshold). It used to be a discrete choice, but now it's continuously adjustable. In addition, such a network also makes the quantization compression technology more simple and of more application value.
- cons
 - The core idea of compressed network, on the one hand, is to obtain higher real-time performance, on the other hand, because the volume of large-scale network is too large. Therefore, this method is equivalent to sacrificing one party to the other. And we cannot use this method with a network unless the uncompressed version of the network is able to be stored in the FPGA, which greatly limits the application.
-The theoretical explanation of confidence is not enough, and there are no corresponding more controlled experiments, so the persuasiveness is not strong enough. Because this article should be based on experiments, but the number of experiments is still relatively small, which may need to be further supplemented, especially more about the comparison of low-precision quantization networks such as 2bit and 3bit with the methods mentioned in the article. The exchange of precision and speed is equivalent to solving the transition problem before the quantization network and the actual network. But is it possible to get the same result by directly quantifying it with a slightly higher number of digits, which is better in storage cumsuming?

review_title: This paper presents a method to balance the precision and latency of Deep Neural Networks, which is good but but not good enough. 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper claims that this is the first work to consider "using a high compressed DNN along with the original DNN in parallel". The idea is to use the compressed DNN when possible and to fall back on the original DNN when the confidence in the prediction is low. 

This is an interesting idea and area, as the authors outline there is considerable scope to improve throughput/latency (or save power) with such an approach. The authors report they can reduce latency by nearly 2x at a low cost. Experiments are performed using an FPGA. The compressed networks are created using binary or ternary neural networks.  

Unfortunately, I believe this is an idea that has been explored by numerous previous works (and in some cases evaluated using FPGAs) e.g.: CascadeCNN ( https://arxiv.org/abs/1807.05053 )

Of course, cascade architectures, e.g. for keyword spotting are also common. The idea of anytime predictors and early exit may also be relevant here and even schemes such as dynamic channel pruning (https://arxiv.org/abs/1810.05331). Interesting work has also been undertaken to provide improved confidence metrics for such approaches.

Some related work:
Big/little DNN: https://ieeexplore.ieee.org/document/7331375
https://arxiv.org/abs/1710.03368
BranchyNet: https://arxiv.org/abs/1709.01686
https://arxiv.org/abs/1708.06832
etc.

It is unclear what specific new contribution the paper makes over previous work?

Would there be an argument to compare to a pruned network? 

It would be useful to indicate the size of each network too (rather than just DSP/LUT counts), again to help comparisons to previous work. 

UPDATE AFTER REBUTTAL: Many thanks to the authors for their comments. I still believe novelty is limited and will leave my score unchanged. 








review_title: Cascade CNN idea (use simple network when possible, if not confident, use larger more accurate network)

final_score: rating: 4: Ok but not good enough - rejection
review_content: SUMMARY:
The paper at hand discusses a compressed network inference scheme, where two networks are trained to solve a give classification task. One network aims at achieving a high accuracy, whereas the other network is a highly compressed network which is able to highly speed-up inference. The compression of the network is performed by quantizing the parameters of the network to two or to three different values. The approach of the authors consists of estimating the uncertainty of the highly-compressed network by thresholding the scores. In the case that one score does not exceed the other scores by the pre-defined threshold, the sample will additionally be feed into the high-accuracy network. Experiments are performed on the CIFAR-10 and the ImageNet dataset using a convolutional neural network, a ResNet and a MobileNetV2 network. The speed-up is evaluated with respect to specialized hardware (FPGAs).

REASONS FOR SCORE:
1. The novelty of the described approach is very limited. The approach of training two different neural networks, one for high accuracy and one for low latency, was already proposed by Mocerino & Calimera (2014). Also estimating the uncertainty of the prediction by analyzing the score vectors should be considered common-knowledge.
2. The description of batch normalization in Section 3.1 is wrong. From the description it seems like the weights and biases are normalized, which is not the case for batch-normalization where activations are normalized!
3. The mathematical writing is flawed at various places, e.g., Equation (1) is not properly integrated in the sentence. The same holds for Equation (2).
4. Overall, the paper does not explain the specialized hardware used to implement the proposed approach. There are many abbreviations like HLS, IPS, FPGA, DPS and LUT that are not even defined. This makes the paper hard to understand for readers that are not familiar with FPGAs. In my opinion the paper needs a rewrite before publication.
5. Missing labels for points in Figure 3 b)-d). It is not clear to the reader what the unlabeled grey dots show. Perhaps, the interpretabilty of this plots could be improved by adding color and a colorbar?
6. Are the numbers in Table 2 test accuracies? If yes, how are the thresholds determined? This should have been done using a validation set.

DECISION:
Overall, I recommend rejecting this paper because of the issues stated above.

UPDATE AFTER REBUTTAL:
I would like to thank the authors for their responses. After reading the updated version I still would recommend to reject the paper. The reason is that the paper is written for a very narrow audience and is hard to understand for readers who are not familiar with this area of research. Also I  feel like some of my concerns where not properly addressed in the updated version (e.g., issue 2 and 6). After reading the authors' response, I still think that the novelty of this paper is very limited. I decided to keep my score at 4.

MINOR REMARKS AND TYPOS:
- The last sentence of introduction: section numbers are roman instead of arabic as in the template and "section" should be capitalized.

review_title: Cascade CNN implemented on FPGAs 

Meta-review: All four knowledgeable referees have indicated reject mainly because the novelty is limited - they thought (and I also agreed) that  it would be difficult to argue the novelty of the proposed framework simply by considering the more recent compressed network training technique, as the reviewer mentioned through rebuttal. In addition, there were concerns about various terms and basics specialized for hardware that are not kindly explained for more diverse audiences in the machine learning field. It improved a little through revision, but I think it needs a more kind explanation. It seems that more thorough experimental verifications  are needed.
Decision: Reject