Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper discusses several algorithms for quantizing neural networks, compares the resulting accuracy, and identifies several challenges in quantization.  
strengths: The addition of a github repo with example code significantly improves the presented benchmark's usefulness.  I have changed the score accordingly.  

* The discussion of some of the nuances involved in NN quantization is useful.  
* The comparison between the algorithms on multiple popular architectures is thorough and instructive. 
* The authors' experiments showing the effects of different approaches to BatchNorm and graph structure are interesting.
* Overall the paper provides a good comparison of several quantization methods.

weaknesses: The weaknesses mostly pertain to use of the submitted work specifically as a benchmark.  

* A good benchmark should be usable by future algorithms and methods.  If someone creates a new quantization algorithm, how should they use this benchmark?  There should be some clear instructions or code examples, which are absent.  Typically I would expect a git repository with code to execute the benchmark, instructions, and an example.  

* It should be clear what the benchmark is measuring. Is the goal to measure how different quantization-aware training regimes perform when targeting a given inference-mode quantization scheme?  Is it meant to compare the accuracy or speed impacts of different inference-mode quantization schemes?  Something else?  Put another way, any benchmark fixes certain aspects of a problem, leaves other aspects of the problem as free design variables, and measures certain results.  It is not clear from the paper what is fixed, what is free, and what is measured.  

* As a quantization overview, there should be some more discussion of the results.  The accuracy and accuracy degradation (FP accuracy - Quantized Accuracy) varies substantially from one model/setting to another, which the authors point out.  But are there some trends, groupings in the data, or underlying intuition to be taken from the results?  Do the data point to some guidance for an engineer choosing a quantization algorithm?
correctness: This reviewer did not find any major factual errors or methodological problems.  This work's weaknesses as a benchmark are discussed in the weaknesses section.
clarity: The paper is generally clearly written.  A few instances of unclear passages and mechanical issues are addressed here:

* Abstract. Several minor issues.  "By aligning up the training settings" should just be "By aligning the training settings." The phrase "about-the-same performance" should have no hyphens.   Line 15: "there is ... still a long way to go."  A long way to to what exactly? 

* Line 41: "paper does not testify the algorithm " => "papers do not test the algorithm"

* L47: "Another noteless problem" Did you mean "notable?"

* L68: "evaluation is performed in 4 dimensions: supported hardware library, ..."  It becomes clear later on what is meant by HW library (a software library for performing NN tasks on given supported HW), but at this point in the paper it is not clear.  Please clarify the term.

* L96: The phrase "identify the quantizer type with learning-based which learns the scale and rule-based which directly computes" needs several corrections: "identify the quantizer type *as* learning-based, which learns the scale, *or* rule-based, which directly computes ..."

* L110: "As for 2-bit quantization, we find most of the algorithm will crash on hardware settings." Elaborate.  Did the QA training process fail to converge?  Did the training software crash?  With fake quantization, the quantities in the graph are still just FP numbers like in any other NN, so it is unclear why 2b quantization would cause a crash.

* Fig. 2(b) why is the value after ReLU ($x^{l+1}$) dequantized?  It is already an INT32.

* Section 3.2.  The discussion on batch-norm folding is confusing.  Folding is typically applied for inference, and computed after training, but several of the points discuss the the impact on training.  Is the benchmark meant to cover quantized training?

* It is not clear how the discussions of BatchNorm and block graph structure (3.2, 3.3) connect to the larger point of the paper.  These are two specific algorithm choices that are part of any quantization algorithm.  If the paper is intended as an overview of different techniques in NN quantization and their effects on accuracy, performance etc., then it makes sense to include these sections.  If the paper is describing a benchmark, then I would expect the focus to be on the benchmark and the included algorithms as they are published, without dissecting them.  If these block structures are design choices that a user needs to make after choosing target hardware and a quantization algorithm, then please clarify this fact.

* Table 7: At the bottom, footnotes 1,2,3,4 should be separated by commas.

* L326: "However, MQBench also has limitations, beyond image classification, quantization faces more challenges in deployments like object detection and NLP application." This sentence is unclear.


relation to prior work: Prior work is adequately addressed.
additional feedback: 
* It seems like a missed opportunity that none of the targeted HW libraries support MCUs/DSPs.  Since edge inference on MCUs/DSPs is one of the biggest use cases for quantizing NNs, it seems like it would be useful to include something like TFLite Micro.

* L214/Table 4:  Can you discuss the differences in accuracy shown in Table 4?  Were the training configurations used to reproduce the results the same as those discussed in the paper?

* L303.  It appears that for the purpose of calculating difficulty, a setting/algorithm combination that does not converge counts as 0%, so that setting will have a "difficulty" equal to the highest accuracy for that setting.  This method makes the numeric difficulty not very meaningful and also not stable to new entrants.  For example, if another algorithm is entered into the benchmark that performs worse on a setting, the difficulty will increase.  This seems like an undesirable characteristic for a benchmark.
review_title: Good overview of NN quantization, but lacking as a benchmark

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This work attempts to reproduce recent quantization research via a benchmarking tool called MQBench. The authors test a variety of algorithms and assess the real-world deployability of each method. MQBench connects algorithms to hardware to assess the performance and accuracy gained or lost through quantization.
strengths: 1. This work evaluates the reproducibility of several methods in a single benchmark tool. Reproducibility is a significant challenge and direct algorithm comparison in a single tool can help advance the state of the art within the community.
2. The introduction presents strong arguments and outlines the need for a benchmarking tool within the community.
3. The paper is well organized with clear tables and diagrams.
4. Clear results and recommendations are made after the data is presented.
5. Detailed evaluation of how BN folding impacts accuracy vs. performance showed interesting results.
weaknesses: 1. Authors claim the code is open source, but the link provided did not have any code posted. Please include a link to the open-source software or remove the open-source check from Table 1.
2. Very few details were provided on how the framework functions (such as input/output/options/system requirements/etc.)
3. How extensible is the framework to new hardware and/or new deep learning layers?
correctness: Results in the supplementary material provide strong evidence to support the finding presented within the paper. I have no reason to believe claims are not accurate.
clarity: The paper is organized in a clear and straightforward way. The tables and diagrams are well-organized and concise.
relation to prior work: This paper benchmarks several methods from prior work. The goal of this research is to reproduce prior work as well as assess each algorithm's ability to deploy in standard hardware. All prior work is referenced or sourced.
additional feedback: Minor grammar errors throughout such as lines 41, 44, 110.
review_title: MQBench

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: Authors propose a novel benchmark called MQBench to analyze and evaluate quantized by quantization-aware training (QAT) models using consistent setups. Their motivation is to fairly compare existing and future QAT methods with the same hyperparameters. Another goal of this paper is to benchmark QAT models with real-world deployments on CPU, GPU, ASIC, DSP (using corresponding vendor libraries). Authors clearly describe the methodology for benchmarking, batch normalization folding, and quantized convolutional layer implementations. Unfortunately, this paper falls short of fulfilling reproducible research goals due to unfinished code documentation and benchmark web-site.
strengths: 1. Reproducibile research with fair evaluations is important for the community.

2. MQBench methodology is clearly described. Also, paper discusses corner cases during training and inference phases of quantized models.

3. Evaluation of existing algorithms show interesting results that can be valuable for broader research community.
weaknesses: 1. The biggest weakness of this paper is lack of documented code and a web-site with public submission form. Looks like some code has been released, but there is no documentation and step-by-step guide how to reproduce MQBench results. Also, it is not possible for the community to submit their QAT methods for fair comparisons. Therefore, the main goal of this paper is not fulfilled unless authors release clearly described documentation how to run their code and/or submission web-site is provided.

2. Currently, only ImageNet classification has been used in MQBench. It means that MQBench is limited to a particular task and a dataset. Authors recognize this limitation and suggest object detection and NLP applications as a potential future work.

3. I believe, comparison to MLPerf/MLCommons benchmarks is missing in this paper.
correctness: Methodology is correct, but has some limitations: error bars are missing, only a single classification task on a single ImageNet dataset has been evaluated.
clarity: This paper is well written. However, some sentences need minor revisions:
#130: a activation -> an activation
Table 5: asynchronized -> asynchronous

relation to prior work: This paper clearly discusses differences with previous contributions. Though some discussion on other benchmarks such as MLPerf/MLCommons, I believe, is missing.
additional feedback: I suggest authors to release documentation for their code with step-by-step guide on how to reproduce each experiment, and finish their web-site with public submission form. Then, I will increase paper score.

Upd 7/13: I increased my score based on author's feedback. They made a good effort to improve benchmark web-site, documentation and code reproducibility.
review_title: This paper introduces a novel benchmark to evaluate quantized models. Its main goal is to benchmark quantization methods using reproducible setups i.e. with the identical hyperparameters, models, tasks, hardware.

Meta-review: The paper introduces a benchmark for model quantization. Reviewers found the paper to be clear and the benchmark useful, but raised concerns primarily about the reproducibility and extensibility of the benchmark. These limitations were mostly addressed through the authors’ response and revised paper, which also include a link to open-source code which makes the proposed benchmark significantly more useful. Congratulations on having your paper accepted to the NeurIPS 2021 Track on Datasets and Benchmarks! The authors are encouraged to take the feedback from reviewers into account when preparing the final version of their paper.
Decision: Accept