Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
This paper proposed a set of methods for post-training quantization of dnns. The methods include AdaQuant (which jointly optimizes quantization steps for weight and activation per output activation of each layer), Integer Programming (which determines bit-precision for all the layers), and the batchnorm tuning. The authors presented promising experimental results on various neural networks to support the proposed methods. 

However, there are serious concerns about these claims as follows:
1) AdaQuant
- It is straightforward to think that the joint optimization of quantization step size for weight and activation would result in better quantization results. But this joint optimization would also increase the search space (at least) quadratically, resulting in significant computational cost. Note that the biggest merit of post-training quantization is its simplicity (cf., QAT incurs full-blown training epochs); thus increased cost for post-training quantization is not desirable. Since AdaQuant is the major claim, the authors should provide more discussion on how they dealt with this increased complexity.

- The authors claim that AdaQuant avoids overfitting, but the reason does not seem to be clear. There is no clear explanation of how AdaQuant increases the generality of the quantized model, and the discussion about the sample size (B) is hard to understand (why there's infinite solution when B << N? how B>= Ck^2/(HW) is derived for the convolution case?)

- Also, it seems that the "per-channel" quantization method is utilized in this work, but the formulation in (2) seems to be for "per-layer" optimization. Why they are different?

- How much time does it take to solve this joint optimization?


2) Integer Programming
- The authors proposed an Integer Programming formulation, but there seem to be missing information: what is the formulation of the penalty function, "deltaL"? The authors described it simply as "Loss", but it is not clear what the exact method it is calculated. In fact, deltaL can be pretty complex functions, which might not be independent terms for each layer; thus the formulation like (3) might not be correct. Note that the impact of quantization in the earlier layers affect the quantization impact in the current layer. Without clear explanation and justification about it, the proposed IP formulation does not make sense.

- The authors mentioned that deltaP should be additive and sum up to the total benefit. How can one guarantee it? 

- Also, it seems that the complexity of the IP optimization increases as the number of layers increases. How much computation time increases if the number of layers are large? 


3) Batch normalization tuning
- Unfortunately, there is a very similar idea proposed by [Sun et al., NeurIPS 19]. Cf. "Sec.3 Trans-Precision Inference in FP8".


Also, there are several suggestions to improve understanding of readers.
- Currently, the ablation study looks very confusing. It is not clear which of the pipeline options (light, advanced?) include what kinds of techniques. Please do specify (maybe in a separate table) the list of techniques covered by different pipeline options. 

- The proposed method is not much evaluated by various neural networks. It would be desirable to expand the coverage of neural nets as much as the prior work did. 

- Currently, the proposed methods only utilized "per-channel" quantization. How much accuracy the proposed methods can maintain if they adopt "per-layer" quantization?

- What is the definition of "compression ratio"? (typically compration RATIO is like 12:1, and compression rate is like 2X, 3X...)

- 



review_title: Confusing claims with lack of details and limited novelty

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose to use many techniques to push the limit of neural quantization, which shows reasonable improvements in some datasets.

+ good performance.
+ clear presentation.
+ easy to read and follow

My main complaint is that this type of combination is better for a technical report rather than a standalone paper. In more detail, the authors claim four proposed components: AdaQuant, Integer programming, Batch-norm tuning, and two pipelines for neural quantization. However, the problem is, are these four components all original or just adapted from existing works? The authors are failed to present the relation of the proposed components with the exitinging ones. For example, in my understanding, the AdaQuant is only an adapted version of AdaRound, but the discussion about it (in Section 3.1) is too superficial. Furthermore, the experiment part is not convincing and I think can not back up the claim. With so many "proposed components", the missing of thorough ablation study is a big problem.
review_title: Overall, the authors propose to use a combination of different twisted techniques for better neural quantization, which lacks meaningful insights.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduces a series of techniques to quantize neural networks, and how to combine them:
* Layer by layer quantization where weights can change as needed (rather than to the nearest quantization error).
* Integer programming to determine the precision required at every layer.
* Tuning batch norm weights by re-computing statistics.

PROS

* Main text is well written (see below for other issues though).
* All components in the proposed method are straightforward.
* Strong results, with very little performance loss after very aggressive quantization (~4 bits).

CONS

* The organization of the paper can and should be easily improved (see below).
* Very basic details, such as the metric used in some tables, are omitted.
* Few experiments/baselines.

---

On the flow of the paper: Fig. 1 is barely introduced. Fig. 2 is introduced much earlier than it is first referred to in the text, in section 5.2, where it is not explained either. The labels in the figure are not defined either at this point (and ip should be IP). Additionally, the colors for the last two lines are too similar.

The experiment of Fig.1  should be introduced with some level of detail (there is none outside the caption). The caption is a bit confusing: what are the "the last layers after"? Why are variance bars always red? They are also hard to see. The labels are not descriptive (light pipeline, advanced pipeline, relaxed advanced pipeline) and hard to find in the text. MN-V2 and B-SQuad1.1 just drop in Table 1 without any context. It can be inferred from earlier sections what this refers to but it should be explained clearly. Same for "min-max" in Fig. 1 and Table 1.

Baselines are barely discussed. Also, I am not very familiar with quantization papers, so I might have missed relevant baselines, but they seem hard to compare. For instance, the authors omit [Nagel et al 2020](https://arxiv.org/pdf/2004.10568.pdf), which seems to do better at similar quantization levels, but I am not sure the results are directly comparable. The authors should discuss this better.

I think section 4 can be titled simply "Quantization flow".

Compression ratio in the plots should indicate %?

---

Typos/grammar:

* "Mobilsnet-V2" -> "Mobilenet-V2
* "we suggest [an] integer-linear programming"
* "AdaRound['s] implicit constraint"
* "MSE distance": shouldn't this just be MSE?
* "Early work by Lin et al. (2016) used [a] convex optimization formulation which results ~~with~~ [in] a simple greedy compression scheme."
* "3. OPTIMIZING [THE] QUANTIZATION PIP[E]LINE"
* "model['s] internal statistic[s]" -> found twice
* "often result with an inferior solution" -> "often result in an inferior solution"
* "Accordingly, researches suggested" -> researchers?
* "where V is a continuous variable V" -> redundant
* "thus enjoy[ing] some of the flexibility"? -> "benefitting from" might be a better phrasing?
* "Quantization-Aware- Training" -> extra space
* "and [is] much less prone to over-fitting"
* "[A] Similar derivation"
* "Even optimizing on [a] single image"
* "MAC operations." -> acronym not previously introduced (unless mistaken)
* Section 4: IP acronym should be introduced in the integer programming section.
* "we investigate [a] mixture"
* "results with high degradation" -> in high degradation

review_title: See below

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work presents a quite comprehensive multi-step scheme for post-training neural quantization that does not rely on large datasets or large computational resources.  

The work is has significance in the domain of post-training neural quantization, especially in cases where only a small calibration set or limited resources are available. It would be interesting to also think about accumulator quantization.


Pros:

The empirical results are relatively strong in this method; 4-bit quantization is a good achievement in the models considered here.

The quantization process covers nicely the various different parts of the errors that  post-training quantization induces and propose somewhat original solutions to them. 

Cons:

The framework is relatively complex and consists of multiple steps.

What is the detailed difference of computational resource use between light and advanced pipeline?

AdaQuant seems like a rather straight-forward step from AdaRound by combining with it some related works.

It is not clear how the BN error compensation differs exactly from related work.

Some details were missing, for example, it is up to the reader to guess how many bits were used in the accumulators.

Some spelling mistakes, e.g., “Optimizing Quantization Pipline”


Overall: An engineering oriented paper with some lack of testable hypotheses and analysis of some parts of the methods, but the 4-bit results justify publication. Edit: I have not seen author reply and further reading of the paper has not clarified the main issues found by all reviewers. I have to lower the score. Edit2: new version of the paper and the author reply cleared some concerns, score raised accordingly.



review_title: Collection of steps to deal with quantization-induced error in post-training quantization

final_score: rating: 7: Good paper, accept
review_content: 

Summary: The paper studies the problem of Post-Training Quantization of NNs, where no fine-tuning is performed to quantize the model. In particular, the authors focus on sub-8 bit quantization and propose a novel integer linear programming formulation to find the optimal bit width for a given model size. Additional approaches are proposed to minimize accuracy degradation after quantization. These include
(i) AdaQuant in which the parameters are quantized layer-by-layer to match the full precision output,
(ii) a batch norm tuning approach to re-adjust the statistics to the quantized model, and
(iii) an advanced pipeline for cases where backpropogation can be performed.

Experiments are performed on ResNet18/50, MobileNetV2, and BERT-SQuaD1.1 showing the feasibility of the proposed method. 

I think the approach in the paper is pretty interesting, and specially the integer linear programming solution.
Overall the paper is strong however, please note the following:


- Page 3 last paragraph: it seems there are errors in the results for calibration data.
1/ it is stated that if "B>> M then we might underfit the data.". Do the authors mean "B>>N"? Similarly it seems the result given for the convolution (i.e. B> cK^2/HW) needs to be revised.
2/ The analysis requires specifying the rank of W and in particular the relationship between M and N. In particular, note that the matrix W can at most have a rank of min(M,N) and as a result the rest of the linear equations for calibrating the data would be redundant. This needs to be taken into account in your result.

- Page 4: "Depending on the needs, our performance metrics P would be either the execution time of the network or its power consumption." This is good but no result on either latency or power consumption is provided in the paper.

- Figure 4 (a,b): Please provide the BOPS for the mixed-precision results. BOPS proposed by (https://arxiv.org/pdf/2005.07093.pdf) is a good metric to measure the total reduction in computations for mixed precision quantization.

- The results section uses weak FP32 baseline. For instance, the baseline accuracy for ResNet50 is 77.2\% and MobileNetV2 is 73\%, while the paper uses 76.1\% and 71.8\%.

- Related to the above, it is stated "on the extensively studies 8bit MobileNet-V2 topology we achieved 71.6% top-1 accuracy". This is a good result but please note that other work in the literature (arxiv:2001.00281) reports 72.91\% for INT8 quantization of MobileNetV2 (this comparison is actually missing from the paper). It is immediately not clear if the lower reported accuracy is due to the weaker FP32 baseline used or if it is an inherent problem with the method (most probably it is the former but it would be to show this).




Minor Comments:

- There were several grammatical/spelling mistakes in the paper. Please proofread the paper thoroughly. Below are some of the errors that I caught:

- 3 OPTIMIZING QUANTIZATION PIPLINE -> PIPELINE

- Page 6:  the our method robustness -> the robustness of our method

- Page 6: manged -> managed

- Page 7: an significant advantage -> a significant advantage

- Page 8: we managed to switched  -> we managed to switch

- Page 8: For instance, on the extensively studies -> For instance, on the extensively studied
review_title: Interesting Approach To Quantization

Meta-review: This paper received mixed reviews, 3 positives (7, 6, 6) and 2 negatives (4, 4). Due to the divergence of the reviews, I carefully read the paper and made my best efforts to understand the paper and the review comments. This paper proposes to learn a quantization network using a small calibration set given a network trained with the full precision. The combination of AdaQuant, integer programming, and batch-norm tuning makes sense although they do not have substantial novelty. The three components are reasonably tightly-coupled and comprise a complete algorithm. However, the sequential-AdaQuant distracts the main claim of this work significantly. This is probably added during the review process but looks ad-hoc to me. Sequential AdaQuant seems to be effective to improve accuracy, but cannot be applied before the bit allocation was set, which makes it require integer programming no more. Because of this issue, the overall presentation becomes confusing and the argument sometimes sounds unfair (please refer to the last posting by R5.). 

In addition, the presentation of this paper could be improved, especially for the details of the integer programming formulation. It is not clear how to define some variables mathematically. The discussion about the size of the calibration set together with the overfitting issue is lacking, and rigorous discussion and analysis would make the paper much stronger. The reviewers are not convinced of the novelty of this paper, and they rather believe that this is an engineering-oriented work. Considering this fact,  the evaluation of this paper is not very comprehensive. The ablation study with respect to the size of the calibration set should be conducted more intensively. The experiment fails to show the benefit of mixed precision quantization effectively and it is limited to presenting the compression ratio in Figure 3. The authors used a small calibration set taken from the training dataset, which looks weird because they claim that the post-training quantization requires only a small "unlabeled" calibration set at the beginning of the abstract; it is more desirable to use arbitrary examples in the same domain.

Despite the interesting aspects, I believe that this paper needs a focus and substantial improvement for publication, and, consequently, recommend rejection.
Decision: Reject