Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: Quick Summary: Based on semi-quantitative analysis the paper first proposes two rules for quantization of DNNs, then extends previous methods based on these rules to propose specific technique for quantizing activations and weights. Experimental results are provided on MobileNet V1/VB2 and ResNet50 and compare favourably against baselines (which are old and unfortunately do not represent recent developments in the literature). Overall I found novelty low and also there are several recent papers already published before this submission which provide results which are at least as good or even better

Details
Specifically the rules are: 
1. To prevent logits from entering saturation region of the cross entropy loss, the effective weight in the last fully connected layer should be small.  
2.To keep the gradient of weights in the same scale across the whole network, either BN layers should be used after linear layers such as convolution and fully- connected layers, or the variance of the effective weights should be on the order of the reciprocal of the number of neurons of the linear layer (n_l).
They use DoReFa for weight quantization and PACT for activation quantization based on the above observations/rules. However the contributions were not really very novel in my opinion. See especially the paper "LEARNED STEP SIZE QUANTIZATION" I linked below when explaining novelty

This reviewer feels that the authors were perhaps unaware of several important contributions to the literature this year which are substantially better than the baselines compared against in the paper. I list a few below for the authors to compare against, and to explain their novelty in contributions in the rebuttal phase. Unfortunately this makes me feel the paper should be a clear reject in its current state, unless the authors can convince us otherwise. 

https://arxiv.org/pdf/1902.08153.pdf
https://arxiv.org/pdf/1903.08066.pdf
https://arxiv.org/pdf/1905.11452.pdf
https://arxiv.org/pdf/1905.13082.pdf

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes two rules for efficient training of quantized networks by investigating the scale of the logit values and gradient flow. The authors claim that accuracy degradation of recent quantization methods results from the violation of these two rules.

One of my main concerns is that the analysis of the rules for weight and activation quantization are separated. E.g., the analysis of weight quantization in Section 3.2 is based on eq (1a)-(1d) where no activation quantization is considered. In this case, does the analysis still hold when applying weight and activation are quantized simultaneously?

Moreover, the analysis is only suited for a limited range of quantization methods. In the proposed SAT, the authors propose to multiplies the normalized weight with the square root of the reciprocal of the number of neurons in the linear layer, to make up for the variance difference caused by quantization. However, this increase indeed depends on the initialization of the weights. If the weights are not sampled from "a Gaussian distribution of zero mean and variance proportional to the reciprocal of the number of neurons" as at the end of page 5, then this recipe may not work any longer. Moreover, the proposed SAT seems to be only suited for the specific quantization function for Dorefa-Net in (5), what about many other recent quantization functions that do not need this kind of clamping?

Others:
1. The citation format is wrong.
2. In the abstract, "Recent quantization approaches violates ... and results ..." => "Recent quantization approaches violate ... and result ..."
3. What is the "scaling factor in Eq. (3)" before the subsection "Efficient Training Rule II (ETR II)"?
4. Keep the same number of decimal places in the tables.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper presents network quantization techniques for weight quantization (SAT) and activation quantization (CG-PACT). The authors first formulate efficient training rules (ETR I and ETR II) and propose a normalization scheme for weight quantization based on the rules. Independently, the authors also propose an activation quantization rule by removing the approximated term in PACT. The authors present the effectiveness of their algorithm in MobileNet V1, V2, and PreResNet-50.

Overall, I think the proposed algorithms lack novelty due to the following reasons.

- SAT suggests a normalization scheme under some assumptions (ETR I and ETR 2), however, I am not sure this is a significant contribution compared to DoReFa. In particular, I have some doubt in assumptions which is listed in ‘Other comments’. 

- CG-PACT seems to be a simple variant of PACT. In methodological contribution, I believe that it could only be a marginal improvement with respect to PACT. In terms of empirical contribution, it was hard to see the improvement made by CG-PACT as PACT and PACT+CG exhibit similar performance in Table 4.

Other comments
- In the derivation of ETR 1, some approximations are not clear for me. For example, the authors drop \gamma in (17). In addition, approximation in (16d) depends on the activation function and BN parameters but they are also ignored. I believe that incorporating all these would not introduce much computational overhead.

- In (3) and (4), I think that equality only holds under some assumptions (e.g., an infinite number of parameters, i.i.d. weights, etc.) but they are not clearly mentioned. 

review_title: Official Blind Review #3

Meta-review: The submission proposes methodology for quantizing neural networks.  The reviewers were unanimous in their opinion that the paper is not suitable for publication at ICLR.  Concerns included novelty over previous works, comparatively weak baseline comparisons, and overly restrictive assumptions.
Decision: Reject