Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors propose a framework towards 4-bit auantization of CNNs. Specifically, during training, the proposed method contains a full precision branch supervised by classification loss for accurate prediction and representation learning, as well as a parameterized quantization branch to approximate the full precision branch. A quantization loss between the full precision branch and the quantization branch is defined to minimize the difference between activation distributions. The authors proposed a series of improvements, including alternative optimization, dynamic scheduling, detach and batch normalization to help boosting the performance to SOTA under 4-bit quantization.

Strengths:
+ Well-written paper with good clarity and technical correctness.
+ Proposed method seems light, sweet and technically correct.
+ Good experimental performance and result on ImageNet.
+ Good and clear ablation study.

Weaknesses:
- Major performance improvement comes from the combination of different incremental improvements.
- Lack of evaluations with variety of datasets (CIFAR-10/MNIST)/configurations (other bitwidth)
- Lack of the citation and comparison to many most recent works on binarized networks (except XNOR-Net)

Comments:
I consider this a well-written paper with great clarity and good empirical performance. I enjoyed reading the paper. The proposed framework seems technically correct and effective. 

However, a major weakness of this work is that most of the performance improvement comes from a combination of add-on improvements, except that the authors put them together into a unified framework and explained elegantly. The vanilla architecture, which is a main contribution and described in Fig. 1, doesn't seem to give that significant improvement. To some extent, the real technical contributions of this work are partly weakened given the add-on combinations and the existence of similar methods. For example, the alternative optimization of W and \theta is similar to alternative re-training in network pruning, although a unified loss/optimization framework is applicable in this case. Others such as dynamic scheduling and gradient detach are also heuristic-driven.

The results on ImageNet under 4-bit quantization are strong and convincing, but the paper could benefit from conducting additional experiments on different datasets and bitwidth configurations. A more comprehensive study similar to Louizos et al., 2019 will be great. Citations and comparisons to more recent binarized networks other than XNOR-Net will be appreciated too.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper propose a new quantization-friendly network training algorithm called GQ (or DQ) net. I addresses the existing issues in the common paradigm, where a floating-point network is trained first, followed by a second-phase training step for the quantized version. It is a well-written paper. Concepts were clearly explained and easy to follow. Below I present my comments about some details in the paper that were not entirely clear for me. 

- The two loss terms conflict each other. If the training algorithm focuses too much on the first term, it will make the network less friendly to the quantization process. On the other hand, the second one is going to enforce too much emphasis on the accuracy from the quantized network. It is natural to involve some hyperparameter search to find the balance between the two blending parameters. The paper suggests a strategy as to how to handle this issue, but it is not comprehensive, and rather controversial. I think the paper will benefit from a more in-depth discussion and analysis on this regularization issue. 

- The schedule for the loss term blending parameters looks drastic to me. It’s more like “train the floating point net first, and then train the quantized one, and then revisit the floating point one, and so on.” I know I simplified, because the floating point network never stops getting updated as it’s \omega_f is always 1. However, it seems to me that this drastic scheduling strategy sounds like very similar to the traditional approach that trains the floating point network first and then finetune the quantized one, except for the fact that this proposed algorithm repeats this process a few times. Hence, I think the authors’ argument about the supremacy of the proposed method to the two-step finetuning approach is not clearly supported. 

- The exponentially decaying learning rate scheduling looks like the one from ResNet. I’m wondering if it should be the best, especially with the drastic introduction and omission of the second loss. 

- In the ablation studies, it seems that some of the suggested training options are conflicting each other and the clear winner seems to be the multi-domain BN. I cannot conclude anything from this analysis as to which one is more important than the other one, except for the Alt{W,\theta} case. 

Some minor things:

- What’s the name of the proposed network? Is it GQ or DQ?
 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This work introduces GQ-Net, a novel technique that trains quantization friendly networks that facilitate for 4 bit weights and activations. This is achieved by introducing a loss function that consists of a linear combination of two components: one that aims to minimize the error of the network on the training labels of the dataset and one that aims to minimize the discrepancy of the model output with respect to the output of the model when the weights and activations are quantized. The authors argue that this has the effect of “guiding” the optimization procedure in finding networks that can be quantized without loss of performance. For the discrepancy metric the authors use the KL divergence from the predictive distribution of the floating point model to the one of the quantized model. The authors then propose several extra techniques that boost the performance of their method: 1. scheduling the weighting coefficients of the two loss terms (something which reminisces iterative pruning methods), 2. stopping the gradient of the floating point model w.r.t. the second loss term, 3. learning the parameters of the uniform quantizer, 4. alternating optimization between the weights and the parameters of the quantizers and 5. using separate batch normalization statistics for the floating point and quantized models. The authors then evaluate their method on Imagenet classification using ResNet-18 and Mobilenet v1 / v2, while also performing an ablation study about the extra tricks that they propose.

This work is well written and in general conveys the main idea in an effective manner. Quantization friendly neural networks in an important subject in order to make deep learning tractable for real world applications. The idea seems on a high level to be interesting and simple; train floating point models that can fit the data well while also encouraging them to be robust to quantization by enforcing the predictive distributions of the fixed and floating point models to be similar in the KL-divergence sense. Nevertheless, I do have some comments that would hopefully help in improving this work:

- It does seem that GQ-Nets need extra tricks in order to perform well, and those tricks come with their own set of hyperparameters that need to be tuned. For example, at section 4.3 you mention that the top-1 accuracy of vanilla GQ-Nets is 60.95, which is lower than the RelaxedQuant baseline (that has 61.52). This raises the question whether the boost in performance is due to the several additional steps employed (which in general can be applied to other quantization techniques as well), and not due to the main idea itself. 
- Do you employ the straight-through estimator (STE) for the weights in the L_q objective? In the second paragraph of the second page you argue that due to the biased gradients of STE the performance is in general reduced, so I was wondering whether STE posed an issue there or whether you used an alternative estimator. 
- How is batch normalization handled? Do you absorb the scale and shifts in the weights / biases before you perform quantization or do you quantize the weights and then apply the BN scale and shift in full precision?
- How do you ensure and ub > lb when you learn the quantizer? In general learning the quantizer can be also done with alternative techniques (e.g. simply learning the scale and offset) so I was wondering whether you noticed benefits from using the ub, lb parametrization compared to others.
- Do you show the pre-quantization distributions at Figure 2b? In the caption you mention quantized but the resolution seems to be higher than the 16 values you should get with 4 bits. Furthermore, it should be noted that the discrepancy in BN in quantized models was, as far as I am aware, firstly noticed at [1] (and subsequently at RelaxedQuant) and both of these methods simply re-estimated the moving averages during the inference time.

Overall, I am on the fence about this work and tend to reject. Having said that, I am of course willing to revise my score after the discussions with the authors / other reviewers.

[1] Probabilistic Binary Neural Networks, Jorn W.T. Peters, Max Welling
review_title: Official Blind Review #2

Meta-review: The paper propose a new quantization-friendly network training algorithm called GQ (or DQ) net. The paper is well-written, and the proposed idea is interesting. Empirical results are also good. However, the major performance improvement comes from the combination of different incremental improvements. Some of these additional steps do seem orthogonal to the proposed idea. Also, it is not clear how robust the method is to the various hyperparameters / schedules. For example, it seems that some of the suggested training options are conflicting each other. More in-depth discussions and analysis on the setting of the regularization parameter and schedule for the loss term blending parameters will be useful.
Decision: Reject