Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper investigated the Pareto efficiency of quantized convolutional neural networks (CNNs). 
The authors pointed out that  when the number of channels of a network are changed in an iso-model size scenario, the lower precision value can Pareto dominate the higher ones for CNNs with standard convolutions. Furthermore, the authors showed that depth-wise convolutions are quantization-unfriendly when going low- precision even in quantization-aware training. We further provide theoretical insights for it and show that the number of input channels per output filter affects the Pareto optimal precision value. The authors then proposed DualPrecision, a simple yet effective heuristic for precision selection of a given network. When applied on ImageNet, DualPrecision outperforms the 8 bit fixed point baseline and prior art in mixed-precision by a significant margin.

1. The wording issues 
- The paper used words like "iso-model" and "Pareto" at many parts of the draft, making it quite hard to read..  

2. The experiments seem fair. However, the improvement seems quite marginal
The proposed method improved the baseline by 0.9% and 2.2% in top-1 accuracy on ImageNet for ResNet50 and MobileNetV2, respectively.

I have to admit that I am unfamiliar with this particular topic and only knows a few classic papers, like for example Xnor-net: Imagenet classification using binary convolutional neural networks
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The author studies the quantization strategy of CNNs in terms of Pareto Efficiency. Through a series of experiments with three standard CNN models (ResNet, VGG11, MobileNetV2), the authors demonstrated that lower precision value can be better than high precision values in term of Pareto efficiency under the iso-model size scenario. They also study cases with and without depth-wise convolution, and propose a new quantization method, DualPrecision. DualPrecision empirically outperformed 8-bit quantization and flexible quantization methods on ImageNet.

Comment:
I am not at all familiar with quantization methods, therefore no knowledge of relevant related works. If, however, the authors did a thorough job of surveying related works and chose sensible baselines, I think the experiments demonstrate the usefulness of the new DualPrecision technique.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: 
This paper studies the accuracy vs model-size trade-off of quantized CNNs under different channel width multipliers. The authors demonstrated that while all-to-all convolution works well under low bit settings, depthwise conv needs a different sweet spot. The authors then proceed to use the insight to design quantized cnns that have two different schemes for depthwise and normal conv.


Strength

The paper is well written and motivated. By adding network width to the search space,  using the simple heuristics, the authors provide a better results than previously DRL based search method.

Weakness

One of my main concerns is the direct usage of total number of bits as an equivalent measurement between models. While it is useful to measure the storage cost for weights. The choices of bit width will likely affect the computing cost in a non-trivial way, depending on the target hardware platform. It is unclear whether equal model size would mean equal inference latency in practice (most likely they would not be). Providing empirical implementations of these models will shed light into this question.

These weakness makes it a borderline paper.

Question:

How do you handle batchnorm layer, do you use floating points?

How many bits did you use for accumulating the results?

Update after rebuttal:

I have read the authors' response. I would like to keep my current review as it is. Also note that the authors uses floating pt for batchnorm and 32 bit for accumulation, such additional cost might out-weights the benefit of choosing ultra low bits in the low bits regime, making the study less relevant for practice reasons



review_title: Official Blind Review #2

Meta-review: This paper studies the trade-off between the model size and quantization levels in quantized CNNs by varying different channel width multipliers. The paper is well  motivated and draws interesting observations but can be improved in terms of evaluation. It is a borderline case and rejection is made due to the high competition. 
Decision: Reject