Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this article, the authors propose a compression-aware method to achieve efficient compression of feature maps during the inference procedure. When in the training step, the authors introduce a regularization term to the target loss that optimizes the entropy of the activation values. At inference time, an entropy encoding method is applied to compress the activations before writing them into memory. The experimental results indicate that the proposed method achieves better compression than other methods while holding accuracy.
There are still some issues as follows:
1.	The authors should carefully check the format of the references in the whole article. For example, in section 2, line 5 from the top and line 8 from the bottom, “Xiao et al. (2017), Xing et al. (2019)” and “(Chmiel et al., 2019)” are in the wrong format.
2.	It is suggested that the authors swap the order of formulation (8) and (9) in section 3.2 so that it will be a good correlation with the formulation (3) and (4).
3.	I am interested in learning the time taken by the proposed method during the inference procedure vs other related methods.
4.	The authors studied two differentiable entropy approximation in the paper, and they stated that they calculate soft entropy only on the part of the batch for the reduction of both memory requirements and time complexity in training. I hope the authors will clarify 1) Whether the accuracy will be affected by other differentiable entropy approximations; 2) what is the impact on accuracy if only part of the batch is considered.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The format of the paper does not meet the requirement of ICLR. Due to this, I will give a 3. I suggest the authors to change it as soon as possible.

Besides that, the main idea of the paper is to regularize the training of a neural network to reduce the entropy of its activations. There are extensive experiments in the paper.

The paper introduce two kinds of method to regularize the entropy. The first method is a soft version of the original entropy, and the second is the compressibility loss. After adding the regularization, the performance drop of the compressed network is reduced. The experiment performance is promising.

I think the method is straightforward and reasonable with only a few questions:
1. Why do you quantize the weight? Seems it's not necessary because the paper only address activation quantization.
2. What will happen if the weights are quantized to lower bits? For example, 4bit?
2. How about adding the regularization to weights?

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: Summary:
The authors propose a method for training easy-to-quantize models that are quantized after training (post-training quantization). They do so by regularizing by the entropy, thereby forcing the weight distribution to be more compressible. They further compress the weights using entropy coding.

Strengths of the paper:
- The paper presents strong experimental results on ResNet, SqueezeNet VGG and MobileNet architectures and provides the code, which looks sensible. 

Weaknesses of the paper:
- The authors could have applied CAT to other tasks such as Image Detection, while proving inference times on CPUs. Indeed, it is unclear to me what would be the influence of the entropic decoder which is claimed to be fast for "efficient implementations" by the authors.
- The idea of regularizing by the entropy is not novel (see for instance "Entropic Regularization", Grandvalet et a.l),  as well as the idea of further encoding the weights using entropic coders (as in "Deep Compression", Han et al.).

Justification of rating:
The authors present an intuitive method (yet not novel) for quantizing the weights of a neural network. My main concern would be about the inference time but I consider that the experimental results suggest strong evidence that CAT performs well on a wide variety of architectures. 
review_title: Official Blind Review #1

Meta-review: This work propose a compression-aware training (CAT) method to allows efficient compression of  feature maps during inference. I read the paper myself. The proposed method is quite straightforward and looks incremental compared with existing approaches based on entropy regularization. 


Decision: Reject