Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Strengths:
- Unlike most previous approaches that suffer from significant accuracy drops for good feature map compression, the proposed method achieves reductions in feature map sizes of 1 order of magnitude at effectively no loss in accuracy.
- Technical approach relates closely to some of the prior approaches (e.g., Iandola et al. 2016) but can be viewed as learning the quantization rather than relying on a predefined one.
- Good results on both large-scale classification and object detection.
- Technical approach is clearly presented.

Weaknesses:
- The primary downside is that the approach requires a specialized architecture to work well (all experiments are done with SqueezeNets). Thus, the approach is less general than prior work, which can be applied to arbitrary architectures.
- From the experiments it is not fully clear what is the performance loss due to having to use the SqueezeNet architecture rather than state-of-the-art models. For example, for the image categorization experiment, the comparative baselines are for AlexNet and NIN, which are outdated and do not represent the state-of-the-art in this field. The object detection experiments are based on a variant of Faster R-CNN where the VGG16 feature extractor is replaced with a SqueezeNet model. However, the drop in accuracy caused by this modification is not discussed in the paper and, in any case, there are now much better models for object detection than Faster R-CNN.
- In my view the strengths of the approach would be more convincingly conveyed visually with a plot reporting accuracy versus memory usage, rather than by the many numerical tables in the paper.


review_title: high compression rate for marginal accuracy loss, but approach requires specialized architecture

final_score: rating: 7: Good paper, accept
review_content: The method of this paper minimizes the memory usage of the activation maps of a CNN. It starts from a representation where activations are compressed with a uniform scalar quantizer and fused to reduce intermediate memory usage. This looses some accuracy, so the contribution of the paper is to add a pair of convolution layers in the binary domain (GF(2)) that are trained to restore the lost precision. 

Overall, this paper seems to be a nice addition to the body of works on network compression. 

+ : interesting approach and effective results. 

+ : well related to the state of the art and good comparison with other works. 

- : somewhat incremental. Most of the claimed 100x compression is due to previous work.

- : impact on runtime is not reported. Since there is a caffe implementation it would be interesting to have an additional column with the comparative execution speeds, even if only on CPU. I would expect the FP32 timings to be hard to beat, despite the claims that it uses only binary operations.

- : the paper is sometimes difficult to understand (see below)

detailed comments: 

Equations (3)-(4) are difficult to understand. If I understand correctly, b just decomposes a \hat{x} in {0..2^B-1} into its B bits \tilda{x} \in {0,1}^B, which can be then considered as an additional dimension in the activation map where \hat{x} comes from. 

It is not stated clearly whether P^l and R^l have binary weights. My understanding is that P^l has but R^l not.

4.1 --> a discussion of the large mini-batch size (1024) could be useful. My understanding is that large mini-batches are required to use averaged gradients and get smooth updates. 

end of 4.1 --> unclear what "equivalent bits" means


review_title: initial review

final_score: rating: 5: Marginally below acceptance threshold
review_content: In order to compress DNN intermediate feature maps the authors covert fixed-point activations into vectors over the smallest finite field, the Galois field of two elements (GF(2)) and use nonlinear dimentionality reduction layers.

The paper reads well and the methods and experiments are generally described in sufficient detail.

My main concern with this paper and approach is the performance achieved. According to Table 1 and Table 2 there is a small accuracy benefit from using the proposed approach over the "quantized" SqueezeNet baseline. If I am weighing in the need to alter the network for the proposed approach in comparison with the "quantized" setting then, from practical point of view, I would prefer the later "quantized" approach.

review_title: DNN Feature Map Compression using Learned Representation

Meta-review: The paper presents a technique for feature map compression at inference time. As noted by reviewers, the main concern is that the method is applied to one NN architecture (SqueezeNet), which severely limits its impact and applicability to better performing state-of-the-art models.
Decision: Reject