Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents an extension of binary networks, and the main idea is to use different bit rates for different layers so we can further reduce bitrate of the overall net, and achieve better performance (speed / memory). The paper addresses a real problem which is meaningful, and provides interesting insights, but it is more of an extension.

The description of the Heterogeneous Bitwidth Binarization algorithm is interesting and simple, and potentially can be practical, However it also adds more complication to real world implementations, and might not be an elegant enough approach for practical usages. 

Experiments wise, the paper has done solid experiments comparing with existing approaches and showed the gain. Results are promising.

Overall, I am leaning towards a rejection mostly due to limited novelty. 


review_title: Review of Heterogeneous Bitwidth Binarization in Convolutional Neural Networks

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper suggests a method for varying the degree of quantization in a neural network during the forward propagation phase.

Though this is an important direction to investigate, there are several issues:

1. Comparison with previous results is misleading:
a.	1-bit weights and floating point activations: Rastegari et al. got 56.8% accuracy on Alexnet, which is better than this paper 1.4bit result of 55.2%.
b.	Hubara et al. got 51% results on 1-bit weights and 2-bit activations included also quantization first and last layer, in contrast to this paper. Therefore, it is not clear if there is a significant benefit in the proposed method which achieves 51.5% when decreasing the activation precision to 1.4bit. 

Therefore, it is not clear that the proposed methods improve over previous approaches.

2. It is not clear to me: in which dimension of the tensors are we saving the scale factor? If it is per feature map, or neuron, this eliminates the main benefits of quantization: doing efficient binarized operations when doing Weight*activation during the forward pass?

3. The review of the literature is inaccurate. For example, it is not true that Courbariaux et al. (2016) “further improved accuracy on small datasets”: the main novelty there was binarizing the activations (which typically decreased the accuracy). Also, it is not clear if the scale factors introduced by XNOR-Net indeed allowed "a significant improvement over previous work" in ImageNet (e.g., see DoReFA and Hubara et al. who got similar results using binarized weigths and activations on ImageNet without scale factors).  Lastly, the statement “Typical approaches include linearly placing the quantization points” is inaccurate: it was observed that logarithmic quantization works better in various cases. For example, see Miyashita, Lee and Murmann 2016, and Hubara et al.

%%% After Author's Clarification %%%
This paper results seem more positive now, and I have therefore have increased my score, assuming the authors will revise the paper accordingly.


review_title: There are flaws in comparisons with previous works

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper tries to maintain the accuracy of 2bits network, while uses possibly less than 2bits weights.

1.  The paper misses some more recent reference, e.g. [a,b]. The author should also have a discussion on them.

2. Indeed, AlexNet is a good seedbed to test binary methods. However, it is more interesting and important to test on more advanced networks. So, I wish to see a section on testing with Resnet and GoogleNet.

Indeed, the authors have commented: "AlexNet with batch-normalization (AlexNet-BN) is the standard model ... acceptance that improvements made to accuracy transfer well to more modern architectures." So, please show that.

3. The paper wants to find a good trade-off on speed and accuracy. The authors have plotted such trade-off on space v.s. accuracy in Figure 3(b), then how about speed v.s. accuracy?

My concern is that one-bit system is already complicated to implement. Indeed, the authors have discussed their implementation in Section 3.3, so, how their method works in practice? One example is Section 4 in [Courbariaux et al. 2016].

4. Is trade-off between 1 to 2 bits really important? 

Compared with 2bits or ternary network, the proposed method at most achieving (1.4/2) compression ratio and (2/1.4) speedup (based on their Table 1). Is such improvement really important?

Reference:
[a]. Trained Ternary Quantization. ICLR 2017
[b]. Extremely low bit neural network: Squeeze the last bit out with ADMM. arvix 2017
review_title: The improvement is not significant

Meta-review: All of the reviewers find the approach interesting, but they have reservations regarding the practical impact and empirical evaluation. The paper needs improvement both on the motivation and on the experimental results by including more baseline methods and neural architectures.

Decision: Reject