Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a method to quantize weights and activations in neural network during propagations.

The residual binarization idea is interesting. However, the experimental results are not sufficiently convincing that this method is meaningfully improving over previous methods. Specifically:

1) In table 2, the 1-st level method is not performing better the FINN, while at the higher levels we pay with a much higher latency (about x2-x3 in figure 7) to get slightly better accuracy. 

2) Even at the highest level, the proposed method is not performing better than BinaryNet in terms of accuracy. The only gain in this comparison is the number of epochs needed for training. However, this is might be due to the size difference between the models, and not due to the proposed method. 

3) In a comment during the review period, the authors mention that "For Imagenet, we can obtain a top-1 accuracy of 28.4%, 32.6%, and 33.6% for an Alexnet architecture with  1-3 levels of residual binarizations, while the Binarynet baseline achieves a top-1 accuracy of 27.9% with the same architecture." However, this is not accurate, BinaryNet actually achieves 41.8% top-1 accuracy for Imagenet with Alexnet (e.g., see BNN on table 2 in Hubara et al.). 

Minor comment regarding novelty:
The temperature adjustment method sounds somewhat similar to previous method of increasing the slope described "Adjustable Bounded Rectifiers: Towards Deep Binary Representations"
review_title: Experimental results seem weak

final_score: rating: 4: Ok but not good enough - rejection
review_content: 1. The idea of multi-level binarization is not new. The author may have a check at  Section "Multiple binarizations" in [a] and Section 3.1 in [b]. The author should also have a discussion on these works.

2. For the second contribution, the authors claim "Temperature Adjustment" significantly improves the convergence speed. This argument is not well supported by the experiments.

I prefer to see two plots: one for Binarynet and one for the proposed method. In these plot, testing accuracy v.s. the number of epoch (or time) should be shown. The total number of epochs in Table 2 does not tell anything.

3. Confusing in Table 2. In ResBinNet, why 1-, 2- and 3- level have the same size? Should more bits required by using higher level?

4. While the performance of the 1-bit system is not good, we can get very good results with 2 bits [a, c]. So, please also include [c] in the experimental comparison.

5. The proposed method can be trained end-to-end. However, a comparison with [b], which is a post-processing method, is still needed (see Question 1). 

6. Could the authors also validate their proposed method on ImageNet? It is better to include GoogleNet and ResNet as well. 

7. Could the authors make tables and figures in the experiment section large? It is hard to read in current size.

Reference
[a] How to Train a Compact Binary Neural Network with High Accuracy. AAAI 2017
[b] Network Sketching: Exploiting Binary Structure in Deep CNNs. CVPR 2017
[c] Trained Ternary Quantization. ICLR 2017
review_title: The experimental comparison are not enough

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes ResBinNet, with residual binarization, and temperature adjustment. It is a reconfigurable binarization method for neural networks. It improves the convergence rate during training. 

I appreciate a lot that the authors were able to validate their idea by building a prototype of an actual hardware accelerator.

I am wondering what are the values of \gamma’s in the residual binarization after learning? What is its advantage over having only one \gamma, and then the rest are just 1/2*\gamma, 1/4* \gamma, … , etc.? The latter is an important baseline for residual binarization because that corresponds to the widely used fixed point format for real numbers. If you can show some results that residual encoding is better than having {\gamma, 1/2*\gamma, 1/4* \gamma, …, } (which contains only one \gamma), it would validate the need of using this relatively complex binarization scheme. Otherwise, we can just use the l-bit fixed point multiplications, which is off-the-shelf and already highly optimized in many hardwares. 

For the temperature adjustment, modifying the tanh() scale has already had a long history, for example, http://yann.lecun.com/exdb/publis/pdf/lecun-89.pdf page 7, which is exactly the same form as in this paper. Adjusting the slope during training has also been explored in some straight-through estimator approaches, such as https://arxiv.org/pdf/1609.01704.pdf. In addition, having this residual binarization and adjustable tanh(), is already adding extra computations for training. Could you provide some data for comparing the computations before and after adding residual binarization and temperature adjustment? 

The authors claimed that ResBinNet converges faster during training, and in table 2 it shows that ResBinNet just needs 1/10 of the training epochs needed by BinaryNet. However, I don’t find it very fair. Given that the accuracy RBN gets is much lower than Binary Net, the readers might suspect that maybe the other two models already reach ResBinNet’s accuracy at an earlier training epochs (like epoch 50), and just take all the remaining epochs to reach a higher accuracy. On the other hand, this comparison is not fair for ResBinNet as well. The model size was much larger in BinaryNet than in ResBinNet. So it makes sense to train a BinaryNet or FINN, in the same size, and then compare the training curves. Lastly, in CIFAR-10 1-level case, it didn’t outperform FINN, which has the same size. Given these experiments, I can’t draw any convincing conclusion.

Apart from that, There is an error in Figure 7 (b), where the baseline has an accuracy of 80.1% but its corresponding bar is lower than RBN1, which has an accuracy of 76%. 
review_title: Interesting work

Meta-review: R1 and R3’s  main concern was that the work was not actually outperforming existing work and therefore its advantages were unclear. R2 brought up several questions on the experiments and asked for clarification with respect to previous work. R3 had several other detailed questions for the authors. The authors did not provide a response.
Decision: Reject