Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: A. Summary

Problem:
Binary NNs promise to make neural networks compatible with devices that only have access to limited computational resources (fundamental to embed NNs in mobiles or IoT devices). However, the loss of computational accuracy comes with a great loss of performances. Designing the right learning algorithm and the right binary architecture remains an open issue.

Contributions:
1. this paper reviews the current literature in Binary Neural Networks and the authors compile the existing methods to build a strong baseline. The strong baseline outperforms existing methods, which is impressive in itself.
2. the authors introduce a novel layer-wise objective that pushes the binary activations to match the real activations, which are given by a teacher model (real-to-binary). This is simpler and more efficient than existing alternatives.
3. they propose to train the real-to-binary model using a multi-stage teacher-student procedure. 
4. the authors introduce a data-dependent re-scaling term for the binary activations. 

Experiments:
1. SOTA on ImageNet for BinaryNNs: The combined methods allow bridging the gap between real-valued and binary-valued classifiers on ImageNet (65.4% vs 69.3% top-1 acc).
2. Comparisons with existing methods: On ImageNet, the model is compared with a complete list of alternative methods (low-bit quantization, larger binary nets, binary nets and real-valued nets). This is however unclear how the method compares with TTQ.
3. An ablation study is performed on CIFAR-100. It tests the gains that come with the attention matching, the data-dependent gating mechanism and the multi-stage teacher-student mechanism.

B. Decision
6: Weak Accept.

C. Argumentation
The paper clearly states the problem and what are the contributions. The solution is mostly iterative but clearly brings the binary NNs one step forward. The claims are supported by a comparison with a great variety of baselines and an ablation study. Furthermore, it is laudable that great efforts were put into designing such a strong baseline. 

However, the paper could be easier to read and some points remain unclear:
1. Is it possible to compare TTQ on the same scale? this is difficult to precisely asses how real-to-binary convolutions compete with this method in the paper.
2. The equation 2. is intuitive yet not perfectly clear. What are the "transfer points"? Why using such a normalization?


D. Feedback
1. Data augmentation and Mix-up: is it necessary to use them here as they should yield improvements for all methods? `
2. table 1: is it possible to include TTQ? suggestion: is it possible to add a 4th column that measures the overall performance (estimated runtime, speedup?)
3. table 3: please define all the abbreviations.

E. Question
1. Could you please draw a more precise comparison between TTQ and your method?
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper studies the problem of training binary neural networks. The authors first provide a strong baseline by assembling a group of training techniques that appeared in recent work that achieves state-of-the-art performance. Then the authors proposed two methods to further boost the performance gain. The first method is to use a teacher-student mechanism that uses a fully real-valued network to teach a binary network. The process is divided into three stages involving two intermediate models to reduce the gap within each teacher-student pair. The second method is to learn a re-scale factor for binary activations using real-valued activations from the previous block. Experiments show that the proposed methods improves the performance on ImageNet and CIFAR-100.
The experimental results seem promising. The proposed model reduces the gap to real-valued network to within 3-5%. However, the novelty of the paper is limited and why the proposed methods would help increase the performance gain is not well demonstrated. The teacher-student model is a well-known technique for vision tasks. The authors observed in Section 4.2 that it is very important for the teacher and student to have similar architectures, but did not explain the more important question that why a real-valued network would be able to teach a binary network, since they have quite different information flow. For re-scaling, the authors did not give a detailed comparison between their approach and previous work, and it is not clear how the data-driven way helps. As the ablation study shows the gating function actually hurts for binary down-sampling layers.
The writing of the paper needs improvement. A workflow/framework/algorithm description is helpful to better understand the whole framework, and the methodology part in Section 4 requires more details. Some notations need to be defined or clarified. For instance, in Figure 1 Left, what is A? The definition is given only in Section 4.2, where it is not stated in detail either. In Figure 1 Right, what is r? In Table 3, what do the abbreviations mean respectively?
Some specific questions:
- Why the real-valued teacher can help train the binary network while they have different information flow? What is the intuition behind the consistency assumption?
- The authors did not visually show the maps of real-valued and binary activations. How are they aligned in the proposed framework? And are they more similar with each other compared with previous approaches?
- In Section 4.1 for Initialization a 2-stage optimization strategy is used, while in Section 4.2 a multi-stage teacher-student optimization strategy is used. How are the two strategies combined?

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper is on building binary network. The steps for building binary network takes several components: traditional strategy to binary/optimize a model (like data augmentation,  binary initialization using 2-stage optimization, etc), real-to-binary attention matching that tries to match the output of real values and binarized model, and data-driven channel rescaling to better approximate real convolutions. All these components together makes a strong binary network.

Although there are so many steps/tricks mentioned in the paper, I think the explanation and reason for each step is easy to understand. The outcome of the model is quite impressive-- 5% improvement over the best binary model. 

It would be interesting to compare with some other compression techniques, like low-rank, sparsity, weight sharing, etc. Or it will be also interesting to see how these techniques can combine with binary model to further compress the model.

Also it would be interesting to see how the latency changes using the proposed binary model. As I can see from Table 1, which outlines the FLOPS and BOPS, to my understanding BOPS is much faster than FLOPS, so in the latency wise, the proposed model will be much faster than the original model for inference. Therefore I am looking forward to the real-timing results. 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper greatly reduces the gap between binarized and real valued imagenet, using a variety of techniques. The most significant contributions of this paper are engineering based, and the careful combination and integration of approaches from previous papers. I believe that this is of significant practical importance to the field. I particularly appreciate the effort put into developing a very strong baseline that combined ideas from many previous papers!

My biggest concern is that ResNet is itself a very wasteful architecture in terms of compute and parameter count. If the goal is to develop a compute- and memory-efficient architecture, it would be good to also consider real-valued-network baselines that were proposed with computational and/or memory efficiency as a design goal.

Additionally, the specific choices for the new student-teacher loss, and new scaling network architecture, seem fairly ad-hoc.

Detailed comments:

"this implies a reduction of 32× in memory usage" , assuming the parameter count is held constant.

Fig 1 right: This is motivated in terms of preserving scaling factors that are lost by the binarization, but the functional form for this makes it look a lot like a learned gating operation. If the sigmoid is dropped from the architecture, does performance worsen? It would be nice to see some discussion of the degree to which this is helpful because it reverses information loss due to binarization, vs. introduces a new architectural feature which is itself helpful.

Add a sentence describing what "double skip connections" are. I wasn't familiar with this phrase.

eq. 2:
This functional form is pretty weird.
Why is Q a square norm rather than a norm? Square error on an already-squared property is an unusual choice.
Why is the denominator itself a norm? Taking the norm of a square norm is similarly an unusual choice. (eg, why not just take an average or sum over Q) 
Say what Q_s and Q_t are (student and teacher network from context)

"thus, at test time, these scaling factors will not be fixed but rather inferred from data" nit: Would not generally call this an inference process. "Inference" typically refers to values that are computed indirectly (eg by Bayesian reasoning), while in this case the values are computed directly. Would rather say that scaling factors are a function of data, or are determined by data, or similar.

"By doing so, more than 1/3 of the remaining gap with the real-valued network is bridged." text is shifting back and forth between using % and fractional gap to describe benefits. Would just use one measure consistently.

Computational cost analysis:
This is very useful.
Note though that ResNet is a very wasteful architecture in terms of compute! It would be good to include a comparison to imagenet architectures that have computational and memory efficiency as a design goal. (eg, MobileNet comes to mind)

Very nice on the ablation studies.
review_title: Official Blind Review #1

Meta-review: This paper proposes methodology to train binary neural networks.

The reviewers and authors engaged in a constructive discussion. All the reviewers like the contributions of the paper.

Acceptance is therefore recommended.
Decision: Accept (Poster)