Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a novel method to make the training Binary Neural Networks with low-memory and low-energy by modifying the backpropagation and forward process. To this end, the paper binarized weight gradients, change batch normalization layer for removing full-precision the inputs, and utilize quantization to accelerate the whole training BNNs procedure.   

The paper targeted one of the problems in Binary Neural Networks and provided experiments as well as source codes to the proof of efficiency. This is the most significant contribution of the paper. 

However, there are the following concerns:
- Need more experiments on the actual training time between two BNNs which are trained with quantized and full-precision gradient weight.
- Comparison with similar works in recent BNNs.
- What does it mean B variables in Algorithm 2 of lines: 6, 9, and 10, and how does it influence the performance of training?
- In figure 2, the legends should be put in the figure (not in the caption). It is better to follow.
- It would be nice to have the training time and accuracy in the large-scale dataset of ImageNet.

In conclusion, the paper addresses the novel idea for the training improvement of Binary Neural Networks in low-memory and low-energy. However, there are many concerns aforementioned.  
review_title: Official Blind Review #1

final_score: rating: 6: Marginally above acceptance threshold
review_content: #### Comments
Summary:

The authors proposed a low-cost binary neural network training strategy exhibiting sizable memory footprint reductions and energy savings. The methods include binarizing weight gradients, modifying the forward and backward batch normalization operations and using power-of-two activation gradients and reduced-precision floating-point data. The experimental results show some improvement memory footprint reductions and energy savings vs standard approach.


Strength:
-- The authors carried out a relatively sufficient experimental analysis, and evaluated across multiple models, data sets, optimizers and batch sizes
-- Storage and energy consumption based on hardware models increases the completeness and credibility of conclusions.
-- The experiment results seem that the proposed method achieves good performance in memory footprint reductions and energy savings.

Weakness:
-- The proposed method seems like a combination of existing technical methods.
-- There is a lack of comparison with other low-cost binary neural network training works.


Comments:
(1)	There is a difference in the memory consumption in Table1 and in section 5.1 (1.67 MiB or 1.41 MiB ?). The authors may check this.
(2)	As for the perspective of overall design, it’s better to emphasize the trade-offs between the importance of different variables to the overall training and the choose of the data type.
(3)	It’s better to add some comparisons with other low-cost binary neural network training works.

review_title: review

final_score: rating: 5: Marginally below acceptance threshold
review_content: I agree with the key contributions listed in the paper, especially the binarization of weight gradients and activations. The paper is well written and clearly articulates a contribution to the literature. The proposed BNN training scheme can have a significant practical impact. The experimental evidence is provided for several standard image classification tasks. Most of the related works are cited. The paper does not contain a theory part, but wherever possible, equations are provided to illustrate how the method works.

Concerns: 
This paper includes a detailed empirical evaluation of the proposed BNN training scheme. The major concern is that the proposed low-cost BNN training scheme can cause a nontrivial accuracy degradation (2.25%) as shown in Table 5. The tradeoff between accuracy and memory footprint/energy consumption is not carefully evaluated. For example, a smaller network model with fewer parameters and activations can be trained using the baseline BNN training scheme to reduce memory and energy consumption. Besides, the baseline networks (BinaryNet and ResNet) used for comparison are out-of-date.  Many recent works [1-3] propose new BNN architectures, which improve the accuracy of BNNs significantly. It is useful to justify the effectiveness of the proposed scheme for these SOTA BNN architectures.

To evaluate the energy consumption of the traditional and the proposed BNN training scheme, the authors assume that the two training schemes have the same convergence rate. Although the traditional BNN training scheme consumes higher power during training, it might take fewer epochs to reach the same accuracy. Therefore, the saving in energy consumption can be lower than the reported numbers. In addition, the estimated energy consumption of BNN training obtained from using QKeras is very rough, which makes the improvement in energy consumption less convincing. The authors can mainly improve the paper's strength by prototyping the proposed BNN training scheme on an embedded CPU and measuring real-world performance and power.

Reasons for score: In general, I like the idea of enabling low-cost BNN training by identifying unnecessary high-precision data. However, the improvement numbers presented in the paper need better justification. I would consider raising my score if the authors could address the aforementioned concerns.

[1] Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm

[2] ProxyBNN: Learning Binarized Neural Networks via Proxy Matrices

[3] ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions

review_title: This work proposes a low-cost BNN training scheme to reduce memory consumption and improve energy efficiency. Compared to the traditional BNN training algorithm, the proposed scheme achieves a significant reduction in memory footprint and energy consumption on multiple datasets.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: I think this a very good contribution to ICLR given the topic and the quality of the submission (originality, contribution to the state of the art, experimental evidence, e

 Some of the strong points of the submission are summarized as follows, along with some points for clarification

1.	Enabling training on any embedded device (SoC, FPGA, micro-controller) is one of the holy grails for edge AI and IoT. As the authors mention, this also intersects with other domains such as federating learning privacy by design systems. The authors provide and ample motivations of the importance of this work, and some of the applications edge AI might enable, as well as the current challenges.
2.	The state of the art (despite the previous comment) contextualizes the subject matter in a succinct but comprehensive manner. Although there are certain aspects that could be improved, such as including a table outlining in a clearer manner the contributions of the authors in this context.
3.	The comparison with the traditional training method is clear. However, I would like to know if the authors have made an ablation study to assess whether or not the use of batch normalization would have an effect on the accuracy of the proposed models. Do you provide experimental evidence of the lack of degradation due to the use of l1 batch normalization? These two aspects are not mentioned in the next nor provided in the supplementary sections.
4.	The experimental design is good, showing a careful analysis to validate the proposal and several ablation studies to assess the memory footprint reductions and its effects on the training of various models
5.	The foundations for the method are presented in great detail in a formalized manner and provides sufficient elements (i.e. experiments) to assess the validity of the proposed approach.



review_title: I think this paper is very interesting and if some minor comments are addressed, it should be accepted for presentation at ICLR21

Meta-review: After reading the paper, reviews and authors’ feedback. The meta-reviewer agrees that this paper addresses an important topic. However, as the reviewers pointed out. The paper mainly builds the technique on simulated setting, and it is unclear how the method will translate to real world speedups. Past work(e.g. [1]) has also shown that many cases there could be a huge gap when the solution is not built carefully.

The paper would benefit from a prototype to demonstrate the applicability of the approach. This paper is therefore rejected.

Thank you for submitting the paper to ICLR. 

[1] Riptide: Fast End-to-End Binarized Neural Networks

Decision: Reject