Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes methods to further reduce energy consumption of binary neural networks by removing or replacing 32-bit components (e.g., skip connections) in SOTA BNNs. More specifically, the proposed architecture (1) reduces the precision of skip connections and activation functions, (2) transforms the BatchNorm layer into a simple sign function, and (3) employs a multi-slice strategy to alleviate the loss of representational capacity incurred by binarizing the feature maps and shortcut connections. The results shows that the new model achieves 4.7x energy reduction with some accuracy degradation compared to SOTA architectures.
main_review: This paper is an attempt to reduce the precision of 32-bit components in BNNs. To this end, a new network architecture is introduced that integrates batchnorm into sign function, modifies downsample block and skip connections to reduce their required precision, and exploit a multi-slice convolution to compensate for the loss incurred by the precision reduction techniques. 

Strengths:
- The paper is well-written and easy to understand. 
- The proposed network architecture is novel and is an attempt to further reduce the complexity of neural networks for their efficient deployment.
- There is a discussion on one employment scenario and its implementation results to support the effectiveness of the proposed architecture.

Weaknesses:
- My first concern is the accuracy performance of the proposed architecture w.r.t. the SOTA architecture (i.e., ReActNet). In the ReActNet paper, there is a network architecture called ReActNet-A which achieves the accuracy of 69.4% while requiring 0.87 OPs. If we compare performance of this architecture given the accuracy and the number of OPs with the results reported in Figure 4(a), we can see ReActNet-A significantly outperforms BoolNets and BaseNets.
- My second concern is the implementation results of BoolNets. In industry, we don't usually design a custom hardware for a specific network architecture. On the other hand, we consider a general custom hardware that can support a wide range of computing units and layer types. Of course, in this scenario, each network architecture will show a different implementation performance on the hardware accelerator. Therefore, I believe the hardware performance of BoolNets and ReActNets should be measured on a specific hardware accelerator for a fair comparison, and the choice of hardware accelerator should be limited to well-know ones that are being used commercially. Otherwise, the design of custom hardware for a specific network can be biased to favor that network and does not constitute a fair comparison in my opinion. 

summary_of_the_review: After carefully reading the paper, I believe while this paper made novel contributions in reducing the precision of 32-bit component of BNNs, it couldn't show its advantages over existing works such as ReActNet. More specifically, ReActNet-A achieves a better accuracy and also requires a lower number of OPs, making the advantage of BoolNet limited to the implementation results. However, the implementation results on a custom hardware designed for a specific network doesn't establish a fair comparison. My suggestion is to report hardware performance of the networks on a commercial accelerator instead for a fair comparison and more reliable results.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to eliminate 32bit features of BNNs as possible. This paper claims that existing BNNs embed 32bit features, which can improve the accuracy of BNNs but must lead to overheads during inference. In the proposed network, full-precision batchnorm layers are replaced with 'shifted-sign' layers, full-precision scaling factors are eliminated, and multi-slice binary convolutions and 1b shortcuts are used. The network achieves 63% accuracy on ImageNet with various techniques and knowledge distillation.
main_review: This reviewer has several concerns about this paper as below:
- This reviewer is curious about the portion of overhead due to full-precision layers during BNN inference. This paper refers to a previous study (Fromm et al., 2020), but in that paper, only squeezenet was evaluated. To strengthen the motivation of this paper, this paper also includes simulation results on energy consumption, but it is a little hard for this reviewer to accept the simulated results because 1) a few networks are compared with the proposed work and 2) simulation results strongly depend on assumptions about h/w implementations. BaseNet/BoolNet with 32-but features can be comparable networks for ablation study to show the overhead of 32-bit features. Since current BNNs show better accuracy than the proposed work, this paper should strengthen and prove the motivation, why we need to eliminate the 32-bit features from BNNs completely, with deeper analysis.
- This paper claims that existing BNNs use various 32-bit features and proposes various techniques to make neural networks that consist of only binary operations. However, some ideas seem to be similar to previous works. In addition, for some ideas, there seems to be a lack of comparisons and citations.
  - The proposed shifted sign idea looks similar to ‘Batchnorm-activation as Threshold’ (Umuroglu et al., 2017). which is not cited and compared. This idea is not surprised for BNN researchers.
  - Multi-slice binary convolution layers seem to make a significant impact on improved accuracy. But this idea looks similar to ABC-net (Lin, 2017) and GroupNet (Zhuang, 2019). While ABC-net is compared with some descriptions, the GroupNet (it may be improved than ABC-net) is not compared and it is simply mentioned just one time. This reviewer wants to know why GroupNet is not compared.
  - Figure 4 uses ReActNet(Bi-Real-based) for comparison. Why don’t you use ReActNet-A that shows better accuracy and lower FLOPS? Indeed, the GitHub repo of ReActNet also includes ReActNet-A.

- Minor
  - Page 4 : with an running → with a running

Umuroglu, Yaman, et al. "Finn: A framework for fast, scalable binarized neural network inference." Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. 2017.

Lin, Xiaofan, Cong Zhao, and Wei Pan. "Towards accurate binary convolutional neural network." arXiv preprint arXiv:1711.11294 (2017).

Zhuang, Bohan, et al. "Structured binary neural networks for accurate image classification and semantic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
summary_of_the_review: As this reviewer mentioned, a more concrete analysis for the motivation should be added and comparisons with previous BNN architectures are required. The current manuscript seems to be lack novelty in this reviewer’s opinion. But, this reviewer is ready to listen to other reviewers’ opinions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper is the first to build fully 1bit neural networks. The proposed BoolNet achieves state-of-the-art performance on the trade-off between accuracy and energy consumption.
main_review: Strengths :
+ The first work for fully 1bit neural networks, which is very important for practical deployment on hardware.
+ The effects of 32-bit layers in commen BNNs are analyzed and removed by specific design.
+ A Multi-slice strategy and other tricks are proposed to alleviate the accuracy loss.
+ State-of-the-art performance on the trade-off between accuracy and energy consumption.

Weaknesses:
- The introduced MS-BConv increases the number of channels. Will it increase the memory usage and inference time?
- MS-BConv increases the number of channels, which is related to [1]. This related work should be included.
[1] Searching for accurate binary neural architectures. ICCVW 2019.
summary_of_the_review: This paper is the first work to build fully 1bit neural networks. I like this work and recommend acceptance for it.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes two novel BNNs (BaseNet and BooLNet) where most of the parameters are represented in the binary formats which is the major difference compared to pervious works where activations in layers are represented in 32-bit floats. Moreover, the proposed networks removes the BN and most of ReLu activations and replaces with sign function. To increase the knowledge capacity, the authors uses Multi-slice Binary Convolution and Local Adaptive Shifting approaches. The inference performance of BaseNet and BooLNet are evaluated and compared with the inference performance of ResNet-34 using 32-bit floating-point format. 
main_review: Strength: 

1- Reducing number of FP32 operations in BNN that reduces the energy consumption. 

2- The author evaluated the effect of various approaches such as Multi-slice Binary Convolution for BNN on accuracy, memory footprint and number of operations ( the ablation study on ImageNet and ResNet-34 is interesting)  

Weakness:

1- The novelty of paper is not obvious. It seems the author uses pervious approaches and combines it with BNN to improve accuracy. 

2- The motivation behind using sign function for BN is not explained in the paper. Please elaborate this. 

3- In the table on the Figure 4, to have a fair comparison, is distillation and long training epoch considered in pervious works?

4- The author did not compare BooLNet with reference [1] approach which also uses Multi-slice Binary Convolution approach. 

[1] Pouransari, Hadi, Zhucheng Tu, and Oncel Tuzel. "Least squares binary quantization of neural networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.

5- The BooLNet is only evaluated on ImageNet and ResNet-34 and it is not obvious that this proposed work can be generalized for others datasets and networks. I recommend evaluate the BoolNet on the others networks such as MobileNet-V2. 
summary_of_the_review: As this reviewer mentioned, the motivation to use only binary network is good research direction, However, the novelty of this paper is not obvious and the BoolNet needs to be evaluated on other benchmarks. I recommend this paper to be rejected.
review_title: null

Meta-review: ### Description
The authors note that the most recent and successful in terms of accuracy binary networks are in fact combining binary and floating-point computations, in particular have residual full-precision paths, with their parameters, that connect all the way to the output. Although such paths are made lightweight, they can be a bottleneck with respect to the energy consumption, memory and latency. The paper proposes a novel binary neural network model that uses only binary operations (except in the first and last layers). It is proposed to estimate the energy efficiency of binary networks more accurately, using hardware design compilers.

### Decision
Reviewers came to a consensus that the proposed BNN architecture claimed in the paper to be the main novelty, while it is indeed quite distinct from the mainstream and best performing BNNs, does not propose novel solutions with respect to the total state of the art. This is our main reason for rejection. The paper makes a great effort in steering the development of BNNs towards more energy efficient models, by carefully estimating the potential energy consumption of different models, using specialized software for design and simulation of hardware needed to run particular models. This is proposed not as a practical solution for industry but rather as a way to measure the potential efficiency of different models. While this was recognized as a great effort, some questions remained regarding fairness of comparison and possibility to reproduce the results by non-experts in order that other developers could estimate and compare potential efficiency of their models. Additionally, it is unclear how power efficiency of a model with $k$ slices (BoolNet) differs from that of a $k$-bit quantized network.

### Details
First of all, I very much like the motivation for the work: to aim at a design of BNN that would be more efficient in terms of speed and energy and to measure that efficiency more precisely. I am not an expert in the hardware, however I do share the concern in this paper that full precision residual paths and blocks would incur a larger latency and higher amount of computation (more chip area, more energy,...). The fact that residual connections in BiRealNet make it necessary to read and write full precision feature maps to the global memory was non-obvious to me. The current state of the art reports binary and floating point operations and largely ignores the necessary memory operations, which as authors argue are the real performance bottleneck.

The main claimed contribution of the paper is the design of a novel model. This was the main point of concert. It is indeed innovative with respect to the current trend of the best performing binary networks to make a "pure" binary network. However it is hard to agree that removing some of the components from BiRealNet or models based on it and going back to simpler architectures which were used before can be a contribution. Specifically, plain non-residual BNNs were the very well-known pioneering works [r1, r2]. The fact that BN in front of binarization can be simplified is trivial and well known, e.g. [r3]. Using multiple binary activations through power-of-two coding or uniform thresholds has been considered multiple times [r2, r6, ABCNet]. Same for group-wise convolutions. I have not seen residual connections in the form of shuffle-net so far, but it can also be considered as a rather standard trick. Using stride instead of max pooling or average pooling is another well known trick. Many of these solutions are in fact used in recent works, e.g. [r4] has no skip-connections, combines BN with sign at the inference time and uses strided convolutions for downsampling. To summarize, the submission does not appear to propose novel modelling solutions relative to the total state of the art.
Furthermore, [r3] is closely related in that they investigated the energy efficiency (however looking at the energy consumption of individual operations only) and with a similar motivation developed binary networks without 32 bit residual connections. Please see also other references pointed out by reviewers.

### Unclear in the paper:
-What is the meaning of $F$ column in Fig4a for BaseNet / Boolnet?
-Why grouped convolution with input channels 256k and k groups has output of 256 channels in fig 3a. 
If the outputs from each group are summed, isn't it equivalent to a full convolution 256k x 256?
-What do the authors mean by 3x3 depth-wise convolution?
- The local adaptive shifting module is discussed inside the paragraph describing MS-BConv. Is it a part of MS-BConv or not?
- It seems that with $k$ slices, there is $k$ times more bits per activation used and $k$ times more bits per weight used (because the channel width is multiplied by $k$).
It must be therefore equivalent in terms of the power consumption to a network that uses $k$ bits per activation and weight. 
Using these $k$ bits to represent uniform slices rather than powers of two slices appears inferior in terms of quantization error and accuracy. Indeed, many works have successfully quantization different models down to $4$ bits without a loss in accuracy. 

### Related work: 

Rather than reviewing different methods for making networks more efficient, a deeper review of BNNs would be more helpful, in particular 
looking at the works that are closer to the hardware, such as "in-memory computing", "neuromorphic computing", [r5, r6].

### Discussion

A well-justified way to measure the potential energy efficiency of BNNs would be an excellent contribution that could standardize comparison and drive the development of BNNs in the energy-efficient direction. Unfortunately,
it does not appear at the moment that non-experts in hardware and design compilers could repeat the compilation and simulation of accelerators as the authors proposed. A simplified estimation method is needed that can be used in python for any model composed of some standard blocks. The authors seem to be in a good position to propose and validate such a simplified estimator. To start with may I suggest to clarify the following questions:
- Do we need power for the operation pipelines for different operation types (cache, global memory) that are not currently used?
- Are the arithmetic operations implemented in hardware to optimize energy or the throughput?
- Can we assume that all latencies can be masked by parallelism?
- Is it a good approximation to assume that a convolution (with an efficient implementation and large enough cache) needs to read the input only once?
- Can any coordinate-wise transform be appended to the preceding transform on the fly, canceling a read-write in between?
- What is a reasonable estimate of a cost for float32 operations? It seems from the quantization literature that all such operations can be safely quantized to e.g. 8 bit representations without a loss of accuracy.

[r1] Courbariaux et al. 2016, Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or −1] 

[r2 Hubara 2018: Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations, JMLR] 

[r3 Ding et al. 2019: Regularizing Activation Distribution for Training Binarized Deep Networks] 

[r4 Livochka et al. Initialization and Transfer Learning of Stochastic Binary Networks From Real-Valued Ones]

[r5 Baskin et al. Streaming Architecture for Large-Scale Quantized Neural Networks on an FPGA-Based Dataflow Platform]

[r6 Umuroglu et al. FINN: A Framework for Fast, Scalable Binarized Neural Network Inference]
Decision: Reject