Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper proposes an improvement of Binary Neural Networks from the work of Bi-RealNet (Liu et al. 2018) by utilizing the linearity of modules. The method removes the necessity of scaling factors and used a novel non-linear layer named Fully Parametric ReLU (FPReLU) to increase the capability of BNNs. In addition, similar to recent works in BNNs, the paper also utilize group convolution layers to reduce the number of parameters and save the computation's cost. The experiments are tested with image classification tasks on the large-scale dataset of ImageNet. The paper is well written and straightforward. 
 
Although the results in ImageNet are promising and the computation is less, It raises concerns about the novelty of the work. The method used alike structure in Bi-RealNet with a bit of modification and group convolution which are already used in previous works of Liu et al. ECCV 2018 and Phan et al. CVPR 2020. 

The paper used novel FPReLU but missed some comparisons with a similar idea work using PReLU[1,2].

For results in ImageNet, it would be nice if we can do more comparison and analysis with the state-of-the-art recent work [3]. 
 
In conclusion, with current manuscripts, the paper is not sufficient enough to present at the conference.

[1].  Tang et al. How to Train a Compact Binary Neural Network with High Accuracy? AAAI 2017.
[2]. Phan et al. MoBiNet: A Mobile Binary Network for Image Classification, WACV 2020.
[3]. Liu et al. ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions, ECCV 2020.
review_title: Official Blind Review #3

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary:
Summary:
The authors note that the poor performance of BNN may be due to the disappearing of the amount of non-linearity in the input-output transformation. They propose a new Bi-Real architecture where they replace the scaling factor with extra non-linearities. The performance of the model is compared with the original Bi-Real Net and further possible improved version of the proposed method. 

Strengths:
Analyzing the role of non-linearity is a very nice idea. Studying BNNs is a clean setup for understanding the well-known predictive power of NN, even in real-valued settings. The computational gains associated with the purification of the convolutional layers look impressive.  

Weaknesses:
The comparison is between binarized but rescaled networks and binary networks without rescaling but added non-linearity. It is not clear if the good performance of the proposed architecture is due to i) the BN step making the scaling redundant or ii) the additional nonlinearities.  In some sense, noting that the inclusion of non-linear transformations can boost the predictive power of a network is not surprising. The extra computational cost associated with the introduction of non-linear activation is not fully discussed. The comparison does not include the case where non-linearities are added to the rescaled version (the one on the left of Figure 1). 
	
Questions:
- what happens if the binarized part of the network (between the FP first and last layer) is completely removed? Often, a two-layer network may achieve good accuracy on simple tasks such as MNIST-digit recognition.
- how does the introduction of extra non-linearity affect the computational budget in the proposed method? 
- is there a figure presenting the performance of the reduced budget model shown in figure 3?
- is the benefit associated with extra non-linearities be expected to extend to other architecture than Bi-Real?
review_title: The benefit of extra non-linearities in BNN

final_score: rating: 5: Marginally below acceptance threshold
review_content: The author proposed an improved binary neural network (BNN) model.  FPReLU which increases the discriminative ability of BNN and group convolution make the proposed BNN architecture achieve better accuracy than prior works under similar operation budget condition. Other techniques like knowledge distillation can further increase the model performance.

4-5 % top-1 accuracy improvement compared to prior models is an impressive result. I think reducing the burden of floating-point operations by editing first convolution layer and downsampling shortcuts is another good point, because the computation of uncompressed first convolution layer is easy to overlooked, but can be the bottleneck in the case of end-to-end inference.

The paper was clearly well-written, but I still have a concern or questions about the parts described below. I hope the authors to solve my concerns and questions so that my evaluation would be changed.

1. According to the sign function in Appendix E, its output value is either 0 or 1 if its input is from ReLU and is either -1 or 1 if its input is from PReLU, FPReLU or not passed through a non-linear function. According to [1],  better performance can be achieved by using [0,1] activation in BNN, so my major concern is that the accuracy improvement of Bi-Real Net V2 compared to Bi-Real Net may come from the fact that both [0,1] and [-1,1] activation are used in a network.  Note that 'Baseline' and 'Partially configured~' models in Table 3 use both [-1,1] and [0,1] activation.    
2. (-1.2,1.2) clipping can seem to be unfamiliar, because usually (-1.0, 1.0) range is used for BNN. I have the question how that range was decided.
3. One of the factor which can decide the model accuracy is the number (or total bits) of parameters, so I think the performance comparison among BNN models will be more clear if the authors add that information of each BNN models.

Minor comment:

In Appendix C, it is stated that original ResNet-18 model is used as a teacher model in all stages, which is different from [2] where the model trained at the previous stage is used as a teacher. This is the difference between using a teacher which has a good classification ability and using a teacher which resembles the student. I question if the authors had any reason to choose the former method and how two training cases affect to BNN performance.

Reference

[1] Peisong Wang, et al. Sparsity-Inducing Binarized Neural Networks. AAAI, 2020.
[2] Brais Martinez, et al. Training binary neural networks with real-to-binary convolutions. ICLR, 2020.
review_title: Nice work, but more analysis is required for acceptance

final_score: rating: 4: Ok but not good enough - rejection
review_content: ---paper summary---:  

This paper proposes to improve the BNN’s discriminative ability by introducing additional non-linearities. In addition, the paper exploits the group convolution to enable a wider network, which can strengthen BNN’s representational ability, while keeping the total overhead unchanged. 
		

---Pros---:  

This paper introduces some practical methods to improve the performance of BNNs. In particular, the additional FPReLU is convincing. Moreover, the paper shows that grouped convolutions can be applied to wider BNNs to increase the representational capability while keeping the same complexity.

---Cons---:

1:  This paper is incremental with limited new technical insights. 

a) Adding the nonlinearity can improve the representational capability of BNNs has been extensively explored in the literature. Specifically, ReActNet [Liu et al. 2020b] inserts additional RPReLU after each binary convolution to increase the non-linearity;  [A1; Martinez et al., 2020; Tang et al. 2017]  add PReLU as the nonlinearity;  Group-Net [Zhuang el al. 2019] and XNOR-Net [Rastegari et al., 2016] argue that adding additional ReLU is important to BNNs performance.  
b) Varying width and/or depth has been studied in previous fixed-point quantization/BNNs literature, especially in NAS-based methods [Bulat et al. 2020;  A2]. The original idea comes from EfficientNet.        
c)  Some other tricks such as replacing the 1x1 downsampling shortcut with pooling have been widely used in the community.

2:   Some arguments need theoretical proofs. For example, the authors argue that “despite the big quantization error made by quantization, the binary model can achieve much higher accuracy than the real-valued model with no quantization error”. In other words, minimizing the quantization error can hinder the discriminative ability of BNNs, which is the main point of this paper. This observation is interesting, but needs further theorems to further explore whether the quantization error has some relations with the predicted accuracy under some constraints. If zero quantization error cannot lead to a good result, then what should be the best trade-off? I encourage the authors to further explore this issue. At the current stage, it is far from enough.

3: The experimental results in Table 4 may have mistakes. The paper claims that BOPs are converted to equivalent FLOPs with a factor of $1/64$. However, why do smaller BOPs correspond to larger FLOPs? 

4:  The necessary “AND-count” operations may have technical issues. The AND-Count with values binarized to {0,1} should be equivalent to XNOR-Count with the values binarized to {-1, 1}, with a scalar difference. The authors can derive by themselves to verify this. However, the formulations in Eq. (3) and Eq. (4) are not equivalent if both values are binarized to {-1, 1}.

5: More experimental results on deeper networks (e.g., ResNet-50, -101) on ImageNet are needed to justify the effectiveness of the method.  In addition, the comparisons with RELU [Zhuang el al. 2019; Rastegari et al., 2016], PReLU [A1; Martinez et al., 2020; Tang et al. 2017],  RPReLU [Liu et al. 2020b] (optional) should be included.

6: Some typos. For example, “inheriting exiting advanced structures” → “inheriting existing advanced structures”; “Base on this consideration” → “Based on this consideration”.

References: 

[A1]: Bayesian Optimized 1-Bit CNNs, in ICCV2019

[A2]: Joint Neural Architecture Search and Quantization, arXiv:1811.09426v1

review_title: Official Blind Review #2

Meta-review: ## Description 

The paper proposes an improvement to binary neural networks with real-valued skip connections between pre-activations, by introducing more flexible learnable non-linearities on the real-valued connections. The parametric non-linearity is actually linear at initialization, which makes the training easier at the beginning. Due to learnable parameters it eventually adjusts to a more complex one, able to refine the accuracy. I think this idea is a good finding.

## Review Process and Decision
The reviewers initially gave low ratings to the paper, indicating that the contribution is incremental and not fully clearly presented. 
There was no detailed discussion with the authors, since the author's response and the rebuttal revision came in the very end of the discussion period. In the subsequent discussion phase the reviewer board has not indicated any major changes to the initial reviews/ranking. The AC checked the paper and supports rejection.

## Details

The authors are encouraged to improve the paper carefully addressing points proposed by reviewers.

I think the argumentation of the paper should be improved. Some explanations are intuitive, but operating with fuzzy notions and may in fact be incorrect or irrelevant. The paper should be made more precise, based on verifiable arguments.

I think the following is crucial and not made clear in the paper:
The non-linearities inserted before the sign function *do not affect the result of sign*. They indeed affect only the residual connections. Furthermore, the structure of residual connections should be fully clarified to reveal that there are complete real-valued paths all the way from the input to the network to its output, made of the residual connections with their own learnable parameters (and 1x1 convolutions) and (learnable) non-linearities and an intake from binary convolutions on the way. The learnable non-linearities can in principle improve performance just because the real-valued paths can learn better.

I paste below feedback by reviewers to author's response (I believe they would agree to share it with authors but did not find a suitable way of doing it):

## Response by R1:
I acknowledge that I read and appreciated the authors' answers to my questions. I think the idea of analyzing the role of non-linearities is nice and I tend to confirm my score. But I also agree with other reviewers that, as it is, the paper has some unclear parts and would not complain if it is rejected.

## Response by R3:
Thanks for your responses to answer my questions for the paper. I agree with the results of the proposed FBTN for improved Binary Neural Networks (BNNs). However, my concern about the novelty of using group convolution modules in BNNs has not been addressed. I think the paper is not sufficient enough to publish at the conference. So, I do not change my rating of the paper.

## Response by R4:
I maintained my rating when combining other reviews and responses to them, despite of their well response. It is still questionable whether FPReLU, one of the main contributions they claimed, actually improves the performance of BNN remarkably. In particular, this is supported by the fact that the performance of BNN on ResNet-34 which the techniques in this paper were applied does not show much difference from 'Real-to-Bin' model.
Decision: Reject