FTBNN: Rethinking Non-linearity for 1-bit CNNs and Going Beyond
Authors: Zhuo Su
Linpu Fang
Deke Guo
Dewen Hu
Matti Pietikäinen
Li Liu
Venue:  (2021)
Keywords: Binary neural networks
network quantization
network compression

Abstract: Binary neural networks (BNNs), where both weights and activations are binarized into 1 bit, have been widely studied in recent years due to its great benefit of highly accelerated computation and substantially reduced memory footprint that appeal to the development of resource constrained devices. In contrast to previous methods tending to reduce the quantization error for training BNN structures, we argue that the binarized convolution process owns an increasing linearity towards the target of minimizing such error, which in turn hampers BNN's discriminative ability. In this paper, we re-investigate and tune proper non-linear modules to fix that contradiction, leading to a strong baseline which achieves state-of-the-art performance on the large-scale ImageNet dataset in terms of accuracy and training efficiency. To go further, we find that the proposed BNN model still has much potential to be compressed by making a better use of the efficient binary operations, without losing accuracy. In addition, the limited capacity of the BNN model can also be increased with the help of group execution. Based on these insights, we are able to improve the baseline with an additional  4$\sim$5% top-1 accuracy gain even with less computational cost. Our code and all trained models will be made public.

Under review as a conference paper at ICLR 2021
FTBNN: RETHINKING NON-LINEARITY FOR 1-BIT CNNS AND GOING BEYOND
Anonymous authors Paper under double-blind review
ABSTRACT
Binary neural networks (BNNs), where both weights and activations are binarized into 1 bit, have been widely studied in recent years due its great benefit of highly accelerated computation and substantially reduced memory footprint that appeal the development of resource constrained devices. contrast to previous methods tending reduce the quantization error for training BNN structures, we argue that the binarized convolution process owns an increasing linearity towards the target of minimizing such error, which in turn hampers BNN's discriminative ability. In this paper, we re-investigate and tune proper non-linear modules to fix that contradiction, leading to strong baseline which achieves of-e-of-the- art performance on the large-scale ImageNet dataset in terms of accuracy and training efficiency. To go further, we find that the proposed BNN model still has much potential to be compressed by making better use of the efficient binary operations, without losing accuracy. In addition, the limited capacity of the BNN model can also be increased with the help of group execution. Based on these insights, we are able to improve the baseline with an additional 4~55% top-1 accuracy gain even with less computational cost. Our code and all trained models will be made public.
INTRODUCTION
In the past decade. Deep Neural Networks (DNNs), in particular Deep Convolutional Neural Networks (DCNNs), has revolutionized computer vision and been ubiquitously applied in various computer vision tasks including image classification (Krizhevsky et al. 2012), object detection (Liu et al. 2020a) and semantic segmentation (Minaee et al.] (020). The top performing DCNNs (He cct al. 2016 Huang et al. 2017) are data and energy hungry, relying on cloud centers with clusters of energy hungry processors to speed up processing, which greatly impedes their deployment in ubiquitous edge devices such as smartphones, automobiles, wearable devices and IoTs which have very limited computing resources. Therefore, in the past few years, numerous research effort has been devoted to developing DNN compression techniques to pursue a satisfactory tradeoff between computational efficiency and prediction accuracy (Deng al. 2020).
Among various DNN compression techniques, Binary Neural Networks (BNNs), firstly appeared in the pioneering work by Hubara al. (2016). have attracted increasing attention due to their favorable properties such as fast inference. low power consumption and memory saving. In BNN, the weights and activations during inference are aggressively quantized into 1-bit (namely two values). which can lead to 32x saving in memory footprint and up to 64x speedup on CPUs (Rastegari al. 2016). However, the main drawback of BNNs that despite recent progress (Liu Gu et al. 2019 Kim et al. 2020b), BNNs have trailed the accuracy of their full-precision counterparts. This is because the binarization inevitably causes serious information loss due to the limited representational capacity with extreme discreteness. Additionally, the discontinuity nature of the binarization operation brings difficulty to the optimization of the deep network (Alizadeh) et al. 2018).
al.] 8
A popular direction on enhancing the predictive performance of a BNN is to make the binary operation mimic the behavior of its full-precision counterpart by reducing the quantization error cuased by the binarization function. For example, XNOR-Net (Rastegari et al.] 2016) firstly introduced scaling factors for both the binary weight and activation such that the output of the

---

Under review as a conference paper at ICLR 2021
(4)
(b)
Block chain Net:
x
Block chain FTEMN:
Sign
Sign
MD
Basic black shown in left
i APReLU We ReLU
0.
Binary weight
Binary weight
FTBNN [Top-l 275
1:62 8%,
0.7
08
Scaling
Traiing
TOP 1A
0.4
BN
Addition
02
Addition
[a]: Basic block
(b): Basic block FTBNY
30
Epoch
Figure I: Left: The basic block in original Bi-Real Net vs. the simplified basic block in FTBNN, where we directly absorb the explicit scaling factors into the BN layer by leveraging BN's scaling factors. Right: The non-linear modules (ReLU or FPReLU) are explicitly added after each basic block in FTBNN. To maximize the model's discriminative power while keeping its training stability, the number of ReLU is controlled and the proposed FPReI.U is connected with most blocks. Training curves on ImageNet of the two 18-layer networks are depicted to show the training efficiency. Solid lines denote Top-1 accuracy on the validation set (y-axis on the right), dashed lines denote training loss (y-axis on the left). Both models are trained from scratch.
binary convolution can be rescaled to closely match the result of the real-valued convolution just like the original full-precision weight and activation are used. The method outperforms its vanilla counterpart BNN (Hubara et al. 2016) by a large margin (44.2% vs. 27.9% in Top-1 accuracy on ImageNet using the AlexNet architecture (Krizhevsky et al.] 2012). Because of the remarkable success of NNe, a series of approaches emerged subsequently with the effort of either finding better scaling factors or proposing novel optimization strategies to further reduce the quantization error. Specifically, XNOR ENOR-Net++ (Bulat & Tzimiropoulos/ 2019) improved the way of calculating the scaling factors by regarding them as model parameters which can be learnt end-to-end from the target loss. While Real-to-Bin (Martinez et al. 2020) proposed to compute the scaling factors on the fly according to individual input samples, which is more flexible. From another perspective, IR- Net (Qin et al. 2020) progressively evolved the backward function for binarization from an identity map into the original Sign function during training, which can avoid big quantization error in the early stage of training. BONN (Gu et al. 2019) added a Bayesian loss to encourage the weight kernel following a Gaussian mixture model with each Gaussian centered at each quantization value, leading to higher accuracy. Other works aiming reduce the quantization error also include ABC-Net (Lin et al. 2017), Ri-Real Net (Liu et al. 2018), ProxyBNN (He et al.) 2020), etc.
However, another problem arises with the quantization error optimized towards 0, especially for the structure like Bi-Real Net ( Fig. [. where the only non-linear function is the binarization function. The non-linearity of the binarization function will be eliminated if the binary convolution with scaling factors can perfectly mimic the real-valued convolution in the extreme case (quantization error equals to 0), thus hindering the discriminative ability of BNNs. Therefore, is necessary to re- r--investigate the non-linear property of BNNs when inheriting existing advanced structures.
Based on this consideration, we conduct the experiment on MNIST dataset (LeCun & Cortes 2010) using 2-layer Bi-Real Net like structure (which begins with an initial real-valued convolution layer and two basic blocks illustrated in Fig. (b), optionally followed by a non-linear module, and ends with fully connected (FC) layer) and some interesting phenomenons can be found as shown in Fig. 2 where we visualized the feature space before the FC layer and calculated the feature discrepancy caused by the binarization process as well as the corresponding classification accuracy (ACC, in %) for each model. Firstly, comparing the first two figures, despite the big quantization error made by binarization, the binary model achieves much higher accuracy than the realev model. which does not have quantization error. This indicates that the binarization function owns potential ability to enhance the model's discriminative power, and also explains why Bi-Real Net
2

---

Under review as a conference paper at ICLR 2021
Discrepancy: 0 ACC: 82.64 + 0.04
Discrepancy; 1.04: 0.05
Discrepancy: 1.66 0.04
Discrepancy: 2.41 2.13-
ACC: 98.45 :: 0.16
ACC: 98.70 0.07
ACC: 98.83 0.07
Linear
Biary
P
RALU
Figure 2: Feature visualization for 4 different models experimenting on MNIST, which are denoted as Linear: using real-valued convolution layers without non-linear module; Binary: using binary convolution layers without non-linear module; PReLU: using binary convolution layers with PReLU module; ReLU: using binary convolution layers with ReLU module. Testing accuracies recorded based on runs for each model. The average discrepancy between output from binary convolutions and that from its corresponding real-valued convolutions (using the proxy weights and original real- valued activations to conduct the convolution) are recorded as well (best viewed in color).
Top-1 accuracy on ImageNet of different BNN models with commonly used training strategies, based the ResNet-18 architecture (He al. 2016). VIx denotes the corresponding strategy is / is not utilized. KD denotes knowledge distillation, MS means using multiple training steps, GA indicates applying gradient approximation in the backward pass. Note that Binary Duo (Kim et al. 20205) improved GA with a coupled ternary model. Scaling represents using explicit scaling factors to reweight the output of binary convolutions. N/A+n means only the number of epochs n in the final step for multi-step training is given. Here Bi-Real Net was trained using two indicates the double skip connections proposed by Liu et al. (2018) were used. It can be seen that FTBNN achieves promising accuracy even with naive training scheme and can be further improved when combining the training strategies.
Table I:
different implementations.
Method
KD
GA
MS
Scaling
Epoch
Top-1
Method
KD GA MS Scaling
Epoch
Top-I
BNN
X
X
x
NIA
42.2%
IR-Net 1
X
N/A
58.1%
XNOR-Net
X
x
v
v
NIA+58
51.2%
BinaryDuo i
X
N/A
120+40
60.9%
XNOR-Net+t
x
57.1%
Real-to-Bin baseline 1
V
75+75
60.9%
Bi Real Net (B) f
X
x
d
256
56.4%
FTBNN
X
x
x
x
60
60.2%
Bi-Real (A) f
N/A+20
56.4%
FTBNN
x
75+75
63.2%
can possibly achieve high accuracy even without any non-linear modules. Such ability should be respected when we design the BNN structures rather than minimizing the quantization error to O. Second, when explicitly introducing a non-linear module, either PRcLU (He et al.) 2015) or ReLU, the discriminative power can be further enhanced, even though the feature discrepancy enlarged. This again shows that there is no strict correlation between the quantization discrepancy and the final predictive performance. Mcanwhile, the ReLU model brings the most discriminative power with clearer class separateness in the feature space. Our following experiments also show the superiority of the traditional ReLU function due stronger non-linearity.
Motivated by the above observation, we propose FTBNN (meaning fast training, see Fig. [. by firstly eliminating the scaling factors after the binary convolutional operation, which are instead fused into the Batch Normalization (BN) (Ioffe & Szegedy] (2015) layers by leveraging BN's scaling factors, and explicitly configuring appropriate non-linear modules after each block. By doing so, we can release the optimization burden for extra scaling layers as well as keep the non-linear property of the Sign function via a weaker scaling mechanism. At the same time, the configuration of explicit non-linear modules (we also proposed more flexible non-linear module, namely Fully Parametric ReLU (FPReLUU) shown in Fig. 3) able to further boost the discriminative ability of the BNN model as well as ensure the training stability. The resulted FTBNN is highly competitive among the state-of-the-at BNN frameworks, with its enhanced performance in terms of both accuracy and training efficiency, as shown in Table T and Fig. [ Note that very recently Liu et al. (2020b) also demonstrated the importance of non- lineariyy by redistributing activations in BNNs.
To go beyond the baseline structure have already obtained, we also notice that two aspects in the literature were rarely investigated. One is to think how far we can go to make use of the fast and light binary operations and avoid the use of expensive real-valued counterparts, to make our
3

---

Under review as a conference paper at ICLR 2021
BNN model more compact and computationally efficient. This is motivated by the fact that the real- valued operations in a BNN structure often take a considerable proportion of computation cost (e.g., 85% in an 18-layer Bi-Real Net. and so does the proposed FTBNN, see Fig. B Second, inspired by Binary MobileNet (Phan et a 2020) which leveraged group convolution (Xie et al.) 2017) to automatically configure optimal binary network architecture in their network search space, consider how group convolution can affect the performance of BNN structures (in representational capacity, discriminative power, etc.) with different configurations (vary in depth and width). To the best of our knowledge this paper is also the first attempt to give such specific investigation and can lead to useful insights on designing better BNN structures with the group mechanism.
As for our experimental outcomes, firstly, we are able to build an easily trainable strong baseline FTBNN by improving the popular Bi-Real Net structure, already achieving the state-of-the-art performance (Table 1 Table B and Table Secondly, we enhanced the FTBNN by circumventing the usage of real-valued operations and incorporating the group mechanism, leading to a series of derived models that not only surpass the baseline in higher accuracy but also have less computational overhead (Table 4). It is also hoped that our exploration can give better understanding of BNN design in the community and derive more efficient binary structures.
2 METHODS
2.1
BACKGROUND AND FTBNN
In order to reduce the quantization error (denote as E) and protect the information flow in BNNs, which has been an important focus of recent years' studies, very basic idea is introducing several scaling factors to reweight the output of the binary convolution, such that the rescaled output can approach closely to the result if using the real-valued operation:
(1) where W, A. Wo and As indicate the real-valued weight and activation, the binarized weight and activation respectively, @ represents a multiplicative convolution and @ represents a convolution using the WNOR-Count operations (or binary convolution), I is the scaling factors and @ element- wise multiplication.
E = (W& @ Ab) or- W & A ~
In XNOR-Net (Rastegari et al.) 2016). I actually consisted of two separate parts, for the weights and activations respectively and both were calculated in an analytical way during the forward pass, by solving an optimization problem, which is therefore time-consuming. XNOR-Net++ (Bulat & Tzimiropoulos 2019) further argued that I can be a single part and learnt during training, showing better performance in terms of both accuracy and efficiency. More recently, Real-to-Bin (Martinez) al. 2020) proposed data-driven version of т, which was computed on the fly according to individual input samples.
In fact, with the purpose of finding satisfactory initialization for T, usually a pretrained model with full-precision weights is needed (Rastegari et al. 2016 Lin et al. 2017). However, it is shown that the scaling factors can be fusable with the parameters in BN (Liu et al. 2018), directly training the BN layers can avoid the above troublesome, without degenerating the whole structure (Bethge et al., 2019). This also matches our assumption on Section I that the BN scaling factors can be a good alternative for an explicit scaling layer.
On the other hand, to maintain the information flow and maximize BNN's expressive ability, another aspect of experience in the literature is the usage of shortcut connections (He et al., 2016). One milestone about that is the Bi-Real Net structure (Liu et al., 2018)), where the authors emphasized using the shortcut which connects the real-valued activations before the Sign function to the output of binary convolution, and proved the almost computation- free shortcut can significantly preserve
BNN's representational capability. It was then repeatedly adopted in other works (Liu et al. 2020b Bulat et al. 2020; Kim et al. (020a) to improve the quality of information propagation.
Standing on the shoulders of giants and according to our discuss in Section I we propose simple block based structure, dubbed as FTBNN" (Fig. [ that both enables training efficiency and encourages non-linearity by discarding the scaling factors which are implicitly fused into BN, and
For reduction block, we adopt the real-valued 1xl convolutional downsampling shortcut.
4

---

Under review as a conference paper at ICLR 2021
Cony
carvelztion laper
100
Last FC layer
operatins
Binary convelation
Rlyny (scarding stanus)
Floz busgee (nhi))
50
Plepe budget: 172M
Flose burgett BOM
TOP-1:
TOP-3:
Binary wesggt (giousp commedlttion
AugPost
80
kyy
Figure Left: The proposed FPReLUU slops in both sides are started from (equal an identity map) and adaptively learned in channel-wise. Middle: Reduction block with group binary convolution and cheap downsampling layer (for normal block, the original binary convolution layer in Fig. [ is also implemented with groups in the derived models). Right: Computational budget distributions for original and binary-purified FTBNN. The computational budget a original floating point multiply operation (FLOP) in convolution is reduced by 64 times if the operation is binarized (using XNOR-Count instead). Based on that, when measuring the computational budget, the budget of binary operations (BOPs) converted to equivalent of FLOPSs with factor of 1 1/64. 15% and 77% denote the corresponding budget percentages for BOPs.
adding specific activation functions. Contrary to Tang et al. (2017) and Bulat et al. (2019), we find that traditional ReLU is still more effective than PReLU with stronger non-linear property. While at the same time, immoderately inserting ReI.Us in the network can also cause divergence and deterioration, as the information collapse phenomenon (Sandler et al.) (018) become serious. To alleviate that, extend PReLU to Fully Parametric ReLU (FPReLU) where the slops for both negative and positive activations are learnable (Fig. 3). to stably maximize BNN's flexibility.
2.2 TOWARDS A MORE COMPACT BNN WITH BINARY PURIFICATION
Real-valued operations BNN are often needed to drive and preserve the information flow (e.g., binarization for the 1xl convolutional shortcuts in the reduction block of Bi-Real Net will interrupt the continuous pull-precision information flow, the damage of which is hard to recover in the following layers (Bethge et al., 2019)).
From Fig. we find that the most weappputttioocoonsuuitti parts among the real-valued operations in FTBNN are the initial convolutional layer and the real-valued 1xl convolutional shortcuts. Thereby we make the following two changes to relieve the computation burden for the baseline.
Initial Conv: Inspired by Bethge et al. (2020), in the full-precision initial convolutional layer, we set the kernel size to 3x3 and constrain the number of output channels (only half of that in the baseline in experiments). To facilitate building BNNs with different width configurations, as will discuss later, we add an extra binary convolutional layer after the initial convolutional layer as a bridge that arbitrarily expand the initial width (see appendix for more detail).
Downsampling shortcuts: Similar to Liu et al. (2020b), we propose copy-and-paste paste strategy to avoid a convolutional downsampling shorteut to computation while preserving the information flow. In a reduction block where the resolutions of feature maps are halved and the number of filters is doubled, firstly concatenate the duplicated input volume extend the input, then employ 3x3 average pooling with stride=2 to construct the shortcut (Fig. 3
2.3 FEATURE AGGREGATION WITH GROUP EXECUTION
As discussed by Phan et al. 2020). binary operation restricts BNN's feature representation only using limited discrete values. Simply enlarging the network by increasing the number of channels can be very expensive since a Nx x widening leads to a 2 growth in computational budget. We exploit the group convolution (Xie al. 2017) enable wider network (Fig. 3), which we believe
5

---

Under review as a conference paper at ICLR 2021
strengthen BNN's representational ability, while keeping the total overhead unchanged. The differences to other similar works are as follows.
Compared with methods using multiple bases or branches: Lin et al. (0117) and Zhuang et al. (219)) gathered results from multiple branches or bases to enrich the feature representation for binary convolutions. Their methods can be equally viewed running several independent networks parallelly followed by a summation, while ours only uses a single pipeline without introducing additional computational overhead. In the form of information aggregation, concatenation used in group convolution, which is different to summation adopted in these methods.
Compared with methods leveraging group convolutions: Group convolution is also incorporated in recent approaches to facilitate binary network architecture search (NAS) (Phan et al. 2020 Bulat ct al. 2020), while unfortunately a proper ablation study is missing, making the benefits from the group convolution unclear under the whole framework where the resulted advantage may be biased towards the NAS technology itself. Our study can be regarded as such an specific investigation to make a better understanding of group convolution when designing BNNs.
3 EXPERIMENTS AND DISCUSSES
FTBNN Baseline: As shown in Fig. to give the model enough non-linearity while maintaining training stability, we insert a ReLU after every four blocks and for other blocks, FPReLU adopted.
Compact and efficient derived models: Based on our baseline and the proposed binary purification pipeline, we set the number of output channels in the initial convolutional layer as 32. A bridge binary convolutional layer is then connected, followed by the blocks with group convolution as shown in Fig. 3 (also see the appendix) and the non-linear modules. Finally, a layer followed by a soft-max layer is applied to do the classification. In order observe the effect of group convolution, we elaborate different computational budgets for BNNs (calculated in FLOPs as stated in Fig. following Liu et al. (2018) and Rastegari et al. (2016) by varying width and depth.
Replacing XNOR-Count when ReLU is used: In Fig. (b), activation values before Sign are either O positive if ReLU is applied at the end of the previous block. Then, all the Os would be assigned to the lower binary value with the Sign function, eliminating the effect of ReLU in the residual branch. In this case, we have designed particular bit operations, which is illustrated in the appendix.
All the models are simply trained using Adam optimizer (Kingma & Ba 2014) for 60 epochs from scratch with starting learning rate 0.01, which is decayed in a multi-step way (at epoch 45 and 55, with the decaying rate 0.1) for the baseline, and cosine shape scheme (towards 0 at the end of training) for the derived models. However we note there is no big difference between this two ways. The weight decay is set to 0 for all the binary convolutional layers. We use STE (Hubara et al. 2016) for backward propagation with value clipping in range (-1.2, 1.2) for both weights and activations. The Pytorch library (Paszke et al. 2019) is applied for implementation. Finally, following Bi-Real Net, we use the large-scale challenging ImageNet benchmark (ILSVRC 2012, Deng et al. 2009)) in experiments. A more detailed description for the implementation can be found in the appendix.
3.1 ABLATION ON FTBNN BASELINE
The most similar one of the proposed baseline is the Bi-Real Net structure. Where, the authors used a 2-stage training scheme get good initialization for BNN, magnitude-aware gradient approximation was also considered during weights updating. Finally, the trained scaling factors are absorbed into the BN layers. The whole training procedure was carefully tuned in order to fully exploit the model's capacity. It can be seen in Table 3 that when applying our naive training scheme, the Bi-Real Net structure (Fig. (a)) deleteriously degraded (from 56.4% to 52.3% on T accuracy). The main deficiency of this structure is the lack of proper non-linearity introduction, which is demonstrated to be indispensable for BNNs. In addition, our structure without any non-linear module performs slightly better than the Bi-Real Net structure (52.5% vs. 52.3%), demonstrating the benefits of weaker scaling mechanism from the BN layers.
The following experiments then focus the effectiveness of different con-linear modules. From Table we can find that the ReLU function is still preferred with a stronger non-linear property, that is able to bring in more discriminative power For example, only inserting 4 ReL.s can outperforms

---

Under review as a conference paper at ICLR 2021
Table 2: Binary convolutions in FTBNN baseline based on ResNer-18 structure. BConvs-y indicates yth block in the æth stage, where the size and number of filters are also given. The learned coefficients of FPReLU after each block are listed in the form of mean [minimum maximumj.
Layer
Learned coefficients
Layer
Learned coefficients
Negative
Positive
Negalive
Positive
BConv 1-1 3x3, 64
1.82 [0.48 4.63]
[0.63 2.19
BConv3.1 3x3, 256
0.84 [0.47 2.67] 0.95 [0.61 1.84]
BConv1-2 3x3, 64
0.95 [0.00 2.69] 0.72 [0.34 1.72]
BConv3-2
3x3, 256
0.86 [0.44 2.85]
0.79 [0.46 1.51]
BConv1-3 3x3, 64
0.87 [0.33 4.49]
0.75 [0.26 1.40]
BConv3-3
3x3,256
0.76
[0.04 1.97]
0.78 [0.28 2.50]
BConv 1-4 3x3, 64
ReLU
BConv3-4 3x3, 256
ReLU
BConv2-1 3x3, 128
0.91 (0,54 0.93 [0.56 2.04]
BConv4-1 312,
0,96
[0.54 2.86] 0.90 [0.14
BConv2-2 3x3, 128
0,93 (0.45 3.16] 0.83 [0.54 1.70]
BConv4-2 3x3.512
1.04
[0.44 4.66] 0.67 [0.00 1.44]
BConv2-3 3x3, 128
0.81 [0.00 1.82] 0.86 (0.46 1.69]
BConv4-3 3x3,
0.68 (0.23 2.80]
0.95 [0.23 3.51]
BConv2-4 3x3, 128
ReLU
BConv4-4 3x3,512
ReLU
Table 3: Ablation study for FTBNN baseline based on the ResNet 18 architecture.
Model
Top-1 (%)
Top-5 (%)
Bi-Real Net reported hy Liu et al. 2018
56.4
79.5
Bi-Real Net with our navve training scheme
52.3
75.9
No explicit non-linear modules
52.5
76.0
All configured with PReLU
58.9
81.1
All configured with FPReLU
59.3
81.3
All configured with ReLU (1r=0.00015, diverge Ir too big)
45.9
70.5
Partially configured with ReI.U
59.6
81.4
Partially configured with ReLU & PReLU in others
59.7
81.6
Baseline (Partially configured with ReI.U & FPReI.U in others)
60.2
82.0
Full-precision of ResNet-18 88 (our implementation)
69.8
89.2
the one with all blocks connected with PReLU by 0.7% in Top-1 accuracy. Note that ReLU is a special case of PReLU, indicating ReLU is already a decent local optima for PReLU by giving strong non-linearity, making the optimization easier. At the same time, if the usage of ReLU is limitless, the BNN would also be prone to divergence, showing that unlike the full-precision counterparts, binarized architectures are essentially vulnerable, an abuse of ReLU which causes severe information collapse in the feature space should be avoided.
A straightforward but effective way control the non-linearity, so that model's 's discriminative power can be maximized, and the stability is also guaranteed, is to control the number of ReLUs and additionally introduce weaker non-linear functions. Here, we show that compared with PReLU, the proposed FPReLU is an ideal solution to meet this requirement. As shown in Fig. 3 FPReLU starts from an identity function and gradually evolves during training to compensate the non-linearity in both sides of activations. Table 2 lists the ultimately learnt coefficients of the FPReLU functions. These values for both side of the functions vary around with a fairly wide range, some even approach to 0 or a big value (i.e,, 5), meaning that FPReLU is competent to tune the network into balanced point automatically, since both slops for negative and positive activations are considered.
3.2 THE DERIVED MODELS
In this section, we evaluate the effectiveness of the derived models with binary purification by adjusting the number of groups. Table 3 left shows the influences of group convlutions on binary- purified BNNs with varying settings of depth and group numbers, by constraining the computation costs similar. It can be found that the simplest derived models without using group convolutions (i.e., G 1) can surpass the proposed FTBNN baseline by large margin (a 1.7% improvement on Top-1 accuracy with computational budget almost halved based on ResNet 18), which strongly demonstrates the effectivenss of the proposed binary purification strategy. It also shows that there is no influence of the group execution for relatively shallow binary networks under different budget constraints. As going deeper, the group convolution begin to diverge the corresponding models with notable characteristics. On one hand, when observing the training accuracy, increasing the number of groups can obviously introduce a better fitting, indicating an increasing representational capacity of the BNN. The phenomenon can be found where the layer 34 regardless of the model complexity.
7

---

Under review as a conference paper at ICLR 2021
Table 4: Left: Experimental results of the derived structures with different group and budget configurations, with T accuracy (in %) recorded, where the widths of networks are adjusted meet the budget requirements. For clearer illustration, we also give the number of parameters for each model: Params number of floating point paramters + number of binary parameters / 32. Right: Comparison with state-of-theearts (W/A) means how many bits used in the weights (W) and activations (A), (1/1) Xn indicates n branches or bases individual models were adopted to expand network capacity for BNN. Our method with * indicates knowledge distillation and multi-step
training used (refer to appendix for detail).
Budgel (in MD
Layer Group
FLOPs
BOPs
Params
Top-1 (Train)
Top-1 (Val)
Mothod
Bitwidth Budget Top- (W/A)
(in M)
(in M)
(in M)
(in M) (in %)
18
19.0
4516
1.66
61.0
61.9
TTQ
et 2017
2/32
18
3
23.1
4580
2.06
61.6
61.9
BWN
Resicpar oral
2016
1/32
60.8
18
5
25.8
4301
2.28
62.2
61.9
RQ ST
ALQ Care 2020)
Lowzzs & 2019
4/4
2/2
NIA
62.5
66.4
~00
18
28.7
3988
2.55
61.5
61.1
1/8
34
19.3
1.32
58.8
61.0
IWWGQ CCl al 2017
1/2
59.6
34
23.8
4005
1.59
60.6
62.0
14
27.0
3829
1.79
61.8
62.5
Net (Lan 2011
(17J)S5
65.0
34
8
30.1
3577
1.97
61.9
62.0
Group-Ser Zinuin (ill 2019
CBCN eL 20719
(11)x4
NIA
66.3
18
21.1
7399
2.42
66.1
64.8
BNN Enserbos
(111)66
61.0
18
26.2
7285
2.88
66.6
64.5
XNOR-Ner Ner [Radiggan cinni 2011
171
51.2
18
5
29.9
7031
3.22
67.2
64.2
Bi-Real Net- (Liu
F/
56.4
3.60
66.4
63.4
XNOR-NON Netert
I/1
$7.1
34
21.6
6994
1.99
63.5
64.3
CI-BCNN (Wang
2019
1/1
161
59.9
34
27.4
6639
2.31
65.8
65.3
BONN (urera 2019
1/1
59.3
135
34
31.5
6412
2.56
66.8
65.5
IR-Ner Qf 2020
1/1
58.1
34
36,0
6090
2.83
67.3
65.0
FTBNN Raseliee (oirs)
FTBNN Baseline (ours)
1/1
63.2
44
22.1
6847
2.07
62.4
63.3
Liu
171
44
28.5
6512
2.33
65.3
64.8
Binury MobileNer |Phin el 2020
1/1
154
60.9
44
32,9
6251
2.55
66.4
65.3
Real-to-BB [Martheez et
44
8
37,5
5947
2.76
66.9
65.3
FTENN derivea (ourS)
1/1
132
65.5
On the other hand, the positive effect for deeper networks can also activate better accuracy on the validation set, meaning that an increase in discriminative power are also accompanied. All the experiments deeper BNNs with Group 5 show clear accuracy boost than those without group execution (Group = 1). E.g., from 61.0% to 62.5% for Layer 34, and from 63.3% to 65.3% for Layer= 44.
However, an increasing representational capacity may also lead to a higher risk dropping in generability. For some cases, the accuracy validation set is higher than that reported the training set, due to the regularization nature of binary operations. With group execution, such effect of binarization is gradually weakened, leading to overfitting issues.
3.3 COMPARISON WITH STATE-OF-THE-ARTSS
Here we compare the proposed FTBNN with existing state-of-the-aar methods as illustrated in the right side of Table including the low-bit quantization methods, multiple binarizations methods, other binary frameworks based on the ResNet-18 18 architecture, etc. Interestingly, we find that some of our derived models are able achieve superior comparable performance when compared with the methods using multiple bases or branches, while with far less computational overhead, again showing the effectiveness and efficiency the proposed method.
4 PERSPECTIVES
Here, we raise some possible insights from this work: a) To further enhance the model's discriminative ability through non-linearity, we wonder is it the best way to combine ReLU and FPReLU? Is it affected by other elements, such as depth, width and block orders? b) To what extent do the binary purification? As still have much proportion of real-valued operations in the derived models. c) As shown in our experiments on BNN, group convolution accompanies with overfitting. How to alleviate overfitting for BNNs should also be considered in future works.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Milad Alizadeh, Javier Mannndez--Maquués, Nicholas D Lane, and Yarin Gal. An empirical study of binary neural networks' optimisation. In International Conference on Learning Representations, 2018.
Joseph Bethge, Haojin Yang. Marvin Bornstein, and Christoph Meinel. Binarydensenet: Developing an architecture for binary neural networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Joseph Bethge, Christian Bartz, Haojin Yang, Ying Chen, and Christoph Meinel. Meliusnet: Can binary neural networks achieve mobilenet-level accuracy? arXiv preprint arXiv:2001.05936, 2020.
Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++: Improved binary neural networks. British Machine Vision Conference, 2019.
Adrian Bulat, Georgios Tzimiropoulos, Jean Kossaifi, and Maja Pantic. Improved training of binary networks for human pose estimation and image recognition. arXiv preprint urXiv:1904.000 2019.
Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. Bats: Binary architecture search. Proceedings of the European Conference on Computer Vision, 2020.
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 5918-5926, 2017.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IFEE Conference on Computer Vision and pattern recognition, pp. 248-244822 225. Icce, 2009.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprchensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Julian Faraone, Nicholas Fraser, Michaela Blott. and Philip HW Leong. Syq: Learning symmetric quantization for efficient deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4300-4309, 2018
Jiaxin Gu, Junhe Zhao, Xiaolong Jiang, Baochang Zhang, Jianzhuang Liu, Guodong Guo, and Rongrong Ji. Bayesian optimized 1-bit cnns. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4909-4917, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Xiangyu He, Zitao Mo, Ke Cheng, Weixiang Xu, Qinghao Hu, Peisong Wang. Qingshan Liu, and Jian Cheng. Proxybnn: Learning binarized neural networks via proxy matrices. In Proceedings of the European Conference on Computer Vision, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Ca-y,, and Yoshua Bengio. Binarized neural networks. In Advances in Neural Information Processing Systems, pp. 4107-4115 2016.

---

Under review as a conference paper at ICLR 2021
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. volume 37 Proceedings of Machine Learning Research, pp. 448 456, Lille, France, 07-09 Jul 2015. PMLR. CRLLhttp://pxoceediin mlr press. 237/ioffe15.htr
Dahyun Kim, Kunal Pratap Singh, and Jonghyun Choi. Learning architectures for binary networks. Proceedings the European Conference on Computer Vision, 2020a.
Hyungjun Kim, Kyungsu Kim, Jinseok Kim, and Jae-Joon Kim. Binaryduo: Reducing gradient
mismatch in binary activation network by coupling binary activations. Conference on Learning Representations, 2020b.
In International
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69800 6.800, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097- 2012.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http::ttp/:/att
lecun. wwtp///bb////sst...//////
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances in Neural Information Processing Systems, pp. 345-353, 2017.
Chunlei Liu, Wenrui Ding, Xin Xia, Baochang Zhang, Jiaxin Gu, Jianzhuang Liu, Rongrong Ji, and David Doermann. Circulant binary convolutional networks: Enhancing the performance of 1-bit denns with circulant back propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2691-2699 2019.
Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen. Deep learning for generic object detection: A survey. International Journal of Computer Vision, 128(2):261-311 2020a.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang. Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European Conference on Computer Vision, pp. 722-737,
2018.
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. Proceedings of the European Conference on Computer Vision, 2020b.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Relaxed quantization for discretized neural networks. In International Conference on Learning Representations, 2019.
Brais Martinez, Jing Yang. Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binary convolutions. In International Conference on Learning Representations, 2020.
Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri
Terzopoulos. Image segmentation using deep learning: A survey. arXiv preprint arXiv:2001.05566, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems, pp- 6026-80337 2019.
Hai Phan, Zechum Liu, Dang Huynh, Marios Savvides. Kwang- Ting Cheng, and Zhiqiang Shen. Binarizing mobilenet via evolution- searching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 13420-13429, 2020.
10

---

Under review as a conference paper at ICLR 2021
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2250-
2259, 2020.
Zhongnan Qu, Zimu Zhou, Yun Cheng, and Lothar Thiele. Adaptive loss-aware quantization for multi-bit networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7988-7997, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Proceedings of the European Conference Computer Vision, pp. 525-542. Springer, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chies Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Wei Tang, Gang Hua, and Liang Wang- How train compact binary neural network with high accuracy? In Proceedings of the AAAI Conference on Artificial Intelligence, 2017.
Ziwei Wang. Jiwen Lu, Chenxin Tao, Jie Zhou, and Qi Tian. Learning channel-wise interactions for binary convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 568-577, 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492-1500, 2017.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference on Computer Vision, pp. 365-382, 2018.
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. In International Conference Learning Representations, 2017.
Shilin Zhu, Xin Dong, and Hao Su. Binary ensemble neural network: More bits per network or more networks per bit? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4923-4932, 2019.
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Linggiao Liu, and Ian Reid. Structured binary neural networks for accurate image classification and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 413 113-422, 2019.

---

Under review as a conference paper at ICLR 2021
A
MORE STRUCTURE INFORMATION
Fig.9 gives more structure information of FTBNN and the derived models.
(c)
X
X
X
Weight (7x7)
Weight (3x3)
Copy
BN
PReLU
Sign
ReLU
BN
Binary weight (3x3)
(Group ceil(w / c))
X
Initial Conv block (derived)
MaxPooo (stride=2)
BN
X_ input image
BN
X output of Initial Conv block
Initial black (baseline)
Addition
oulput Bridge Conv block with arbitrary width
Bridge Conv block (derived)
(d)
(e)
Copy
Sign
Sign
Binary weight
Binary weight (group convolution)
AvgPool stride=2
(group convolution)
BN
BN
Addition
Additonn
Normel block (derived)
Reduction block (derived)
Figure 4: (a): Initial convolutional block for FTBNN baseline with a 7x7 convolutional layer, the output of which has 64 channels. (b): Initial convolutional block for the derived models with 3x3 convolutional layer, the number of channels the output of which is halved compared with the baseline. (c): Bridge convolutional block for the derived models, the number of channels in the output of which can arbitrary. e and w represent the channel numbers. derived models. (e): Reduction block for the derived models (in our experiments, we set the actual group number to in the reduction block for settings Group I and Group 2 in Section 3.2)
(d): Normal block for
B REPLACING XNOR- Count OPERATIONS IN FTBNN AS NEEDED
Suppose we are using the following Sign function to do binarization when doploying our models:
T. e, > 0 F. U, <0'
(v.)
where U, and Us represents the real-valued and binarized values (can be either of weights or activations) respectively. As we mentioned in Section the input of a basic block will be truncated such that the values of which either 0 or positive, if ReLU is connected after the previous block.
12

---

Under review as a conference paper at ICLR 2021
Supposing the binarized weights in the current block is ws before truncation 2, = |-3, -2, 1.5, 2, -1.2], and truncated to z, = [0,0. 0, 1.5, 2, 0] after ReLU.
[T. and the input activations
According to the Sign function, the activations will then be binarized to 2 F.T.T.F..
ANOR-Coum operation is used, the result of binary convolution is (the popcount function counts the bits of T): To popcount(17 xnor wh))
[F xnor [T,
Dopcount[F,77T,,)
2.
The result is the same as if the ReLU is not used, because without ReLU, the original input activations will still be binarized 2₁ [F, F. T, F..
Actually, our goal is to make the model trained in PyTorch be deployed using bit operations. Thus, the output of convolution in PyTorch should be the same when using bit operations. Noting the definition of torch.sign() in Appendix is:
{
if x < 0,
torch.sign(1)
if 0,
(4)
if
In the above case, the process of convolution in PyTorch is:
Step 1, based on Eq. and Appendix with the presence of ReLU, v, will be binarized to: xf [0, 1, 0);
Step 2, convolution between o and w (+ is the convolution function):
w [1, 1,, 1, x x wff 0.
(5)
Obviously, the output from Eq. 3 does not match the output x.. To match xg when deploying with bit operations, the following calculation:
To = popcount( (r and wa) popcount(13, and !wb).
(6)
Therefore, the output of bit operations is:
popcount([F) F, F.T.T.K F] and [T, T.F,F,,FTT). - (F F,F,T,F and [FT,F.T,FF) popcount( ([,, F, popcount( F. F))
= 0 o
In addition, Eq. may double the number BOPs than the original binary convolution which only uses XNOR-Cowne We have considered that when calculating the computational budgets for our models using doubled BOPs when ReLU used.
C EXPERIMENTAL CONFIGURATION FOR TRAINING
Data augmentation: The large-scale ImageNet dataset (ILSVRC 2012, Deng et (2009) used in our experiments as the benchmark, consisting of 1.28 million images for training and 50,000 images for validation. We adopt the standard data augmentation scheme as in He et al. (2016) during training, including random scale augmentation, random crop to 224x222 random horizontal flip with probability 0.5 and per-pixt per-pixel mean value subtraction. During testing, we resize the image such that the shorter side is 256 while keeping the original aspect ratio, crop 224x224 patch in the center, and apply the same per-pixel value subtraction as the training set.
Multi-step training: This training scheme is only used to enhance the proposed FTBNN baseline, demonstrating that the baseline is essentially orthogonal to other training insights for BNNs (for other cases in experiments, the naive training scheme with 60 epochs mentioned in Section
13

---

Under review as a conference paper at ICLR 2021
0.3
031
0.45
10 EEEERR satE
0.7
0.7
0.30
0.88
0.22
0.5
04
Triiriing
Tranning
03
0.8
EPOCH
Groupe1
Groupi3
Groupe5
Gm.pp8
EPOC1
Figure 5: Left: Training curves for Layer= = 18 with computational budget ~90. Right: Training curves for Layer 34 with computational budget 135M. Group execution has a positive effect for deeper BNN structures as discussed in Section 3.2
applied). Following Martinez et al. 2020 adopt the following teacher-student knowledge distillation with attention mapping for multi-step training. Specifically, we firstly train a full- precision model by keeping the weights real-valued and Tanh function instead of Sign function for activations. Then, a two-stage training scheme is used. In stage 1, we adopt Sign function for activations, while keeping the weights real-valued. In stage both weights and activations are binarized with the Sign function. In both stages, we use the output of the original ResNet- 18 model as the soft labels when calculating cross-entropy loss.
D TRAINING CURVES IN GROUP EXECUTION
The training curves for FTBNN with group execution are shown in Fig.
E CORE CODE USED IN OUR IMPLEMENTATION WITH PYTORCH
import torch import torch. nn as nn import torch nn. functional as
#*##8#
Binarization for activations #
class Sign ( nn. Module):
def -.init__ bound=1.2):
super (Sign self). -nni... self. bound bound
def forward(self x):
out torch clamp -self. bound. self bound) out forward torch sigch.sign(x y out.forward ()tach + out out detach () return
######
Binarization for weights and binary convolution
class BConv nn. Module)
def inint__ (self. in.chn, cut. kernel.size-;, stride=l, padding=l, groups=1, bound=1.2):
super (BConv .init.. self. stride = stride self, padding padding
14

---

Under review as a conference paper at ICLR 2021
self. groups groups self. bound bound
shape (out_chn, in_chn groups,
groups,, kernel-size, kernel_size)
self. weights = mn. Parameter (torch. (e torch. init xavier (self. weights, gain=2.0)
def (self, x):
weipped.weights = torch clamp (self. weights.
-self. bound, self bound)
binary.weighs.noograaa
torch sign self. weights)
binary.weights binary weights no grad.detach(
\
- clipped clipped_weightt weights detach ()
conv2d(( binary weights stride=self. stride. padding=self, padding. groups=self groups) .ropsps
return out
#*#0** FPReLU #RR###
class FPReLU nn. Module):
def -sel chn):
super self). self)...init... O
self I_weights nn Parameter (torch, ones (1, chn, 1, 1)) self r_wiights nn Parameter ( (torch... ones (1, chn, 1, 1))
def forwf x):
= torch, where (x > 0, x self rr.. weights, x self Iweights) return
15


Reviews:
[{"review_id": "rXEN3-UyRhI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---paper summary---:  \n\nThis paper proposes to improve the BNN’s discriminative ability by introducing additional non-linearities. In addition, the paper exploits the group convolution to enable a wider network, which can strengthen BNN’s representational ability, while keeping the total overhead unchanged. \n\t\t\n\n---Pros---:  \n\nThis paper introduces some practical methods to improve the performance of BNNs. In particular, the additional FPReLU is convincing. Moreover, the paper shows that grouped convolutions can be applied to wider BNNs to increase the representational capability while keeping the same complexity.\n\n---Cons---:\n\n1:  This paper is incremental with limited new technical insights. \n\na) Adding the nonlinearity can improve the representational capability of BNNs has been extensively explored in the literature. Specifically, ReActNet [Liu et al. 2020b] inserts additional RPReLU after each binary convolution to increase the non-linearity;  [A1; Martinez et al., 2020; Tang et al. 2017]  add PReLU as the nonlinearity;  Group-Net [Zhuang el al. 2019] and XNOR-Net [Rastegari et al., 2016] argue that adding additional ReLU is important to BNNs performance.  \nb) Varying width and/or depth has been studied in previous fixed-point quantization/BNNs literature, especially in NAS-based methods [Bulat et al. 2020;  A2]. The original idea comes from EfficientNet.        \nc)  Some other tricks such as replacing the 1x1 downsampling shortcut with pooling have been widely used in the community.\n\n2:   Some arguments need theoretical proofs. For example, the authors argue that “despite the big quantization error made by quantization, the binary model can achieve much higher accuracy than the real-valued model with no quantization error”. In other words, minimizing the quantization error can hinder the discriminative ability of BNNs, which is the main point of this paper. This observation is interesting, but needs further theorems to further explore whether the quantization error has some relations with the predicted accuracy under some constraints. If zero quantization error cannot lead to a good result, then what should be the best trade-off? I encourage the authors to further explore this issue. At the current stage, it is far from enough.\n\n3: The experimental results in Table 4 may have mistakes. The paper claims that BOPs are converted to equivalent FLOPs with a factor of $1/64$. However, why do smaller BOPs correspond to larger FLOPs? \n\n4:  The necessary “AND-count” operations may have technical issues. The AND-Count with values binarized to {0,1} should be equivalent to XNOR-Count with the values binarized to {-1, 1}, with a scalar difference. The authors can derive by themselves to verify this. However, the formulations in Eq. (3) and Eq. (4) are not equivalent if both values are binarized to {-1, 1}.\n\n5: More experimental results on deeper networks (e.g., ResNet-50, -101) on ImageNet are needed to justify the effectiveness of the method.  In addition, the comparisons with RELU [Zhuang el al. 2019; Rastegari et al., 2016], PReLU [A1; Martinez et al., 2020; Tang et al. 2017],  RPReLU [Liu et al. 2020b] (optional) should be included.\n\n6: Some typos. For example, “inheriting exiting advanced structures” → “inheriting existing advanced structures”; “Base on this consideration” → “Based on this consideration”.\n\nReferences: \n\n[A1]: Bayesian Optimized 1-Bit CNNs, in ICCV2019\n\n[A2]: Joint Neural Architecture Search and Quantization, arXiv:1811.09426v1\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "pDjOChx-dnd", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The author proposed an improved binary neural network (BNN) model.  FPReLU which increases the discriminative ability of BNN and group convolution make the proposed BNN architecture achieve better accuracy than prior works under similar operation budget condition. Other techniques like knowledge distillation can further increase the model performance.\n\n4-5 % top-1 accuracy improvement compared to prior models is an impressive result. I think reducing the burden of floating-point operations by editing first convolution layer and downsampling shortcuts is another good point, because the computation of uncompressed first convolution layer is easy to overlooked, but can be the bottleneck in the case of end-to-end inference.\n\nThe paper was clearly well-written, but I still have a concern or questions about the parts described below. I hope the authors to solve my concerns and questions so that my evaluation would be changed.\n\n1. According to the sign function in Appendix E, its output value is either 0 or 1 if its input is from ReLU and is either -1 or 1 if its input is from PReLU, FPReLU or not passed through a non-linear function. According to [1],  better performance can be achieved by using [0,1] activation in BNN, so my major concern is that the accuracy improvement of Bi-Real Net V2 compared to Bi-Real Net may come from the fact that both [0,1] and [-1,1] activation are used in a network.  Note that 'Baseline' and 'Partially configured~' models in Table 3 use both [-1,1] and [0,1] activation.    \n2. (-1.2,1.2) clipping can seem to be unfamiliar, because usually (-1.0, 1.0) range is used for BNN. I have the question how that range was decided.\n3. One of the factor which can decide the model accuracy is the number (or total bits) of parameters, so I think the performance comparison among BNN models will be more clear if the authors add that information of each BNN models.\n\nMinor comment:\n\nIn Appendix C, it is stated that original ResNet-18 model is used as a teacher model in all stages, which is different from [2] where the model trained at the previous stage is used as a teacher. This is the difference between using a teacher which has a good classification ability and using a teacher which resembles the student. I question if the authors had any reason to choose the former method and how two training cases affect to BNN performance.\n\nReference\n\n[1] Peisong Wang, et al. Sparsity-Inducing Binarized Neural Networks. AAAI, 2020.\n[2] Brais Martinez, et al. Training binary neural networks with real-to-binary convolutions. ICLR, 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "d34oPIiXv0c", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an improvement of Binary Neural Networks from the work of Bi-RealNet (Liu et al. 2018) by utilizing the linearity of modules. The method removes the necessity of scaling factors and used a novel non-linear layer named Fully Parametric ReLU (FPReLU) to increase the capability of BNNs. In addition, similar to recent works in BNNs, the paper also utilize group convolution layers to reduce the number of parameters and save the computation's cost. The experiments are tested with image classification tasks on the large-scale dataset of ImageNet. The paper is well written and straightforward. \n \nAlthough the results in ImageNet are promising and the computation is less, It raises concerns about the novelty of the work. The method used alike structure in Bi-RealNet with a bit of modification and group convolution which are already used in previous works of Liu et al. ECCV 2018 and Phan et al. CVPR 2020. \n\nThe paper used novel FPReLU but missed some comparisons with a similar idea work using PReLU[1,2].\n\nFor results in ImageNet, it would be nice if we can do more comparison and analysis with the state-of-the-art recent work [3]. \n \nIn conclusion, with current manuscripts, the paper is not sufficient enough to present at the conference.\n\n[1].  Tang et al. How to Train a Compact Binary Neural Network with High Accuracy? AAAI 2017.\n[2]. Phan et al. MoBiNet: A Mobile Binary Network for Image Classification, WACV 2020.\n[3]. Liu et al. ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions, ECCV 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "a1eY0LfjIyY", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "summary:\nSummary:\nThe authors note that the poor performance of BNN may be due to the disappearing of the amount of non-linearity in the input-output transformation. They propose a new Bi-Real architecture where they replace the scaling factor with extra non-linearities. The performance of the model is compared with the original Bi-Real Net and further possible improved version of the proposed method. \n\nStrengths:\nAnalyzing the role of non-linearity is a very nice idea. Studying BNNs is a clean setup for understanding the well-known predictive power of NN, even in real-valued settings. The computational gains associated with the purification of the convolutional layers look impressive.  \n\nWeaknesses:\nThe comparison is between binarized but rescaled networks and binary networks without rescaling but added non-linearity. It is not clear if the good performance of the proposed architecture is due to i) the BN step making the scaling redundant or ii) the additional nonlinearities.  In some sense, noting that the inclusion of non-linear transformations can boost the predictive power of a network is not surprising. The extra computational cost associated with the introduction of non-linear activation is not fully discussed. The comparison does not include the case where non-linearities are added to the rescaled version (the one on the left of Figure 1). \n\t\nQuestions:\n- what happens if the binarized part of the network (between the FP first and last layer) is completely removed? Often, a two-layer network may achieve good accuracy on simple tasks such as MNIST-digit recognition.\n- how does the introduction of extra non-linearity affect the computational budget in the proposed method? \n- is there a figure presenting the performance of the reduced budget model shown in figure 3?\n- is the benefit associated with extra non-linearities be expected to extend to other architecture than Bi-Real?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: ## Description 

The paper proposes an improvement to binary neural networks with real-valued skip connections between pre-activations, by introducing more flexible learnable non-linearities on the real-valued connections. The parametric non-linearity is actually linear at initialization, which makes the training easier at the beginning. Due to learnable parameters it eventually adjusts to a more complex one, able to refine the accuracy. I think this idea is a good finding.

## Review Process and Decision
The reviewers initially gave low ratings to the paper, indicating that the contribution is incremental and not fully clearly presented. 
There was no detailed discussion with the authors, since the author's response and the rebuttal revision came in the very end of the discussion period. In the subsequent discussion phase the reviewer board has not indicated any major changes to the initial reviews/ranking. The AC checked the paper and supports rejection.

## Details

The authors are encouraged to improve the paper carefully addressing points proposed by reviewers.

I think the argumentation of the paper should be improved. Some explanations are intuitive, but operating with fuzzy notions and may in fact be incorrect or irrelevant. The paper should be made more precise, based on verifiable arguments.

I think the following is crucial and not made clear in the paper:
The non-linearities inserted before the sign function *do not affect the result of sign*. They indeed affect only the residual connections. Furthermore, the structure of residual connections should be fully clarified to reveal that there are complete real-valued paths all the way from the input to the network to its output, made of the residual connections with their own learnable parameters (and 1x1 convolutions) and (learnable) non-linearities and an intake from binary convolutions on the way. The learnable non-linearities can in principle improve performance just because the real-valued paths can learn better.

I paste below feedback by reviewers to author's response (I believe they would agree to share it with authors but did not find a suitable way of doing it):

## Response by R1:
I acknowledge that I read and appreciated the authors' answers to my questions. I think the idea of analyzing the role of non-linearities is nice and I tend to confirm my score. But I also agree with other reviewers that, as it is, the paper has some unclear parts and would not complain if it is rejected.

## Response by R3:
Thanks for your responses to answer my questions for the paper. I agree with the results of the proposed FBTN for improved Binary Neural Networks (BNNs). However, my concern about the novelty of using group convolution modules in BNNs has not been addressed. I think the paper is not sufficient enough to publish at the conference. So, I do not change my rating of the paper.

## Response by R4:
I maintained my rating when combining other reviews and responses to them, despite of their well response. It is still questionable whether FPReLU, one of the main contributions they claimed, actually improves the performance of BNN remarkably. In particular, this is supported by the fact that the performance of BNN on ResNet-34 which the techniques in this paper were applied does not show much difference from 'Real-to-Bin' model.
Decision: Reject