Training binary neural networks with real-to-binary convolutions
Authors: Brais Martinez
Jing Yang
Adrian Bulat
Georgios Tzimiropoulos
Venue:  (2020)
Keywords: binary networks

Abstract: This paper shows how to train binary networks to within a few percent points (~3-5%) of the full precision counterpart. We first show how to build a strong baseline, which already achieves state-of-the-art accuracy, by combining recently proposed advances and carefully adjusting the optimization procedure. Secondly, we show that by attempting to minimize the discrepancy between the output of the binary and the corresponding real-valued convolution, additional significant accuracy gains can be obtained. We materialize this idea in two complementary ways: (1) with a loss function, during training, by matching the spatial attention maps computed at the output of the binary and real-valued convolutions, and (2) in a data-driven manner, by using the real-valued activations, available during inference prior to the binarization process, for re-scaling the activations right after the binary convolution. Finally, we show that, when putting all of our improvements together, the proposed model beats the current state of the art by more than 5% top-1 accuracy on ImageNet and reduces the gap to its real-valued counterpart to less than 3% and 5% top-1 accuracy on CIFAR-100 and ImageNet respectively when using a ResNet-18 architecture. Code available at https://github.com/brais-martinez/real2binary

Published as a conference paper at ICLR 2020
TRAINING BINARY NEURAL NETWORKS WITH REAL- TO- TO-BINARY CONVOLUTIONS
Brais Martinez', Jiig Yang! 12,, Adrian Bulat! & Georgios Tzimiropoulos!?
Samsung AI Research Center, Cambridge, UK 2 Computer Vision Laboratory, The University of Nottingham, UK
(brais.a adrian. bulat, georgios. . samssmggggg.... com
ABSTRACT
This paper shows how train binary networks to within few percent points (~~ 3 5%) of the full precision counterpart. We first show how to build a strong baseline, which already achieves ofate-of-the-ar accuracy, by combining recently proposed advances and carefully adjusting the optimization procedure. Secondly. we show that by attempting to minimize the discrepancy between the output of the binary and the corresponding real-valued convolution, additional significant accuracy gains can be obtained. We materialize this idea in two complemen- tary ways: (I) with loss function, during training, by matching the spatial at- tention maps computed at the output of the binary and real-valued convolutions, and (2) in a data-driven manner, by using the real-valued activations, available during inference prior to the binarization process, for re-scaling the activations right after the binary convolution. Finally, we show that, when putting all of our improvements together, the proposed model beats the current state of the art by more than 5% top-1 accuracy ImageNet and reduces the gap its real- valued counterpart to less than 3% and 5% top-1 accuracy on CIFAR- --00 and ImageNet respectively when using ResNet-18 architecture. Code available https::/ com/ mara wtw.eiaattteeeeeelaaaaaaaa
INTRODUCTION
Following the introduction of the BinaryNeuralNet (BNN) algorithm (Courbariaux al., 2016 binary neural networks emerged as one of the most promising approaches for obtaining highly effi cient neural networks that can be deployed convolutions are appealing mainly for two reasons: (a) Model compression: if the weights of the network are stored as bits in 32-bit float, this implies reduction of 32x in memory usage. (b) Computational speed-up: computationally intensive floating-poin multiply and add operations are replaced by efficient xnor and pop-count operations. which have been shown provide practi- cal speed-ups of up to 58x on CPU (Rastegari et al. (016) and, as opposed to general low bit-width operations, are amenable to standard hardware. Despite these appealing properties, binary neural networks have been criticized as binarization typically results in large accuracy drops. Thus, their deployment in practical scenarios is uncommon. For example, on ImageNet classification, there is a 18% gap in top-1 accuracy between a ResNet-18 and its binary counterpart when binarized with XNOR-Net (Rastegari et al. 2016), which is the method of choice for neural network binarization.
devices with limited computational resources. Binary
But how far are we from training binary neural networks that are powerful enough to become viable alternative to real-valued networks? Our first contribution in this work is to take stock of recent advances binary neural networks and train very strong baseline which already results in state-of-thee -otte----the-rtt performance. Our second contribution is method for bridging most of the remain- ing gap, which boils down to minimizing the discrepancy between the output of the binary and the corresponding real-valued convolution. This idea is materialized in our work in two complemen- tary ways: Firstly, we use an attention matching strategy so that the real-valued network can more
Denotes equal contribution

---

Published as a conference paper at ICLR 2020
Binary block
BatchNorm
XW XE
GlobalAvgPool
Sign
Linear
1 x
XC
ReLU
181 x
)
Convolution
Linear
1 1X1
AM
Sigmoid
1 XC
KW XC
Activation
Real block (teacher)
The proposed real-to-binary block. The diagram shows how spatial attention maps computed from a teacher real-valued network are matched with the ones computed from the binary network. Supervision is injected at the end of each binary block. See also section 4.2 Right: The proposed data-driven channel re-scaling approach. The left-hand side branch corresponds to the standard binary convolution module. The right-hand side branch corresponds to the proposed gating function that computes the channel-scaling factors from output of the batch normalization. The factor controls the compression ratio on the gating function, and H, W and C' indicate the two spatial and the channel dimensions the activation tensors. See also section 4.3
Figure 1: Left:
closely guide the binary network during optimization. However, show that due to the architec- tural discrepancies between the real and the binary networks, a direct application of teacher-student produces sub-optimal performance. Instead, propose use sequence of teacher-student pairs that progressively bridges the architectural gap. Secondly, we further propose to use the real-valued activations of the binary network, available prior the binarization preceding convolution, com- pute scale factors that are used to re-scale the activations right after the application of the binary convolution. This in line with recent works which have shown that re-scaling the binary convo- lution output can result in large performance gains (Rastegari et al. 2016) Bulat & Tzimiropoulos, 2019). However, unlike prior work, we compute the scaling factors data-driven manner based on the real-valued activations of each layer prior to binarization, which results in superior performance.
Overall, we make the following contributions:
We construct a very strong baseline by combining some recent insights on training binary networks and by performing thorough experimentation find the most well-suited opti- mization techniques. We show that this baseline already achieves state-of-the-ar accuracy ImageNet, surpassing all previously published works on binary networks.
We propose real-to-binary attention matching: this entails that matching spatial attention maps computed at the output of the binary and real-valued convolutions is particularly suited for training binary neural networks (see Fig. leet and section We also devise an approach in which the architectural gap between real and binary networks is progressively bridged through a sequence of teacher-student pairs.
We propose a data-driven channel re-scaling: this entails using the real-valued activations of the binary network prior to their binarization to compute the scale factors used scale the activations produced right after the application of the binary convolution. See Fig. I right, and section 4.3
We show that combined contributions provide, for the first time, competitive results two standard datasets, achieving 76.2% top-1 performance on CIFAR-100 100 and 65.4% top-1 performance ImageNet when using ResNet-18 18 gap bellow 3% and 5% respectively compared to their full precision counterparts.
2

---

Published as a conference paper at ICLR 2020
2
RELATED WORK
While being pre-dated by other works on binary networks (Soudry al. 2014), the BNN algo- rithm (Courbariaux et al., 2016) established how to train networks with binary weights within the familiar back-propagation paradigm. The training method relies on real-valued copy of the net- work weights which is binarized during the forward pass, but is updated during back proppaaaatttionn ignoring the binarization step. Unfortunately, BNN resulted in a staggering 28% gap in top-1 accuracy compared to the full precision ResNet- 18 on ImageNet.
worth noting that binary networks do have a number of floating point operations. fact, the output of a binary convolution is not binary (values are integers resulting from the count). Also, in accordance to other low bit-width quantization methodologies, the first convolution (a costly 7 x kernel in ResNet), the fully connected layer and the batch normalization layers are all real-valued. In consequence, a line of research has focused on developing methodologies that add a fractional amount of real-valued operations in exchange for significant accuracy gains. For example, the sem- inal work of XNOR-Net (Rastegari et al.] 2016) proposed to add real-valued scaling factor to each output channel of a binary convolution, technique that has become standard for binary networks. Similarly, Bi-Real Net (Liu et al.] 2018) argued that skip connections are fundamental for binary networks and observed that the flow of full precision activations provided by the skip connections is interrupted by the binary downsample convolutions. This degrades the signal and make subsequent skip connections less effective. To alleviate this, they proposed making the downsample layers real valued, obtaining around 3% accuracy increase in exchange for a small increase in computational complexity.
Improving the optimization algorithm for binary networks has been another fundamental line of research. Examples include the use of smooth approximations of the gradient, the use of PReL.U Bulat et al. 2019), a two-stage training which binarizes the weights first and then the activations (ullat et al. 2019) and progressive quantization (Gong et al. 2019 Bulat et al., (201)). The work in WWan et al. 2019) proposed to learn channel correlations through reinforcement learning to better preserve the sign of a convolution output. A set of regularizers are added to the loss term in (Ding jet al., 2019) so as to control the range of values of the activations, and guarantee good gradient flow. Other optimization aspects, such the effect of gradient clipping or batch-norm momentum, were empirically tested in (Alizadeh al.,, 2019). In section |. we show how to combine many of the insights provided in these works with standard optimization techniques to obtain a very strong baseline that already achieves state-of-the-an accuracy.
While the aforementioned works either maintain the same computational cost, or increase it by a fractional amount, other research has focused instead on relaxing the problem constraints by in- creasing the number of binary operations by a large amount, typically factor of 2 to 8 times. Examples include ABC-Net (Lin et al. 2017), the structure approximation of (Zhuang et al. 2019), the circulant CNN of (Liu et al.] (2019). and the binary ensemble of (Zhu et al.) (201)). Note that the large increase of binary operations diminishes the efficiency claim that justifies the use of binary networks in first place. Furthermore, we will show that there is still lot of margin in order to bridge the accuracy gap prior to resorting to scaling up the network capacity]
The methodology proposed in this paper has some relations with prior work: our use of atten- tion matching as described in section 4.2 is somewhat related to the feature distillation approach of (Zhuang et al., 2018). However, (Zhuang et al., 2018) tries to match whole feature maps of the to oo-be-quantized network with the quantized feature maps of a real-valued network that is trained in parallel with the to-be-quantized network. Such approach is shown improve training of low-bitwidth quantized models but not binary networks. Notably, our approach based on matching attention maps is much simpler and shown be effective for the case of binary networks
Our data-driven channel re-scaling approach, described in section 4.3 is related to the channel re- scaling approach of XNOR-Net, and also that of (Xu & Cheung 2019 Bulat & Tzimiropoulos/ 2019), 1, which propose to learn the scale factors discriminatively through backpropagation. Contrary to (Xu& & Cheung 2019 Bulat & Tzimiropoulos 2019), our method is data-driven and avoids using
'There is also a large body of work focusing other low-bit quantization strategies, but review of these techniques goes beyond the scope of this section.
3

---

Published as a conference paper at ICLR 2020
fixed scale factors learnt during training. Contrary to XNOR-Net, our method discriminatively leams how to produce the data-driven scale factors so that they are optimal for the task hand.
3 BACKGROUND
This section reviews the binarization process proposed in (Courbariaux et al., 2 and its improved version from (Rastegari et al. (016). which is the method of choice for neural network binarization.
We denote by W E Roxexkx x and A Rexw,, xhin the weights and input features of a CNN layer, where o and с represent the number of output and input channels, k the width and height of kernel, and Win. and hin represent the spatial dimension of the input features A. In (Courbariaux (et al., 2016). both weights and activations are binarized using the sign function and then convolution performed as A * W x sign(AA) sign(W) where ( denotes the binary convolution, which can be implemented using bit-wise operations.
However, this direct binarization approach introduces high quantization error that leads to low accuracy. To alleviate this, XNOR-Net (Rastegari et al.] 2016) proposes to use real-valued scaling factors re-scale the output of the binary convolution
A W (Ai O sign(W)) KCa,
where denotes the element-wise multiplication, and IC are the weight and activation scaling factors, respectively, calculated in Rastegari et al. (2016) in an analytic manner. More recently, Bulat & Tzimiropoulos (2019) proposed fuse and K. into single factor T that is learned via backpropagation, resulting in further accuracy gains.
4 METHOD
This section firstly introduces strong baseline. Then, we present ways improve the ap- proximation Eq. [ Firstly, we use loss based on matching attention maps computed from the binary and real-valued network (see section 4.2p Secondly, we make the scaling factor function of the real-valued input activations A (see section 4.3)
4.1 BUILDING A STRONG BASELINE
Currently, almost all works on binary networks use XNOR-Net and BNN as baselines. In this sec- tion, show how construct strong baseline by incorporating insights and techniques described in recent works as well as standard optimization techniques. We show that our baseline already achieves state-of-the-att accuracy. We believe this is an important contribution towards understand- ing the true impact of proposed methodologies and towards assessing the true gap with real-valued networks. Following prior work in binary networks, we focus on the ResNet-18 architecture and apply the improvements listed below:
well-known that a modified ResNet block must be used to obtain optimal results for binary networks. We found the widely-used setting where the operations are ordered as BatchNorm - Binarization + BinaryConv + Activation to be the best. The skip connection the last operation of the block (Rastegari et al. 2016). Note that we use the sign function to binarize the activations. However, the BatchNorm layer includes an affine transformation and this ordering of the blocks allows its bias term act learnable binarization threshold.
Block structure:
Residual learning:
We used double skip connections, as proposed (Liu et al. 2018).
Activation: We used PReLU (He et al. 2015) it is known to facilitate the training of binary networks (Bulat et al. 2019).
Scaling factors: We used discriminatively learnt scaling factors via backpropagation in BBulat & Tzimiropoulos 2019).
Downsample layers: We used real-valued downsample layers (Liu al. 2018 We found the large accuracy boost to be consistent across our experiments (around 3 4% top improvement on ImageNet).
4

---

Published as a conference paper at ICLR 2020
We used the following training strategies to train our strong baseline:
When training binary networks, it is crucial to use a 2-stage optimization strat- egy (Bulat et al. 2019). In particular, we first train network using binary activations and real-valued weights, and then use the resulting model as initialization to train a network where both weights and activations are binarized.
Initialization:
Weight decay: Setting up weight decay carefully is surprisingly important. We use 1e- when training stage (binary activation and real weights network), and set it to 0 stage 2 (Bethge al.) 2019). Note that weights at stage 2 are either or 11, so applying an L2 regularization term to them does not make sense.
Data augmentation: For CIFAR-100 we use the standard random crop, horizontal flip and rotation (=15%). For ImageNet, we found that random cropping, flipping and colour jitter augmentation worked best. However, colour jitter is disabled for stage 2.
Mix-up: We found that mix-up (Zhang al. 2017) crucial for CIFAR-100, while it slightly hurts performance for ImageNet- this is due to the higher risk of overfitting on CIFAR-100 100.
Warm-up: We used warm-up for epochs during stage and no warm-up for stage 2.
Optimizer: We used Adam (Kingma & Ba 2014 with a stepwise scheduler. The learning rate is set to le- for stage and 2e 4 for stage 2. For CIFAR-100, we trained for 350 epochs, with steps at epochs 150, 250 and 320. For ImageNet, we train for 75 epochs, with steps at epochs 40, 60 and 70. Batch sizes are 256 for ImageNet and 128 for CIFAR-1000
4.2 REAL-TO-BINAN ATTENTION MATCHING
We make the reasonable assumption that if a binary network trained so that the output of each binary convolution more closely matches the output of real convolution in the corresponding layer of a real-valued network, then significant accuracy gains can be obtained. Notably, a similar as- sumption was made in (Rastegari et al.) 2016) where analytic scale factors were calculated so that the error between binary and real convolutions is minimized. Instead, and inspired by the attention transfer method of (Zagoruyko Komodakis 2017). we propose to enforce such a constraint via a loss term at the end of each convolutional block by comparing attention maps calculated from the binary and real-valued activations. Such supervisory signals provide the binary network with much- needed extra guidance. It is also well-known that backpropagation for binary networks is not as effective as for real-valued ones. By introducing such loss terms at the end of each block, gradients do not have to traverse the whole network and suffer degraded signal.
Assuming that attention matching is applied at a set of J transfer points within the network, the total loss can be expressed as:
J
Cast
1.
2
1
j=1
where Q² = Ei=1 | and A; is the i-th channel of activation map A. Moreover, at the end of the network, we apply standard logit matching loss (Hinton et al. 2015).
Progressive teacher-student: We observed that teacher and student having as similar architecture as possible is very important in our case. We thus train a sequence of teacher-student pairs that progressively bridges the differences between the real network and the binary network in small increments:
Step 1: the teacher is the real-valued network with the standard ResNet architecture. The student is another vealee network, but with the same architecture as the binary ResNet-18 (e.g. double skip connection, layer ordering, PReLU activations, etc). Furthermore, a soft binarization (a Tanh function) is applied to the activations instead the binarization (sign) function. In this way the network is still real-valued, but it behaves more closely network with binary activations.
Step 2: The network resulting from the previous step is used as the teacher. A network with binary activations and real-valued weights is used the student.
Step 3: The network resulting from step 2 is used as the teacher and the network with binary weights and binary activations is the student. In this stage, only logit matching is used.
5

---

Published as a conference paper at ICLR 2020
4.3 DATA-DRIVEN CHANNEL RE-SCALING
While the approach of the previous section provides better guidance for the training of binary net- works, the representation power of binary convolutions still limited, hindering its capacity to approximate the real-valued network. Here we describe how to boost the representation capability of binary neural network and yet incur in only a negligible increment on the number of operations.
Previous works have shown the effectiveness of re-scaling binary convolutions with the goal of better approximating real convolutions. XNOR-Net (Rastegari et al., 2016) proposed to compute these scale factors analytically while (Bulat & Tzimiropoulos 2019 Xu & Cheung| (2019) proposed to learn them discriminatively in an end-to-end manner, showing additional accuracy gains. For the latter case, during training. the optimization aims to find set of fixed scaling factors that minimize the average expected loss for the training set. We propose instead to go beyond this and obtain discriminatively traineed input-dependent scaling factors thus, at test time, these scaling factors will not be fixed but rather inferred from data.
Let us first recall what the signal flow is when going through binary block. The activations entering a binary block are actually real-valued. Batch normalization centers the activations, which are then binarized, losing large amount of information. Binary convolution, re-scaling and PReLU follow. We propose to use the full-precision activation signal, available prior to the large information loss incurred by the binarization operation, to predict the scaling factors used to re-scale the output of the binary convolution channel-wise Specifically, we propose to approximate the real convolution as follows:
A W= = (sign(A) + sig) o a O G(A:Wc).
where Wo are the parameters of the gating function G. Such function computes the scale factors used re-scale the output of the binary convolution, and uses the pre-convolution real-valued acti- vations as input. Fig. shows our implementation of function G. The design is inspired by Hu et al.) 20188) but
the gating function predict ahead rather than self-attention mechanism.
An optimal mechanism to modulate the output of the binary convolution clearly should not be the same for all examples as in Bulat & Tzimiropoulos 2019) or Xu & Cheung (2019). Note that in Rastegari et al. (2016) the computation of the scale factors depends on the input activations. However the analytic calculation is sub-optimal with respect to the task at hand. To circumvent the aforementioned problems, our method learns, via backpropagation for the task at hand, to predict the modulating factors using the real-valued input activations. By doing so, more than 1/3 of the remaining gap with the valued network is bridged.
4.4 COMPUTATIONAL COST ANALYSIS
Table details the computational cost of the different binary network methodologies. We differen- tiate between the number of binary and floating point operations, including operations such as skip connections, pooling layers, etc. It shows that our method leaves the number of binary operations constant, and that the number of FLOPs increases by only 1% of the total floating point operation count. This is assuming factor of 8, which is the one used in all of our experiments. To put this into perspective, the magnitude is similar to the operation increase incurred by the XNOR-Net with respect to its predecessor, BNN. Similarly, the double skip connections proposed in (Liu al. 2018) adds again a comparable amount of operations. Note however that in order to fully exploit the computational efficiency of binary convolutions during inference, specialized engine such as (Zhang et al. 2019 Yang et al. 2017) is required.
5 RESULTS
We present two main sets of experiments. We used ImageNet (Russakovsky et al.] 2015) a bench- mark to compare our method against other most widely used dataset to report results binary networks and, at the same time, allows to show for the first time that binary networks can perform competitively on a large-scale dataset. We further used CIFAR-100 100 (Krizhvskyy & Hinton 2009) conduct ablation studies (Sec. 5.2
approaches in Sec. [511 ImageNet the

---

Published as a conference paper at ICLR 2020
Method
BOPS
FLOPS
BNN (Courbariaux et al. 60
1.695x10
1.314x10
XNOR-Net (Rastegari et al.
2016)
1.695x10'
1.333x
Double Skip ((Liu et 2018
1.695x10'
x.351x10
Bi-Real (Liu et al. 2018
16676x109
x
Ours
1.676x109
x,564x10*
Full Precision
1.826x 10
Table 1: Breakdown of floating point and binary operations for variants of binary ResNet- 18.
5.1 COMPARISON WITH THE STATE-OF-THE-ARY
Table 2 shows a comparison between our method and relevant state-of-the-an methods, including low-bit quantization methods other than binary.
Vs. other binary networks: Our strong baseline already comfortably achieves state-of-the art results, surpassing the previously best-reported result by about 1% (Wang al. (019).. Our full method further improves over the state-of- oot by 5.5% top-1 accuracy. When comparing to
binary models that scale the capacity of the network (second set of results on Tab. 2 only (Zhuang et al. 2019) outperforms our method, surpassing it by 0.9% top-1 accuracy yet, this is achieved using 4 times the number of binary blocks.
Vs. real-valued networks: Our method reduces the performance gap with its real-valued counter- part 4% top-1 accuracy, 5% if we compare against a real-valued network trained with attention transfer.
Vs. other low-bit quantization: Table 2 also shows comparison to the state-of-the-arr for low-bit quantization methods (first set of results). It can be seen that our method surpasses the performance of all methods, except for TTQ (Zhu et al. 2017). which uses 2-bit weights, full full-precision activations and 1.5 the channel width at each layer.
5.2 ABLATION STUDIES
In order to conduct a more detailed ablation study we provide results on CIFAR-100. We thoroughly optimized R full precision network serve the real-valued baseline.
Teacher-Student effectiveness: We trained real-valued ResNet- 18 using ResNet-34 its teacher, yielding ~ 1% top-1 accuracy increase. Instead, our progressive teacher-student strategy yields 5% top-1 accuracy gain, showing that it is fundamental tool when training binary networks, and that its impact is much larger than for real-valued networks, where the baseline optimization is already healthier.
Performance gap to real-valued: We observe that, for CIFAR-100, we close the gap with real- valued networks to about 2% when comparing with the full full-precision ResNet-18, and to about 3% when optimized using teacher supervision. The gap is consistent to that on ImageNet in relative terms: 13% and 10% relative degradation on ImageNet and CIFAR- 100 respectively.
Binary vs real downsample: Our proposed method achieves similar performance increase irrespec- tive of whether binary or real-valued downsample layers are used, the improvement being 5.5% and 6.6% top-1 accuracy gain respectively. It is also interesting to note that the results the ablation study are consistent for all entries on both cases.
Scaling factors and attention matching: It is also noteworthy that the gating module is not effec- tive in the absence of attention matching (see SB+G entries). It seems clear from this result that both interconnected: the extra supervisory signal is necessary to properly guide the training, while the extra flexibility added through the gating mechanism boosts the capacity of the network to mimic the attention map.

---

Published as a conference paper at ICLR 2020
ImageNet
Method
Bitwidth (W/A)
Top-
Top-5
BWN (Rastegari al. 2016
1/32
60.8
83.0
TTQ
(Zuu et al. 2017
2/32
66.6
87.2
HWGQ (Cai et al. 2017
1/2
59.6
82.2
LQ-Net (Zhang et 2018
1/2
62.6
84.3
SYQ Faraone et al. 2018
1/2
55.4
78.6
DOREFA-Net Zhou et al.
2016
2/2
62.6
84.4
ABC-Net (Lin et al. (0017)
(1/1)x55
65.0
85.9
Circulant CNN (Liu el al.
2019)
(1/1)x4
61.4
82.8
Struct Appr Zhuang et al.
2019
(1/1)x4
64.2
85.6
Struct Appro (Zhuang et
(1/1)x4
66.3
86.6
Ensemble (Zhu et al. (2099)
(1/1)x6
61.0
BNN (Courbariaux et al. (2016)
1/1
42.2
69.2
XNOR-Net (Rastegari al.
2016
1/1
51.2
73.2
Trained Bin (uu & Cheung
2019
1/1
54.2
77.9
Bi-Real Net (Lau 2019)
2018)
1/1
56.4
79.5
CI-Net (Wang et al.
1/1
56.7
80.1
XNOR-Net++ (Bulat T Tzimiropoulos 2019)
1/1
57.1
79.9
CI-Net (Wang et al. 2019
1/1
59.9
84.2
Strong Baseline (ours)æ*
60.9
83.0
Real-to- Real-to-Biin (ours)**
1/1
65.4
86.2
Real valued
32/32
69.3
89.2
Real valued T-S
32/32
70.7
90.0
Table 2: Comparison with statc-of-the-att methods on ImageNet. indicates real-valued down-
sample. The second column indicates the number of bits used to represent weights and activations. Methods include low-bit quantization (upper section), and methods multiplying the capacity of the network (second section). For the latter case, the second column includes the multiplicative factor of the network capacity used.
Stage
Stage 2
Method
Top-1 / Top-5
Top-1
Top-5
Strong Baseline
69.3 88.7
68.0 88.3
SB Att Trans
72.2 / 99.3
71.1 / 90.1
SB Att Trans HKD
73.1
71.9 90.9
SB +
/
66.2 / 86.8
SB Progressive TS
73.8/913
72.3 / 89.8
Real-to-Bin
/ /2.0/
73.5 / 91.6
Strong Baseline**
72.1/89.9
69.6 89.2
SB Att Trans
74.3/
72.6 / 91.4
SB Trans + *KD**
75.4/92.2
73.9/91.
SB G**
72.0 89.8
70.9 / 89.3
SB + Progressive TS**
75.7 /92.1
74.6 91.8
Real-to-Bin"*
76.5 92.8
76.2 / 92.7
Full Prec (our impl.)
78.3
93.6
Full Prec TS (our impl.)
79.3
94.4
Table 3: Top-1 and Top-5 classification accuracy using ResNet-18 on CIFAR-100,
indicates
real-valued downsample layers. indicates that the gating function of Sec. 4.3 is used.
8

---

Published as a conference paper at ICLR 2020
6 CONCLUSION
In this work we showed how to train binary networks to within a few percent points of their real- valued counterpart, turning binary networks from hopeful research into compelling alternative to real-valued networks. We did so by training binary network to not only predict training labels, but also mimic the behaviour of real-valued networks. To this end, we devised a progressive attention matching strategy to drive optimization, and combined it with gating strategy for scaling the output of binary convolutions, increasing the representation power the convolutional block, The two strategies combine perfectly to boost the state-of-the- of-t-o---tth----- art of binary networks by 5.5 top-1 accuracy on ImageNet, the standard benchmark for binary networks.
REFERENCES
Milad Alizadeh, Javier Fernández-Marqués, Nicholas D. Lane, and Yarin Gal. An empirical study of binary neural networks' optimisation. In International Conference on Learning Representations, 2019.
Joseph Bethge, Haojin Yang, Marvin Bornstein, and Christoph Meinel. Back to simplicity: How to train accurate BNNs from scratch? arXiv preprint
08637, 2019.
Adrian Bulat and Georgios Tzimiropoulos. XNOR-Net++: Improved binary neural networks. In
British Machine Vision Conference, 2019.
Adrian Bulat, Georgios Tzimiropoulos, Jean Kossaifi, and Maja Pantic. Improved training of binary networks for human pose estimation and image recognition. arXiv preprint arXiv: 19X4::9004.00 05868, 2019.
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. IEEE Conference on Computer Vision and Pattern Recogni- fion, 2017.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yanix and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or-1. arXiv, 2016
Ruizhou Ding, Ting-Wi Wu Chin, Zeye Liu, and Diana Marculeseu. Regularizing activation distribu- tion for training binarized deep networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Julian Faraone, Nicholas J. Fraser, Michaela Blott, and Philip H. W. Leong. SYQ: learning symmet- ric quantization for efficient deep neural networks. In IEEE Conference Computer Vision and Pattern Recognition, 2018.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. arXiv, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance imagenet classification. In IEEE International Conference on Computer Vision, pp. 1026-1034, 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253/ 2015.
Jie Hu, Li Shen, and Gang Sun. seuezze-and-xxcitaiin networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 2014.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.

---

Published as a conference paper at ICLR 2020
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances on Neural Information Processing Systems, 2017.
Chunlei Liu, Wenrui Ding, Xin Xia, Baochang Zhang, Jiaxin Gu, Jianzhuang Liu, Rongrong Ji, and David Doermann. Circulant binary convolutional networks: Enhancing the performance of 1-bit denns with circulant back propagation. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-Real Net: Enhancing the performance of 1-bit CNNs with improved representational capability and advanced training algorithm. In European Conference on Computer Vision, 2018.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-Net: Imagenet classification using binary convolutional neural networks. European Conference on Computer Vision, 2016.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal on Computer Vision, 115(3):111-222,22) 2015.
Daniel Soudry, Itay Hubara, and Ron Meir. Expectation backpropagation: Parameter-free train- ing of multilayer neural networks with continuous or discrete weights. In Advances on Neural Information Processing Systems, 2014.
Ziwei Wang, Jiwen Lu, Chenxin Tao, Jie Zhou, and Qi Tian. Learning channel-wise interactions for binary convolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Zhe Xu and Ray C.C. Cheung. Accurate and compact convolutional neural networks with trained binarization. In British Machine Vision Conference, 2019.
Haojin Yang, Martin Fritzsche, Christian Bartz, and Christoph Meinel. BMXNet: An open-source binary neural network implementation based on MXNet. In ACM International Conference on Multimedia, 2017.
Sergey Zagonuyko and Nikos Komodakis. Paying more attention to attention: Improving the per- formance of convolutional neural networks via attention transfer. International Conference on Learning Representations, 2017.
Dongging Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. LQ-Nets: Learned quantization for highly accurate and compact deep neural networks. In European Conference on Computer Vision, 2018
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Mixup: Beyond empiri- cal risk minimization. arXiv preprint 17X1v:1710.09412, 2017.
Jianhao Zhang, Yingwei Pan, Ting Yao, He Zhao, and Tao Mei. dabnn: A super fast inference framework for binary neural networks on ARM devices. In ACM International Conference on Multimedia, 2019.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv, 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. Interna- tional Conference on Learning Representattons, 2017.
Shilin Zhu, Xin Dong, and Hao Su. Binary ensemble neural network: More bits per network or more networks per bit? In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and lan D. Reid. Towards effective bow-bitwidth convolutional neural networks. IEEE Conference on Computer Vision and Pattern Recognition, 2018
10

---

Published as a conference paper at ICLR 2020
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and lan Reid. Structured binary neural networks for accurate image classification and semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.


Reviews:
[{"review_id": "rygS_Rv15B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper greatly reduces the gap between binarized and real valued imagenet, using a variety of techniques. The most significant contributions of this paper are engineering based, and the careful combination and integration of approaches from previous papers. I believe that this is of significant practical importance to the field. I particularly appreciate the effort put into developing a very strong baseline that combined ideas from many previous papers!\n\nMy biggest concern is that ResNet is itself a very wasteful architecture in terms of compute and parameter count. If the goal is to develop a compute- and memory-efficient architecture, it would be good to also consider real-valued-network baselines that were proposed with computational and/or memory efficiency as a design goal.\n\nAdditionally, the specific choices for the new student-teacher loss, and new scaling network architecture, seem fairly ad-hoc.\n\nDetailed comments:\n\n\"this implies a reduction of 32× in memory usage\" , assuming the parameter count is held constant.\n\nFig 1 right: This is motivated in terms of preserving scaling factors that are lost by the binarization, but the functional form for this makes it look a lot like a learned gating operation. If the sigmoid is dropped from the architecture, does performance worsen? It would be nice to see some discussion of the degree to which this is helpful because it reverses information loss due to binarization, vs. introduces a new architectural feature which is itself helpful.\n\nAdd a sentence describing what \"double skip connections\" are. I wasn't familiar with this phrase.\n\neq. 2:\nThis functional form is pretty weird.\nWhy is Q a square norm rather than a norm? Square error on an already-squared property is an unusual choice.\nWhy is the denominator itself a norm? Taking the norm of a square norm is similarly an unusual choice. (eg, why not just take an average or sum over Q) \nSay what Q_s and Q_t are (student and teacher network from context)\n\n\"thus, at test time, these scaling factors will not be fixed but rather inferred from data\" nit: Would not generally call this an inference process. \"Inference\" typically refers to values that are computed indirectly (eg by Bayesian reasoning), while in this case the values are computed directly. Would rather say that scaling factors are a function of data, or are determined by data, or similar.\n\n\"By doing so, more than 1/3 of the remaining gap with the real-valued network is bridged.\" text is shifting back and forth between using % and fractional gap to describe benefits. Would just use one measure consistently.\n\nComputational cost analysis:\nThis is very useful.\nNote though that ResNet is a very wasteful architecture in terms of compute! It would be good to include a comparison to imagenet architectures that have computational and memory efficiency as a design goal. (eg, MobileNet comes to mind)\n\nVery nice on the ablation studies.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hkls16xPqS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of training binary neural networks. The authors first provide a strong baseline by assembling a group of training techniques that appeared in recent work that achieves state-of-the-art performance. Then the authors proposed two methods to further boost the performance gain. The first method is to use a teacher-student mechanism that uses a fully real-valued network to teach a binary network. The process is divided into three stages involving two intermediate models to reduce the gap within each teacher-student pair. The second method is to learn a re-scale factor for binary activations using real-valued activations from the previous block. Experiments show that the proposed methods improves the performance on ImageNet and CIFAR-100.\nThe experimental results seem promising. The proposed model reduces the gap to real-valued network to within 3-5%. However, the novelty of the paper is limited and why the proposed methods would help increase the performance gain is not well demonstrated. The teacher-student model is a well-known technique for vision tasks. The authors observed in Section 4.2 that it is very important for the teacher and student to have similar architectures, but did not explain the more important question that why a real-valued network would be able to teach a binary network, since they have quite different information flow. For re-scaling, the authors did not give a detailed comparison between their approach and previous work, and it is not clear how the data-driven way helps. As the ablation study shows the gating function actually hurts for binary down-sampling layers.\nThe writing of the paper needs improvement. A workflow/framework/algorithm description is helpful to better understand the whole framework, and the methodology part in Section 4 requires more details. Some notations need to be defined or clarified. For instance, in Figure 1 Left, what is A? The definition is given only in Section 4.2, where it is not stated in detail either. In Figure 1 Right, what is r? In Table 3, what do the abbreviations mean respectively?\nSome specific questions:\n- Why the real-valued teacher can help train the binary network while they have different information flow? What is the intuition behind the consistency assumption?\n- The authors did not visually show the maps of real-valued and binary activations. How are they aligned in the proposed framework? And are they more similar with each other compared with previous approaches?\n- In Section 4.1 for Initialization a 2-stage optimization strategy is used, while in Section 4.2 a multi-stage teacher-student optimization strategy is used. How are the two strategies combined?\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJx2ejZS5r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is on building binary network. The steps for building binary network takes several components: traditional strategy to binary/optimize a model (like data augmentation,  binary initialization using 2-stage optimization, etc), real-to-binary attention matching that tries to match the output of real values and binarized model, and data-driven channel rescaling to better approximate real convolutions. All these components together makes a strong binary network.\n\nAlthough there are so many steps/tricks mentioned in the paper, I think the explanation and reason for each step is easy to understand. The outcome of the model is quite impressive-- 5% improvement over the best binary model. \n\nIt would be interesting to compare with some other compression techniques, like low-rank, sparsity, weight sharing, etc. Or it will be also interesting to see how these techniques can combine with binary model to further compress the model.\n\nAlso it would be interesting to see how the latency changes using the proposed binary model. As I can see from Table 1, which outlines the FLOPS and BOPS, to my understanding BOPS is much faster than FLOPS, so in the latency wise, the proposed model will be much faster than the original model for inference. Therefore I am looking forward to the real-timing results. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lneTXwcH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A. Summary\n\nProblem:\nBinary NNs promise to make neural networks compatible with devices that only have access to limited computational resources (fundamental to embed NNs in mobiles or IoT devices). However, the loss of computational accuracy comes with a great loss of performances. Designing the right learning algorithm and the right binary architecture remains an open issue.\n\nContributions:\n1. this paper reviews the current literature in Binary Neural Networks and the authors compile the existing methods to build a strong baseline. The strong baseline outperforms existing methods, which is impressive in itself.\n2. the authors introduce a novel layer-wise objective that pushes the binary activations to match the real activations, which are given by a teacher model (real-to-binary). This is simpler and more efficient than existing alternatives.\n3. they propose to train the real-to-binary model using a multi-stage teacher-student procedure. \n4. the authors introduce a data-dependent re-scaling term for the binary activations. \n\nExperiments:\n1. SOTA on ImageNet for BinaryNNs: The combined methods allow bridging the gap between real-valued and binary-valued classifiers on ImageNet (65.4% vs 69.3% top-1 acc).\n2. Comparisons with existing methods: On ImageNet, the model is compared with a complete list of alternative methods (low-bit quantization, larger binary nets, binary nets and real-valued nets). This is however unclear how the method compares with TTQ.\n3. An ablation study is performed on CIFAR-100. It tests the gains that come with the attention matching, the data-dependent gating mechanism and the multi-stage teacher-student mechanism.\n\nB. Decision\n6: Weak Accept.\n\nC. Argumentation\nThe paper clearly states the problem and what are the contributions. The solution is mostly iterative but clearly brings the binary NNs one step forward. The claims are supported by a comparison with a great variety of baselines and an ablation study. Furthermore, it is laudable that great efforts were put into designing such a strong baseline. \n\nHowever, the paper could be easier to read and some points remain unclear:\n1. Is it possible to compare TTQ on the same scale? this is difficult to precisely asses how real-to-binary convolutions compete with this method in the paper.\n2. The equation 2. is intuitive yet not perfectly clear. What are the \"transfer points\"? Why using such a normalization?\n\n\nD. Feedback\n1. Data augmentation and Mix-up: is it necessary to use them here as they should yield improvements for all methods? `\n2. table 1: is it possible to include TTQ? suggestion: is it possible to add a 4th column that measures the overall performance (estimated runtime, speedup?)\n3. table 3: please define all the abbreviations.\n\nE. Question\n1. Could you please draw a more precise comparison between TTQ and your method?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes methodology to train binary neural networks.

The reviewers and authors engaged in a constructive discussion. All the reviewers like the contributions of the paper.

Acceptance is therefore recommended.
Decision: Accept (Poster)