Training binary neural networks with real-to-binary convolutions

1 INTRODUCTION . Following the introduction of the BinaryNeuralNet ( BNN ) algorithm ( Courbariaux et al. , 2016 ) , binary neural networks emerged as one of the most promising approaches for obtaining highly efficient neural networks that can be deployed on devices with limited computational resources . Binary convolutions are appealing mainly for two reasons : ( a ) Model compression : if the weights of the network are stored as bits in a 32-bit float , this implies a reduction of 32× in memory usage . ( b ) Computational speed-up : computationally intensive floating-point multiply and add operations are replaced by efficient xnor and pop-count operations , which have been shown to provide practical speed-ups of up to 58× on CPU ( Rastegari et al. , 2016 ) and , as opposed to general low bit-width operations , are amenable to standard hardware . Despite these appealing properties , binary neural networks have been criticized as binarization typically results in large accuracy drops . Thus , their deployment in practical scenarios is uncommon . For example , on ImageNet classification , there is a ∼ 18 % gap in top-1 accuracy between a ResNet-18 and its binary counterpart when binarized with XNOR-Net ( Rastegari et al. , 2016 ) , which is the method of choice for neural network binarization . But how far are we from training binary neural networks that are powerful enough to become a viable alternative to real-valued networks ? Our first contribution in this work is to take stock of recent advances on binary neural networks and train a very strong baseline which already results in state-of-the-art performance . Our second contribution is a method for bridging most of the remaining gap , which boils down to minimizing the discrepancy between the output of the binary and the corresponding real-valued convolution . This idea is materialized in our work in two complementary ways : Firstly , we use an attention matching strategy so that the real-valued network can more * Denotes equal contribution closely guide the binary network during optimization . However , we show that due to the architectural discrepancies between the real and the binary networks , a direct application of teacher-student produces sub-optimal performance . Instead , we propose to use a sequence of teacher-student pairs that progressively bridges the architectural gap . Secondly , we further propose to use the real-valued activations of the binary network , available prior to the binarization preceding convolution , to compute scale factors that are used to re-scale the activations right after the application of the binary convolution . This is in line with recent works which have shown that re-scaling the binary convolution output can result in large performance gains ( Rastegari et al. , 2016 ; Bulat & Tzimiropoulos , 2019 ) . However , unlike prior work , we compute the scaling factors in a data-driven manner based on the real-valued activations of each layer prior to binarization , which results in superior performance . Overall , we make the following contributions : • We construct a very strong baseline by combining some recent insights on training binary networks and by performing a thorough experimentation to find the most well-suited optimization techniques . We show that this baseline already achieves state-of-the-art accuracy on ImageNet , surpassing all previously published works on binary networks . • We propose a real-to-binary attention matching : this entails that matching spatial attention maps computed at the output of the binary and real-valued convolutions is particularly suited for training binary neural networks ( see Fig . 1 left and section 4.2 ) . We also devise an approach in which the architectural gap between real and binary networks is progressively bridged through a sequence of teacher-student pairs . • We propose a data-driven channel re-scaling : this entails using the real-valued activations of the binary network prior to their binarization to compute the scale factors used to rescale the activations produced right after the application of the binary convolution . See Fig . 1 , right , and section 4.3 . • We show that our combined contributions provide , for the first time , competitive results on two standard datasets , achieving 76.2 % top-1 performance on CIFAR-100 and 65.4 % top-1 performance on ImageNet when using a ResNet-18 –a gap bellow 3 % and 5 % respectively compared to their full precision counterparts . 2 RELATED WORK . While being pre-dated by other works on binary networks ( Soudry et al. , 2014 ) , the BNN algorithm ( Courbariaux et al. , 2016 ) established how to train networks with binary weights within the familiar back-propagation paradigm . The training method relies on a real-valued copy of the network weights which is binarized during the forward pass , but is updated during back-propagation ignoring the binarization step . Unfortunately , BNN resulted in a staggering ∼ 28 % gap in top-1 accuracy compared to the full precision ResNet-18 on ImageNet . It is worth noting that binary networks do have a number of floating point operations . In fact , the output of a binary convolution is not binary ( values are integers resulting from the count ) . Also , in accordance to other low bit-width quantization methodologies , the first convolution ( a costly 7 × 7 kernel in ResNet ) , the fully connected layer and the batch normalization layers are all real-valued . In consequence , a line of research has focused on developing methodologies that add a fractional amount of real-valued operations in exchange for significant accuracy gains . For example , the seminal work of XNOR-Net ( Rastegari et al. , 2016 ) proposed to add a real-valued scaling factor to each output channel of a binary convolution , a technique that has become standard for binary networks . Similarly , Bi-Real Net ( Liu et al. , 2018 ) argued that skip connections are fundamental for binary networks and observed that the flow of full precision activations provided by the skip connections is interrupted by the binary downsample convolutions . This degrades the signal and make subsequent skip connections less effective . To alleviate this , they proposed making the downsample layers real valued , obtaining around 3 % accuracy increase in exchange for a small increase in computational complexity . Improving the optimization algorithm for binary networks has been another fundamental line of research . Examples include the use of smooth approximations of the gradient , the use of PReLU ( Bulat et al. , 2019 ) , a two-stage training which binarizes the weights first and then the activations ( Bulat et al. , 2019 ) and progressive quantization ( Gong et al. , 2019 ; Bulat et al. , 2019 ) . The work in ( Wang et al. , 2019 ) proposed to learn channel correlations through reinforcement learning to better preserve the sign of a convolution output . A set of regularizers are added to the loss term in ( Ding et al. , 2019 ) so as to control the range of values of the activations , and guarantee good gradient flow . Other optimization aspects , such the effect of gradient clipping or batch-norm momentum , were empirically tested in ( Alizadeh et al. , 2019 ) . In section 4.1 , we show how to combine many of the insights provided in these works with standard optimization techniques to obtain a very strong baseline that already achieves state-of-the-art accuracy . While the aforementioned works either maintain the same computational cost , or increase it by a fractional amount , other research has focused instead on relaxing the problem constraints by increasing the number of binary operations by a large amount , typically a factor of 2 to 8 times . Examples include ABC-Net ( Lin et al. , 2017 ) , the structure approximation of ( Zhuang et al. , 2019 ) , the circulant CNN of ( Liu et al. , 2019 ) , and the binary ensemble of ( Zhu et al. , 2019 ) . Note that the large increase of binary operations diminishes the efficiency claim that justifies the use of binary networks in first place . Furthermore , we will show that there is still a lot of margin in order to bridge the accuracy gap prior to resorting to scaling up the network capacity1 . The methodology proposed in this paper has some relations with prior work : our use of attention matching as described in section 4.2 is somewhat related to the feature distillation approach of ( Zhuang et al. , 2018 ) . However , ( Zhuang et al. , 2018 ) tries to match whole feature maps of the to-be-quantized network with the quantized feature maps of a real-valued network that is trained in parallel with the to-be-quantized network . Such an approach is shown to improve training of low-bitwidth quantized models but not binary networks . Notably , our approach based on matching attention maps is much simpler and shown to be effective for the case of binary networks . Our data-driven channel re-scaling approach , described in section 4.3 , is related to the channel rescaling approach of XNOR-Net , and also that of ( Xu & Cheung , 2019 ; Bulat & Tzimiropoulos , 2019 ) , which propose to learn the scale factors discriminatively through backpropagation . Contrary to ( Xu & Cheung , 2019 ; Bulat & Tzimiropoulos , 2019 ) , our method is data-driven and avoids using 1There is also a large body of work focusing on other low-bit quantization strategies , but a review of these techniques goes beyond the scope of this section . fixed scale factors learnt during training . Contrary to XNOR-Net , our method discriminatively learns how to produce the data-driven scale factors so that they are optimal for the task in hand . 3 BACKGROUND . This section reviews the binarization process proposed in ( Courbariaux et al. , 2016 ) and its improved version from ( Rastegari et al. , 2016 ) , which is the method of choice for neural network binarization . We denote byW ∈ Ro×c×k×k andA ∈ Rc×win×hin the weights and input features of a CNN layer , where o and c represent the number of output and input channels , k the width and height of the kernel , and win and hin represent the spatial dimension of the input features A . In ( Courbariaux et al. , 2016 ) , both weights and activations are binarized using the sign function and then convolution is performed as A ∗W ≈ sign ( A ) ©∗ sign ( W ) where©∗ denotes the binary convolution , which can be implemented using bit-wise operations . However , this direct binarization approach introduces a high quantization error that leads to low accuracy . To alleviate this , XNOR-Net ( Rastegari et al. , 2016 ) proposes to use real-valued scaling factors to re-scale the output of the binary convolution as A ∗W ≈ ( sign ( A ) ©∗ sign ( W ) ) Kα , ( 1 ) where denotes the element-wise multiplication , α and K are the weight and activation scaling factors , respectively , calculated in Rastegari et al . ( 2016 ) in an analytic manner . More recently , Bulat & Tzimiropoulos ( 2019 ) proposed to fuse α and K into a single factor Γ that is learned via backpropagation , resulting in further accuracy gains . 4 METHOD . This section firstly introduces our strong baseline . Then , we present two ways to improve the approximation of Eq . 1 : Firstly , we use a loss based on matching attention maps computed from the binary and a real-valued network ( see section 4.2 ) . Secondly , we make the scaling factor a function of the real-valued input activations A ( see section 4.3 ) .

Summary: This paper studies the problem of training binary neural networks. The authors first provide a strong baseline by assembling a group of training techniques that appeared in recent work that achieves state-of-the-art performance. Then the authors proposed two methods to further boost the performance gain. The first method is to use a teacher-student mechanism that uses a fully real-valued network to teach a binary network. The process is divided into three stages involving two intermediate models to reduce the gap within each teacher-student pair. The second method is to learn a re-scale factor for binary activations using real-valued activations from the previous block. Experiments show that the proposed methods improves the performance on ImageNet and CIFAR-100.