Weights Having Stable Signs Are Important: Finding Primary Subnetworks and Kernels to Compress Binary Weight Networks

1 INTRODUCTION . Convolutional Neural Networks ( CNNs ) have achieved great success in many computer vision tasks such as image classification ( Krizhevsky et al. , 2012 ) , object detection ( Girshick et al. , 2014 ) and semantic segmentation ( Long et al. , 2015 ) . However , modern CNNs usually have large number of parameters , posing heavy costs on memory and computation . To ease their deployment in resourceconstrained environments , different types of neural network compression and acceleration techniques have been proposed in recent years , such as network pruning ( Han et al. , 2015 ; Li et al. , 2017 ) , network quantization ( Hubara et al. , 2016 ; Rastegari et al. , 2016 ; Zhou et al. , 2016 ) , knowledge distillation ( Ba & Caruana , 2014 ; Hinton et al. , 2015 ) , efficient CNN architecture engineering and searching ( Howard et al. , 2017 ; Zhang et al. , 2018b ; Zoph & Le , 2017 ) . Comparatively , network quantization is more commercially attractive as it can not only benefit specialized hardware accelerator designs ( Sze et al. , 2017 ) , but also can be readily combined with other techniques to get further improved compression and acceleration performance ( Mishra & Marr , 2018 ; Han et al. , 2016 ; Zhou et al. , 2017 ) . Quantization methods aim to approximate fullprecision ( 32-bit floating-point ) neural networks with low-precision ( low-bit ) ones . In particular , the extremely quantized models called Binarized Neural Networks ( BNNs ) ( Courbariaux et al. , 2015 ; 2016 ; Rastegari et al. , 2016 ) which force the weights or even weights and activations to have 1-bit values ( +1 and −1 ) , bringing 32× reduction in model size and making costly 32-bit floating-point multiplications can be replaced by much cheaper binary bit-wise operations . Because of this , how to train accurate BNNs either in a post-training manner or in a training from scratch manner has attracted increasing attention . However , training BNNs poses a non-differentiable issue as converting full-precision weights into binary values leads to zero gradients . To combat this issue , most existing methods use the Straight-Through-Estimator ( STE ) . Although there are few attempts ( Achterhold et al. , 2018 ; Chen et al. , 2019 ; Bai et al. , 2019 ; Hou et al. , 2017 ) to learn BNNs without STE by using proximal gradient methods or meta-learning methods , they suffer from worse accuracy and heavier parameter tuning compared to STE based methods . In STE based methods , full-precision weights are retained during training , and the gradients w.r.t . them and their binarized ones are assumed to be the same . In the forward pass of the training , the full-precision weights of the currently learnt model are quantized to binary values for predication loss calculation . In the backward pass , the gradients w.r.t . full-precision weights instead of binary ones are used for model update . To compensating for drastic information loss and training more accurate BNNs , most state of the art STE based methods follow the formulation of ( Rastegari et al. , 2016 ) in which the binary weights are represented as a combination of scaling factors and weight signs to approximate 32-bit floating-point weight values layer-by-layer , yet also present a lot of modifications . These modifications include but are not limited to expanding binary weights to have multiple binary bases ( Lin et al. , 2017 ; Guo et al. , 2017 ) , replacing hand-crafted scaling factors with learnable ones ( Zhang et al. , 2018a ) , making an ensemble of multiple binary models ( Zhu et al. , 2019 ) , searching high-performance binary network architectures ( Kim et al. , 2020 ) , and designing improved regularization objectives , optimizers and activation functions ( Cai et al. , 2017 ; Liu et al. , 2018 ; Helwegen et al. , 2019 ; Martinez et al. , 2020 ) . There are also a few works , trying to make a better understanding of the training of BNNs with STE . In ( Alizadeh et al. , 2019 ) , the authors evaluate some of the widely used tricks , showing that adapting learning rate with a second-moment optimizer is crucial to train BNNs with STE based methods while other tricks such as weights and gradients clipping are less important . Bethge et al . ( 2019 ) shows the commonly used techniques such as hand-crafted scaling factors and custom gradients are also not crucial . Sajad et al . ( 2019 ) demonstrates learnable scaling factors combined into a modified sign function can enhance the accuracy of BNNs . Anderson & Berg ( 2018 ) makes an interpretation of why binary models can approximate their full-precision references in terms of high-dimensional geometry . Galloway et al . ( 2018 ) validates that BNNs have surprisingly improved robustness against some adversarial attacks compared to their full-precision counterparts . In this paper , we revisit the training of BNNs , particularly Binary Weight Networks ( BWNs ) with STE , but in a new perspective , exploring structural weight behaviors in training BWNs . Our main contributions are summarized as follows : • We use two popular methods ( Rastegari et al. , 2016 ) and ( Zhang et al. , 2018a ) for an empirical study , showing both hand-crafted and learnable scaling factors are not that important , while the change of weight signs plays the key role in the training of BWNs , under the settings of using common techniques and tricks . • More importantly , we observe two astonishing training phenomena : ( 1 ) the training of BWNs demonstrates the process of seeking primary binary sub-networks whose weight signs are determined and fixed at the early training stage , which is akin to recent findings of the lottery ticket hypothesis ( Frankle & Carbin , 2019 ) for training sparse neural networks ; ( 2 ) binary kernels in the convolutional layers ( Conv layers ) of final BWNs tend to be centered on a limited number of binary kernels , showing binary weight networks may has the potential to be further compressed . This breaks the common understanding that representing each weight with a single bit puts the quantization to the extreme compression . • We propose a binary kernel quantization method to compress BWNs , bringing a new type of BWNs called Quantized Binary-Kernel Networks ( QBNs ) . 2 AN EMPIRICAL STUDY ON UNDERSTANDING BWNS ’ TRAINING . In this section we will briefly describe BWNs we use in experiments , implementation details , scaling factors in BWNs , full-precision weight norm , weight sign , and sub-networks in BWNs . 2.1 DIFFERENT BINARY WEIGHT NETWORKS . BWNs generally represents those networks with binary weights , and there are several different BWNs existing . Overall they use αB to replace full-precision weight W , where B = sign ( W ) and α is proposed to minimize ||αB − W || in an either learnable or calculated way . In following experiments , we use the one implemented in XNor-Net ( Rastegari et al. , 2016 ) and denote it as XNor-BWN , and the one implemented in LQ-Net ( Zhang et al. , 2018a ) and denote it as LQ-BWN which is 1-bit weight 32-bit activation version of LQ-Net . Other popular BWN methods like DoReFa-Net and BinaryConnect are similar to these two methods . Both XNor-BWN and LQ-BWN use STE framework , and XNor-BWN uses hand-crafted calculated scaling factors , and LQ-BWN uses learnable scaling factors . 2.2 IMPLEMENTATION DETAILS AND NOTATION . Quantization : We directly use open source codes of BWN released by authors , including XNorBWN1 and LQ-BWN2 . Dataset and Network Structure : CIFAR-10 ( Krizhevsky & Hinton , 2009 ) and ImageNet ( Russakovsky et al . ) are used in our experiments . We use VGG-7 ( Simonyan & Zisserman , 2015 ) and ResNet-20 ( He et al. , 2016 ) on CIFAR-10 , and ResNet18 on ImageNet . The strcuture is the same as original ones . Hyper-parameters : We use the same training parameters on each network . The network is trained for 200 epochs . The learning rate is set initially to 0.02 and divided by 10 at 80 and 160 epochs . For random crop , we first use zero pad to resize the image into 40× 40 , and random crop into 32× 32 . For BWN trained on ImageNet , each is trained for 100 epochs . The initial learning rate is 0.1 and decays 0.1 at 30 , 60 , 90 epoch . The image is rescaled into 256×256 and then randomly cropped into 224× 224 . No additional data augmentations are used . For all networks , weight decay is applied to all Conv layers set to 4× 10−5 . Notations : In figures and tables , we will use the following abbreviations for clearer expression . BN : BatchNormalization , LR : Learning Rate . WD : Weight Decay . SF : Scaling Factors . FP : Fullprecision . VGG-7 XNor-BWN : a VGG-7 network using the binarization algorithm of XNor-BWN . ResNet-20 Baseline : a full-precision ResNet-20 only using data augmentation and weight decay without any additional tricks . Other network structures with certain methods are similar to this . Large weights , large magnitude weights , and weights with larger norm have the same meaning to indicate those weights having relatively large absolute values . 2.3 SCALING FACTORS . According to previous methods , scaling factors are one essential element in obtaining BWNs . However , according to our experiments and analysis , we find scaling factors are not so important in training BWNs , and they can somehow be ignored without the drop in performance . Here we list four reasons to explain why scaling factors are unimportant . A simple proof : BN is a common practice to be used in training BWNs . It contains two operations , Normalization and Affine as shown in Equation1 . γ and β are the affine parameters used in BN . = 5e− 4 is used in PyTorch to avoid the error of dividing zero . We use a simple proof to demonstrate that BN can absorb scaling factors as shown in Equation2 . This is correct during training when one scaling factor is applied to each output channel under the Conv-BN-ReLU structure . x′ = Normalize ( x ) = x− x̄√ σ2 + y = Affine ( x′ ) = γx′ + β ( 1 ) yα = γ αx− αx̄√ α2σ2 + + β ≈ γ x− x̄√ σ2 + + β = y ( 2 ) 1We use the codes of DoReFa-Net to realize XNor-BWN which is the same as the original implementation . https : //github.com/tensorpack/tensorpack/tree/master/examples/DoReFa-Net 2LQ-BWN is the 1-bit weight 32-bit activation version of LQ-Nets . https : //github.com/microsoft/LQ-Nets Experimental Results : In the second aspect , we directly go to experimental results . As shown Table.2 in Appendix.B , we train different networks with and without scaling factors . The test accuracy on Cifar-10 and validation accuracy on ImageNet do not show a large difference between the two methods . Later we fix scaling factors of all layers to a certain value and magnify their learning rate according to the fixed scaling factors ’ magnitude . The performance does not change when fixing scaling factors . Thus , we conclude with proper learning rate , scaling factors are not essential to train BWNs . Compare learnable SF and γ in BN : LQ-BWN uses channel-wise scaling factors . From the experiments in Appendix.C , we find that these channel-wise scaling factors having a high correlation with γ in the BN after corresponding binary Conv . This finding indicates that BN ’ s γ can replace channel-wise SF to some extent . Quantization Error Curve : Another purpose using scaling factors is to reduce the quantization error between full-precision weights and binary weights according to a BNN survey ( Qin et al. , 2020 ) . By using experiments in Appendix.D we prove that the quantization error is not actually reduced by scaling factors , but weight decay helps on this reduction .

Summary: The Authors show that scaling factors with hand-crafted or learnable methods are not so important when training Binary Weight Networks (BWNs), while the change of weight signs is crucial. They make two observations: The weight signs of the primary binary sub-networks are determined and fixed at the early training stage. Binary kernels in the convolutional layers of final models tend to be centered on a limited number of fixed structural patterns. Based on these observations, they propose a new method called binary kernel quantization to further compress BWNs.