Weights Having Stable Signs Are Important: Finding Primary Subnetworks and Kernels to Compress Binary Weight Networks

1 INTRODUCTION . Convolutional Neural Networks ( CNNs ) have achieved great success in many computer vision tasks such as image classification ( Krizhevsky et al. , 2012 ) , object detection ( Girshick et al. , 2014 ) and semantic segmentation ( Long et al. , 2015 ) . However , modern CNNs usually have large number of parameters , posing heavy costs on memory and computation . To ease their deployment in resourceconstrained environments , different types of neural network compression and acceleration techniques have been proposed in recent years , such as network pruning ( Han et al. , 2015 ; Li et al. , 2017 ) , network quantization ( Hubara et al. , 2016 ; Rastegari et al. , 2016 ; Zhou et al. , 2016 ) , knowledge distillation ( Ba & Caruana , 2014 ; Hinton et al. , 2015 ) , efficient CNN architecture engineering and searching ( Howard et al. , 2017 ; Zhang et al. , 2018b ; Zoph & Le , 2017 ) . Comparatively , network quantization is more commercially attractive as it can not only benefit specialized hardware accelerator designs ( Sze et al. , 2017 ) , but also can be readily combined with other techniques to get further improved compression and acceleration performance ( Mishra & Marr , 2018 ; Han et al. , 2016 ; Zhou et al. , 2017 ) . Quantization methods aim to approximate fullprecision ( 32-bit floating-point ) neural networks with low-precision ( low-bit ) ones . In particular , the extremely quantized models called Binarized Neural Networks ( BNNs ) ( Courbariaux et al. , 2015 ; 2016 ; Rastegari et al. , 2016 ) which force the weights or even weights and activations to have 1-bit values ( +1 and −1 ) , bringing 32× reduction in model size and making costly 32-bit floating-point multiplications can be replaced by much cheaper binary bit-wise operations . Because of this , how to train accurate BNNs either in a post-training manner or in a training from scratch manner has attracted increasing attention . However , training BNNs poses a non-differentiable issue as converting full-precision weights into binary values leads to zero gradients . To combat this issue , most existing methods use the Straight-Through-Estimator ( STE ) . Although there are few attempts ( Achterhold et al. , 2018 ; Chen et al. , 2019 ; Bai et al. , 2019 ; Hou et al. , 2017 ) to learn BNNs without STE by using proximal gradient methods or meta-learning methods , they suffer from worse accuracy and heavier parameter tuning compared to STE based methods . In STE based methods , full-precision weights are retained during training , and the gradients w.r.t . them and their binarized ones are assumed to be the same . In the forward pass of the training , the full-precision weights of the currently learnt model are quantized to binary values for predication loss calculation . In the backward pass , the gradients w.r.t . full-precision weights instead of binary ones are used for model update . To compensating for drastic information loss and training more accurate BNNs , most state of the art STE based methods follow the formulation of ( Rastegari et al. , 2016 ) in which the binary weights are represented as a combination of scaling factors and weight signs to approximate 32-bit floating-point weight values layer-by-layer , yet also present a lot of modifications . These modifications include but are not limited to expanding binary weights to have multiple binary bases ( Lin et al. , 2017 ; Guo et al. , 2017 ) , replacing hand-crafted scaling factors with learnable ones ( Zhang et al. , 2018a ) , making an ensemble of multiple binary models ( Zhu et al. , 2019 ) , searching high-performance binary network architectures ( Kim et al. , 2020 ) , and designing improved regularization objectives , optimizers and activation functions ( Cai et al. , 2017 ; Liu et al. , 2018 ; Helwegen et al. , 2019 ; Martinez et al. , 2020 ) . There are also a few works , trying to make a better understanding of the training of BNNs with STE . In ( Alizadeh et al. , 2019 ) , the authors evaluate some of the widely used tricks , showing that adapting learning rate with a second-moment optimizer is crucial to train BNNs with STE based methods while other tricks such as weights and gradients clipping are less important . Bethge et al . ( 2019 ) shows the commonly used techniques such as hand-crafted scaling factors and custom gradients are also not crucial . Sajad et al . ( 2019 ) demonstrates learnable scaling factors combined into a modified sign function can enhance the accuracy of BNNs . Anderson & Berg ( 2018 ) makes an interpretation of why binary models can approximate their full-precision references in terms of high-dimensional geometry . Galloway et al . ( 2018 ) validates that BNNs have surprisingly improved robustness against some adversarial attacks compared to their full-precision counterparts . In this paper , we revisit the training of BNNs , particularly Binary Weight Networks ( BWNs ) with STE , but in a new perspective , exploring structural weight behaviors in training BWNs . Our main contributions are summarized as follows : • We use two popular methods ( Rastegari et al. , 2016 ) and ( Zhang et al. , 2018a ) for an empirical study , showing both hand-crafted and learnable scaling factors are not that important , while the change of weight signs plays the key role in the training of BWNs , under the settings of using common techniques and tricks . • More importantly , we observe two astonishing training phenomena : ( 1 ) the training of BWNs demonstrates the process of seeking primary binary sub-networks whose weight signs are determined and fixed at the early training stage , which is akin to recent findings of the lottery ticket hypothesis ( Frankle & Carbin , 2019 ) for training sparse neural networks ; ( 2 ) binary kernels in the convolutional layers ( Conv layers ) of final BWNs tend to be centered on a limited number of binary kernels , showing binary weight networks may has the potential to be further compressed . This breaks the common understanding that representing each weight with a single bit puts the quantization to the extreme compression . • We propose a binary kernel quantization method to compress BWNs , bringing a new type of BWNs called Quantized Binary-Kernel Networks ( QBNs ) . 2 AN EMPIRICAL STUDY ON UNDERSTANDING BWNS ’ TRAINING . In this section we will briefly describe BWNs we use in experiments , implementation details , scaling factors in BWNs , full-precision weight norm , weight sign , and sub-networks in BWNs . 2.1 DIFFERENT BINARY WEIGHT NETWORKS . BWNs generally represents those networks with binary weights , and there are several different BWNs existing . Overall they use αB to replace full-precision weight W , where B = sign ( W ) and α is proposed to minimize ||αB − W || in an either learnable or calculated way . In following experiments , we use the one implemented in XNor-Net ( Rastegari et al. , 2016 ) and denote it as XNor-BWN , and the one implemented in LQ-Net ( Zhang et al. , 2018a ) and denote it as LQ-BWN which is 1-bit weight 32-bit activation version of LQ-Net . Other popular BWN methods like DoReFa-Net and BinaryConnect are similar to these two methods . Both XNor-BWN and LQ-BWN use STE framework , and XNor-BWN uses hand-crafted calculated scaling factors , and LQ-BWN uses learnable scaling factors . 2.2 IMPLEMENTATION DETAILS AND NOTATION . Quantization : We directly use open source codes of BWN released by authors , including XNorBWN1 and LQ-BWN2 . Dataset and Network Structure : CIFAR-10 ( Krizhevsky & Hinton , 2009 ) and ImageNet ( Russakovsky et al . ) are used in our experiments . We use VGG-7 ( Simonyan & Zisserman , 2015 ) and ResNet-20 ( He et al. , 2016 ) on CIFAR-10 , and ResNet18 on ImageNet . The strcuture is the same as original ones . Hyper-parameters : We use the same training parameters on each network . The network is trained for 200 epochs . The learning rate is set initially to 0.02 and divided by 10 at 80 and 160 epochs . For random crop , we first use zero pad to resize the image into 40× 40 , and random crop into 32× 32 . For BWN trained on ImageNet , each is trained for 100 epochs . The initial learning rate is 0.1 and decays 0.1 at 30 , 60 , 90 epoch . The image is rescaled into 256×256 and then randomly cropped into 224× 224 . No additional data augmentations are used . For all networks , weight decay is applied to all Conv layers set to 4× 10−5 . Notations : In figures and tables , we will use the following abbreviations for clearer expression . BN : BatchNormalization , LR : Learning Rate . WD : Weight Decay . SF : Scaling Factors . FP : Fullprecision . VGG-7 XNor-BWN : a VGG-7 network using the binarization algorithm of XNor-BWN . ResNet-20 Baseline : a full-precision ResNet-20 only using data augmentation and weight decay without any additional tricks . Other network structures with certain methods are similar to this . Large weights , large magnitude weights , and weights with larger norm have the same meaning to indicate those weights having relatively large absolute values . 2.3 SCALING FACTORS . According to previous methods , scaling factors are one essential element in obtaining BWNs . However , according to our experiments and analysis , we find scaling factors are not so important in training BWNs , and they can somehow be ignored without the drop in performance . Here we list four reasons to explain why scaling factors are unimportant . A simple proof : BN is a common practice to be used in training BWNs . It contains two operations , Normalization and Affine as shown in Equation1 . γ and β are the affine parameters used in BN . = 5e− 4 is used in PyTorch to avoid the error of dividing zero . We use a simple proof to demonstrate that BN can absorb scaling factors as shown in Equation2 . This is correct during training when one scaling factor is applied to each output channel under the Conv-BN-ReLU structure . x′ = Normalize ( x ) = x− x̄√ σ2 + y = Affine ( x′ ) = γx′ + β ( 1 ) yα = γ αx− αx̄√ α2σ2 + + β ≈ γ x− x̄√ σ2 + + β = y ( 2 ) 1We use the codes of DoReFa-Net to realize XNor-BWN which is the same as the original implementation . https : //github.com/tensorpack/tensorpack/tree/master/examples/DoReFa-Net 2LQ-BWN is the 1-bit weight 32-bit activation version of LQ-Nets . https : //github.com/microsoft/LQ-Nets Experimental Results : In the second aspect , we directly go to experimental results . As shown Table.2 in Appendix.B , we train different networks with and without scaling factors . The test accuracy on Cifar-10 and validation accuracy on ImageNet do not show a large difference between the two methods . Later we fix scaling factors of all layers to a certain value and magnify their learning rate according to the fixed scaling factors ’ magnitude . The performance does not change when fixing scaling factors . Thus , we conclude with proper learning rate , scaling factors are not essential to train BWNs . Compare learnable SF and γ in BN : LQ-BWN uses channel-wise scaling factors . From the experiments in Appendix.C , we find that these channel-wise scaling factors having a high correlation with γ in the BN after corresponding binary Conv . This finding indicates that BN ’ s γ can replace channel-wise SF to some extent . Quantization Error Curve : Another purpose using scaling factors is to reduce the quantization error between full-precision weights and binary weights according to a BNN survey ( Qin et al. , 2020 ) . By using experiments in Appendix.D we prove that the quantization error is not actually reduced by scaling factors , but weight decay helps on this reduction .

Summary: This paper proposes some interesting observations for training BWNs. 1: The scaling factors can be removed with batch normalization used. 2: The signs of the weights with large norms are determined and fixed at the early training stage. 3:  The binary weight networks can be further compressed. Moreover, the authors provide some empirical visualizations and results to demonstrate its analysis. However, the paper seems to be incomplete and needs to be further improved.