BNN+: Improved Binary Network Training

1 INTRODUCTION . Deep neural networks ( DNNs ) have demonstrated success for many supervised learning tasks ranging from voice recognition to object detection ( Szegedy et al. , 2015 ; Simonyan & Zisserman , 2014 ; Iandola et al. , 2016 ) . The focus has been on increasing accuracy , in particular for image tasks , where deep convolutional neural networks ( CNNs ) are widely used . However , their increasing complexity poses a new challenge , and has become an impediment to widespread deployment in many applications ; specifically when trying to deploy such models to resource constrained and lower-power devices . A typical DNN architecture contains tens to thousands of layers , resulting in millions of parameters . As an example , AlexNet ( Krizhevsky et al. , 2012 ) requires 200MB of memory , VGGNet ( Simonyan & Zisserman , 2014 ) requires 500MB memory . Large model sizes are further exacerbated by their computational cost , requiring GPU implementation to allow real-time inference . Such requirements evidently can not be accustomed by edge devices as they have limited memory , computation power , and battery . This motivated the community to investigate methods for compressing and reducing computation cost of DNNs . To make DNNs compatible with the resource constraints of low power devices , there have been several approaches developed , such as network pruning ( LeCun et al. , 1990 ) , architecture design ( Sandler et al. , 2018 ) , and quantization ( Courbariaux et al. , 2015 ; Han et al. , 2015 ) . In particular , weight compression using quantization can achieve very large savings in memory , where binary ( 1- bit ) , and ternary ( 2-bit ) approaches have been shown to obtain competitive accuracy ( Hubara et al. , 2016 ; Zhu et al. , 2016 ; Tang et al. , 2017 ) . Using such schemes reduces model sizes by 8x to 32x depending on the bit resolution used for computations . In addition to this , the speed by quantizing the activation layers . In this way , both the weights and activations are quantized so that one can replace the expensive dot products and activation function evaluations with bitwise operations . This reduction in bit-width benefits hardware accelerators such as FPGAs and neural network chips . An issue with using low-bit DNNs is the drastic drop in accuracy compared to its full precision counterpart , and this is made even more severe upon quantizing the activations . This problem is largely due to noise and lack of precision in the training objective of the networks during back-propagation ( Lin et al. , 2017 ) . Although , quantizing the weights and activations have been attracting large interests thanks to their computational benefits , closing the gap in accuracy between the full precision and the quantized version remains a challenge . Indeed , quantizing weights cause drastic information loss and make neural networks harder to train due to a large number of sign fluctuations in the weights . Therefore , how to control the stability of this training procedure is of high importance . In theory , it is infeasible to back-propagate in a quantized setting as the weights and activations employed are discontinuous and discrete . Instead , heuristics and approximations are proposed to match the forward and backward passes . Often weights at different layers of DNNs follow a certain structure . How to quantize the weights locally , and maintaining a global structure to minimize a common cost function is important ( Li et al. , 2017 ) . Our contribution consists of three ideas that can be easily implemented in the binary training framework presented by Hubara et al . ( 2016 ) to improve convergence and generalization accuracy of binary networks . First , the activation function is modified to better approximate the sign function in the backward pass , second we propose two regularization functions that encourage training weights around binary values , and lastly a scaling factor is introduced in both the regularization term as well as network building blocks to mitigate accuracy drop due to hard binarization . Our method is evaluated on CIFAR-10 and ImageNet datasets and compared to other binary methods . We show accuracy gains to traditional binary training . 2 RELATED WORK . We focus on challenges present in training binary networks . The training procedure emulates binary operations by restricting the weights and activations to single-bit so that computations of neural networks can be implemented using arithmetic logic units ( ALU ) using XNOR and popcount operations . More specifically , XNOR and popcount instructions are readily available on most CPU and GPU processing units . Custom hardware would have to be implemented to take advantage of operations with higher bits such as 2 to 4 bits . The goal of this binary training is to reduce the model size and gain inference speedups without performance degradation . Primary work done by Courbariaux et al . ( 2015 ) ( BinaryConnect ) trains deep neural networks with binary weights { −1 , +1 } . They propose to quantize real values using the sign function . The propagated gradient applies update to weights |w| ≤ 1 . Once the weights are outside of this region they are no longer updated , this is done by clipping weights between { −1 , +1 } . In that work , they did not consider binarizing the activation functions . BNN ( Hubara et al. , 2016 ) is the first purely binary network quantizing both the weights and activations . They achieve comparable accuracy to their prior work on BinaryConnect , and achieve significantly close performance to full-precision , by using large and deep networks . Although , they performed poorly on large datasets like ImageNet ( Russakovsky et al. , 2015 ) . The resulting network presented in their work obtains 32× compression rate and approximately 7× increase in inference speed . To alleviate the accuracy drop of BNN on larger datasets , Rastegari et al . ( 2016 ) proposed XNORNet , where they strike a trade-off between compression and accuracy through the use of scaling factors for both weights and activation functions . In their work , they show performance gains compared to BNN on ImageNet classification . The scaling factors for both the weights and activations are computed dynamically , which slows down training performance . Further , they introduced an additional complexity in implementing the convolution operations on the hardware , slightly reducing compression rate and speed up gains . DoReFa-Net ( Zhou et al. , 2016 ) further improves XNOR-Net by approximating the activations with more bits . The proposed rounding mechanism allows for low bit back-propagation as well . Although they perform multi-bit quantization , their model still suffers from large accuracy drop upon quantizing the last layer . Later in ABC-Net , Tang et al . ( 2017 ) propose several strategies , adjusting the learning rate for larger datasets . They show BNN achieve similar accuracy as XNOR-Net without the scaling overhead by adding a regularizer term which allows binary networks to generalize better . They also suggest a modified BNN , where they adopted the strategy of increasing the number of filters , to compensate for accuracy loss similar to wide reduced-precision networks ( Mishra et al. , 2017 ) . More recently , Liu et al . ( 2018 ) developed a second-order approximation to the sign activation function for a more accurate backward update . In addition to this , they pre-train the network in which they want to binarize in full precision using the hard tangent hyperbolic ( htanh ) activation , see Figure 2 . They use the pre-trained network weights as an initialization for the binary network to obtain state of the art performance . 3 IMPROVED BINARY TRAINING . Training a binary neural network faces two major challenges : on weights , and on activation functions . As both weights and activations are binary , the traditional continuous optimization methods such as SGD can not be directly applied . Instead , a continuous approximation is used for the sign activation during the backward pass . Further , the gradient of the loss with respect to the weights are small . So as training progresses weight sign remains unchanged . These are both addressed in our proposed method . In this section , we present our approach to training 1-bit CNNs in detail . 3.1 BINARY TRAINING . We quickly revisit quantization through binary training as first presented by ( Courbariaux et al. , 2015 ) . In ( Hubara et al. , 2016 ) , the weights are quantized by using the sign function which is +1 if w > 0 and −1 otherwise . In the forward pass , the real-valued weights are binarized to wb , and the resulting loss is computed using binary weights throughout the network . For hidden units , the sign function non-linearity is used to obtain binary activations . Prior to binarizing , the real weights are stored in a temporary variable w. The variables w are stored because one can not back-propagate through the sign operation as its gradient is zero everywhere , and hence disturbs learning . To alleviate this problem the authors suggest using a straight through estimator ( Hinton , 2012 ) for the gradient of the sign function . This method is a heuristic way of approximating the gradient of a neuron , dL ( w ) dw ≈ dL dw ∣∣∣∣ w=wb 1 { |w|≤1 } ( 1 ) where L is the loss function and 1 ( . ) is the indicator function . The gradients in the backward pass are then applied to weights that are within [ −1 , +1 ] . The training process is summarized in Figure 1 . As weights undergo gradient updates , they are eventually pushed out of the center region and instead make two modes , one at −1 and another at +1 . This progression is also shown in Figure 1 . 3.2 IMPROVED TRAINING METHOD . Our first modification is on closing the discrepancy between the forward pass and backward pass . Originally , the sign derivative is approximated using the htanh ( x ) activation , as in Figure 2 . Instead , we modify the Swish-like activation ( Ramachandran et al. , 2017 ; Elfwing et al. , 2018 ; Hendrycks & Gimpel , 2016 ) , where it has shown to outperform other activations on various tasks . The modifications are performed by taking its derivative and centering it around 0 SSβ ( x ) = 2σ ( βx ) [ 1 + βx { 1− σ ( βx ) } ] − 1 , ( 2 ) where σ ( z ) is the sigmoid function and the scale β > 0 controls how fast the activation function asymptotes to −1 and +1 . The β parameter can be learned by the network or be hand-tuned as a hyperparameter . As opposed to the Swish function , where it is unbounded on the right side , the modification makes it bounded and a valid approximator of the sign function . As a result , we call this activation SignSwish , and its gradient is dSSβ ( x ) dx = β { 2− βx tanh ( βx2 ) } 1 + cosh ( βx ) ( 3 ) which is a closer approximation function compared to the htanh activation . Comparisons are made in Figure 2 . Hubara et al . ( 2016 ) noted that the STE fails to learn weights near the borders of −1 and +1 . As depicted in Figure 2 , our proposed SignSwish activation alleviates this , as it remains differentiable near −1 and +1 allowing weights to change signs during training if necessary . Note that the derivative ddxSSβ ( x ) is zero at two points , controlled by β . Indeed , it is simple to show that the derivative is zero for x ≈ ±2.4/β . By adjusting this parameter beta , it is possible to adjust the location at which the gradients start saturating . In contrast to the STE estimators , where it is fixed . Thus , the larger β is , the closer the approximation is to the derivative of the sign function .

Summary: The authors of this paper aim to reduce the constraints required by neural networks so they can be evaluated on lower-power devices. Their approach is to quantize weights, i.e. rounding weights and hidden units so they can be evaluated using bit operations. There are many challenges in this approach, namely that one cannot back-propagate through discrete weights or discrete sign functions. The authors introduce an approximation of the sign function, which they call the SignSwish, and they back-propagate through this, quantizing the weights during the forward pass. Further, they introduce a regularization term to encourage weights to be around learned scales. They evaluate on CIFAR-10 and Imagenet, surpassing most other quantization methods.