BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations

1 INTRODUCTION . Deep neural networks ( DNNs ) have been achieving remarkable performance improvement in various cognitive tasks . However , the performance improvement generally comes from the increase in the size of the DNN model , which also greatly increases the burden on memory usage and computing cost . While server-level environment with large number of GPUs can easily handle such a large model , deploying a large-size DNN model on resource-constrained platforms such as mobile application is still a challenge . To address the challenge , several network compression techniques such as quantization ( Wu et al. , 2018b ; Zhou et al. , 2016 ; Lin et al. , 2016 ) , pruning ( Han et al. , 2015 ; Hu et al. , 2016 ; Wen et al. , 2016 ) , and efficient architecture design ( Howard et al. , 2017 ; Wu et al. , 2018a ; Tan & Le , 2019 ) have been introduced . For quantization , previous works tried to reduce the precision of the weights and/or activations of a DNN model to even 1-bit ( Courbariaux et al. , 2016 ; Rastegari et al. , 2016 ) . In Binary Neural Networks ( BNNs ) , in which both weights and activations have 1-bit precision , high-precision multiplications and accumulations can be replaced by XNOR and pop-count logics which are much cheaper in terms of hardware cost . In addition , the model parameter size of a BNN is 32x less than its full-precision counterpart . Although BNNs can exploit various hardware-friendly features , their main drawback is the accuracy loss . An interesting observation is that binarization of activation causes much larger accuracy drop than the binarization of weights ( Cai et al. , 2017 ; Mishra et al. , 2018 ) . In particular , it can be observed that the performance degradation is much larger when the bit precision is reduced from 2-bit to 1-bit than other multi-bit cases ( > 2-bit ) as shown in Table 1 . To explain the poor performance of binary activation networks , the gradient mismatch concept has been introduced ( Lin & Talathi , 2016 ; Cai et al. , 2017 ) and several works proposed to modify the backward pass of activation function to reduce the gradient mismatch problem ( Cai et al. , 2017 ; Darabi et al. , 2018 ; Liu et al. , 2018 ) . ∗Hyungjun Kim and Kyungsu Kim equally contributed to this work . In this work , we first show that previous approaches to minimize the gradient mismatch have not been very successful in reducing accuracy loss because the target measure which the approaches tried to minimize is not effective . To back up the claim , we propose a new measure for better estimation of gradient mismatch . Then , we propose a new training scheme for binary activation network called BinaryDuo based on the observation that ternary ( or higher bitwidth ) activation suffers much less from gradient mismatch problem than the binary activation . We show that BinaryDuo can achieve state-of-the-art performance on various benchmarks ( VGG-7 , AlexNet and ResNet ) . We also provide our reference implementation code online1 . 2 RELATED WORKS . Binary Neural Network A BNN was first introduced by Courbariaux et al . ( 2016 ) . In BNNs , both weights and activations are constrained to +1 or -1 and sign function is used to binarize weights and activations . Since sign function is not differentiable at a single point and has zero gradient everywhere else , the concept of straight-through estimator ( STE ) has been used for backpropagation ( Bengio et al. , 2013 ) . Courbariaux et al . ( 2016 ) used HardTanh function as differentiable approximation of the sign function so that its derivative is used instead of delta function . Rastegari et al . ( 2016 ) proposed a similar form of BNN , called XNOR-Net . XNOR-Net used +α/ − α instead of +1/-1 , where α is a channel-wise full-precision scaling factor . Sophisticated STEs Although using STE enabled the back-propagation of the quantized activation function , gradient mismatch caused by the approximation occurs across the input range of binary activation function . To mitigate the gradient mismatch problem , Darabi et al . ( 2018 ) proposed to use SwishSign function as a differentiable approximation of the binary activation function , while Liu et al . ( 2018 ) proposed to use the polynomial function . Continuous binarization Few recent work proposed to use a continuous activation function that increasingly resembles a binary activation function during training , thereby eliminating approximation process across activation function . Sakr et al . ( 2018 ) used a piecewise linear function of which the slope gradually increases , while Yang et al . ( 2019 ) and Gong et al . ( 2019 ) proposed to use sigmoid and tanh function , respectively . Additional shortcut Most recent models have shortcut connections across layers . Since the parameter size and computing cost of shortcut connections is negligible , shortcut data often maintains full-precision even in BNN . Bi-Real-Net ( Liu et al. , 2018 ) pointed out that these information-rich shortcut paths help improving the performance of BNNs , and hence proposed to increase the number of shortcut connections in ResNet architecture . Since additional shortcuts increase the element-wise additions only , the overhead is negligible while the effect on performance is noticeable . 3 SHARP ACCURACY DROP FROM 2-BIT TO 1-BIT ACTIVATION . 3.1 EXPERIMENTAL CONFIRMATION OF THE POOR PERFORMANCE OF BINARY ACTIVATION . In this section , we first verify that performance drop of a neural network is particularly large when the bit-precision of the activation is reduced from 2-bit to 1-bit ( Table 1 ) . We used the VGG7 ( Simonyan & Zisserman , 2014 ) network on CIFAR-10 dataset for our experiment . We trained the benchmark network with full-precision , 4-bit , 3-bit , 2-bit and 1-bit activation while keeping all weights in full-precision to solely investigate the effect of the activation quantization . Activation 1https : //github.com/Hyungjun-K1m/BinaryDuo quantization was carried out by clipping inputs outside of [ 0,1 ] and then applying uniform quantization as in Eq . 1 . Q ( x ) = 1 2k − 1 · round ( clip ( x , 0 , 1 ) · ( 2k − 1 ) ) ( 1 ) Here , x and Q ( x ) are the input and output of the quantization function , and k is number of bits . For consistency , we used Eq . 1 for binary activation function instead of the sign function which is another widely used option ( Cai et al. , 2017 ; Zhou et al. , 2016 ; Mishra et al. , 2018 ) . We trained each network for 200 epochs and detailed experimental setups are described in Appendix A . The best test accuracy from each precision is plotted in Figure 1a . It is clearly shown that reducing activation bitwidth from 2-bit to 1-bit results in the substantial drop in the test accuracy . One might argue that the sharp accuracy drop occurred because the capacity of the binary activation network was not large enough for the given task while that of the network with 2-bit or higher bitwidth was . To verify , we increased the model size by modifying the network width ( not depth ) for each activation precisions ( Figure 1b ) . We found that the large accuracy gap still existed between 2-bit activation network and 1-bit activation network regardless of the network width . For example , 1-bit activation network with 2 times wider layers needs more parameters and more computations , but shows much worse accuracy than 2-bit activation network with original width . In addition , poor performance of binary activation was also observed in different networks ( e.g . AlexNet and ResNet ) and different datasets as in Table 1 . Therefore , we argue that the sharp accuracy drop for the binary activation stems from the inefficient training method , not the capacity of the model . 3.2 GRADIENT MISMATCH AND SOPHISTICATED STES . When training a neural network with quantized activation function such as thresholding function , STE is often used for back-propagation . In such case , we call the back-propagated gradient using STE as coarse gradient following Yin et al . ( 2019 ) . However , training with the coarse gradient suffers from gradient mismatch problem caused by the discrepancy between the presumed and the actual activation function ( Lin & Talathi , 2016 ) . The gradient mismatch makes the training process noisy thereby resulting in poor performance . Unfortunately , it is not useful to measure the amount of gradient mismatch directly because the true gradient2 of a quantized activation function is zero almost everywhere . Therefore , previous works used an indirect criterion to measure gradient mismatch based on the discrepancy between the actual activation function and its differentiable approximation ( Darabi et al. , 2018 ; Liu et al. , 2018 ) . In Liu et al . ( 2018 ) , the cumulative difference between actual activation function ( f ) and its differentiable approximation ( g ) ( Eq.2 ) was used as a criterion for gradient mismatch . cumulative difference = ∫ ∞ −∞ |f ( x ) − g ( x ) |dx ( 2 ) Previous works tried to improve the accuracy by designing the differentiable approximation of binary activation function which has the small cumulative difference from the binary activation function . Figure 2 shows different options for the approximation functions proposed by previous 2We call the original gradient as ‘ true gradient ’ to distinguish with the coarse gradient . works ( Darabi et al. , 2018 ; Liu et al. , 2018 ) . The shaded area in each activation function represents cumulative difference with thresholding function . We conducted the same experiment for a binary activation network as in previous section with various sophisticated STEs . For each case , the cumulative difference and test accuracy are shown in Figure 2 . As proposed in previous works , cumulative differences when using sophisticated STEs are much lower than conventional case ( ReLU1 ) . However , its effect on network performance is quite limited in our experiment . As shown in the Figure 2 , large accuracy gap still exists between binary activation network and 2-bit activation network even if the sophisticated STEs are used . From this observation , we think that current method of measuring gradient mismatch using the cumulative difference can not reliably estimate the amount of gradient mismatch problem . The same observation was also reported by Bethge et al . ( 2019 ) recently . Therefore , in the following section , we introduce a better method for estimating the effect of gradient mismatch problem . 4 ESTIMATING THE GRADIENT MISMATCH . 4.1 GRADIENT OF SMOOTHED LOSS FUNCTION . Since the true gradient of quantized activation network is zero almost everywhere , using the value of the true gradient does not provide useful measure of the gradient mismatch problem . To mitigate the issue with zero true gradient values , Yin et al . ( 2019 ) recently introduced a method using the gradient for the expected loss given the distribution of the data to analyze the effect of different STEs . However , the distribution of the dataset is usually unknown and the gradient values of the layers other than the first layer are also zero in multi-layered networks . In this work , we propose to use the gradient of the smoothed loss function for gradient mismatch analysis . Using the simplest rectangular smoothing srect , the gradient of the smoothed loss function ( srect ∗ f ) along each parameter can be formulated as follows : ∇x ( srect ∗ f ) = ( ∂ ∂x1 ∫ ε −ε f ( x+ s · e ( 1 ) ) 2ε ds , ... , ∂ ∂xn ∫ ε −ε f ( x+ s · e ( n ) ) 2ε ds ) = ( ∂ ∂x1 ∫ x1+ε x1−ε f ( s , x2 , ... , xn ) 2ε ds , ... , ∂ ∂xn ∫ xn+ε xn−ε f ( x1 , ... , xn−1 , s ) 2ε ds ) = ( f ( x+ ε · e ( 1 ) ) − f ( x− ε · e ( 1 ) ) 2ε , ... , f ( x+ ε · e ( n ) ) − f ( x− ε · e ( n ) ) 2ε ) = ∇ε , xf ( 3 ) where e ( i ) refers to standard basis vector with value one at position i . We named∇ε , xf as Coordinate Discrete Gradient ( CDG ) since it calculates the difference between discrete points along each parameter dimension . Note that the formula for CDG is very close to the definition of the true gradient which is limε→0∇ε , xf . While the true gradient indicates the instant rate of change of loss ( ε → 0 ) , CDG is based on the average rate of change in each coordinate . We provide a graphical explanation on CDG in Appendix B . Note that the evolutionary strategy-style gradient estimation is another good candidate which can be used for the same purpose as it provides an unbiased gradient estimate for a smoothed function in a similar way ( Choromanski et al. , 2018 ) . Please refer to Appendix H for more information .

Summary: This paper studies activation quantization in deep networks. The authors first compare the coordinate discrete gradient and those obtained by various kinds of straight-through estimators, and found 1-bit activation networks have much poorer gradient estimation than 2-bit ones. Thus they speculate that this explains the poorer performance of 1-bit activation networks than 2-bit ones. To utilize higher precision of activation, the authors then propose to decouple a ternary activation into two binary ones, and achieve competitive results on typical image classification data sets CIFAR-10 and ImageNet.