An Empirical study of Binary Neural Networks' Optimisation

1 INTRODUCTION . There is great interest in expanding usage of Deep Neural Networks ( DNNs ) from running remotely in the cloud to performing local on-device inference on resource-constrained devices ( Sze et al. , 2017 ; Lane & Warden , 2018 ) . Examples of such devices are mobile phones , wearables , IoT devices , and robots . This is motivated by privacy implications of sharing data and models with remote machines , and the appetite to apply DNNs in new environments and scenarios where cloud-inference is not viable . However , requirements of such devices are very demanding : there are stringent compute , storage , memory and bandwidth limitations ; many applications need to work in real-time ; many devices require long battery life for all-day or always-on use ; and there is a thermal ceiling to consider when designing thin and light devices . On the other hand , the quest for more accurate DNNs has resulted in deeper , more compute-intensive models . This is particularly the case for CNNs . For instance , while the convolutional layers of AlexNet ( Krizhevsky et al. , 2012 ) make up only 4 % of the model parameters , they are accountable for 91 % of the computations at inference time ( Louizos et al. , 2017 ) . Compression and efficient implementation of DNNs are therefore more important than ever . There has been a spate of recent work proposing training and post-training schemes that aim to compress models without significant loss in their performance . Main examples of these techniques are pruning , weight sharing , low-rank approximation , knowledge distillation and perhaps most importantly quantisation to lower precisions ( Han et al. , 2017 ; Ullrich et al. , 2017 ; Hinton et al. , 2015 ) . Quantisation is widely used in commercial deployments and its trade-offs and performance improvements for CNNs is well-studied in the literature ( Krishnamoorthi , 2018 ) . One appealing training-time quantisation scheme ( Courbariaux et al. , 2015 ) pushed it to the extreme , by representing each weight with a single bit , while maintaining respectable model accuracy . This paved the way for emergence Binary Neural Networks ( BNNs ) . Courbariaux et al . ( 2016 ) and Rastegari et al . ( 2016 ) expanded BNNs by using the sign function as the non-linearity to achieve binary activations in addition to binary parameters . With this approach , full-precision MAC operations in convolution layers can be replaced with cheap XNOR and POPCOUNT binary operations . This results in 58× ( Rastegari et al. , 2016 ) improvement in compute-time in addition to the inherit 32× saving in model size that comes from replacing 32-bit floating point parameters with binary ones . However , as we will describe in Section 2 , the common optimisation process used in BNNs is still not fully understood . Moreover , state-of-the-art binary models employ various modifications to conventional training settings in order to squeeze the best performance from the models . Some examples of these modifications are : applying constraints to weights and gradients , changing typical order of operations in a convolutional block , scaling learning rates based on Xavier ( Glorot & Bengio , 2010 ) initialisation values , learning additional parameters for affine transformations of kernels , changing momentum hyper-parameters in Batch Normalisation ( Ioffe & Szegedy , 2015 ) and the choice of optimiser , loss function , learning rate and number of training epochs . In the absence of rigorous mathematical understanding as of yet , it is imperative to empirically study the sensitivity of the optimisation process and the performance of BNNs to these settings and tweaks . Such an empirical understanding of the tools will greatly aid any development of solid mathematical foundations for the field . To that end , the main contributions of this work are as follows : • We identify the essential techniques required for successful optimisation of binary models and show that end-to-end training of binary networks crucially relies on the optimiser taking advantage of second-moment gradient estimates . • We show that most of the commonly used tricks in training binary models , such as gradient and weight clipping , are only required during the final stages of the training to achieve the best performance . Further , we demonstrate that these tricks lead to much slower convergence in the early stages of optimisation . • We propose new procedures for training , making optimisation notably faster by delaying these tricks , or by training a full-precision model first and fine-tune it into a binary model . • We provide our reference implementations and training-evaluation source code online 1 . 2 BACKGROUND . Early quantised models were derived by quantisating full-precision weights of pre-trained models ( Gong et al. , 2014 ) . This approach is widely used in real deployments and enjoys advantages such as the flexibility to apply different levels of quantisation based on the target model size , and not requiring knowledge of model internals . However , it suffers from a significant loss of accuracy . Hubara et al . ( 2017 ) showed that in order to maintain model performance , quantisation must be incorporated as part of the training process . This is done by either performing additional training steps to fine-tune a quantised model or by directly learning quantised parameters . This is essential for BNNs where binarising weights of pre-trained models result in significant loss in accuracy . The first successful binarisation-aware training method was proposed in BinaryConnect by Courbariaux et al . ( 2015 ) . In their work , the binary weights are not learned directly ; instead , full-precision weights are maintained and learned during the training as proxies for the binary weights . These proxies are only required during training . During the forward path , binary weights are computed by applying sign function to their corresponding full-precision proxies . Since the sign function is not differentiable BinaryConnect employs Straight-Through-Estimator ( STE ) ( Bengio et al. , 2013 ) for back-propagating gradient estimates to full-precision proxies . The STE estimator simply passes the gradients along as if the non-differentiable operator was not present . In practice , BinaryConnect applied two additional restrictions on vanilla STE : ( 1 ) Gradient clipping stops gradient flow if the weight ’ s magnitude is larger than 1 . This effectively means gradients are computed with respect to hard tanh function . ( 2 ) Weight clipping is applied to weights after gradients have been applied to keep them within a range . To formalise this consider wr to be a full-precision proxy for binary weight wb . During the forward path ( and at the end of the training ) : wb = sign ( wr ) 1https : //github.com/mi-lad/studying-binary-neural-networks STE with gradient clipping provides an estimate for the gradient of this operation : ∂wb ∂wr = 1|r|≤1 ( 1 ) In a back-propagation context , we assume the gradient of the cost C at the output ( ∂C∂wb ) is available where in computing it the same STE estimator above has been used wherever required . Eq 1 enables us to estimate the gradient of the cost at the input ( ∂C∂wr ) and update the proxies : ∂C ∂wr = ∂C ∂wb 1|r|≤1 The estimator passes gradients backwards unchanged when proxies are within the { -1,1 } range and cancels the gradient flow when the proxy weight has got too positive or too negative . Figure 1 depicts how this works for a convolutional kernel in a CNN . There have been several extensions to BinaryConnect ’ s core idea of using STE estimator in binary models . BinaryConnect showed slightly better results when STE was used in stochastic binary neurons . BinaryNet ( Courbariaux et al. , 2016 ) used binary activations in addition to binary parameters ( as described in Section 1 ) and made the convolution operation more efficient by using a custom GPU kernel . XNOR-Net and BWN ( Rastegari et al. , 2016 ) managed to scale up BNNs to achieve competitive results on the much bigger ImageNet ( Deng et al. , 2009 ) dataset by learning additional full-precision scale factors per-layer . DoReFa-Net ( Zhou et al. , 2016 ) used STE in the backpropagation path to quantise gradients and achieve faster training . TernaryNet ( Zhu et al. , 2016 ) quantised parameters to one and a half bits and represented weights using { -1,0 , +1 } . Having zero allows efficient hardware implementations when kernels are sparse . Lin et al . ( 2017 ) achieved state-of-the-art performance by learning a combination of very few binary kernels in each layer . 3 A SYSTEMATIC STUDY OF EXISTING METHODOLOGIES IN BNNS . There have been several non-empirical attempts to formalise STE and BNNs . Anderson & Berg ( 2018 ) took a high-dimensional geometric point-of-view to justify the existence of binary solutions irrespective of the optimisation process . Li et al . ( 2017 ) provided accuracy guarantees for training binary models under convexity assumptions . However , STE still has not been shown to find the solution of any particular loss function . In the meantime , binary models are achieving acceptable levels of accuracy in practice . Table 1 lists some of the recent binary architectures and their commonly-used training setup . In this section , we provide an empirical analysis of the main approaches used in these models and help the researchers and practitioners navigate this space . We explore two classes of architectures in our study of binary networks : A CNN inspired by VGG-10 ( Simonyan & Zisserman , 2015 ) on CIFAR-10 ( Krizhevsky & Hinton , 2009 ) dataset and an MLP with three hidden layers with 2048 units and rectified linear units ( ReLUs ) for MNIST ( LeCun , 1998 ) dataset . We make use of gradient and weight clipping and squared hinge loss unless stated otherwise . We use the last 10 % of the training set for validation and report the best accuracy on the test set associated with the highest validation accuracy achieved during training . The results shown are the average of five runs . We have not used early stopping or finite time budget in any of the experiments . The remainder of this section is organised as follows : We first show that the choice of optimiser matters considerably . We then show the impact of clipping gradients and weights followed by batch normalisation hyper-parameters on convergence speed and accuracy of BNNs . We finish by testing the effectiveness of some of the other commonly used tweaks used in training binary models . 3.1 IMPACT OF OPTIMISER . The majority of recent binary models use an adaptive optimiser in their implementations : BinaryConnect uses ADAM ( Kingma & Ba , 2015 ) for CIFAR-10 and vanilla SGD for MNIST ( although in their released source code they used ADAM for both datasets ) , DoReFa-Net and XNOR-Net use ADAM in their experiments and ABC-Net ( Lin et al. , 2017 ) uses SGD with momentum . In this section , we show that this is not accidental and investigate how the optimiser type and its associated hyper-parameters affects the viability of the STE estimator . For experiments in this section , we looked at optimisers from four classes in order of increasing complexity : ( 1 ) history-free optimisers such as mini-batch SGD that do not take previous jumps or gradients into account , ( 2 ) momentum optimisers that maintain and use a running average of previous jumps such as Momentum and Nesterov ( Sutskever et al. , 2013 ) , ( 3 ) Adaptive optimisers that adjust learning rate for each parameter separately such as AdaGrad ( Duchi et al. , 2011 ) and AdaDelta ( Zeiler , 2012 ) , and finally , ( 4 ) optimisers that combine elements from categories above such as ADAM which combines momentum with adaptive learning rate . Table 2 summarises the best accuracies we achieved using different optimisers . We ran experiments for more epochs than typically required for the datasets ( up to 500 epochs depending on the experiment ) . In each experiment , the relevant hyper-parameters were tuned for best results . We observed great variance in convergence speed and model performance as a result of optimiser choice that goes beyond differences seen when training non-binary models . Our first observation is that vanilla SGD generally fails in optimising binary models using STE . We note that reducing SGD ’ s stochasticity ( by increasing batch size ) improves performance initially . However , it still fails to obtain the best possible accuracy . SGD momentum and Nesterov optimisers perform better than SGD when they are carefully fine-tuned . However , they perform significantly slower compared to optimsing non-binary models and have to be used for many more epochs than normally used for CIFAR-10 and MNIST datasets . Similar to SGD , increasing momentum rate improves training speed significantly but results in worse final model accuracy . In Appendix A we include results for the equivalent non-binary models that show the effect of batch size and momentum are far less substantial . A possible hypothesis is that early stages of training binary models require more averaging for the optimiser to proceed in presence of binarisaton operation . On the other hand , in the late stages of the training , we rely on noisier sources to increase exploration power of the optimiser . This is reinforced by our observation that binary models are often trained long after the training or validation accuracy stop showing improvements . Reducing the learning rate in these epochs does not improve things either . Yet , the best validations are often found in these epochs . In other words , using early stopping for training binary models would terminate the training early on and would result in suboptimal accuracies . Finally , adaptive optimisers , and specifically ADAM , consistently perform faster and able to achieve better accuracy levels . We experimented with different hyper-parameters in ADAM optimiser ( see Figure 2c ) and found the decay rate for the second moment estimate to play a significant role .

Summary: The authors made several claims and provide suggestions on training binary networks, however, they are not proved or theoretically analyzed.  The empirical verification of the proposed hypothesis was viewed as weak as the only two datasets used are small datasets MNIST and CIFAR-10, and the used network architectures are also limited. Much more rigorous and thorough testing is required for an empirical paper which proposes new claims.