Fast Binarized Neural Network Training with Partial Pre-training

1 INTRODUCTION . Quantizing neural networks ( Gupta et al. , 2015 ) , constraining weights and activations to take on values within some small fixed set , is a popular set of techniques for reducing the storage ( Han et al. , 2016 ) or compute ( Fromm et al. , 2020 ) requirements of deep neural networks . Weights and activations can often be quantized down to as few as 8 bits with no loss in accuracy compared to a full-precision model . Further quantization often comes at the expense of accuracy : it is possible to binarize neural networks ( Hubara et al. , 2016 ; Rastegari et al. , 2016 ) , constraining weights and activations to take on values within a set of two elements ( often { −1 , 1 } ) , but such binarization often lowers the accuracy of the resultant network , necessitating a tradeoff between desired compression and accuracy . In the literature , there are two primary techniques for obtaining a quantized neural network : quantizing a pre-trained full-precision network ( Banner et al. , 2019 ; Han et al. , 2016 ) , and training a quantized network from scratch ( Hubara et al. , 2016 ; Gupta et al. , 2015 ) . Full-precision training . Quantizing a full-precision network requires few or even no additional training epochs on top of training that full-precision network . Typical procedures for quantizing a full-precision network range from data-blind procedures like selecting quantization bins to minimize distance from the original weights ( Banner et al. , 2019 ) , to data-intensive procedures such as retraining the network to be more amenable to quantization ( Han et al. , 2016 ) . However without significant additional training time , quantizing a pre-trained network often does not reach the highest accuracy possible for the quantized network architecture ( Alizadeh et al. , 2019 ) . Further , achieving high accuracy with heavy quantization , such as binarization , often requires changing the network architecture , for instance by adding skip connections ( Bethge et al. , 2019 ) ; such architectural changes mean that the weights of a pre-trained full-precision network may not transfer to the new architecture . Low-precision training . Alternatively , training a quantized network from scratch allows for achieving high accuracy regardless of the availability of pre-trained full-precision weights ( Alizadeh et al. , 2019 ) . Typical procedures for training a quantized network from scratch involve tracking and optimizing latent weights , weights which are quantized during the forward pass but treated as fullprecision during the backward pass ( Hubara et al. , 2016 ) . However , training a quantized network from scratch can be costly . Quantized networks typically require more training iterations to plateau in accuracy ( Hubara et al. , 2016 , Figure 1 ; Bethge et al. , 2019 , Figure 2 ) . Further , since quantized networks are often trained by simulating the quantized operations in floating-point ( Zhang et al. , 2019 ) , low-precision training can be even more computationally expensive than the full-precision equivalent . Research question . In this paper , we explore the question : Can we accelerate training a binarized neural network from scratch to a given target accuracy ? Concretely , we assume that a network architecture and standard training schedule are provided , but that pre-trained full-precision networks are not available . We also specifically focus on achieving accuracy in the early phase of training , exposing the tradeoff between training cost and accuracy . Partial pre-training . To answer the above research question , we evaluate a technique , partial pre-training , that allows for faster training of binarized neural networks by first training the network as a standard floating point network with standard full-precision training for a short amount of time , then converting the network to a binarized neural network and continuing to train from there with standard low-precision training for the remainder of the budgeted training time . We specifically evaluate partial pre-training ’ s speedup over standard low-precision training , when training a binarized neural network from scratch . We find that partial pre-training can train VGG , ResNet , and Neural Collaborative Filtering networks on CIFAR-10 , ImageNet , and MovieLens-20m between 1.26× and 1.61× faster than standard low-precision training . Contributions . • We present partial pre-training , which can train binarized neural networks from scratch between 1.26× and 1.61× faster than standard low-precision training . • We find that partial pre-training both requires fewer iterations to train to a given accuracy , and also that partial pre-training takes on average less time per iteration than standard low-precision training . • We analyze the sensitivity of partial pre-training to the choice of split between full-precision and low-precision training finding that an even split , though not always optimal , nearly matches the highest accuracy achievable by any other choice of split . All together , we find that partial pre-training is a simple and effective approach for accelerating binarized neural network training . Partial pre-training is a step towards the goal of binarized neural network training procedures that can match the efficiency gains of binarized neural network inference . 2 BACKGROUND . Binarized neural networks trade off accuracy for inference efficiency . However , binarized neural networks often take longer to train than the full-precision versions of the same network architecture , both in terms of training iterations until convergence and wall-clock time per iteration . Training iterations . Binarized neural networks tend to take more iterations to train than the fullprecision versions of the same network architecture . For instance , Hubara et al . ( 2016 , Figure 1 ) show a binarized neural network with a custom architecture requiring 4× as many training iterations to plateau in accuracy as a full-precision baseline on CIFAR-10 . Bethge et al . ( 2019 , Figure 2 ) similarly show a binarized ResNet-18 taking 2× as many training iterations to plateau in accuracy as a full-precision baseline on ImageNet . Wall-clock time . Beyond requiring more iterations to train , binarized neural networks tend to take more wall-clock time to complete each iteration of training than full-precision networks do . This is because binarized neural networks are often trained by simulating low-precision operations with standard floating point operations ( Zhang et al. , 2019 ; Fromm et al. , 2020 ) , and require additional bookkeeping that full-precision networks do not require , such as performing the binarization of weights and activations ( Hubara et al. , 2016 ) and calculating scaling factors ( Rastegari et al. , 2016 ) . While it is theoretically possible to accelerate binarized neural network training , we are not aware of any effort to exploit binarization during the training phase . It is also not clear what the maximum speedup possible from accelerating binarized neural network training would be : Fromm et al . ( 2020 ) show an acceleration of 6.33× for a VGG during inference ; with the additional bookkeeping of binarized neural network training and potentially requiring higher precision gradients in the backward pass ( Zhou et al. , 2016 ) , real training speedups would likely be lower . 3 PARTIAL PRE-TRAINING . This paper focuses on accelerating the training time of binarized neural networks . Specifically , we aim to reduce both the training iterations and wall clock time of training binarized neural networks . We achieve this by leveraging the faster training time—both iteration-count and time-per-iteration—of full-precision networks . This section formalizes the design space of partial pre-training algorithms , describing the exact training methodology used in the experiments in Sections 5 and 6 . Partial pre-training splits training into multiple phases . First , partial pre-training trains the network for a short amount of time at full precision ( 32 bits ) using no quantization methods : weights , activations , and gradients are not quantized or clipped . Next , the binarization operators are added to the network ( binarizing weights and activations ) . Partial pre-training then continues to train the binarized neural network using standard low-precision training . To avoid requiring hyperparameter search for each different network , we prescribe a standard training schedule based on the original training schedule of the full-precision network ( i.e. , learning rate and associated decay , which is assumed to be provided ) . Each step of partial pre-training takes 50 % of the allotted training time . Within the training time for each step of partial pre-training , the network is trained using the original learning rate schedule compressed to the allotted training time . The partial pre-training algorithm is presented in Algorithm 1 : Algorithm 1 Partial pre-training . 1 . Train the network at full precision , using the original learning rate schedule compressed down to half of the desired training time . 2 . Binarize the network , inserting quantization operators into the network . 3 . Train the binarized network , using the original learning rate schedule compressed down to the remaining half of the desired training time . 4 EXPERIMENTAL METHODOLOGY . 4.1 DATASETS AND NETWORKS . We evaluate partial pre-training across a variety of datasets and networks . Specifically , we evaluate partial pre-training on a CIFAR-10 ( Krizhevsky , 2009 ) ResNet-20 ( He et al. , 2016 ) , a CIFAR-10 VGG-16 ( Simonyan & Zisserman , 2014 ) , an ImageNet ( Russakovsky et al. , 2015 ) ResNet-34 , and a MovieLens 20M ( Harper & Konstan , 2015 ) Neural Collaborative Filtering ( NCF ) ( He et al. , 2017 ) model . Following Bethge et al . ( 2019 ) , our ResNet-20 and ResNet-34 are extended with additional skip connections past every quantized convolution , rather than past every block of quantized convolutions as is standard for ResNets ; this architectural change facilitates training binarized ResNets from scratch . Our NCF model additionally has BatchNorm ( Ioffe & Szegedy , 2015 ) layers after each binarized layer . The networks otherwise use standard architectures , data augmentation schemes , and training schedules drawn from the literature . Details about the networks and their respective training regimes are presented in Table 1 . 4.2 TRAINING DETAILS . All networks were trained on AWS GPU instances . The CIFAR-10 and MovieLens-20M networks were trained on p3.2xlarge instances , using an NVIDIA V100 GPU . The ImageNet networks were trained on p3.8xlarge instances , training with 4 data-parallel NVIDIA V100 GPUs . The vision networks were trained using custom implementations of each network in TensorFlow 2.2.0 ( Abadi et al. , 2016 ) . The NCF was trained using the PyTorch 0.4 ( Paszke et al. , 2019 ) implementation included in the MLPerf 0.5 training benchmark suite ( Mattson et al. , 2020 ) . 4.3 BINARIZATION . We evaluate partial pre-training using standard approaches to neural network binarization : Inputs . We binarize input activations using PACT ( Choi et al. , 2018 ) , a gradient-based method of determining activation scale which is used in place of the ReLU activation in binarized neural networks . PACT introduces one trainable parameter per layer , α , which controls the scale of the activations . The PACT activation on binarized networks has the following form : PACT ( x ) = { 0 , x ∈ ( −∞ , α2 ] α , x ∈ ( α2 , ∞ ) ∂ PACT ( x ) ∂α = { 0 , x ∈ ( −∞ , α ) 1 , x ∈ [ α , ∞ ) ∂ PACT ( x ) ∂x = { 1 , x ∈ [ 0 , α ] 0 , otherwise We initialize α = 3 for each layer , and control its magnitude with an L2 regularization penalty with coefficient 0.0002 . Weights . We binarize weights using the sign of the weight and the straight-through estimator : sign ( x ) = { −1 , x ≤ 0 1 , x > 0 ∂ sign ( x ) x = { 1 , x ∈ [ −1 , 1 ] 0 , otherwise Full precision layers . Following standard practice ( Hubara et al. , 2016 ; Simons & Lee , 2019 ; Qin et al. , 2020 ) , we do not binarize inputs or weights to the first , last , or batch normalization layers , nor do we binarize projection layers in ResNets ( Bethge et al. , 2019 ) .

Summary: The paper suggests a method for training binary neural networks. The proposed method is to partially train with full precision and then continue with binarized training using the straight-through estimator. The method is very simple and there is very limited technical contribution, so in order to be worthy of publication it needs to be supported with compelling experimental results. Unfortunately this is not the case.