Toward Efficient Low-Precision Training: Data Format Optimization and Hysteresis Quantization

1 INTRODUCTION . Deep neural networks have been used in various fields such as vision , audio , natural language processing , and reinforcement learning . As larger and more complex neural networks are adopted , the energy and time consumed for training have become a critical issue in hardware implementation . Using low-bit representations in training significantly reduces hardware overhead and memory footprint ; hence , neural network training with limited precision has been extensively studied recently . For instance , 16-bit formats are already adopted in commercial devices such as FP16 ( IEEE , 2019 ) in NVidia GPU and bfloat16 ( Kalamkar et al. , 2019 ) in Google TPU ( Wang et al. , 2019 ) . Also , Köster et al . ( 2017 ) suggested a new data format using a shared exponent suitable for low-precision training . Recently , it has been demonstrated that even 8-bit formats could be adopted in deep neural network training with reasonable accuracy ( Sun et al. , 2019 ; Fox et al. , 2020 ) . However , there are various issues in realizing low-precision training in practical applications as detailed below . Optimal data format for low-precision training : Training performance is susceptible to the data format we use to represent variables in the network . When a value is represented using a floatingpoint format with a fixed number of bits , there is a trade-off between dynamic range and precision . For instance , allocating more bits to the exponent part in a floating-point format enlarges the dynamic range but lowers precision due to fewer bits in the mantissa part . Recent studies on 8-bit training suggest various ways to reduce the dynamic range required for number representation to enhance representation precision . Early work on 8-bit training ( Wang et al. , 2018 ) adopts a 5-bit exponent to represent different variables using a single format , but Sun et al . ( 2019 ) examine the statistics of each variable and optimize the numeric formats separately . Specifically , the values used in the forward path ( weight and activation ) have a relatively narrow dynamic range , and only 4 bits are allocated to the exponent . Fox et al . ( 2020 ) propose to divide data into smaller blocks and assign a shared exponent bias to each block . Since the values in a block tend to exhibit similar statistics , the forward ( weight and activation ) and backward ( error ) paths could be represented using only 2-bit and 4-bit exponents , respectively . Note that the shared exponent bias is effectively identical to the scaling factor . If a variable has a value of m · 2e and a shared exponent bias of b , then its actual value is m · 2e+bias , which is identical to the scaling factor of 2bias . However , these approaches are difficult to generalize since we should empirically decide numeric formats for each task , neural network structure , and quantization scheme ( Fig . 1 ) . Furthermore , analyzing the statistics of each variable is not enough to determine an optimal format . Their distributions often have a long tail , and hence the dynamic range of the numeric format should be experimentally selected through many trial-and-errors in actual training . Performance degradation in from-scratch training : Previous studies on quantized models show that a model could achieve comparable accuracy to full-precision models even using 1- or 2-bit weights ( Choi et al. , 2019 ; Martinez et al. , 2020 ) through fine-tuning a pre-trained model . However , in low-precision training where a neural network is trained from scratch using low-precision values and computations , the trained model typically shows a noticeable accuracy drop ( Elhoushi et al. , 2021 ) . Fig . 1 ( b ) shows the Top-1 validation accuracy of ResNet-18 ( He et al. , 2016 ) trained on ImageNet ( Deng et al. , 2009 ) for different training schemes . The weights are quantized into a 4- bit base-2 logarithmic format . From-scratch training of the model with quantized weights results in a 2.1 % accuracy drop , whereas only 1.0 % degradation is observed if we fine-tune a pre-trained model . This suggests that even though a better solution exists for a given format , it can not be reached through from-scratch training . To formalize the issues above , here we divide quantization in low-precision training into two types : network quantization and data flow quantization . Network quantization refers to the quantization of the neural network model . An example of this type of quantization is weight quantization . In network quantization , we need to reduce the performance difference between from-scratch training and fine-tuning ( Yang et al. , 2019b ) . On the other hand , data flow quantization refers to the on-the-fly quantization that occurs when data propagate through the network in low-precision training . Examples include activation , error , and weight gradient quantizations . Additional errors are introduced in weight update computation due to this type of quantization , which leads to performance degradation . Hence , we need to find an optimal format to minimize accuracy drop due to computation errors in data flow quantization . In this paper , we present a systematic approach to implementing low-precision training on various models and tasks . First , we present a method to efficiently find an optimal format for data flow quantization . In addition , we introduce a hysteresis quantization technique , a new quantization method for network quantization that can mitigate the issues of from-scratch training . Our main contributions are : • We present a method that can predict the training performance of various numeric formats for data flow quantization . This method allows us to determine an appropriate data format for different neural network structures , datasets , and tasks efficiently . • Using the method above , we propose an optimal 8-bit format suitable for low-precision training of various models , which enables quantization of BatchNorm layer input and improves hardware efficiency with minimal performance degradation . • We propose a new quantization scheme that utilizes the hysteresis effect to improve the performance of from-scratch training . This scheme enables ultra-low-precision training using 4-bit logarithmic weights . 2 DATA FLOW QUANTIZATION . 2.1 NUMERIC FORMATS . There are many numeric formats that can be constructed with n bits depending on how much dynamic range is required and how many valid bits are used for representing a value . For example , using 8 bits we could implement 8-bit fixed point integer format , 8-bit floating-point formats such as FP152 , FP143 , and FP125 ( FP1xy represents 1 sign bit , x exponent bits , y mantissa bits ) , 8-bit posit format ( Gustafson & Yonemoto , 2017 ) , and 8-bit float-fix format ( Han et al. , 2019 ) . Since the diversity of formats that could be formulated using n bits is nearly unlimited , here we assume some constraints to limit candidates while still including widely used formats such as fixed-point and floating-point formats as below : • The MSB ( Most Significant Bit ) is used as a sign bit and other bits represent magnitude . Accordingly , only symmetric formats that have identical representable ranges for positive and negative numbers are considered . Two ’ s complement representation is slightly asymmetric since it can represent one more negative value , but it does not incur a significant difference . • The number of valid bits of a larger value is greater than or equal to the number of valid bits of a smaller value . The valid bits stand for significant digits in binary representation . • The ratio between consecutive representable values does not exceed 2 . For example , the base-4 logarithmic format is excluded . We could obtain 166 8-bit formats that meet these constraints . Then , we reduce 1 and 2 valid bits in each format to obtain 7- and 6-bit formats , resulting in 498 formats in total . More information on the numeric formats considered in our experiments is provided in Appendix A.1 . 2.2 ACTIVATION AND ERROR QUANTIZATION . In a neural network consisting of n layers , the training process is described by Al+1 = fl ( W t l , Al ) ( 1 ) El = gl ( W t l , El+1 ) ( 2 ) Gwl = hl ( Al , El+1 ) ( 3 ) W t+1l = o ( Gwl , W t l ) ( 4 ) where A , E , W , and Gw are activation , error , weight , and weight gradient , respectively . f , g , h , and o are forward , backward , gradient , and update functions . l and t represent the layer number and time step . We follow the quantized training scheme suggested by Fox et al . ( 2020 ) , but with the following modifications to reduce hardware implementation costs . A and E are quantized not only for the GEMM input but also for the BatchNorm layer input . BatchNorm layer normalizes input using the mean and variance of each channel , but these values are obtained only after observing all the inputs from the previous layer , necessitating that all input values are temporarily stored in memory . Therefore , quantizing the BatchNorm layer ’ s input significantly reduces memory footprint and memory access overhead . Additionally , the scope of sharing exponent bias is extended to a layer ( Al and El ) to avoid the overhead of aligning partial sums from different blocks in block-wise exponent sharing . Finally , instead of determining the shared exponent bias by analyzing all values in the layer , we conservatively update it by detecting overflow and underutilization that occurred in the previous mini-batch . 2.3 INDICATORS OF TRAINING PERFORMANCE . Effect of quantized error : Quantizing the error E in the backward path is independent of how the forward path behaves since the loss surface of the model does not change . Therefore , the optimalW that the network needs to reach through training remains the same regardless of the error quantization scheme . However , when the error is quantized , a quantization error ∆E is introduced in E , which incurs a noiseN∆E inGw through the gradient function in Eq . 3 and potentially updates each weight in the wrong direction . While some amount of noise may improve the training performance through regularization , using low-precision formats already introduces a large noise in the network , incurring performance degradation ( see Appendix A.8 ) . Therefore , we suggest that the weight gradient error N∆E could be a good indicator of degradation in training performance . One way to implement this is predicting performance using the magnitude ofN∆E ; however , if the noise is in the same direction as Gw , it would only change the amount of each update and result in a less severe effect . Instead , we could measure the misalignment between Gw + N∆E and Gw for performance prediction . The misalignment between two vectors is estimated by ∠ ( A , B ) = cos−1 { A ·B ‖A‖2 · ‖B‖2 } ( 5 ) Then , the change in the update direction due to N∆E is ∠ ( Gw , Gw +N∆E ) . We can expect that the smaller ∠ ( Gw , Gw +N∆E ) , the better the training performance . Effect of quantized activation : Contrary to error quantization , activation quantization affects the way the forward path operates , and the loss surface of the model changes . Hence , the global optima of weight parameters shift , where the amount of shift would be proportional to the quantization noise . The displacement of global optima can be indirectly estimated using the direction of the weight gradients Gw . If the angle ∠ ( Gw , Gw + N∆A ) is small , the deviation of the global optima is expected to be small as well , suggesting a better training performance . In the discussions above , we assumed that the angles ∠ ( Gw , Gw +N∆E ) and ∠ ( Gw , Gw +N∆A ) could be used to predict training performance . We experimentally prove this by comparing the training performance of different numeric formats . For 498 numeric formats in 6 to 8 bits , we compare the loss obtained from training with the proposed performance indicators ( Fig . 2 ) . Training loss is obtained by training ResNet-18 on CIFAR-10 dataset using SGD with a momentum of 0.9 for 60 epochs . The batch size is 128 images and the initial learning rate is 0.1 , which is decayed by a cosine scheduler . We average angles from 100 mini-batches after quantizing a pre-trained model . Note that we use Gw of the first layer since it can reflect quantization errors that occur in the activations and errors of all the layers in the network . The weight gradients from the full-precision network , the network with quantized activations , and the network with quantized errors are Gw , Gw +N∆A , and GW +N∆E , respectively . Fig . 2 shows that using the misalignment angle results in not only a higher Spearman ’ s correlation but also a more distinct shape for low training losses , making it a better metric for data format selection . For instance , using the error magnitude would predict the best format for transformer incorrectly ( see Fig . 7 ( e ) in Appendix A.3 ) . While obtaining the misalignment angle requires additional computations , its overhead is negligible since the part that requires the most time and computation is to obtain Gw , Gw + ∆E , and Gw + N∆A , which is still significantly lower than actual training . Using this method , we could determine the optimal format for a specific neural network model , dataset , and task very efficiently as we only need to measure the misalignment angle without time-consuming network training . For experiments in Fig . 2 , the amount of computation is reduced by 99.6 % , and the reduction will be even larger for larger datasets and complex networks that need more epochs for training .

Summary: This paper proposes a method to find an optimal quantization format based on error angle estimation and hardware overhead. The authors also present an hysteresis-based quantization method to reduce fluctuation of exponent values such that training (from the scratch) using only 4-bit weights can result in negligible amount of accuracy degradation for ResNet-18 on ImageNet. Experimental results are provided for ResNet-18, MobileNetv2, 2-layer LSTM, Transformer, and MobileNetV2+SSDLite. For 8-bit quantization, FP134 is chosen and such quantization is also applied to BatchNorm layers to reduce memory consumption.