Toward Efficient Low-Precision Training: Data Format Optimization and Hysteresis Quantization

1 INTRODUCTION . Deep neural networks have been used in various fields such as vision , audio , natural language processing , and reinforcement learning . As larger and more complex neural networks are adopted , the energy and time consumed for training have become a critical issue in hardware implementation . Using low-bit representations in training significantly reduces hardware overhead and memory footprint ; hence , neural network training with limited precision has been extensively studied recently . For instance , 16-bit formats are already adopted in commercial devices such as FP16 ( IEEE , 2019 ) in NVidia GPU and bfloat16 ( Kalamkar et al. , 2019 ) in Google TPU ( Wang et al. , 2019 ) . Also , Köster et al . ( 2017 ) suggested a new data format using a shared exponent suitable for low-precision training . Recently , it has been demonstrated that even 8-bit formats could be adopted in deep neural network training with reasonable accuracy ( Sun et al. , 2019 ; Fox et al. , 2020 ) . However , there are various issues in realizing low-precision training in practical applications as detailed below . Optimal data format for low-precision training : Training performance is susceptible to the data format we use to represent variables in the network . When a value is represented using a floatingpoint format with a fixed number of bits , there is a trade-off between dynamic range and precision . For instance , allocating more bits to the exponent part in a floating-point format enlarges the dynamic range but lowers precision due to fewer bits in the mantissa part . Recent studies on 8-bit training suggest various ways to reduce the dynamic range required for number representation to enhance representation precision . Early work on 8-bit training ( Wang et al. , 2018 ) adopts a 5-bit exponent to represent different variables using a single format , but Sun et al . ( 2019 ) examine the statistics of each variable and optimize the numeric formats separately . Specifically , the values used in the forward path ( weight and activation ) have a relatively narrow dynamic range , and only 4 bits are allocated to the exponent . Fox et al . ( 2020 ) propose to divide data into smaller blocks and assign a shared exponent bias to each block . Since the values in a block tend to exhibit similar statistics , the forward ( weight and activation ) and backward ( error ) paths could be represented using only 2-bit and 4-bit exponents , respectively . Note that the shared exponent bias is effectively identical to the scaling factor . If a variable has a value of m · 2e and a shared exponent bias of b , then its actual value is m · 2e+bias , which is identical to the scaling factor of 2bias . However , these approaches are difficult to generalize since we should empirically decide numeric formats for each task , neural network structure , and quantization scheme ( Fig . 1 ) . Furthermore , analyzing the statistics of each variable is not enough to determine an optimal format . Their distributions often have a long tail , and hence the dynamic range of the numeric format should be experimentally selected through many trial-and-errors in actual training . Performance degradation in from-scratch training : Previous studies on quantized models show that a model could achieve comparable accuracy to full-precision models even using 1- or 2-bit weights ( Choi et al. , 2019 ; Martinez et al. , 2020 ) through fine-tuning a pre-trained model . However , in low-precision training where a neural network is trained from scratch using low-precision values and computations , the trained model typically shows a noticeable accuracy drop ( Elhoushi et al. , 2021 ) . Fig . 1 ( b ) shows the Top-1 validation accuracy of ResNet-18 ( He et al. , 2016 ) trained on ImageNet ( Deng et al. , 2009 ) for different training schemes . The weights are quantized into a 4- bit base-2 logarithmic format . From-scratch training of the model with quantized weights results in a 2.1 % accuracy drop , whereas only 1.0 % degradation is observed if we fine-tune a pre-trained model . This suggests that even though a better solution exists for a given format , it can not be reached through from-scratch training . To formalize the issues above , here we divide quantization in low-precision training into two types : network quantization and data flow quantization . Network quantization refers to the quantization of the neural network model . An example of this type of quantization is weight quantization . In network quantization , we need to reduce the performance difference between from-scratch training and fine-tuning ( Yang et al. , 2019b ) . On the other hand , data flow quantization refers to the on-the-fly quantization that occurs when data propagate through the network in low-precision training . Examples include activation , error , and weight gradient quantizations . Additional errors are introduced in weight update computation due to this type of quantization , which leads to performance degradation . Hence , we need to find an optimal format to minimize accuracy drop due to computation errors in data flow quantization . In this paper , we present a systematic approach to implementing low-precision training on various models and tasks . First , we present a method to efficiently find an optimal format for data flow quantization . In addition , we introduce a hysteresis quantization technique , a new quantization method for network quantization that can mitigate the issues of from-scratch training . Our main contributions are : • We present a method that can predict the training performance of various numeric formats for data flow quantization . This method allows us to determine an appropriate data format for different neural network structures , datasets , and tasks efficiently . • Using the method above , we propose an optimal 8-bit format suitable for low-precision training of various models , which enables quantization of BatchNorm layer input and improves hardware efficiency with minimal performance degradation . • We propose a new quantization scheme that utilizes the hysteresis effect to improve the performance of from-scratch training . This scheme enables ultra-low-precision training using 4-bit logarithmic weights . 2 DATA FLOW QUANTIZATION . 2.1 NUMERIC FORMATS . There are many numeric formats that can be constructed with n bits depending on how much dynamic range is required and how many valid bits are used for representing a value . For example , using 8 bits we could implement 8-bit fixed point integer format , 8-bit floating-point formats such as FP152 , FP143 , and FP125 ( FP1xy represents 1 sign bit , x exponent bits , y mantissa bits ) , 8-bit posit format ( Gustafson & Yonemoto , 2017 ) , and 8-bit float-fix format ( Han et al. , 2019 ) . Since the diversity of formats that could be formulated using n bits is nearly unlimited , here we assume some constraints to limit candidates while still including widely used formats such as fixed-point and floating-point formats as below : • The MSB ( Most Significant Bit ) is used as a sign bit and other bits represent magnitude . Accordingly , only symmetric formats that have identical representable ranges for positive and negative numbers are considered . Two ’ s complement representation is slightly asymmetric since it can represent one more negative value , but it does not incur a significant difference . • The number of valid bits of a larger value is greater than or equal to the number of valid bits of a smaller value . The valid bits stand for significant digits in binary representation . • The ratio between consecutive representable values does not exceed 2 . For example , the base-4 logarithmic format is excluded . We could obtain 166 8-bit formats that meet these constraints . Then , we reduce 1 and 2 valid bits in each format to obtain 7- and 6-bit formats , resulting in 498 formats in total . More information on the numeric formats considered in our experiments is provided in Appendix A.1 . 2.2 ACTIVATION AND ERROR QUANTIZATION . In a neural network consisting of n layers , the training process is described by Al+1 = fl ( W t l , Al ) ( 1 ) El = gl ( W t l , El+1 ) ( 2 ) Gwl = hl ( Al , El+1 ) ( 3 ) W t+1l = o ( Gwl , W t l ) ( 4 ) where A , E , W , and Gw are activation , error , weight , and weight gradient , respectively . f , g , h , and o are forward , backward , gradient , and update functions . l and t represent the layer number and time step . We follow the quantized training scheme suggested by Fox et al . ( 2020 ) , but with the following modifications to reduce hardware implementation costs . A and E are quantized not only for the GEMM input but also for the BatchNorm layer input . BatchNorm layer normalizes input using the mean and variance of each channel , but these values are obtained only after observing all the inputs from the previous layer , necessitating that all input values are temporarily stored in memory . Therefore , quantizing the BatchNorm layer ’ s input significantly reduces memory footprint and memory access overhead . Additionally , the scope of sharing exponent bias is extended to a layer ( Al and El ) to avoid the overhead of aligning partial sums from different blocks in block-wise exponent sharing . Finally , instead of determining the shared exponent bias by analyzing all values in the layer , we conservatively update it by detecting overflow and underutilization that occurred in the previous mini-batch . 2.3 INDICATORS OF TRAINING PERFORMANCE . Effect of quantized error : Quantizing the error E in the backward path is independent of how the forward path behaves since the loss surface of the model does not change . Therefore , the optimalW that the network needs to reach through training remains the same regardless of the error quantization scheme . However , when the error is quantized , a quantization error ∆E is introduced in E , which incurs a noiseN∆E inGw through the gradient function in Eq . 3 and potentially updates each weight in the wrong direction . While some amount of noise may improve the training performance through regularization , using low-precision formats already introduces a large noise in the network , incurring performance degradation ( see Appendix A.8 ) . Therefore , we suggest that the weight gradient error N∆E could be a good indicator of degradation in training performance . One way to implement this is predicting performance using the magnitude ofN∆E ; however , if the noise is in the same direction as Gw , it would only change the amount of each update and result in a less severe effect . Instead , we could measure the misalignment between Gw + N∆E and Gw for performance prediction . The misalignment between two vectors is estimated by ∠ ( A , B ) = cos−1 { A ·B ‖A‖2 · ‖B‖2 } ( 5 ) Then , the change in the update direction due to N∆E is ∠ ( Gw , Gw +N∆E ) . We can expect that the smaller ∠ ( Gw , Gw +N∆E ) , the better the training performance . Effect of quantized activation : Contrary to error quantization , activation quantization affects the way the forward path operates , and the loss surface of the model changes . Hence , the global optima of weight parameters shift , where the amount of shift would be proportional to the quantization noise . The displacement of global optima can be indirectly estimated using the direction of the weight gradients Gw . If the angle ∠ ( Gw , Gw + N∆A ) is small , the deviation of the global optima is expected to be small as well , suggesting a better training performance . In the discussions above , we assumed that the angles ∠ ( Gw , Gw +N∆E ) and ∠ ( Gw , Gw +N∆A ) could be used to predict training performance . We experimentally prove this by comparing the training performance of different numeric formats . For 498 numeric formats in 6 to 8 bits , we compare the loss obtained from training with the proposed performance indicators ( Fig . 2 ) . Training loss is obtained by training ResNet-18 on CIFAR-10 dataset using SGD with a momentum of 0.9 for 60 epochs . The batch size is 128 images and the initial learning rate is 0.1 , which is decayed by a cosine scheduler . We average angles from 100 mini-batches after quantizing a pre-trained model . Note that we use Gw of the first layer since it can reflect quantization errors that occur in the activations and errors of all the layers in the network . The weight gradients from the full-precision network , the network with quantized activations , and the network with quantized errors are Gw , Gw +N∆A , and GW +N∆E , respectively . Fig . 2 shows that using the misalignment angle results in not only a higher Spearman ’ s correlation but also a more distinct shape for low training losses , making it a better metric for data format selection . For instance , using the error magnitude would predict the best format for transformer incorrectly ( see Fig . 7 ( e ) in Appendix A.3 ) . While obtaining the misalignment angle requires additional computations , its overhead is negligible since the part that requires the most time and computation is to obtain Gw , Gw + ∆E , and Gw + N∆A , which is still significantly lower than actual training . Using this method , we could determine the optimal format for a specific neural network model , dataset , and task very efficiently as we only need to measure the misalignment angle without time-consuming network training . For experiments in Fig . 2 , the amount of computation is reduced by 99.6 % , and the reduction will be even larger for larger datasets and complex networks that need more epochs for training .

Summary: The authors propose a method to predict the performance of different numeric formats, which allows determining the optimal data format for various neural network architectures, datasets, and tasks efficiently. By comparing 498 formats in total, the authors find an optimal 8-bit format suitable for various models. To improve the performance of from-scratch training, the authors further propose hysteresis quantization to mitigate the ﬂuctuation issue. Experiments on 8-bit and 4-bit training demonstrate the effectiveness of the proposed method.