A Dynamic Approach to Accelerate Deep Learning Training

1 INTRODUCTION . The use of Deep Neural Networks ( DNNs ) is becoming ubiquitous in areas like computer vision ( Krizhevsky et al. , 2012 ; Szegedy et al. , 2015 ) , speech recognition ( Hinton et al. , 2012 ) , or language translation ( Wu et al. , 2016 ) . DNNs display very remarkable pattern detection capacities and , more specifically , Convolutional Neural Networks ( CNNs ) are able to accurately detect and classify objects over very large image sets ( Krizhevsky et al. , 2012 ) . Despite this success , a large amount of samples must be exposed to the model for tens or even hundreds of times during training until an acceptable accuracy threshold is reached , which drives up training costs in terms of resources like memory storage or computing time . To mitigate these very large training costs , approaches based on data representation formats simpler than the Floating Point 32-bit ( FP32 ) standard have been proposed ( Courbariaux et al. , 2014 ; Gupta et al. , 2015 ) . These approaches successfully mitigate the enormous training costs of DNNs by using data representation formats that either reduce computing costs or diminish the requirements in terms of memory storage and bandwidth . In particular , some of these proposals have shown the benefits of combining half-precision and single-precision compute during training in terms of keeping model accuracy and reducing compute and memory costs ( Micikevicius et al. , 2017 ; Kalamkar et al. , 2019 ) . These approaches accelerate linear algebra operations by accumulating half-precision input operands to generate 32-bit outputs . While this mixed-precision ( MP ) arithmetic can successfully reduce the use of resources like memory bandwidth or hardware components like register file size , it has a very limited capacity for diminishing computing costs and it is unable to reduce output data size . In this paper we propose new training methodologies able to exclusively use half-precision for a large part of the training process , which constitutes a very significant improvement over mixedprecision approaches in terms of compute and memory bandwidth requirements . We propose two different approaches , the first one statically assigns either the Brain Floating Point 16-bit ( BF16 ) or the FP32 format to the model parameters involved in the training process , while the second dynamically switches between BF16 and MP during training depending on its progress . Our approaches do not require mixed-precision arithmetic while computing linear algebra operations for a large portion of the training process , which enables them to deliver the same performance as if they were operating with half-precision arithmetic during the whole training while providing the same model accuracy as if FP32 was used . This paper is the first in demonstrating that half-precision can be extensively used during DNNs training without the need for mixed-precision arithmetic . We made our code available1 . 2 BACKGROUND ON MIXED-PRECISION APPROACHES AND MOTIVATION . Mixed-Precision training has been extensively explored in recent years . Approaches mixing Floating Point 16-bit ( FP16 ) and FP32 datatypes have been proposed ( Micikevicius et al. , 2017 ) . In these approaches , multiplications of FP16 parameters are accumulated in FP32 registers to minimize data representation range and precision issues . Importantly , relevant phases of the training process like computing weight updates ( WU ) or dealing with batch normalization ( BN ) layers entirely use FP32 , which implies that a FP32 representation of network weights and biases is kept during the whole training . This approach requires some additional computations to enforce that FP32 values are converted to FP16 without data representation range issues . This approach is used by Nvidia Tesla V100 GPUs via mixed-precision computing units called tensor cores , which are able to multiply FP16 parameters and store the results in FP32 . Figure 1a displays the most fundamental operation of this approach combining FP16 and FP32 , the mixed-precision Fused Multiply-Add ( FMA ) instruction , which computes D = A · B + C. Input parameters A and B are represented in the FP16 format . The result of the A ·B operation is kept in FP32 and added to the C parameter , which is represented in FP32 as well . The final output D is also represented in FP32 . FMA instructions constitute around 60 % of the whole training workload for several relevant CNN models , as Section 3 shows . A more recent approach proposes mixed-precision arithmetic combining BF16 and FP32 ( Kalamkar et al. , 2019 ) . It is very close to its FP16 counterpart with the exception of the full to half precision conversion . Since BF16 has the same data representation range as FP32 , conversion from full to half precision is very simple in this case since it just requires applying the Round to Nearest Even ( RNE ) technique . This approach also processes WU and BN layers with FP32 . Figure 1b shows a representation of a mixed-precision FMA combining BF16 and FP32 . It is very close to the previously described FP16-FP32 FMA with the only difference being the data representation format of input parameters A and B . While mixed-precision FMA instructions bring significant benefits since they require less memory bandwidth and register storage than FP32 FMAs , there is still a large margin for improvement if an entirely BF16 FMA like the one represented in Figure 1c could be extensively used for training purposes . First , since BF16 FMA requires exactly one half of the register storage of a FP32 FMA , it doubles its Single Instruction Multiple Data ( SIMD ) vectorization capacity and , therefore , it may significantly increase its FMA instructions per cycle ratio . Extensions of Instruction Set Architectures ( ISA ) to allow SIMD parallelism are becoming a key element for floating-point performance , which has motivated major hardware vendors to include them in their products ( Jeffers et al. , 2016 ; Stephens et al. , 2017 ) . Finally , BF16 FMA instructions also bring significant reductions in terms of memory bandwidth since they involve 50 % and 25 % less data than FP32 and MP FMAs , respectively . While half-precision arithmetic has not been used to train DNNs due to its lack of training convergence , this paper describes two techniques to fully use it while keeping the same convergence properties as FP32 . This paper analyzes in detail 3 relevant training workloads in Section 3 , and applies these findings to build its two main contributions in Section 4 . 3 WORKLOAD ANALYSIS . We consider three CNN models : AlexNet , Inception V2 , and ResNet-50 . Section 5 describes the exact way we use these models and the methodology we follow to analyze their training workload . Figure 2a shows an instruction breakdown of processing one batch on these networks . This figure shows how floating-point instructions constitute a large portion of these workloads . For example , they represent 58.44 % of the total in the case of AlexNet . A very large portion of these floatingpoint instructions , 57.42 % of the total , are FMA instructions . In the cases of Inception and ResNet- 1https : //github.com/dynamicprec/dynamic 50 , FMA instructions represent 60.93 % and 62.95 % of the total , respectively . Therefore , FMA instructions constitute a large portion of the whole training workload , while other FP32 instructions represent a small instruction count that remain below 1.1 % for these three CNNs . This justifies to focus on FMA instructions , as executing them in half-precision has a large potential for performance improvement . Prior research ( Micikevicius et al. , 2017 ; Kalamkar et al. , 2019 ) describes the need for using 32- bit arithmetic in Weight Updates ( WU ) and Batch Normalization ( BN ) layers when using training approaches based on mixed-precision arithmetic . We run an experimental campaign to confirm this observation and to measure the number of instructions devoted to WU and BN . For the case of ResNet-50 , this instruction count is around 30 million instructions per batch , that is , just 0.04 % of the FP instructions . AlexNet and Inception V2 produce similar results . In conclusion , reducing the cost of FMA instructions has a high potential for very significant performance improvements even if WU and BN layers are computed using full precision arithmetic . Processing one training batch for the cases of AlexNet , Inception and ResNet-50 requires running 53.3 , 37.2 , and 70.0 billion dynamic instructions per batch , respectively . The number of model parameters drives the size of these workloads . AlexNet was trained with a batch size of 256 , while Inception and ResNet use a batch size of 64 . 4 PROPOSAL . We propose two training methodologies that rely exclusively on half-precision BF16 for a large portion of the training process , i.e. , a large portion of FMA instructions . Prior mixed-precision approaches preclude large gains in computing costs as some of the data elements remain in FP32 . However , an FMA entirely relying on BF16 can potentially double the SIMD vectorization throughput of current processors and alleviate memory bandwidth requirements . We first propose a scheme that performs all FMA instructions in BF16 ( see Figure 1c ) except those involved in computing WU and processing BN layers , which are entirely performed in FP32 . While this method might not deliver the desired level of accuracy for all CNNs , Section 6 shows how it behaves remarkably well for the Inception V2 model , since it obtains the same level of accuracy as state-of-the training using MP and FP32 . However , some CNNs can not entirely rely on half-precision arithmetic during training . For example , Figure 2b shows top1 accuracy achieved by three training techniques during 15 epochs for ResNet-50 . The first technique ( referred as FP32 in Figure 2b ) entirely relies in FP32 arithmetic , the second approach ( referred as MP in Figure 2b ) represents the state-of-the art Mixed-Precision training ( Kalamkar et al. , 2019 ) , and the third approach ( referred as BF16 in Figure 2b ) performs all FMA instructions in BF16 except for WU and BN . While the BF16 approach behaves relatively well , it displays lower accuracy than MP and FP32 for all the epochs , which indicates the need for an approach able to take advantage of BF16 arithmetic while delivering the same accuracy results as mixed- or full-precision approaches . The methodology we use to generate Figure 2b is described in Section 5 . Our second contribution dynamically switches between MP and BF16 to deliver the same accuracy as MP while relying in BF16 FMAs during a large portion of the training process . Algorithm 1 displays a high level pseudo-code of our proposal . It starts the training process using the state-of-the art mixed-precision approach ( Kalamkar et al. , 2019 ) for several batches , defined by numBatchesMP parameter . Then , it computes the Exponential Moving Average ( EMA ) ( Lawrance & Lewis , 1977 ) of the training loss and , if its reduction is larger than a certain threshold ( emaThreshold parameter ) , it computes the next numBatchesBF16 using BF16 FMAs , except for WU and BN . Once training has gone through these numBatchesBF16 batches , our algorithm checks EMA and compares its reduction with the emaThreshold parameter . If this reduction is not large enough , the algorithm switches back to MP arithmetic . Otherwise , it keeps using BF16 arithmetic for numBatchesBF16 batches before checking EMA again . 5 EXPERIMENTAL METHODOLOGY . 5.1 EXPERIMENTAL SETUP . Our experiments are performed on Intel Xeon Platinum 8160 processors , which include the AVX512 ISA . We use the Intel-Caffe ( Intel , a ) framework ( version 1.1.6a ) . We use the Intel MKLDNN ( Intel , c ) ( version 0.18.0 ) Deep Neural Network library and the Intel MKL library ( Intel , b ) ( version 2019.0.3 ) to run numerical kernels since both libraries are optimized to perform well on our testing infrastructure . Finally , to define and run the experiments we use the pyCaffe python interface , which takes care of loading the data and orchestrating the execution .

Summary: This proposes two techniques to replace mixed-precision arithmetic with half-precision training for a large part of the training process. In the first approach, the authors simply switch all mixed-precision operations with half-precision operations, and can achieve performances slightly lower than SOTA. In the second approach, the authors propose to dynamically switch between mixed-operations and half-precision operations during training. The authors claim that this second approach can match SOTA results while using half-precision arithmetic for more than 94% of training.