A Dynamic Approach to Accelerate Deep Learning Training
Authors: John Osorio
Adrià Armejach
Eric Petit
Marc Casas
Venue:  (2020)
Keywords: reduced precision
bfloat16
CNN
DNN
dynamic precision
mixed precision

Abstract: Mixed-precision arithmetic combining both single- and half-precision operands in the same operation have been successfully applied to train deep neural networks. Despite the advantages of mixed-precision arithmetic in terms of reducing the need for key resources like memory bandwidth or register file size, it has a limited capacity for diminishing computing costs and requires 32 bits to represent its output operands. This paper proposes two approaches to replace mixed-precision for half-precision arithmetic during a large portion of the training. The first approach achieves accuracy ratios slightly slower than the state-of-the-art by using half-precision arithmetic during more than 99% of training. The second approach reaches the same accuracy as the state-of-the-art by dynamically switching between half- and mixed-precision arithmetic during training. It uses half-precision during more than 94% of the training process. This paper is the first in demonstrating that half-precision can be used for a very large portion of DNNs training and still reach state-of-the-art accuracy.

Under review as a conference paper at ICLR 2020
DYNAMICALLY ADAPTING FLOATING-POINT PRECI- SION TO ACCELERATE DEEP NEURAL NETWORK TRAINING
Anonymous authors Paper under houble-blind review
ABSTRACT
Mixed-precision arithmetic combining both single- and half-precision operands the same operation has been successfully applied to train deep neural net- works. Despite the advantages of mixxddppreissio arithmetic in terms reducing the need for key resourees like memory bandwidth register file size, it has a lim- ited capacity for diminishing computing costs and requires 32-bits to represent its output operands. This paper proposes two approaches to replace mixed-precistoe for half- half-precision arithmetic during a large portion of the training. The first ap- proach achieves accuracy ratios slightly slower than the state-of-the-art by using half-precision arithmetic during more than 99% of training. The second approach reaches the same accuracy the state-of-the-ar by dynamically switching be- tween half- and mixed-precision arithmetic during training. It uses half celf--rrecsioon during more than 94% of the training process. This paper is the first in demon- strating that half-precision can be used for very large portion of DNNs training and still reach state-of-the-arr accuracy.
INTRODUCTION
The use of Deep Neural Networks (DNNs) is becoming ubiquitous in areas like computer vi- sion (Krizhevsky et al. [0122) Szegedy et al. 2015), speech recognition (Hinton et al.) [2012), or language translation (Wu et al. 2016). DNNs display very remarkable pattern detection capacities and, more specifically, Convolutional Neural Networks (CNNs) are able to accurately detect and classify objects over very large image sets (Krizhessky et al. 2012). Despite this success, a large amount samples must be exposed to the model for tens or even hundreds of times during train- ing until an acceptable accuracy threshold is reached, which drives up training costs in terms of resources like memory storage or computing time.
To mitigate these very large training costs, approaches based data representation formats simpler than the Floating Point 32-bit (FP32) standard have been proposed (Courbariaux et al. 2014:) Gupta al. 2015). These approaches successfully mitigate the enormous training costs of DNNs by using data representation formats that either reduce computing costs or diminish the requirements in terms of memory storage and bandwidth. In particular, some of these proposals have shown the bene- fits of combining half-precision and single-precision compute during training in terms of keeping model accuracy and reducing compute and memory costs (Micikevicius al., 2017 Kalamkar al., 2019). These approaches accelerate linear algebra operations by accumulating half- caff-precision input operands to generate 32-bit outputs. While this mixed- peeeediieeiiion (MP) arithmetic can successfully reduce the use of resources like memory bandwidth or hardware components like register file size, has a very limited capacity for diminishing computing costs and is unable to reduce output data size.
In this paper we propose new training methodologies able to exclusively use half-precision for a large part of the training process, which constitutes a very significant improvement over mixed- precision approaches in terms of compute and memory bandwidth requirements. We propose two different approaches, the first the FP32 format to the model parameters involved in the training process, while the second dynam- ically switches between BF16 and MP during training depending on its progress. Our approaches
statically assigns either the Brain Floating Point 16-bit (BF16) or

---

Under review as a conference paper at ICLR 2020
do not require mixed-prees preeeis-reiiioon arithmetic while computing linear algebra operations for large por- tion of the training process, which enables them to deliver the same performance as they were operating with half-precision arithmetic during the whole training while providing the same model accuracy as if FP32 was used. This paper is the first in demonstrating that half-precision can be extensively used during DNNs training without the need for mixed-precision arithmetic. We made our code available
2 BACKGROUND ON MIXED-PRECISS APPROACHES AND MOTIVATION
Mixed-Precision training has been extensively explored in recent years. Approaches mixing Float- ing Point 16-bit (FP16) and FP32 datatypes have been proposed (Micikevicius al. 2017). In these approaches, multiplications of FP16 parameters are accumulated in HP32 registers to minimize data representation range and precision issues. Importantly, relevant phases of the training process like computing weight updates (WU) or dealing with batch normalization (BN) layers entirely use FP32, which implies that FP32 representation of network weights and biases is kept during the whole training. This approach requires some additional computations to enforce that FP32 values are con- verted FP16 without data representation range issues. This approach is used by Nvidia Tesla V100 GPUs via mixed- -peediipe computing units called tensor cores, which are able to multiply FP16 pa- rameters and store the results in FP32. Figure Ia displays the most fundamental operation of this approach combining FP16 and FP32, the mixed-preciston
Fused Multiply-Add (FMA) instruction, which computes D= = A. + C. Input parameters A and B are represented in the FP16 format. The result of the A B operation is kept in FP32 and added to the C' parameter, which is represented in FP32 as well. The final output D is also represented in FP32. FMA instructions constitute around 60% of the whole training workload for several relevant CNN models, as Section 3 shows.
A more recent approach proposes mixed-precisse arithmetic combining BF16 and FP32 (Kalamkar et al. 2019). It is very close its FP16 counterpart with the exception of the full half precision conversion. Since BF16 has the same data representation range as FP32, conversion from full to half precision is very simple in this case since it just requires applying the Round to Nearest Even (RNE) technique. This approach also processes WU and BN layers with FP32. Figure Tb shows representation of mixed-precision FMA combining BF16 and FP32. It very close to the previously described FP16-FP32 FMA with the only difference being the data representation format of input parameters A and B.
While mixed-precision FMA instructions bring significant benefits since they require less memory bandwidth and register storage than FP32 FMAs, there is still a large margin for improvement if an entirely BF16 FMA like the one represented in Figure IC could be extensively used for training purposes. First, since BF16 FMA requires exactly one half of the register storage a FP32 FMA, it doubles its Single Instruction Multiple Data (SIMD) vectorization capacity and, therefore, it may significantly increase its FMA instructions per cycle ratio. Extensions of Instruction Set Architec- tures (ISA) to allow SIMD parallelism are becoming key element for floating- tontingg-piitt performance, which has motivated major hardware vendors to include them in their products (Jeffers et al.) 2016 Stephens et al. 2017). Finally, BF16 FMA instructions also bring significant reductions in terms of memory bandwidth since they involve 50% and 25% less data than FP32 and MP FMAs, respec- tively. While half-precision arithmetic has not been used to train DNNs due to its lack of training convergence, this paper describes two techniques to fully use it while keeping the same convergence properties FP32. This paper analyzes in detail 3 relevant training workloads in Section 3 and applies these findings to build its two main contributions in Section
3 WORKLOAD ANALYSIS
We consider three CNN models: AlexNet, Inception V2, and ResNet-50. Section 5 describes the exact way use these models and the methodology we follow to analyze their training workload. Figure 2a shows an instruction breakdown of processing one batch on these networks. This figure shows how floating-pos -ponnt instructions constitute large portion of these workloads. For example, they represent 58 34.4 of the total in the case of AlexNet. A very large portion of these floating- point instructions, 57.42% of the total, are FMA instructions. In the cases of Inception and ResNet-
https:///iiii http::ttww.cccooooooaaaaeeeeeeee
2

---

Under review as a conference paper at ICLR 2020
11
11
1I
(PP32)
(FP16)
(FP32)
(BF16)
(FP32)
(BF16)
(BF16)
(a) FMA with FP32 and FP16 (MP)
(b) FMA with FP32 and BF16 (MP)
(c) Full BFI6 precision
FMA
Figure 1: Approaches to compute Fused Multiply-Ade (FMA) instructions
Top1-FP82
0.7
Topl-MF
Top1-BP16
Percentagg of Instrne tions
80
0.6
00-
0.4
0.3
0.1
0.0
AlexNet
Inception
ResNet
Other FT33
FMA
No FP
2
Epocha
10
(a) Instruction breakdown
(b) Static techniques ResNet-50.
Figure 2: CNN Analysis
50, FMA instructions represent 60.93% and 62.95% of the total, respectively. Therefore, FMA instructions constitute large portion of the whole training workload, while other FP32 instructions represent a small instruction count that remain below 1.1% for these three CNNs. This justifies to focus FMA instructions, executing them in half-precision has large potential for performance improvement.
Prior research (Micikevicius al., 2017; Kalamkar al., 2019) describes the need for using 32- bit arithmetic in Weight Updates (WO) and Batch Normalization (BN) layers when using training approaches based on mixed-precision arithmetic. We run an experimental campaign to confirm this observation and to measure the number of instructions devoted to WU and BN. For the case of ResNet-50, this instruction count around 30 million instructions per batch, that is, just 0.04% of the FP instructions. AlexNet and Inception V2 produce similar results. In conclusion, reducing the cost of FMA instructions has a high potential for very significant performance improvements even if WU and BN layers computed using full precision arithmetic.
Processing one training batch for the cases of AlexNet, Inception and ResNet-50 requires running 53.3, 37.2, and 70.0 billion dynamic instructions per batch, respectively. The number of model parameters drives the size of these workloads. AlexNet was trained with a batch size 256, while Inception and ResNet batch size of 64.
PROPOSAL
We propose two training methodologies that rely exclusively on half-precision BF16 for large portion of the training process, i.e., a large portion of FMA instructions. Prior mixed-precision approaches preclude large gains in computing costs some of the data elements remain in FP32. However, an FMA entirely relying on BF16 can potentially double the SIMD vectorization through- put of current processors and alleviate memory bandwidth requirements.
3

---

Under review as a conference paper at ICLR 2020
We first propose scheme that performs all FMA instructions in BF16 (see Figure 1c except those involved in computing WU and processing BN layers, which are entirely performed in FP32. While this method might not deliver the desired level of accuracy for all CNNs, Section 6 shows how it behaves remarkably well for the Inception V2 model, since it obtains the same level of accuracy as state-of-the training using MP and FP32.
However, some CNNs cannot entirely rely on half-precision arithmetic during training. For ex- ample, Figure 2b shows top/ accuracy achieved by three training techniques during 15 epochs for ResNet-50. The first technique (referred as FP32 in Figure 2b) entirely relies in FP32 arithmetic, the second approach (referred as MP in Figure 2b) represents the state-of-the art -recd-Prr training (Kalamkar et al. 2019), and the third approach (referred as BF16 in Figure 2bb performs all FMA instructions BF16 except for WU and BN. While the BF16 approach behaves relatively well, it displays lower accuracy than MP and FP32 for all the epochs, which indicates the need for an approach able to take advantage of BF16 arithmetic while delivering the same accuracy results as mixed- or full-precision approaches. The methodology we use to generate Figure 2b is described in Section 5
Our second contribution dynamically switches between MP and BF16 to deliver the same accuracy as MP while relying in BF16 FMAs during a large portion of the training process. Algorithm [ dis- plays high level pscudo-code of our proposal. It starts the training process using the state-of-iee the art mixeed-recisioon approach (Kalamkar et al. 2019) for several batches, defined by num Batches M P parameter. Then, it computes the Exponential Moving Average (EMA) (Lawrance & Lewis 1977) of the training loss and, its reduction is larger than a certain threshold (ema'l hrrehhld parameter), it computes the next numBatches 1616 using BF16 FMAs, except for WU and BN. Once training has gone through these num Batches BF 16 batches, our algorithm checks EMA and compares its reduction with the maaThreshold parameter. If this reduction is not large enough, the algorithm switches back to MP arithmetic. Otherwise, it keeps using BF16 arithmetic for num num Batches FF16 batches before checking EMA again.
5 EXPERIMENTAL METHODOLOGY
5.1 EXPERIMENTAL SETUP
Our experiments are performed on Intel Xeon Platinum 8160 processors, which include the AVX512 ISA. We use the Intel-Caffe (Intel a) framework (version 1.1. We use the Intel MKLDNN Intell ch (version 0.18.0) Deep Neural Network library and the Intel MKL library (Intel b) (version 2019.0.3) to run numerical kernels since both libraries are optimized to perform well on our testing infrastructure. Finally, to define and run the experiments use the pyCaffe python interface, which takes care of loading the data and orchestrating the execution.
5.2 EMULATION OF BF16 USING DYNAMIC BINARY INSTRUMENTATION
Due to the lack of available hardware implementing the BF16 numerical format, we rely on an em- ulation technique to perform our experiments. Several approaches have been used in the past to emulate the behaviour of reduced floating-point representations, most notably via libraries that per- form transformations like truncation and rounding (Chatelain et al. 2019 Dawson & Düben 2017] Kalamkar et al.) 2019h. We develop a binary analysis tool based PIN 3.7 (Luk et al. 2005). Our tool captures and instruments dynamic instructions, which enables adaptating numerical operands to the targeted numerical data format. Our approach seamlessly works complex frameworks like Py PyTorch, Tensorflow, or Caffe, with interpreted languages, and is able to instrument instructions triggered by dynamically linked libraries. Our binary analysis tool performs the following steps:
It checks the current operation mode, which can be FP32, MP, BF16 (see Figure 1
It checks the current execution routine to determine if we are executing routines that belong to WU or BN layers. If that is the case, computation proceeds with FP32.
The tool intercepts the dynamic instructions of the workload and detects all floating-poit operations, including FMAs. For each FMA instruction, operands that need to be rounded to BF16, depending on the current operation mode, are rounded using the RNE algorithm.
4

---

Under review as a conference paper at ICLR 2020
Algorithm Dynamic Precision
1: MumBatchesMPP P <- 10
Number of consecutive MP batches Number of consecutive BF16 batches Defines EMA reduction threshold
2: numBaltches P6 +- 1000
3: emaTThreshold 0.04
4:
5: precision.MModeBB166 <- False
Indicates current precision mode, I True means BFI6 & Counts how many numBatchesBF 16 have been executed MP Number of batches per training loop iteration
6: countBatchesBF16 +
7: numBatcchsTTrain
8:
9: for i 0 to niter do
Training Loop: niter value depends on the number of epochs
10:
crii.steeynumBaachsSsrriii > Execute numBatchesTrain batches in precisionModeBFII Lossji t train.trainingLoss
12:
if (i 5) then
Initial history to calculate EMA
EMA < averoge(traininggoss
14:
if (i (i>3 5) then
EM Aprev + EMA
16:
EMA + ema www.aaalcccciiotrnnnnnneess if Mpre FF True) then
Aprew)
> Each numBatchesMP
18:
if ((EM Aprev FM A) > emaThreeholdd then
If training loss goes down
Precision F16 True changeToBF166
20:
Switch precision to BF16
21:
else
22:
CountBatchesBB116 + CountBatchesBB16 + BamchessssTrain (count numBatchesBF116 then if Aprev - EM A) > emaThreshold) then
24:
If training loss goes down b Stay BF16 precision If training loss stagnates
countBatches F16 else oodd + False chungeToMPP( countBatches +
26:
27:
28:
Switch precision to MP
29:
The tool can dynamically change its operation mode anytime via simple inter-process communication method that can be invoked from the python high-level interface.
To mitigate the overhead of our binary analysis tool, we implement two optimizations: First, we vec- torize the truncation and rounding routines via AVX512 instructions. Second, we avoid redundant rounding and truncation operations by identifying instructions belonging to the same basic block sharing some input operands already stored in the register file. These two optimizations reduce the overhead of the tool from 100x to 25x with respect to native runs of the binary real hardware.
5.3 DYNAMIC AND STATIC TECHNIQUES
This paper considers two different types of training techniques: static schemes and dynamic schemes. When using static schemes, the training procedure uses the same data representation form for given parameter during its complete execution. For example, the three techniques displayed in Figure 2b are statie. We define the following schemes:
MP: FMA instructions belonging to WU and BN layers always use FP32 precision. The remaining FMA instructions use the mixed-precision approach represented in Figure lb]
This scheme replicates prior work on mixed- precision (Kalnmaar et al.) 2019).
BF16: FMA instructions belonging to WU and BN layers always use FP32 precision. The remaining FMA instructions use BF16 operands to multiply and to accumulate (Figure Ic
The BF16 method is the first contribution of this paper. It extensively uses half half-precision arithmetic while displaying good convergence properties.
The Dynamic scheme we propose in this paper switches between the MP and BFI6 static techniques during training, as explained in Section 4 and detailed in Algorithm [ This dynamic method im-
5

---

Under review as a conference paper at ICLR 2020
proves the training convergence properties of BF16 while still relying in half half-precision FMAs for very large portion of the execution.
The EMA threshold (emaTreshold) is set at 4%. This value is computed as the average EMA reduc- tion when using FP32 computations. The minimum number of batches to be performed in BFI6, defined by the num.Batchest B F16 parameter is set to 1,000, which precludes frequent unnecessary transitions between the two schemes. We set the num Batches parameter to 10, which keeps the number of batches using the MP regime low while keeping its benefits in terms of convergence.
5.4 CONVOLUTIONAL NEURAL NETWORK MODELS
To evaluate our proposals we consider the AlexNet (Krizhessky et al. 2012). Inception V2 (Szegedy et al. 2013) and ResNet50 (He et al. 2015b) models. They are representative CNN state-of oaae--of-thee the-art.. art.
We use the ImageNet database (Degg et al. 2009) as training input. To keep execution times man- ageable when using binary instrumentation tool, we the experiments using reduced Im- ageNet Database, similar to the Tiny ImageNet Visual Recognition challenge data set (Fei-Fei). Therefore, we use 256,000 images divided into 200 categories for training, and 10,000 images for validation. The images have no modifications in terms of the size. All the evaluated CNN models remain unmodified, the only change loading a reduced dataset.
AlexNet is selected due to its simplicity in terms of structure and amount of required computations. To train AlexNet we consider batch size of 256 and the base learning rate is 0.01, which is adjusted every 20 epochs taking into account weight decay of 0.0005 and momentum of 0.9. This model trained for 32 epochs.
We use Inception because it is model conceived to reduce computational costs via cheap 1x1 convolutions. To train we use a batch size of 64 and a base learning rate of 0.045, which updated every 427 steps (0.11 11 epochs). The gamma, momentum and weight decay are set to 0.96, 0.9, and 0.0002, respectively. The training process is executed for 16 epochs.
Finally we use ResNet-50. It is network that delivers good accuracy and avoids the vanishing gradients issue by using residual blocks and the MSRA initializer (He et al. 2015a).. We train using a multi-step approach. The batch size is 64 and the base learning rate is 0.05, which is updated every 30 epochs. The gamma hyperparameter, momentum value, and weight decay are set to 0.1, 0.9, and 0.0001, respectively. The training process runs for total of 32 epochs.
6 EVALUATION
Figure and Table show results from our evaluation campaign. The x-axis of the three plots belonging Figure represent the epochs of the training process while the y-axis represents the accuracy reached by the model over the validation set. Table shows the test accuracy we reach for the three network models when using the FP32 and MP baselines and our two contributions: BF16 and Dynamic.
The AlexNet model, due to its structure, shows a good response when lower precision numerical data types are used. As can be seen in Figure 3aIal all techniques converge, although the BF16 approach shows the worse accuracy when compared to the Dynamic or the MP techniques. Table I shows that FP32, MP, Dynamic, and BF16 reach top-5 accuracies of 84.50% 84.43%, 84.02% and 82.56% for AlexNet after 32 epochs. Importantly, Dynamic reaches the same accuracy as FP32 and MP while using the BF16 approach for 94.60% of the FMAs. In contrast, the BF16 static technique does
Table 1: Accuracy and percentage of FMAs executed in BF16 precision for evaluated CNNs
FP32
MP
Dynamic
BF16
Model
Epoch
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
BF16FMA
Top-1 1
Top-5
BF16FMA
AlexNet
60.79%
84.50%
84.43%
60.32%
84.02%
94.60%
57.80%
82.56%
99.93%
Inception
16
74.01%
92.36%
73.73%
92.67%
72.80%
92.02%
95.55%
72.03%
92.05%
99 90%
ResNet-50
75.97%
93.37%
75.76%
93.20%
74.20%
92.70%
73.00%
92.30%

---

Under review as a conference paper at ICLR 2020
0.8
0.6
Accuracy
Topi-FFe FP32
Top
0.4
Top RF16
Topt-Dyn
Top5-FP32
0.2
Top5-MP
Top5-BF16
Top5- Dyn
0.0
5
10
15
20
25
30
Epochs
(a) AlexNet
0.8
0.6
Accuracy
Topl-FP32
Topl-MP
0.4
Topl-BF1
Topl-Dyn
Top5-FP32
0.2
Top5-M1P
Top5-BF
Top5-Dyn
0.0
2
1
6
8
10
12
14
16
Epochs
(b) Inception
0.8
Accurney
0.6
Top1-FP92
Top IMP
0.4
Topl-Bi BF16
Top1-Dyn
Top5-FP32
0.2
Top5-MP
Top5-BF1 BF16
Top5-Dyn
0.0
15
20
Epochs
(c) ResNet-50
Figure 3: Test accuracy achieved by the different training approaches.

---

Under review as a conference paper at ICLR 2020
99.93% of the FMAs in full BF16 precision (0.07% are in WU and BN layers), but the accuracy drops by almost 3% in Top-1 and 2% in Top-5. This drop in accuracy happens just by doing an additional 5% BF16 FMAs. This give the percentage of BF16 FMAS with the objective to increase the accuracy of the model.
space to improve the Dynamic approach by reducing
Figure 3b shows the validation accuracy during 16 epochs for the Inception V2 model. It shows fluctuation on the accuracy evaluation during training due to its structure and hyperparameters tun- robust way these changes, which highlights its general applicability. Table shows that FP32, MP, Dynamic, and BFI6 reach top-5 accuracies of 93.36%, 92.67%. % 92.02%, and 92.05% for Inception V2 after 16 epochs.
ing. Dynamic responds
Finally, the evaluation on ResNet-50 demonstrates that the Dynamic approach is effective when applied deeper CNNs. In this case, the precision of the model reaches state-of-the-ar levels while using half- calf-precision for 96.4% of the FMA instructions. Figure 3c and Table [ display the exact accuracy numbers we get from our evaluation after 32 epochs. this experiment the Top-1 accuracy drops just 1.2% comparing the BF16 and dynamic approaches, however we could improve the dynamic technique relaxing the quantity of BF16 FMA executed to gain more accuracy.
6.1 SENSITIVITY ANALYSIS FOR DYNAMIC PRECISION ALGORITHM
We provide sensitivity analysis for the parameters employed in Algorithm The objective is to show that for a range of reasonable parameters the algorithm behaves as expected. To do this analysis set one of the parameters to the currently used value (NumBatchesMPP 10) to have manage- able number of combinations. We then test all the possible combinations using numBatchesBF (500, 1000, 2000} and emaThreshold (0.02, 0.04, 0.08). that total of different combina- tions. As stated in Section 5.3 during our evaluation we used the configuration (numBatchesMP, numBatchesBF16. emaThreshold) {10, 1000, 0.04} for all the evaluated networks.
Figure shows, for number of ResNet-50 epochs, the accuracy obtained for each of the 9 tested configurations as part of the sensitivity analysis. The name convention for these configurations is www.m///hhoolddncunneeeeeee
16>> In addition, we include accuracy for BF16, MP, and
FP32 executions.
As shown in the figure. the accuracies obtained at each epoch are always above that of the BF16 technique. For early epochs (i.e., 2 and 4) the dynamic configurations remain between BF16 and FP32 accuracy, even slightly above FP32, due to initial noise. As training advances all dynamic techniques behave similarly and present accuracies that are above BF16 and similar to those obtained with MP and FP32, as would expect. The most important parameter is the emaThreshold, it decides when a precision change occurs. As long as this parameter reasonably set to detect training loss improvement degradation the algorithm is bound to behave expected.
BF16
Dya-0.02 500
Dya-0.02
Dya-0.04 2K
Dyn-0.08 Dyn-0.08 2K
X MP
04_KK
Dym-D.02_1K
Dyn-D 04_500
Dyn-0.08_500
FP32
09
0.55
04
Anemury
0.50
OR1
Toa
08
00:
0.45
D0
08
20
08
DOR
TOR1
08
O8
0.40
DO
O8
T8
0.35
100
9
0.30
12
Fpochs
Figure 4: Sensitivity analysis to dynamic precision algorithm parameters on ResNet-50.
8

---

Under review as a conference paper at ICLR 2020
7
RELATED WORK
Prior work indicates that dynamic fixed-point is effective to train deep neural networks with low precision multipliers (Courbariaux et al. 2014). This approach obtains state-of-the-art results by uniformly applying the dynamic fixed point format with different scaling factors, which are driven by the overflow rate displayed by the ixed-poin numbers. Our proposals target deeper neural net- works than this approach and do not uniformly apply the same format all network parameters. In- stead, we differentiate between computations requiring FP32 during the whole training, like weight updates, from the that are well-suited for dynamic data representation schemes.
Previous approaches show the benefits of applying stochastic rounding to 16-bit fixed-point multiply and add operators Gupta et al. (2015). This previous work rely on FPGA emulation to show the benefits of stochastic rounding when applied to a custom fully connected neural network dealing with the MNIST dataset. The authors also consider CNN similar LeNet-5 enlarge their experimental campaign.
Previous approaches propose a training process DNN using 8-bit floating point numbers (Wang et al. 2018). They rely on a combination of 8-bit and 16-bit and additionally using stochastic rounding to obtain state-of-the-an results. The neural networks used in this previous approach are much simpler than the ones we consider in this paper, which do not allow 8-bit arithmetic.
The BF16 numerical format has been applied to specific-purpose hardware targeting deep neural networks (Wang & Kanwar (019).. This specific hardware used approach very similar to the mixed-precision techniques described in the paper by Kalamkar et al. (2019) and, consequently, our Dynamic approach can be applied top of them to reduce computing costs.
CONCLUSIONS AND FUTURE WORK
This paper analyzes the instruction breakdown workloads focused on deep neural network training that rely mixed-precision training. We show that sxeeedprecision FMAS constitute around 60% of these workloads and propose two approaches based on half-precision FMAs to accelerate the training process without hurting accuracy.
The first approach uses BF16 FMAS for most of the training workload, except routines involved in weight updates or batch normalization layers. This approach uses BF16 for more than 99% of the FMAs, which has very strong potential for performance improvement, while reaching slightly smaller accuracy than the ofate-of-thh-arr We propose second approach that dynamically switches between different data representation formats. This dynamic approach uses BF16 for around 96% of the FMAs while reaching the same precision levels as the standard single-precision and mixed- precision approaches.
Our two proposals are evaluated considering three state-of-the-ar deep neural networks and binary analysis tool that applies the required precision for each instruction. To the best of our knowledge, this is the first paper that demonstrates that half-precision can be used extensively on 294% of all FMAs during the training of very deep models without the need for mixed-precision arithmetic.
REFERENCES
Yohan Chatelain, Eric Petit, Pablo de Oliveira Castro, Ghislain Lartigue, and David Defour. Au- tomatic exploration of reduced floating-point representations in iterative methods. In Euro-Par 2019 Parallel Processing 25th International Conference, Lecture Notes in Computer Science. Springer, 2019.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. (Section 5):1-10, 2014. URL 1412. 7024
A. Dawson and P. D. Düben. rpe v5: emulator for reduced floating- precision in large numerical simulations. Geoscientific Model Development, 10(6):2221-222 -230,, 2017. doi: 10.5194/ gmd- 10-2222-017. URI. htt::://wwwwww.. http:://wwwwwwwwmeeeeeeeeeeo dev httpa//deeeeeeeeeeee///////////0

---

Under review as a conference paper at ICLR 2020
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Lcrge-Scale Hierarchical Image Database. In CVPR09, 2009.
L. Fei-Fei. Tiny imagenet visual recognition challenge. herokuapp.com
URL https:/tiny-immaaee
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. CoRR, abs/1502.02551, 2015. URL htt.:tttt///xxi..rrrgg..
15C2. 02551
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. Proceedings of the IEEE International Conference on Computer Vision, 2015 International Conference on Computer Vision, ICCV 2015: 1026-1034, 2015a. ISSN 15505499, doi: 10.1 10.100//CC.22015.222
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, pp. 1-12, 2015b.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups 1EEE Signal Processing Magazine, 29(6) 2012.
Intel. Intel caffe framework optimization, a. http:s//githuub.ccm//ee////////s
Intel. Intel math kernel library, b. URL.LLpp://eEEw.e.enee......
Intel. Intel deep neural network library. c. URL
James Jeffers, James Reinders, and Avinash Sodani. Intel Xeon Phi Processor High Performance Progrumming: Knights Landing Edition 2Nd Edition. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2nd edition, 2016. ISBN 0128091940, 9780128091944.
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka. Jianyu Huang. Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Ab- hisck Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey, Study of BFLOATI6 for Deep Learning Training, pp. 1-10, 2019. URL httt:://axiv..ott// abs 1905 12322
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097-1105. Curran Associates, Inc., 2012. URL dl. acm. org
cfm http::/:....mmmdd999e9e111 2999222
A. J. Lawrance and P. A. W. Lewis. An exponential moving-average sequence and point process (emal). Journal of Applied Probability, 14(1):98-11) 1977. doi: 10.2307/3213263.
Chi Keung Luk, Robert Cohn, Robert Muth, Harish Patil, Artur Klauser, Geoff Lowney, Steven Wal- lace, Vijay Janapa Reddi, and Kim Hazelwood. Pin: Building customized program analysis tools with dynamic instrumentation. ACM SIGPLAN Notices, 40(6):190-200 2005. ISSN 03621340. doi: 10.114551064778.06663
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training. pp- 1 1-12, 2017. URL http::httt//ittt wwtp///ivoorraaooooooo http:/bbss/1111/ 07744000000000
Nigel Stephens, Stuart Biles, Matthias Boettcher, Jacob Eapen, Mbou Eyole, Giacomo Gabrielli, Matt Horsnell, Grigorios Magklis, Alejandro Martinez, Nathanael Premillieu, Alastair Reid, Ale- jandro Rico, and Paul Walker. The arm scalable vector extension. IEEE Micro, 37(2):26-33 39, March 2017. ISSN 0272-1732. doi: 10.1109//..001..35 URL htt.::ttp://det /oi. org/10. 1109 /0MM 2017 35
10

---

Under review as a conference paper at ICLR 2020
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Computer Vision and Pattern Recognition (CVPR), 2015. URL http://arxiv www.:oorgaaaaa
1409. 4842
Naigang Wang, Jungwook Choi, Daniel Brand, Chia Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point numbers. Advances Neural Information Processing Systems, 2018 8- DecemberNeer1PP,77775----66
2018. ISSN 10495258.
Shibo Wang and Pankaj Kanwar. Bfloat The secret to high performance cloud tpus. 2019. hww.aaa--nnncceeeeee.nnnn. learning/
URL
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint
arXiv:1609.08144, 2016.


Reviews:
[{"review_id": "r1g83xPntS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The author(s) propose to accelerate the training of deep neural networks while also maintain the performance of the trained model by switching between fully half-precision computation and mixed-precision computation. Compared to the commonly-used mixed-precision training strategy, the proposed method can accelerate the training speed. Besides, on an image classification task, models trained by the proposed method achieve comparable performance with those trained by mix-precision training or full-precision training.\n\nStrength:\n1.\tSection 3 provides useful information for the workloads of deep neural networks.\n\nWeakness:\n1.\tThe overall idea is not novel. The proposed method simply switches between two existing training strategies, i.e., the mixed-precision training and half-precision training. The claim that \"this paper is the first in demonstrating that  half-precision can be used for a very large portion of DNNs training and still reach state-of-the-art accuracy\" may not correct, in fact, Nvidia's apex has already supported using mixed-precision or entirely half-precision to train DNNs, and there is no clear evidence that the proposed method is better than theirs due to the lack of experiments on more tasks and datasets.\n\n\n2.\tFrom Table 1, the Dynamic strategy outperforms BF16 in terms of classification accuracy. However, from the experiments, it’s unable to tell that the gains actually come from this Dynamic strategy. Maybe similar gains can be obtained once the same amount of MP iterations are executed at any period of the training process. For example, consider a simpler strategy: first train the model with BF16 for K% of the total training iterations, then for the last (100-K)% iterations, train it with MP. K can be tuned so that the proportion of BF16FMA is close to those in Table 1. Models trained with this strategy might achieve similar performance with the proposed Dynamic strategy. \n\n\n3.\tIt’s hard to apply the proposed method in real applications, since BF16 is only supported by very few kinds of hardware.\n\n\n4.\tTo demonstrate the effectiveness of the proposed method more clearly, it could be better to provide the exact proportion of reductions in terms of memory/computation/bandwidth in the experiments. \n\n\n5.\tFrom Table 1, there still exists a large performance gap in terms of accuracy (1.56% for ResNet-50) between the model trained by the proposed method and the model trained by state-of-the-art MP.\n\n\n6.\tThe organization of this paper can be improved. For example, Section 5.2 spends too much space introducing the emulation of BF16, which I think is not very relevant to the topic of this paper. And Figure 3 takes too much space.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJg3QUIy5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This proposes two techniques to replace mixed-precision arithmetic with half-precision training for a large part of the training process. In the first approach, the authors simply switch all mixed-precision operations with half-precision operations, and can achieve performances slightly lower than SOTA. In the second approach, the authors propose to dynamically switch between mixed-operations and half-precision operations during training. The authors claim that this second approach can match SOTA results while using half-precision arithmetic for more than 94% of training.\n\nOverall the paper is fairly well written, and easy to follow. The proposed techniques seem to empirically work well. However, I have a number of concerns about the paper, which explains my score. I list these concerns below.\n\n1. The proposed approach has a number of additional hyperparameters, which makes it less likely for the algorithm to be widely used if the algorithm is very sensitive to the values of this. For very extreme values of these hyperparameters, I would expect the algorithm to start behaving quite poorly. But it would help a lot to provide some sensitivity analysis to these hyperparameters for reasonable values of these hyperparameters.\n\n2. How much do the optimal hyperparameters (like numBatchesMP, numBatchesBF16, emaT) vary across problems?\n\n3. How much do the above-mentioned optimal hyperparameters vary with mini batch size?\n\n4. How are the other hyperparameters like learning rates selected? Is the learning rate tuned?\n\n5. Are the experiments repeated multiple times?\n\n6. It seems a bit weird to call a modification that simply uses half-precision arithmetic for most FMA operations a significant contribution of the paper, especially since it can't reach SOTA performance.\n\n7. Algorithm 1 should be written out in a better way that shows the training loop. It is slightly confusing the way it is written up right now.\n\nOverall I think the paper would significantly benefit from a more thorough empirical evaluation.\n\n=============================\n\nEdit after rebuttal:\nI thank the reviewers for their response and for running the additional experiments. However, I find the updated version of the paper to be inadequate in fully answering my concerns. While the authors have included a hyperparameter sensititivity analysis, I find the experiment to be unconvincing. Only two of the three hyperparameters are swept over a very small range of values, and the results presented are only for the first 12 epochs, while the actual model is typically trained for 90 epochs. While I appreciate the added experiment and realize that 10 days is too short a time to put in a proper sensitivity analysis, based on the current draft of the paper, I cannot recommend accepting this paper. I am however raising my score to a weak reject.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HklngrP6YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose approaches to accelerate deep neural network training with mixed-precision arithmetic.\nObserved that relying purely on half-precision arithmetic results in lower accuracy, the authors developed a method \nto dynamically switch between mixed-precision arithmetic (MP) and half-precision arithmetic (BF16 FMA).\nEmpirical results show that the dynamic approach can achieve similar accuracy as MP and FP32 algorithms.\n\nAlthough this paper shows the possibility to accelerate DNN training without great loss in performance, there are many issues with the paper itself. First, the title is ambiguous. The dynamic approach could mean a lot of things while training \nDNNs and one cannot tell what the paper is about simply relying on the title.\nAlso, the dynamic algorithm itself is not well presented. For example, how to choose the hyperparameters? What is the\noverhead to switch between MP and BF16FMA? \n\nApart from the algorithm itself, I also have questions regarding the experimental results.\nIs there a reason why the performance of the half-precision arithmetic \nvaries across different neural networks (Inception > Resnet)?\nSpecifically, what is the key factor that influences the sensitivity of a neural network towards precision?\n\nOverall I think this paper should be improved in its experiments and presentation.\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes a dynamic approach to training a neural net which switches between half and full-precision operations while maintaining the same classifier accuracy, resulting in a speed up in training time. Empirical results show the value of the approach, and the authors have added additional sensitivity analysis by sweeping over hyperparameters. 

The reviewers were concerned about the novelty of the approach as well as the robustness of the claims that accuracy can be maintained even in the accelerated, dynamic regime. After discussion there were still concerns about the sensitivity analysis and the significance of the results.

The recommendation is to reject the paper at this time.
Decision: Reject